Article disordered grabbing method and system based on pose estimation

Abstract

The invention provides an article disordered grabbing method and system based on pose estimation. The article disordered grabbing method based on pose estimation comprises the following steps: generating an image data set; constructing a convolutional neural network model based on the image data set; and utilizing the convolutional neural network model to perform feature extraction and pose estimation on a to-be-grabbed article. According to the technical scheme, the image data set is generated; constructing a convolutional neural network model based on the image data set; using the convolutional neural network model to perform feature extraction and pose estimation on a to-be-grabbed article; and the grabbing precision and success rate are improved.

Description

Background Art

[0002] In industrial automation and robotics, automatic grasping of objects is a crucial task, especially for disorderly stacked objects. Traditional grasping methods often encounter many challenges when facing such scenarios, resulting in low grasping accuracy and success rate. Here are some of the main reasons:

[0003] 1. Object occlusion and overlap

[0004] There is often occlusion and overlap between disorderly stacked items. When some items are blocked by other items, it is difficult for the robot's visual system to accurately identify the position and posture of the blocked items. In addition, overlapping items may also lead to a lack of depth information, making it difficult for the robot to judge the spatial relationship between items when grasping.

[0005] 2. Changes in lighting conditions

[0006] In industrial automation environments, lighting conditions may change due to factors such as time, weather, or equipment status. Changes in lighting conditions can affect the quality and clarity of images, which in turn affects the robot's ability to identify and locate objects. In particular, in dim light or in the presence of reflections, the robot's visual system may not be able to accurately capture the object's feature information.

[0007] 3. Diversity of object shapes and sizes

[0008] Different objects can vary greatly in shape and size, which makes traditional grasping methods difficult to adapt to all situations. For example, some objects may have complex geometric shapes or irregular surfaces, making it difficult for the robot to accurately determine their position and posture when grasping. In addition, the size of the object may also affect the robot's grasping strategy and motion planning.

[0009] 4. Single crawling strategy

[0010] Traditional grasping methods often use fixed grasping strategies, such as fixed grasping points or fixed grasping paths. However, for disorderly stacked objects, fixed grasping strategies may not be able to adapt to all situations. For example, when the position or posture of the object changes, the fixed grasping point may no longer be applicable, resulting in grasping failure.

[0011] 5. Sensor accuracy and resolution limitations

[0012] The robot's sensors (such as cameras, lidar, etc.) have limited accuracy and resolution and may not be able to accurately capture all the details of the object. When the object is small or has a complex shape, the accuracy and resolution of the sensor may become a key factor limiting the grasping accuracy and success rate.

[0013] 6. Dynamic environment and interference factors

[0014] In an industrial automation environment, there may be various dynamic factors (such as the movement of other robots, the movement of people, etc.) and interference factors (such as noise, vibration, etc.), all of which may affect the robot's grasping accuracy and success rate. For example, when other robots or people move in the robot's working area, it may interfere with the robot's visual system or motion planning. Summary of the invention

[0015] The present application provides a method and system for unordered object grasping based on pose estimation, so as to improve the grasping accuracy and success rate.

[0016] In a first aspect, a method for unordered grasping of objects based on pose estimation is provided, comprising the following steps:

[0017] Generate image dataset;

[0018] Based on the image dataset, construct a convolutional neural network model;

[0019] The convolutional neural network model is used to extract features and estimate the position and posture of the object to be grasped.

[0020] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0021] In a specific embodiment, it also includes:

[0022] Based on the results of pose estimation, the grasping points are determined and corrected.

[0023] In a specific implementation scheme, the steps of extracting features and estimating the position and posture of the object to be grasped using the convolutional neural network model specifically include:

[0024] Importing the two-dimensional image of the object to be grasped into the convolutional neural network model, and extracting the corresponding confidence map and vector field;

[0025] The predicted translation and rotation of the object to be grasped are calculated using the extracted confidence map and vector field to obtain the result of the pose estimation.

[0026] In a specific implementation scheme, based on the result of the pose estimation, the steps of determining and correcting the grasping point specifically include:

[0027] Based on the result of the pose estimation, an optimal grasping point of the object to be grasped is obtained, and a measurement translation amount of the depth camera is calculated;

[0028] The grasping safety distance correction is performed according to the predicted translation amount and the measured translation amount; if the correction is successful, the correction data is captured; if the correction fails, iterative optimization is performed.

[0029] In a specific implementation scheme, the convolutional neural network model includes a VGG deep convolutional neural network model.

[0030] In a second aspect, a disordered object grasping system based on pose estimation is provided, comprising:

[0031] Dataset module, used to generate image datasets;

[0032] A model module, used to construct a convolutional neural network model based on the image dataset;

[0033] A posture estimation module is used to use the convolutional neural network model to perform feature extraction and posture estimation on the object to be grasped.

[0034] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0035] In a specific embodiment, it also includes:

[0036] The correction module is used to determine and correct the grasping points based on the results of pose estimation.

[0037] In a specific implementation scheme, the convolutional neural network model includes a VGG deep convolutional neural network model.

[0038] In a third aspect, an electronic device is provided, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device can implement any one of the methods for unordered object grasping based on pose estimation.

[0039] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0040] In a fourth aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that the computer-readable storage medium implements any one of the methods for unordered object grasping based on pose estimation.

[0041] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A method block diagram of a method for unordered object grasping based on pose estimation provided in an embodiment of the present application;

[0043] Figure 2 A structural block diagram of an unordered object grasping system based on pose estimation provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of the structure of the grasping robot provided in an embodiment of the present application.

[0045] Among them, 1-RGB-D camera, 2-grasping robot arm, 3-manipulator, 4-carrying box, 5-snake tube, 6-central controller, 7-universal wheel. DETAILED DESCRIPTION

[0046] The present application is further described in detail below through the accompanying drawings and embodiments. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific.

[0047] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0048] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0049] To facilitate understanding of the method and system for unordered grasping of objects based on pose estimation provided by the embodiment of the present application, its application scenario is first explained. The method and system for unordered grasping of objects based on pose estimation provided by the embodiment of the present application are used to improve the accuracy and success rate of grasping. In industrial automation and robotics, the automatic grasping of objects is a crucial task, especially for objects that are stacked in disorder. Traditional grasping methods often encounter many challenges when facing such scenarios, resulting in low grasping accuracy and success rate. The following are some of the main reasons: 1. Object occlusion and overlap: There is often occlusion and overlap between disorderly stacked objects. When some objects are blocked by other objects, it is difficult for the robot's visual system to accurately identify the position and posture of the blocked objects. In addition, overlapping objects may also lead to the loss of depth information, making it difficult for the robot to judge the spatial relationship between objects when grasping. 2. Changes in lighting conditions: In an industrial automation environment, lighting conditions may change due to factors such as time, weather, or equipment status. Changes in lighting conditions will affect the quality and clarity of the image, and thus affect the robot's recognition and positioning of objects. Especially in dim light or reflective conditions, the robot's visual system may not be able to accurately capture the characteristic information of the object. 3. Diversity of object shape and size: The shapes and sizes of different objects may vary greatly, which makes it difficult for traditional grasping methods to adapt to all situations. For example, some objects may have complex geometric shapes or irregular surfaces, making it difficult for the robot to accurately judge their position and posture when grasping. In addition, the size change of objects may also affect the robot's grasping strategy and motion planning. 4. Single grasping strategy: Traditional grasping methods often adopt fixed grasping strategies, such as fixed grasping points or fixed grasping paths. However, for disorderly stacked objects, fixed grasping strategies may not be able to adapt to all situations. For example, when the position or posture of the object changes, the fixed grasping point may no longer be applicable, resulting in grasping failure. 5. Sensor accuracy and resolution limitations: The accuracy and resolution of the robot's sensors (such as cameras, lidar, etc.) are limited and may not accurately capture all the details of the object. When the size of the object is small or the shape is complex, the accuracy and resolution of the sensor may become a key factor limiting the grasping accuracy and success rate. 6. Dynamic environment and interference factors: In an industrial automation environment, there may be various dynamic factors (such as the movement of other robots, the movement of people, etc.) and interference factors (such as noise, vibration, etc.), all of which may affect the robot's grasping accuracy and success rate. For example, when other robots or people move in the robot's working area, it may interfere with the robot's visual system or motion planning. For this reason, the embodiment of the present application provides a method and system for disordered grasping of objects based on pose estimation to improve the accuracy and success rate of grasping. The following is a detailed description of the embodiment in conjunction with specific drawings.

[0050] refer to Figures 1 to 3 , Figure 1 A method block diagram of a method for unordered object grasping based on pose estimation provided in an embodiment of the present application; Figure 2 A structural block diagram of an unordered object grasping system based on pose estimation provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of the grasping robot provided in an embodiment of the present application.

[0051] exist Figure 1 In the embodiment of the present application, a method for grabbing objects in an unordered manner based on pose estimation is provided, comprising the following steps:

[0052] Generate image dataset;

[0053] Based on the image dataset, construct a convolutional neural network model;

[0054] The convolutional neural network model is used to extract features and estimate the position and posture of the object to be grasped.

[0055] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0056] Specifically, the steps of generating an image data set by three-dimensional reconstruction in this application include:

[0057] 1. 3D reconstruction of scenes

[0058] First, use an RGB-D camera to collect RGB images and depth images of the captured scene; use the RGB images and depth images to perform three-dimensional reconstruction of the captured scene through a multi-view stereo vision (MVS) algorithm to generate a high-precision three-dimensional model of the scene. Specifically, it includes:

[0059] Data preprocessing: The collected RGB-D data is preprocessed, including denoising, alignment, and filling holes in the depth map.

[0060] Feature matching: Perform feature matching on images from different perspectives to find corresponding feature points.

[0061] 3D point cloud generation: Generate a 3D point cloud of the scene through triangulation using matched feature points.

[0062] Surface reconstruction: Reconstruct the surface of the point cloud to generate a mesh model of the scene.

[0063] The captured scene is reconstructed in 3D through the multi-view stereo vision (MVS) algorithm to generate a high-precision 3D model of the scene. The advantages include:

[0064] 1. High Precision and Completeness

[0065] High-precision reconstruction: The MVS algorithm can recover the dense three-dimensional structure of the scene from images from multiple perspectives and generate a high-precision three-dimensional model; making the generated model more realistic and accurate in details and able to reflect the true shape and structure of the scene.

[0066] Integrity preservation: Compared with other 3D reconstruction methods, the MVS algorithm can restore the 3D information of the scene more completely; it can handle scenes with complex shapes and structures and generate 3D models with rich details.

[0067] 2. Adaptability and robustness

[0068] Strong adaptability: The MVS algorithm can adapt to complex scenes such as different lighting conditions, perspective changes, occlusion, etc.; it can process images with problems such as texture loss and uneven lighting, and generate stable three-dimensional models.

[0069] High robustness: The MVS algorithm has high robustness in key links such as image feature matching and depth estimation; it can accurately match feature points in the image, generate reliable depth maps, and then generate stable three-dimensional models.

[0070] 3. Automation and efficiency

[0071] High degree of automation: The MVS algorithm can realize the automated 3D reconstruction process and reduce manual intervention; it can automatically process input image data, generate 3D models, and improve reconstruction efficiency.

[0072] Significant efficiency: The MVS algorithm has high computational efficiency in the 3D reconstruction process; it can quickly process large amounts of image data and generate high-precision 3D models to meet the needs of practical applications.

[0073] 2. Synthetic Image Generation

[0074] Based on the generated 3D model, we can render a 2D image from any perspective. In order to generate diverse image data, objects in the scene are randomly transformed (including translation and rotation) and rendered under different lighting conditions. The specific steps include:

[0075] View selection: Render the 3D model from multiple viewpoints to generate RGB images and depth images from different viewpoints.

[0076] Pose transformation: Randomly transform the pose of objects to generate scenes with different poses.

[0077] Lighting simulation: Simulate image rendering under different lighting conditions (such as day and night).

[0078] 3. Automatic Labeling

[0079] While rendering the image, the annotation information required for the grasping detection task is automatically generated, including:

[0080] Object Bounding Box: Automatically generate a 2D bounding box for an object based on the size and position of the 3D model.

[0081] Grasp point annotation: Calculate the best grasp point based on the geometric properties of the object (such as normal vector, curvature) and generate a grasp rectangle annotation.

[0082] 4. Generate image dataset

[0083] An image dataset of the object to be grasped is generated according to the object bounding box and the marked grasping points.

[0084] Preferably, in order to perform random pose transformation on the objects in the grasping scene in three-dimensional space, a transformation matrix is ​​constructed, which combines translation, rotation and scaling. Specifically, it includes:

[0085] In three-dimensional space, the position of an object is represented by a 4x4 homogeneous transformation matrix, which combines rotation, translation and scaling. The formula is as follows:

[0086]

[0087] Among them: R is a 3x3 rotation matrix, which represents the rotation of the object; t is a 3x1 translation vector, which represents the position of the object; the last line is [0,0,0,1], which makes the matrix in homogeneous coordinate form.

[0088] Translation transformation represents the change in the position of an object in three-dimensional space. Suppose we want to translate the object along the x, y, and z axes by t x ,t y ,t z units, the translation vector t can be expressed as:

[0089]

[0090] For random pose transformation, t x ,t y ,t z Randomly generated based on the scene's boundaries or the original position of the object. For example, randomly sample these values ​​within a certain range (such as [-1,1] meters).

[0091] The rotation matrix (R) corresponding to the quaternion (q = (w, x, y, z)) can be fully expressed as:

[0092]

[0093] For random pose transformations, randomly generate quaternions. Generate a random unit quaternion (expressed as (q = (w, x, y, z))) in the following way: Generate four random numbers (u_1, u_2, u_3) in the range ([0, 1]); Use the following formula to generate the quaternion:

[0094]

[0095] The quaternion (q) generated in this way is the unit quaternion and is used to construct a random rotation matrix.

[0096] If you need to scale the object, introduce a scaling factor. The scaling matrix S is a diagonal matrix with the following form:

[0097]

[0098] Among them (s x ,s y ,s z ) are the scaling factors along the x, y, and z axes. For random pose transformations, these scaling factors are randomly generated within a certain range.

[0099] In the above technical solution, the advantages of generating an image dataset through three-dimensional reconstruction include:

[0100] 1. Intuitiveness and Accuracy

[0101] Three-dimensional images are more intuitive:

[0102] 3D reconstruction can generate realistic 3D stereo images that provide richer spatial information and more intuitive visual effects than traditional 2D images; this enables observers to more easily understand the shape, structure, and spatial relationship of objects.

[0103] Improved accuracy: 3D reconstruction enables precise identification of defects and damage in objects.

[0104] 2. Multi-angle observation and analysis

[0105] All-round observation: 3D reconstructed images allow observers to observe and analyze from multiple angles and levels. This helps to discover details and features that are missed in 2D images.

[0106] Enhanced understanding: By observing from multiple angles, observers can gain a more comprehensive understanding of the shape, structure, and function of an object.

[0107] 3. Efficient Data Generation and Processing

[0108] Automated generation: 3D reconstruction technology can automatically generate image data sets, greatly improving the efficiency of data generation; it can obtain large amounts of high-quality image data more quickly for training and testing algorithms such as machine learning and deep learning.

[0109] Flexible data processing: The 3D reconstructed image data set can be easily scaled, rotated, translated, and so on to meet different analysis requirements.

[0110] In a specific embodiment, it also includes:

[0111] Based on the results of pose estimation, the grasping points are determined and corrected.

[0112] In a specific implementation scheme, the steps of extracting features and estimating the position and posture of the object to be grasped using the convolutional neural network model specifically include:

[0113] Importing the two-dimensional image of the object to be grasped into the convolutional neural network model, and extracting the corresponding confidence map and vector field;

[0114] The predicted translation and rotation of the object to be grasped are calculated using the extracted confidence map and vector field to obtain the result of the pose estimation.

[0115] Specifically, the process of importing the two-dimensional image of the object to be grasped into the convolutional neural network (CNN) model for pose estimation includes:

[0116] 1. Image preprocessing

[0117] Image input: First, the 2D image of the object to be grasped is input into the CNN model and the image is resized to the input size required by the model.

[0118] Normalization: In order to improve the convergence speed and performance of the model, the image is usually normalized so that its pixel values ​​are distributed in a standard range (such as 0 to 1 or -1 to 1).

[0119] 2. Feature extraction

[0120] Convolutional layer processing: The image passes through the convolutional layer of CNN, which contains multiple convolution kernels to extract local features in the image;

[0121] Pooling layer processing: The image passes through the pooling layer to reduce the dimension of the feature map while retaining important information;

[0122] Fully connected layer processing: The feature maps are flattened and passed through fully connected layers, which map the features to the desired output space.

[0123] The CNN model outputs two key components:

[0124] 1. Confidence map: This is a two-dimensional image in which each pixel value indicates the confidence that the location is a part of the object to be grasped (such as the center point or key point);

[0125] 2. Vector field: This is a two-dimensional image of the same size as the confidence map, but each pixel value is a vector representing the direction and size from the pixel position to a certain part of the object to be grasped (such as the center point or key point).

[0126] 3. Pose Estimation

[0127] Predicted translation: Use the confidence map to find the pixel position with the highest confidence, which corresponds to the center point or a key point of the object to be grasped. Then, combined with the information in the vector field, calculate the predicted translation from the image plane to the point in the actual physical space;

[0128] Predict the amount of rotation: By analyzing the direction and size of the vector field, the rotation angle of the object to be grasped is estimated; some form of integration or averaging is performed on the vector field to obtain the overall rotation direction.

[0129] Using a deep learning-based pose estimation algorithm for pose estimation has the following advantages:

[0130] High precision: Deep learning-based algorithms can learn the mapping relationship between complex features and poses of objects by training large amounts of data, thereby achieving high-precision pose estimation; deep learning models can capture subtle changes in images, such as the edges and textures of objects, which are crucial for pose estimation.

[0131] Robustness: Deep learning algorithms are highly adaptable to interference factors such as lighting changes, occlusion, and noise, and can work stably in complex environments. By training data under different scenarios and conditions, deep learning models can learn more generalized feature representations and improve the robustness of the algorithm.

[0132] Wide applicability: The pose estimation algorithm based on deep learning can be applied to different types of objects and scenes and has strong versatility; by adjusting the model structure and training data, the algorithm can be easily applied to new objects or scenes.

[0133] High degree of automation: Deep learning algorithms can automatically learn feature representation and pose estimation methods from data, reducing the workload of manual design and parameter adjustment; the algorithm has a high degree of automation, which can greatly improve the efficiency and accuracy of pose estimation.

[0134] Strong scalability: Deep learning-based algorithms can be combined with other technologies, such as sensor fusion and optimization algorithms, to further improve the performance of pose estimation. With the continuous development of deep learning technology, the performance and scope of application of the algorithm will continue to expand.

[0135] 4. Get the pose estimation result

[0136] Combined translation and rotation: The predicted translation and predicted rotation are combined to obtain a complete pose estimate of the object to be grasped in 3D space.

[0137] Post-processing: Perform post-processing steps such as smoothing, filtering, or denoising to improve the accuracy and stability of the pose estimate.

[0138] In a specific implementation scheme, based on the result of the pose estimation, the steps of determining and correcting the grasping point specifically include:

[0139] Based on the result of the pose estimation, an optimal grasping point of the object to be grasped is obtained, and a measurement translation amount of the depth camera is calculated;

[0140] The grasping safety distance correction is performed according to the predicted translation amount and the measured translation amount; if the correction is successful, the correction data is captured; if the correction fails, iterative optimization is performed.

[0141] Specifically, based on the result of the pose estimation, the optimal grasping point of the object to be grasped is determined, and the measured translation of the depth camera is calculated, including:

[0142] 1. Determine the optimal grab point

[0143] Analysis of pose estimation results:

[0144] First, the pose estimation results obtained by the deep learning algorithm include the position (expressed in three-dimensional coordinates) and posture (expressed in rotation matrix) of the object. The matrix of the three-dimensional reconstruction model is used to save calculation and improve the efficiency of the system.

[0145] Grab point selection:

[0146] According to the shape and size of the object and the characteristics of the grasping tool (such as a robotic arm), the optimal grasping point is selected based on the pose estimation results. This grasping point ensures the stability of the object and the effectiveness of the grasping tool during the grasping process.

[0147] 2. Calculate the measured translation of the depth camera

[0148] In-depth information acquisition:

[0149] A depth camera is used to capture the depth image of the object to be grasped, which provides the distance information from each point on the object surface to the camera.

[0150] Calculation of measured translation:

[0151] Combining the intrinsic parameters (such as focal length, optical center position, etc.) and extrinsic parameters (such as the position of the camera relative to the end of the robotic arm) of the depth camera, as well as the pixel coordinates in the depth image, the three-dimensional coordinates of a specific point on the object (such as the optimal grasping point) relative to the camera are calculated using the principle of triangulation.

[0152] Then, based on the current position and attitude of the camera, the measured translation from the camera to that point is calculated.

[0153] 3. Grasping safety distance correction

[0154] Comparison of predicted and measured translations:

[0155] The predicted translation calculated based on the pose estimation results is compared with the measured translation measured by the depth camera.

[0156] Safety distance correction:

[0157] Based on the comparison results, the position of the grasping point is fine-tuned to ensure that the robot arm or gripper maintains an appropriate safety distance from the object during the grasping process. This safety distance should be determined based on the material, shape, characteristics of the grasping tool, and specific requirements of the application scenario.

[0158] During the calibration process, it may be necessary to consider the motion state of the object (such as stationary, moving, etc.) and the motion trajectory planning of the grasping tool to ensure the smoothness and safety of the grasping process.

[0159] To perform a crawl:

[0160] After completing the safety distance correction, the robot arm or gripper moves to the grasping point according to the planned trajectory and speed and performs the grasping operation.

[0161] In the above technical solution, the beneficial effects produced are:

[0162] 1. Improve crawling accuracy

[0163] Precise positioning: Pose estimation can provide the precise position and posture information of an object in three-dimensional space, which enables the robot to accurately find the grasping point of the object to be grasped; through pose estimation, the robot can take into account the shape, size and posture of the object to select the most appropriate grasping position and angle.

[0164] Reduce errors: Traditional grasping methods may be affected by interference factors such as lighting, occlusion, and noise, resulting in inaccurate grasping point positioning; the grasping point determination method based on pose estimation can reduce the errors caused by these interference factors and improve grasping accuracy.

[0165] 2. Enhance crawling stability

[0166] Adaptability to different objects: Pose estimation can handle objects of different shapes, sizes, and materials, allowing the robot to select the best grasping strategy based on the characteristics of different objects; this helps maintain stable grasping performance when facing complex and changing objects.

[0167] Collision avoidance: Through pose estimation, the distance between the grasping point and the surrounding environment can be accurately calculated, thereby avoiding collisions with obstacles during the grasping process; this helps to protect the safety of the robot and the surrounding environment while improving the success rate of grasping.

[0168] 3. Improve crawling efficiency

[0169] Fast response: Pose estimation algorithms are usually computationally efficient and can provide accurate grasping point information in a short time. This enables the robot to respond quickly to grasping instructions and improve overall work efficiency.

[0170] Reduce repeated attempts: Traditional grasping methods may require multiple attempts to find a suitable grasping point, resulting in a waste of time and resources. However, the grasping point determination method based on pose estimation can accurately find the grasping point at one time, reducing the number of repeated attempts and improving grasping efficiency.

[0171] 4. Optimize crawling strategy

[0172] Intelligent planning: Combined with the results of posture estimation, the robot can intelligently plan the grasping path and angle to complete the grasping task in the best way. This helps reduce energy consumption and wear during the grasping process and increase the service life of the robot.

[0173] Adapting to dynamic environments: In a dynamic environment, the position and posture of an object may change. The grasping point determination method based on pose estimation can update the grasping point information in real time, allowing the robot to adapt to these changes and make corresponding adjustments.

[0174] In a specific implementation scheme, the convolutional neural network model includes a VGG deep convolutional neural network model.

[0175] The advantages of using the VGG deep convolutional neural network model include:

[0176] 1. The network structure is simple and consistent

[0177] The network structure of the VGG model is very simple. The entire network uses the same convolution kernel size and maximum pooling size, which makes the network structure consistent and easy to understand and implement. At the same time, since the size of the convolution kernel and pooling layer is fixed, the complexity of the model design is reduced.

[0178] 2. Increased depth improves performance

[0179] The VGG model successfully built a 16-19-layer deep convolutional neural network, proving that increasing the depth of the network can affect the final performance of the network to a certain extent. By increasing the number of network layers, the VGG model can learn more complex feature representations, thereby improving the accuracy of image recognition. In addition, the increase in depth also helps to improve the generalization ability of the model, enabling it to better adapt to different data sets and tasks.

[0180] 3. Small convolution kernels reduce parameters

[0181] The VGG model uses multiple consecutive 3x3 convolution kernels to build its network layer. This design allows the depth of the network to be increased without significantly increasing the number of parameters. Small 3x3 filters help capture local features, and a larger receptive field can be constructed by stacking. Compared with using large convolution kernels (such as 7x7 or 5x5), small convolution kernels can reduce the number of parameters and reduce the computational complexity of the model while maintaining or even improving the performance of the model.

[0182] 4. Strong transfer learning ability

[0183] The VGG model outperforms other models in multiple transfer learning tasks, such as GoogLeNet. This is due to the VGG model's powerful feature extraction capabilities, which enable it to adapt well to different data sets and tasks. Therefore, the VGG model is often used as the basic module of subsequent convolutional neural network models for feature extraction and transfer learning.

[0184] 5. Easy to implement and optimize

[0185] Since the network structure of the VGG model is simple and consistent, it is easy to implement and optimize. At the same time, since the VGG model has achieved excellent performance on multiple data sets and tasks, a large amount of optimization experience and skills have been accumulated. These experiences and skills can further improve the performance of the VGG model, making it more reliable and efficient in practical applications.

[0186] 6. Good scalability

[0187] The VGG model has good scalability and can be easily expanded to larger network sizes or more complex tasks. For example, a deeper network structure can be built by increasing the number of convolutional layers or fully connected layers, or the performance of the model can be further improved by introducing other advanced algorithms and technologies.

[0188] exist Figure 2 In the embodiment of the present application, a disordered object grasping system based on pose estimation is provided, comprising:

[0189] Dataset module, used to generate image datasets;

[0190] A model module, used to construct a convolutional neural network model based on the image dataset;

[0191] A posture estimation module is used to use the convolutional neural network model to perform feature extraction and posture estimation on the object to be grasped.

[0192] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0193] In a specific embodiment, it also includes:

[0194] The correction module is used to determine and correct the grasping points based on the results of pose estimation.

[0195] In a specific implementation scheme, the convolutional neural network model includes a VGG deep convolutional neural network model.

[0196] refer to Figure 3 The disordered object grasping system based on posture estimation includes a grasping robot, and the grasping robot includes an RGB-D camera 1, a grasping robot arm 2, a manipulator 3, a carrying box 4, a serpentine tube 5, a central controller 6, and a universal wheel 7.

[0197] Specifically, the universal wheel is connected to the carrying box through a hub motor, and is used to drive the carrying box to any specified position, so as to facilitate image acquisition of the object to be grasped.

[0198] The RGB-D camera is connected to one end of the serpentine tube, and the other end of the serpentine tube is connected to the carrying box through a rotating rod, which is used to adjust the position of the RGB-D camera. In a specific implementation, two sets of the serpentine tube and the rotating rod are provided to adjust the position of the RGB-D camera in both horizontal and vertical directions.

[0199] The central controller includes a carrying box control module and an image acquisition control module; the carrying box control module is used to control the movement and obstacle avoidance of the carrying box; the image acquisition control module is used to control the rotation of the rotating rod and the image acquisition of the object to be grasped.

[0200] Preferably, after the central controller controls the carrying box to reach a specified position through the hub motor, the central controller adjusts the position of the RGB-D camera in the horizontal and vertical directions through the serpentine tube and the rotating rod to achieve image capture of the object to be grasped.

[0201] An embodiment of the present application also provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device can implement any one of the methods for unordered object grasping based on pose estimation.

[0202] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0203] An embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor so that the computer-readable storage medium implements any one of the methods for unordered object grasping based on pose estimation.

[0204] In the above technical solution, an image data set is generated; a convolutional neural network model is constructed based on the image data set; and the convolutional neural network model is used to perform feature extraction and pose estimation on the object to be grasped, thereby improving the grasping accuracy and success rate.

[0205] Those skilled in the art will appreciate that the present application may be implemented as a system, method or computer program product.

[0206] Therefore, the present disclosure may be specifically implemented in the following forms, namely: it may be completely hardware, it may be completely software (including firmware, resident software, microcode, etc.), or it may be a combination of hardware and software, generally referred to herein as a "circuit", "module" or "system". In addition, in some embodiments, the present application may also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium may contain computer-readable program code.

[0207] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0208] Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art can change, modify, replace and modify the above embodiments within the scope of the present application. On this basis, a variety of replacements and improvements can be made to the present application, all of which fall within the scope of protection of the present application.

Claims

1. A method for unordered object grasping based on pose estimation, characterized in that: The following steps are involved: Generate image dataset; Based on the image dataset, construct a convolutional neural network model; The convolutional neural network model is used to extract features and estimate the position and posture of the object to be grasped.

2. The method for unordered object grasping based on pose estimation according to claim 1, characterized in that: Also includes: Based on the results of pose estimation, the grasping points are determined and corrected.

3. The method for unordered object grasping based on pose estimation according to claim 2, characterized in that: The steps of using the convolutional neural network model to extract features and estimate the position and posture of the object to be grasped specifically include: Importing the two-dimensional image of the object to be grasped into the convolutional neural network model, and extracting the corresponding confidence map and vector field; The predicted translation and rotation of the object to be grasped are calculated using the extracted confidence map and vector field to obtain the result of the pose estimation.

4. The method for unordered object grasping based on pose estimation according to claim 3, characterized in that: Based on the results of pose estimation, the steps of determining and correcting the grasping points include: Based on the result of the pose estimation, an optimal grasping point of the object to be grasped is obtained, and a measurement translation amount of the depth camera is calculated; The grasping safety distance correction is performed according to the predicted translation amount and the measured translation amount; if the correction is successful, the correction data is captured; if the correction fails, iterative optimization is performed.

5. The method for unordered object grasping based on pose estimation according to claim 4, characterized in that: The convolutional neural network model includes a VGG deep convolutional neural network model.

6. An object disorderly grasping system based on pose estimation, characterized in that: include: Dataset module, used to generate image datasets; A model module, used to construct a convolutional neural network model based on the image dataset; A posture estimation module is used to use the convolutional neural network model to perform feature extraction and posture estimation on the object to be grasped.

7. The object disorderly grasping system based on pose estimation according to claim 6, characterized in that: Also includes: The correction module is used to determine and correct the grasping points based on the results of pose estimation.

8. The object disorderly grasping system based on pose estimation according to claim 7, characterized in that: The convolutional neural network model includes a VGG deep convolutional neural network model.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method for unordered object grasping based on pose estimation as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the method for unordered object grasping based on pose estimation as described in any one of claims 1 to 5.

Images