AI robot grabbing training method with close-range identification and calibration
By combining high-resolution cameras and depth sensors, using deep learning technology for multimodal recognition and grab point prediction, the problems of insufficient depth perception and adaptability in AI robot grasping technology are solved, and higher recognition accuracy and Grab accuracy, adaptability and robustness.
Patent Information
- Application Number
- CN202411307062.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-19
- Publication Date
- 2024-11-19
AI Technical Summary
The existing AI robot grasping technology relies on two-dimensional image recognition and cannot accurately sense the depth and shape of objects, resulting in insufficient grasping inaccuracy and adaptability, and lack of real-time calibration capabilities, which affects production efficiency and safety.
Using an AI robot grasping training method with close-range recognition calibration, images and depth information are collected through high-resolution cameras and depth sensors, multi-modal recognition model is built with deep learning technology, and three-dimensional object recognition and grasp point prediction are carried out, and Identification and calibration capabilities are optimized in real time through feedback mechanisms and online learning mechanisms.
It improves the accuracy of object recognition and the accuracy of grabbing, enhances the adaptability and robustness of the robot, and can accurately identify and grab objects of different types and shapes in a dynamic environment, meeting the needs of practical applications.
Smart Images

Figure CN118977249A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of automation control technology, and in particular to an AI robot grasping training method with close-range recognition and calibration. Background Art
[0002] With the continuous development of science and technology, AI robots are increasingly used in industrial production, logistics and warehousing, etc. Among them, the grasping operation of AI robots is a key task, which is of great significance for improving production efficiency and realizing automated operations.
[0003] In traditional robot grasping technology, there are often the following problems:
[0004] Early robots mainly relied on two-dimensional image recognition to perceive objects. However, two-dimensional images can only provide the plane information of objects and cannot accurately reflect the depth and true three-dimensional shape of objects. For example, when faced with objects with complex shapes or stacked together, it is difficult to accurately judge the actual position and relationship of objects based on two-dimensional images alone, which may cause inaccurate positioning problems when the robot grasps, affecting the success rate of grasping.
[0005] Changes in lighting conditions have a significant impact on 2D image-based recognition. Different lighting intensities and angles change the appearance of objects in the image, which may cause color distortion, shadow interference and other problems, thereby reducing the accuracy of object recognition. For example, in strong light, some features of the object may be overexposed and difficult to recognize, while in low light, the image may be blurred and affect the recognition effect.
[0006] Without accurate depth information, it is difficult for the robot to accurately control the force and position of the grasp. For example, when grasping fragile objects, if the distance between the object and the robot's hand cannot be accurately sensed, the object may be damaged due to excessive grasping force; when grasping irregularly shaped objects, if the depth of the grasping point cannot be accurately determined, the grasp may be unstable and the object may slip.
[0007] Traditional robot grasping methods may not be accurate enough when selecting grasping points. They do not fully consider factors such as the shape and weight distribution of the object, resulting in the grasping point being located in an unstable area of the object. For example, for an object whose center of gravity deviates from the geometric center, if an inappropriate grasping point is selected, the object may tilt or even fall during the grasping and handling process, affecting the robot's operating efficiency and safety.
[0008] Many existing robotic grasping systems use methods based on pre-set models and parameters, which are less adaptable when faced with dynamically changing environments and different types of objects. For example, in a logistics warehouse, the types and placement of goods often change, and traditional static models may not be able to adapt to new situations in a timely manner, resulting in an increased failure rate of robotic grasping.
[0009] When the robot encounters slight movement of objects or environmental interference in actual operation, it cannot adjust its perception and motion parameters in real time. For example, on a production line, if the position of an object changes slightly due to machine vibration or other reasons, if the robot cannot calibrate the recognition and grasping parameters in time, it may not be able to grasp the object accurately, affecting the smooth progress of the entire production process.
[0010] In order to solve the above problems, the present invention proposes an AI robot grasping training method with close-range recognition calibration. Summary of the invention
[0011] The purpose of the present invention is to propose an AI robot grasping training method with close range recognition calibration to solve the problems raised in the background technology:
[0012] Existing AI robots have limited recognition accuracy due to the limitations of two-dimensional images and the influence of environmental factors; lack of depth perception and improper selection of grasping points result in low grasping accuracy; limitations of static models and lack of real-time calibration capabilities lead to insufficient adaptability of robots.
[0013] In order to achieve the above object, the present invention adopts the following technical solutions:
[0014] The AI robot grasping training method with close range recognition calibration includes the following steps:
[0015] S1: Use high-resolution cameras to collect image samples of different environmental conditions and different object features, and use depth sensors to collect depth information corresponding to the object image information, and also collect the action sequence data of the AI robot;
[0016] S2: pre-process the collected images and depth data, annotate the collected images, and annotate the set of points to be grasped according to the object features;
[0017] S3: Based on deep learning technology, a recognition model that integrates multimodal data information is constructed to calibrate the recognition of three-dimensional objects and predict the grasping points;
[0018] S4: Set up precise calibration markers or objects so that the AI robot can repeatedly measure and calibrate the markers through vision and depth sensors;
[0019] S5: Conduct a comprehensive evaluation of the model and analyze the problems with the model based on the evaluation results. Continue to collect new data and perform incremental learning to continuously improve the performance of the model.
[0020] Preferably, the different environmental conditions in S1 include different lighting conditions, different angles and different distance ranges; and the different object features include different shapes, different colors and different materials.
[0021] Preferably, in S2, the shape, color and material of the object are annotated, and its outline in the image is outlined and checked. According to the shape, size and structure of each object, a set of points to be grasped is annotated, and the annotated data is divided into a training set, a verification set and a test set.
[0022] Preferably, in S3, an image feature extraction submodule and a depth information processing submodule are designed to extract image features and depth information features respectively, and finally the features output by the two submodules are fused, identified and predicted through a multimodal feature fusion module. A time series processing submodule is also provided to extract time series features and learn the time dependency between the actions of the AI robot. The time series processing submodule learns the time dependency between the action sequences of the AI robot based on a bidirectional Bi-LSTM network and obtains an output feature vector. In the actual grasping process, the grasping action is dynamically adjusted according to the state of the object output by the multimodal feature fusion module.
[0023] The image feature extraction submodule collects feature vectors of the input image based on the ResNet-50 model;
[0024] The depth information processing submodule generates point cloud data based on the depth information data, and based on the improved PointNet for the input point cloud data P = {p i ,i=1,Λ,N} downsample the point cloud through the farthest point sampling algorithm, where N is the number of point cloud data, and then filter the neighborhood points through the KNN algorithm:
[0025] Take sampling point o i As the center of the sphere, a spherical local area is constructed with r as the radius. The local area contains K neighboring points. The corresponding jth neighboring point o i,j The coordinates are converted into relative coordinates with the center point as the origin, and then MLP is used to extract the relative features p of the point cloud in the local area i,j :
[0026] d=||o i,j -o i ||
[0027] Where d is point o i,j With oi The Euclidean distance of
[0028] p i,j =MLP[αarctan(βd)]
[0029] Among them, α and β are adjustment parameters;
[0030] Calculate the characteristic deviation value σ of the neighborhood:
[0031]
[0032] Where n is the number of sampling point clouds;
[0033] Get the corrected point cloud feature p i ' ,j :
[0034]
[0035] Among them, a and b are learning parameters; ε is a stable parameter;
[0036] Based on another MLP, the local features f of the point cloud obtained by the local feature extractor are extracted. i :
[0037]
[0038] Based on the fully connected layer, local features are integrated, and then the corresponding attention score scoal is calculated through softmax i :
[0039] scoal i =softmax(f i )
[0040] The obtained attention score is weighted and summed with the input features to obtain the local aggregation features. Finally, the convolution function and batch normalization operation are used to obtain the output feature f of the module. out :
[0041]
[0042] The multimodal feature fusion module performs feature splicing operation on the output features of each submodule, and inputs the spliced features into the fully connected layer network for classification and grasping point prediction. When classifying objects, the output dimension of the last fully connected layer is the number of object categories C, and then the classification task is trained based on the focal loss function; when predicting grasping points, the output dimension of the last fully connected layer is the coordinate dimension D of the grasping points, and then the grasping point prediction is trained based on the mean square error loss function. Finally, the weights and biases of the fully connected layer are continuously adjusted through the back propagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function.
[0043] Preferably, in S4, based on the feedback mechanism, after the AI robot grasps an object, the grasping effect is fed back in real time through the force sensor and vision sensor installed on the hand of the AI robot.
[0044] Preferably, in S4, dynamic environmental changes in actual work are simulated, grasping training is performed under changing conditions, and the recognition and calibration capabilities of the AI robot are continuously updated.
[0045] Preferably, in S4, an online learning mechanism is introduced to update the model in real time when the AI robot encounters new objects or situations.
[0046] Compared with the prior art, the present invention provides an AI robot grasping training method with close-range recognition calibration, which has the following beneficial effects:
[0047] By fusing image information and depth information under different environmental conditions, the present invention enables the robot to obtain more comprehensive object features and maintain good recognition effects. It also combines various aspects of object information to mark annotation points, makes accurate predictions through deep learning models, and precisely controls the grasping force in combination with depth information. It also simulates dynamic environmental changes in actual work and adopts an online learning mechanism to update and optimize recognition calibration capabilities in real time, and adapts to object and environmental changes with data updates and incremental learning, thereby better meeting the needs of actual applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is a flow chart of the method mentioned in Example 1 of the present invention;
[0049] Figure 2 This is a processing flow chart of the depth information processing submodule mentioned in Example 1 of the present invention. DETAILED DESCRIPTION
[0050] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0051] The present invention fuses image information and depth information under different environmental conditions, so that the robot can obtain more comprehensive object features and maintain a good recognition effect; it also combines various aspects of object information to mark the annotation points, and makes accurate predictions through deep learning models, and also combines depth information to accurately control the grasping force; it also simulates the dynamic environmental changes in actual work and adopts an online learning mechanism to update and optimize the recognition calibration capabilities in real time, and adapts to object and environmental changes with data updates and incremental learning, so as to better meet the needs of practical applications. Specifically include the following contents.
[0052] Embodiment 1:
[0053] See also Figure 1-2 The present invention provides an AI robot grasping training method with close range recognition and calibration, comprising the following steps:
[0054] S1: The high-resolution camera collects image samples of different environmental conditions and different object features, and the depth sensor collects the depth information corresponding to the object image information, and also collects the action sequence data of the AI robot; the details are as follows:
[0055] First, install a high-resolution camera on the AI robot to collect images of various target objects under different lighting conditions (such as strong light, weak light, natural light, etc.), different angles (0 to 360 degrees, at certain angle intervals such as 15 degrees) and different distance ranges (for example, 10 cm to 50 cm, at 5 cm intervals). The target objects should cover different shapes (circular, square, triangle, etc.), different colors (common colors such as red, blue, green and mixed colors), and different materials (plastic, metal, wood, etc.). For each object, collect multiple image samples to increase the diversity and richness of the data, and collect at least 100 images of each object in different states.
[0056] Depth sensors (such as structured light sensors or time-of-flight sensors) are also used to simultaneously collect the depth information of the object, corresponding to the image information. The exact distance from the object to the AI robot is recorded with an accuracy of millimeters so that the AI robot can accurately perceive the three-dimensional position of the object.
[0057] The action sequence data of the AI robot from collecting features to executing and completing the grasping task is also collected, which corresponds to the time sequence of the grasping task, so as to facilitate the learning of the grasping action of the AI robot.
[0058] S2: Preprocess the collected images and depth data, annotate the collected images, and annotate the set of points to be grasped according to the object features; the details are as follows:
[0059] The collected images are labeled using professional data labeling tools to mark the category, outline and other information of the objects. For example, a round red plastic object is labeled as "round red plastic" and its outline in the image is accurately outlined. The labeling of each image must be checked by at least two professional labelers to ensure the accuracy of the labeling, and the labeling accuracy should reach more than 95%.
[0060] For each object, mark the appropriate grasping points according to its shape, size and structure. For example, for a rectangular box, mark its four corners or center positions and other possible stable grasping points. The marked grasping points should take into account the structure and grasping ability of the AI robot's hand to ensure that the AI robot can effectively grasp the object. The labeler needs to have relevant AI robot engineering knowledge to ensure the rationality of the labeling.
[0061] S3: Based on deep learning technology, a recognition model that integrates multimodal data information is constructed to calibrate the recognition of three-dimensional objects and predict the grasping points; the details are as follows:
[0062] An image feature extraction submodule and a depth information processing submodule are designed to extract image features and depth information features respectively. Finally, the features output by the two submodules are fused, identified and predicted through a multimodal feature fusion module. A time series processing submodule is also set to extract time series features and learn the time dependency between the actions of the AI robot. The time series processing submodule learns the time dependency between the action sequences of the AI robot based on a bidirectional Bi-LSTM network and obtains the output feature vector. In the actual grasping process, the grasping action is dynamically adjusted according to the state of the object output by the multimodal feature fusion module. The continuous grasping action of the AI robot is often not isolated, but has a temporal sequence and correlation. For example, when the AI robot grasps an object, it may first approach the object, then adjust the hand posture, and then implement the grasping action. The bidirectional Bi-LSTM can learn the continuity of this action sequence and understand the temporal sequence of each action. The output feature vector can encode this continuity information. This helps to more accurately describe the action state of the AI robot, rather than just looking at each moment of action in isolation. Therefore, in the actual grasping process, the AI robot's actions can be dynamically adjusted according to factors such as the object's position, shape, and weight. The bidirectional Bi-LSTM can capture the dynamic change patterns of these actions. For example, if an object moves suddenly, the AI robot needs to quickly adjust its actions to adapt to the new situation. The feature vector output by the bidirectional Bi-LSTM can reflect the characteristics of this action adjustment and provide a basis for subsequent decision-making.
[0063] The image feature extraction submodule collects the feature vector of the input image based on the ResNet-50 model; initializes the ResNet-50 model structure and configures related parameters, such as the size of the input image (usually set according to the actual size of the collected image, such as the common 224×224 or 256×256).
[0064] Load pre-trained weights (if there are available weights pre-trained on a large-scale image dataset), which can accelerate the convergence of the model and improve performance. If there are no suitable pre-trained weights, you can also randomly initialize the model weights, but it may take more training time and data to achieve good results.
[0065] Define the forward propagation process of the model, that is, pass the input image through each convolution layer and pooling layer of ResNet-50, and finally obtain the feature vector of the image. For example, in ResNet-50, after the image passes through a series of residual blocks (ResidualBlock) and average pooling layers, a feature vector with a dimension of 2048 (assuming that the final feature vector dimension is 2048) is obtained.
[0066] Reference Figure 2 The depth information processing submodule generates point cloud data based on the depth information data, and based on the improved PointNet for the input point cloud data P = {p i ,i=1,Λ,N} downsample the point cloud through the farthest point sampling algorithm, where N is the number of point cloud data, and then filter the neighborhood points through the KNN algorithm:
[0067] Take sampling point o i As the center of the sphere, a spherical local area is constructed with r as the radius. The local area contains K neighboring points. The corresponding jth neighboring point o i,j The coordinates are converted into relative coordinates with the center point as the origin, and then MLP is used to extract the relative features p of the point cloud in the local area i,j :
[0068] d=||o i,j -o i ||
[0069] Where d is point o i,j With o i The Euclidean distance of
[0070] p i,j =MLP[αarctan(βd)]
[0071] Among them, α and β are adjustment parameters;
[0072] Calculate the characteristic deviation value σ of the neighborhood:
[0073]
[0074] Where n is the number of sampling point clouds;
[0075] Get the corrected point cloud feature p i ' ,j :
[0076]
[0077] Among them, a and b are learning parameters; ε is a stable parameter;
[0078] Based on another MLP, the local features f of the point cloud obtained by the local feature extractor are extracted. i :
[0079]
[0080] Based on the fully connected layer, local features are integrated, and then the corresponding attention score scoal is calculated through softmax i :
[0081] scoal i =softmax(f i )
[0082] The obtained attention score is weighted and summed with the input features to obtain the local aggregation features. Finally, the convolution function and batch normalization operation are used to obtain the output feature f of the module. out :
[0083]
[0084] The multimodal feature fusion module performs feature splicing on the output features of each submodule, and inputs the spliced features into the fully connected layer network for classification and grasping point prediction. When classifying objects, the output dimension of the last fully connected layer is the number of object categories C, and the classification task is trained based on the focal loss function; when predicting grasping points, the output dimension of the last fully connected layer is the coordinate dimension D of the grasping points, and the grasping point prediction is trained based on the mean square error loss function.
[0085] Taking the three-layer fully connected network as an example, the input of the first fully connected layer is the concatenated feature vector F, with a dimension of h1+h2, where h1 is the dimension of the image feature vector; h2 is the dimension of the depth information feature; suppose this layer has m 01 neurons, the dimension of the weight matrix of this layer is w 01 =(h1+h2)×m 01 , bias vector b 01 The dimension is m 01 ; The second fully connected layer has m02 neurons, whose weight matrix w 02 The dimension is m 01 ×m 02 , bias vector b 02 The dimension is m 02 ; The output dimension of the third fully connected layer, that is, the last fully connected layer, is the number of object categories C or the coordinate dimension D of the grasping point, and the corresponding weight matrix w 03 The dimension is m 02 ×C or m 02 ×D, bias vector b 02 The dimension is C or D, and the output vector y is finally obtained:
[0086] y=w 03 (w 02 (w 01 F+b 01 )+b 02 )+b 03
[0087] Finally, a suitable loss function is used to train the classification task. The weights and biases of the fully connected layer are continuously adjusted through the back-propagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function and improve the classification accuracy.
[0088] S4: Set up precise calibration markers or objects so that the AI robot can repeatedly measure and calibrate the markers through vision and depth sensors; the details are as follows:
[0089] In the actual operating environment of the AI robot, set some precise calibration markers or objects, and let the AI robot repeatedly measure and calibrate these markers through vision and depth sensors. For example, place some small cubes with known precise size and position. The AI robot continuously adjusts its perception and motion parameters through multiple grasping and measurement, so that its perception error of distance and position is controlled within 1 mm.
[0090] Using the feedback mechanism, when the AI robot grasps an object, it will provide real-time feedback on the grasping effect through the force sensors and visual sensors installed on its hands. If the grasping position is inaccurate or the grasping force is inappropriate (the grasping force can be measured by the force sensor, and the appropriate grasping force should be determined according to the weight and material of the object. For example, for a 100-gram plastic object, the grasping force may be between 1 and 2 Newtons), the recognition calibration model will be fine-tuned based on the feedback information, and the model weights and parameters will be adjusted to improve the accuracy of the next grasp.
[0091] Simulate dynamic environmental changes in actual work, such as slight movement of objects, changes in light, etc. Let the AI robot perform grasping training under these changing conditions, and continuously update and optimize its recognition calibration capabilities. For example, use a fan to simulate the effect of breeze on the position of an object. The AI robot needs to be able to accurately recognize and grasp the object even when the object is blown by the wind. By constantly adapting to this dynamic change, the robustness of the AI robot in actual applications can be improved.
[0092] By introducing an online learning mechanism, the AI robot can update the model in real time when it encounters a new object or situation. For example, when the AI robot encounters an object of a shape or material that it has never seen before, it can grab and collect new data through a small number of attempts, and then update the model online so that it can quickly adapt to the new situation. The learning rate of online learning can be set slightly lower than that of initial training to avoid excessive forgetting of existing knowledge. It can generally be set to 1 / 10 to 1 / 5 of the initial learning rate.
[0093] S5: Conduct a comprehensive evaluation of the model, analyze the existing problems of the model based on the evaluation results, and continue to collect new data and conduct incremental learning to continuously improve the performance of the model. The details are as follows:
[0094] The trained model is fully evaluated using the test set, and the average evaluation value is calculated. The evaluation indicators include object recognition accuracy, grasping point prediction accuracy, grasping success rate, etc. Tests are also conducted in different scenarios, such as in a cluttered environment (multiple objects mixed together), in the case of partial occlusion of objects (part of the object is blocked by other objects), etc., to observe the performance of the AI robot and record its performance.
[0095] The traditional CNN model is taken as model 1, the model built in S3 is taken as model 2, and the model optimized in S4 is taken as model 3. The test results of the three models are evaluated. The specific evaluation results can be found in Table 1:
[0096] Table 1 Comprehensive evaluation results based on different evaluation indicators
[0097] Evaluation Metrics Model 1 Model 2 Model 3 Object recognition accuracy (%) 89.09 92.59 95.44 Grasp point prediction accuracy (%) 83.85 88.47 92.40 Crawl success rate (%) 87.08 91.45 94.86 Object recognition accuracy in cluttered environments (%) 87.61 91.17 94.29 Object recognition accuracy when the object is partially occluded (%) 88.29 91.12 94.51
[0098] As can be seen from the table, the test results based on the model in S3 are significantly higher than the test results of the ordinary CNN network. The test accuracy of the model optimized by S4 is also higher than that of model 3 in S3. The accuracy of AI robot grasping can be further improved through repeated measurement and calibration.
[0099] Based on the evaluation results, we can further analyze the problems of the model. For example, if the grasp point prediction accuracy is relatively low, we can improve the quality of grasp point annotation or adjust the structure of the regression model. If the object recognition accuracy is low, we may need to further optimize the image feature extraction part, add more convolutional layers or adjust the filter parameters.
[0100] Continuously collect new data and perform incremental learning to continuously improve the performance of the model. For example, let the AI robot collect new object images and depth information every once in a while (such as a week), then add this new data to the training set and retrain the model to adapt to changes in the environment and objects.
[0101] Apply the trained AI robots to actual production or life scenarios, such as parts grabbing in factories, cargo sorting in logistics warehouses, etc. In actual applications, closely observe the operation of the AI robots and record any problems and errors that occur.
[0102] For example, in a logistics warehouse, AI robots may encounter packages of various shapes and sizes, as well as different stacking methods. Record the success rate and problems encountered by AI robots when grabbing different types of packages, such as whether they can accurately identify the type and grabbing point of the package, and whether they can adapt to changes in the stacking height and position of the package.
[0103] According to the problems found in actual applications, the model and AI robot parameters are further optimized. For example, if the AI robot often fails to grasp certain objects of special shapes, more data can be collected for these objects for special training, or the structure and grasping method of the AI robot's hand can be adjusted.
[0104] Combined with the needs of actual application scenarios, the performance and adaptability of AI robots are continuously improved. For example, on a factory production line, if production efficiency needs to be improved, the motion path planning and grasping speed of the AI robot can be optimized while ensuring the accuracy and stability of grasping. Through continuous optimization and improvement, AI robots can better adapt to the needs of various actual application scenarios.
[0105] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. An AI robot grasping training method with close range recognition calibration, characterized in that: The steps include: S1: Use high-resolution cameras to collect image samples of different environmental conditions and different object features, and use depth sensors to collect depth information corresponding to the object image information, and also collect the action sequence data of the AI robot; S2: pre-process the collected images and depth data, annotate the collected images, and annotate the set of points to be grasped according to the object features; S3: Based on deep learning technology, a recognition model that integrates multimodal data information is constructed to calibrate the recognition of three-dimensional objects and predict the grasping points; S4: Set up precise calibration markers or objects so that the AI robot can repeatedly measure and calibrate the markers through vision and depth sensors; S5: Conduct a comprehensive evaluation of the model and analyze the problems with the model based on the evaluation results. Continue to collect new data and perform incremental learning to continuously improve the performance of the model.
2. The AI robot grasping training method with close range recognition calibration according to claim 1 is characterized in that: The different environmental conditions in S1 include different lighting conditions, different angles and different distance ranges; the different object features include different shapes, different colors and different materials.
3. The AI robot grasping training method with close range recognition calibration according to claim 1 is characterized in that: In S2, the shape, color and material of the object are annotated, and its outline in the image is outlined and checked. According to the shape, size and structure of each object, a set of points to be grasped is annotated, and the annotated data is divided into a training set, a verification set and a test set.
4. The AI robot grasping training method with close range recognition calibration according to claim 1, characterized in that: In the S3, an image feature extraction submodule and a depth information processing submodule are designed to extract image features and depth information features respectively, and finally the features output by the two submodules are fused, identified and predicted through a multimodal feature fusion module. A time series processing submodule is also provided to extract time series features and learn the time dependency between the actions of the AI robot. The time series processing submodule learns the time dependency between the action sequences of the AI robot based on a bidirectional Bi-LSTM network and obtains an output feature vector. In the actual grasping process, the grasping action is dynamically adjusted according to the state of the object output by the multimodal feature fusion module. The image feature extraction submodule collects feature vectors of the input image based on the ResNet-50 model; The depth information processing submodule generates point cloud data based on the depth information data, and based on the improved PointNet for the input point cloud data P = {p i ,i=1,Λ,N} downsample the point cloud through the farthest point sampling algorithm, where N is the number of point cloud data, and then filter the neighborhood points through the KNN algorithm: Take sampling point o i As the center of the sphere, a spherical local area is constructed with r as the radius. The local area contains K neighboring points. The corresponding jth neighboring point o i,j The coordinates are converted into relative coordinates with the center point as the origin, and then MLP is used to extract the relative features p of the point cloud in the local area i,j : d=o i,j -o i Where d is point o i,j With o i The Euclidean distance of p i,j =MLP[αarctan(βd)] Among them, α and β are adjustment parameters; Calculate the characteristic deviation value σ of the neighborhood: Where n is the number of sampling point clouds; Get the corrected point cloud feature p i ' ,j : Among them, a and b are learning parameters; ε is a stable parameter; Based on another MLP, the local features f of the point cloud obtained by the local feature extractor are extracted. i : f i =max[MLP(p i ' ,j )] Based on the fully connected layer, local features are integrated, and then the corresponding attention score scoal is calculated through softmax i : scoal i =softmax(f i ) The obtained attention score is weighted and summed with the input features to obtain the local aggregation features. Finally, the convolution function and batch normalization operation are used to obtain the output feature f of the module. out : The multimodal feature fusion module performs feature splicing operation on the output features of each submodule, and inputs the spliced features into the fully connected layer network for classification and grasping point prediction. When classifying objects, the output dimension of the last fully connected layer is the number of object categories C, and then the classification task is trained based on the focal loss function; when predicting grasping points, the output dimension of the last fully connected layer is the coordinate dimension D of the grasping points, and then the grasping point prediction is trained based on the mean square error loss function. Finally, the weights and biases of the fully connected layer are continuously adjusted through the back propagation algorithm and the stochastic gradient descent optimization algorithm to minimize the loss function.
5. The AI robot grasping training method with close range recognition calibration according to claim 1, characterized in that: In S4, based on the feedback mechanism, after the AI robot grasps an object, the grasping effect is fed back in real time through the force sensor and vision sensor installed on the hand of the AI robot.
6. The AI robot grasping training method with close range recognition calibration according to claim 5 is characterized in that: In S4, dynamic environmental changes in actual work are simulated, grasping training is performed under changing conditions, and the recognition and calibration capabilities of the AI robot are continuously updated.
7. The AI robot grasping training method with close range recognition calibration according to claim 6 is characterized in that: In S4, an online learning mechanism is introduced to update the model in real time when the AI robot encounters new objects or situations.
Citation Information
Cited By
3D vision automatic discharging system based on neural network adaptive learning
CN120588211A
