Multimodal depth perception and grasping system based on transparent objects

Abstract

The application discloses a multi-modal depth perception and grasping system based on transparent objects, and relates to the field of robot operation, comprising a multi-spectral perception module, a depth correction module, a grasping posture generation module and a control module. The application comprehensively acquires visual information and thermal radiation information of the transparent object by combining two perception methods of RGB-D image and thermal imaging (TIR) image, and analyzes systematic errors of the depth map to detect error sources of the RGB-D camera and the TIR camera. The system adopts an encoder-decoder model as a depth correction core, the model extracts complementary features of the RGB-D image and the TIR image through a modal exclusive encoder, and utilizes a feature fusion module to complete feature alignment and integration, thereby effectively improving the depth estimation accuracy on the transparent surface. Meanwhile, a Bayesian optimization method is adopted to optimize hyperparameters of the depth correction model, through hyperparameter optimization and model fitting processing, the system effectively avoids overfitting and underfitting problems, and further improves the robustness of the depth correction model and the grasping accuracy of the transparent object.

Description

Technical Field

[0001] This invention relates to the field of robotic operations, specifically to a multimodal depth perception and grasping system based on transparent objects. Background Technology

[0002] Transparent objects (such as glasses and bottles) pose significant challenges to robot perception and manipulation due to their specular reflection and refraction properties. Traditional RGB-D sensors often struggle to capture accurate depth information when dealing with transparent objects or highly reflective surfaces. Reflections from transparent surfaces can cause noise or loss of depth data in the sensor, resulting in unreliable grasping and manipulation.

[0003] Existing technologies use TIR (thermal imaging) cameras to identify transparent objects. However, due to the different mounting positions of RGB and TIR cameras, there are spatial alignment errors in the acquired images and depth data. Without precise alignment, subsequent depth restoration and grasping pose estimation will be affected, leading to inaccurate final grasping posture. Precise alignment of sensor data is particularly crucial during robotic grasping.

[0004] Existing technologies, such as the invention patent with announcement number CN118700130B, are methods for estimating the pose of a handheld transparent object and controlling a robot's grasping ability. The method includes: acquiring RGB and depth images; an RGB-D feature encoder (Enco R) extracting RGB-D image features, a Deco R1 decoding conventional features, and a Deco R2 decoding geometric information and hand segmentation maps to assist in pose estimation; using pixel multiplication to filter transparent objects and background interference in the depth map, and extracting hand depth features through Enco D and Deco D; fusing and stacking the RGB and depth features; and a pose decoding module decoding the fused features to obtain the precise 6D pose information of the transparent object.

[0005] Existing technology, such as the invention patent with publication number CN114619447B, relates to a grasping method, grasping device, and robot, and is in the field of robot technology. The method is used to grasp an object to be grasped in an object container, the object container including an edge object located at the edge of the object container. The method includes: when the object to be grasped is an edge object, determining a target direction according to a specified direction of the pose coordinate system of the edge object; the angle between the target direction and the specified direction does not exceed a preset value, the specified direction is perpendicular to the central axis of the object container and points from the central axis to a direction away from the central axis, the central axis of the object container is perpendicular to the bottom of the object container; controlling the gripper to grasp the edge object with the target direction as the opening direction.

[0006] As can be seen from the above, in the field of robotic grasping, although pixel multiplication techniques are used to filter background interference, this technique may struggle to completely eliminate incomplete depth information when dealing with transparent objects. Furthermore, it primarily relies on a combination of RGB-D images and depth maps. While this method can extract conventional features and geometric information relatively well, in complex backgrounds or occluded environments, numerous interfering factors (such as reflections and shadows) may appear in the image, causing the model to be unable to accurately estimate the object's pose and affecting the system's robustness. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a multimodal depth sensing and grasping system based on transparent objects. To achieve the above objectives, this invention employs the following technical solution: a multimodal depth sensing and grasping system based on transparent objects, comprising:

[0008] The multispectral sensing module is used to capture RGB-D and TIR images of transparent objects. It performs spatial alignment of the RGB-D and TIR images through hand-eye calibration, and extracts features from the RGB-D and TIR images respectively to obtain RGB-D feature maps and TIR feature maps.

[0009] The depth correction module is used to fine-tune the depth correction model. It uses an encoder-decoder to process RGB-D feature maps and TIR feature maps to repair the depth map of transparent objects.

[0010] The grasping pose generation module is used to input the depth map of the transparent object, output the six-DOF grasping pose of the transparent object, and pass it to the control module.

[0011] The control module is used to obtain the robot's grasping path based on the six-degree-of-freedom grasping pose, control the robot's end effector to perform the grasping action, and adjust the grasping strategy based on real-time feedback.

[0012] Compared with the prior art, the embodiments of the present invention have at least the following beneficial effects:

[0013] (1) This invention provides a multimodal depth perception and grasping system based on transparent objects, which combines two perception methods: RGB-D images and thermal imaging (TIR) ​​images, and can comprehensively acquire visual and thermal radiation information of transparent objects. RGB-D images provide color, shape, and depth information of transparent objects, while TIR images capture the temperature distribution and temperature gradient characteristics of transparent objects. By fusing these two types of information, the system can perceive the morphological features of objects more comprehensively, especially the details of transparent objects that are easily overlooked in ordinary RGB-D images. Using this multimodal perception method can effectively improve the grasping accuracy of transparent objects in complex environments and overcome the information loss or bias problems that may be caused by traditional single sensors.

[0014] (2) This invention detects the error sources of RGB-D and TIR cameras through systematic error analysis of depth maps and performs automatic correction through a repair network. This process greatly improves the accuracy of depth maps, especially in the grasping of complex transparent objects, which can reduce the deviation introduced by sensor errors and enhance the stability and reliability of the system.

[0015] (3) This invention employs a Bayesian optimization method on the depth correction model, continuously adjusting the hyperparameter combination and performing feedback tuning based on the initial training results, gradually converging to the optimal hyperparameters. The Bayesian optimization method estimates the objective function through a surrogate model and intelligently selects the next hyperparameters for training using the acquisition function, which not only accelerates the model tuning process but also improves the model's performance. By combining hyperparameter optimization and model balancing fitting, the system can effectively avoid overfitting and underfitting, thereby improving the robustness of the depth correction model and ensuring the accuracy of grasping transparent objects and the depth map repair effect.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the above advantages at the same time. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the system modules of the present invention.

[0018] Figure 2 This is a schematic diagram of the logic flow of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In the description of this invention, it should be understood that the terms "opening", "upper", "lower", "thickness", "top", "middle", "length", "inner", "around", etc., which indicate orientation or positional relationship, are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this invention.

[0021] Please see Figure 1 The diagram shows the system modules, including a multispectral sensing module, a depth correction module, a grasping posture generation module, and a control module. This embodiment of the invention provides a multimodal depth sensing and grasping system based on transparent objects, specifically including:

[0022] like Figure 2 The diagram illustrates the logic flow of this invention. The multispectral sensing module captures visual information from different dimensions of the environment using RGB-D and TIR images. This information is then passed to the depth correction module for adjustment and optimization of the depth information. The feature extraction module extracts key features from the RGB-D and TIR images, which are then passed to the grasping posture generation module to generate a suitable grasping posture. The control module receives instructions for grasping path planning and makes real-time feedback adjustments to ensure grasping accuracy and efficiency. During this process, the gripper force is adjusted based on real-time feedback. The diagram demonstrates the system's workflow and the relationships between the modules. It clearly depicts how the various sub-modules within the system work together and indicates how data flows between the modules.

[0023] The multispectral sensing module is used to capture RGB-D and TIR images of transparent objects. It performs spatial alignment of the RGB-D and TIR images through hand-eye calibration, and extracts features from the RGB-D and TIR images respectively to obtain RGB-D feature maps and TIR feature maps.

[0024] It integrates an RGB-D camera and a TIR camera. The RGB-D camera is used to capture RGB-D images of transparent objects. The RGB-D images include color and depth information of the transparent objects.

[0025] TIR cameras are used to capture TIR images of transparent objects, which include thermal radiation information of the transparent objects.

[0026] It's important to note that RGB-D cameras typically have higher sampling rates (e.g., 30 fps or higher) and faster response times. However, the quality and accuracy of their depth maps are often affected by ambient light, especially in low-light or reflective environments. TIR cameras, on the other hand, generally have longer response times and lower frame rates (e.g., 15 fps or lower). This means that TIR images update less quickly than RGB-D images, and thermal imaging data experiences greater latency.

[0027] The relative positions of the RGB-D camera and the TIR camera are obtained by calibration board. The coordinate systems of the RGB-D camera and the TIR camera are aligned by hand-eye calibration. Based on the results of hand-eye calibration, the RGB-D image and the TIR image are registered. During the registration process, the TIR image is compensated for delay.

[0028] The registration process involves spatially aligning image data from different sources—in this invention, RGB-D images and TIR images—using mathematical algorithms to enable comparison or fusion within the same coordinate system. The specific steps are as follows:

[0029] The relative positions of the RGB-D camera and the TIR camera are obtained using a calibration board. The purpose of calibration is to determine the spatial pose relationship between the two cameras. Based on the relative position parameters obtained from hand-eye calibration, a transformation matrix is ​​calculated to map one image (such as a TIR image) into the coordinate system of the other image. Specifically, the registration method in this embodiment is based on feature point matching.

[0030] Multi-level feature extraction of RGB-D images is performed using convolutional neural networks (CNNs) to obtain RGB-D features, including color, edge, angle, and shape features of transparent objects, forming an RGB-D feature map. The formation process involves multi-level feature extraction of the RGB-D image from the input RGB-D image, which contains color and depth information of the transparent objects. The initial layers of the CNN extract low-level features such as edges, textures, and color patches through convolution operations. These low-level features are gradually transformed into more complex features, including the shape and angles of the transparent objects, through further processing by the CNN. Deep convolutional layers learn the geometric features of the transparent objects and their relationship with the viewpoint. Pooling layers downsample the feature map after the convolutional layers, reducing computation and improving generalization ability for transparent objects. Through multi-level feature extraction and fusion, an RGB-D feature map is generated.

[0031] It's worth noting that color information from RGB-D images is one of the most intuitive features of an object's appearance. Through multi-layer convolutional operations in convolutional neural networks, the color patterns and textures of an object can be extracted. For transparent objects, color features can also help distinguish surface color variations and translucent areas.

[0032] Temperature features, including the temperature distribution and temperature gradient features of the transparent object's surface, are extracted from TIR images using thermal radiation pattern analysis to form a TIR feature map. Thermal radiation information of the transparent object is obtained from the TIR image through thermal radiation pattern analysis. Based on the extracted temperature distribution and temperature gradient features, further processing and fusion are performed using a convolutional neural network (CNN) to form the TIR feature map. The TIR feature map contains temperature information about the transparent object's surface.

[0033] The surface of a transparent object may exhibit temperature variations due to ambient temperature, material properties, or reflective characteristics. TIR images can provide temperature information about a transparent object, including the temperature distribution across different regions of its surface. By analyzing the temperature information in TIR images, models can extract the temperature characteristics of the transparent object's surface, thereby helping to identify the object's boundaries, surface conditions, or other thermal response characteristics.

[0034] Delay compensation is performed on the TIR image during the registration process. The specific process is as follows:

[0035] Registration is achieved by matching feature points in RGB-D and TIR images. After registration, noise and artifacts introduced during the registration process are removed. Feature descriptors are used to match feature points in RGB-D and TIR images. The random sampling consensus algorithm is used to eliminate erroneous matching points. Finally, the transformation matrix is ​​estimated using the least squares method to achieve spatial alignment between RGB-D and TIR images.

[0036] During latency compensation, computing unit load parameters, including CPU utilization, CPU temperature, and CPU cache utilization, are retrieved from the database. If any computing unit load parameter exceeds a preset threshold, including CPU utilization threshold, CPU temperature threshold, and CPU cache utilization threshold, the GPU thread count, block size, and memory bandwidth are adjusted. Specifically, this includes:

[0037] Extract the computing unit load parameters that exceed the preset threshold of computing unit load parameters, and calculate the difference between the computing unit load parameters that exceed the preset threshold and the corresponding threshold of computing unit load parameters to obtain the load parameter difference value. Input the difference value into the preset lookup table of load parameter difference - control parameter set in the database to find the corresponding control parameter set, including the number of GPU threads, block size and memory bandwidth.

[0038] Simultaneously, the number of CPU threads is adjusted based on the number of CPU cores to enable multi-threaded processing. Specifically, the adjustment process involves multiplying the number of CPU cores by the CPU utilization rate and rounding down to obtain the number of CPU threads.

[0039] A feature descriptor is a representation method used in computer vision to describe the local features of keypoints in an image. Essentially, it is a vector or a set of values ​​used to encode image information surrounding the keypoint, ensuring that the keypoint's features can still be identified and matched even when the image is rotated, scaled, illuminated, or viewed from different angles. Feature descriptors describe local image information; each feature point typically corresponds to a corner, edge, or textured area in the image. The descriptor encodes the grayscale, color, gradient direction, texture, and other information of this area into a vector. Common feature descriptors include SIFT (Scale Invariant Feature Transform), SURF (Speed-Up Robust Feature Transform), and ORB (Oriented Fast and Rotated BRIEF). In the RGB-D image and TIR image registration scenario of this invention embodiment, SIFT is used as the feature descriptor.

[0040] In the registration process between RGB-D and TIR images, due to the differences in characteristics between different modalities, erroneous matching points exist in the initial matching. Therefore, RANSAC (Random Sample Consensus Algorithm) is needed for filtering. RANSAC randomly selects a subset of matching points, calculates candidate transformation matrices, and substitutes the remaining matching points into the matrix to calculate the projection error, thereby determining which points are inliers. This process iterates multiple times, selecting the matrix that generates the most inliers as a candidate each time, and finally eliminating erroneous matching points that were not identified as inliers to ensure the robustness and accuracy of the matching. Subsequently, based on the filtered inliers, the transformation matrix is ​​optimized using the least squares method, specifically including:

[0041] ;

[0042] Where E is the error function, Let be the coordinates of the i-th interior point in the TIR image. Let R be the coordinates of the i-th inlier in the RBG-D image, R be the rotation matrix, t be the translation vector, i be the inlier number, i = 1, 2, 3, ..., N, and N be the total number of inliers.

[0043] By minimizing the error function, the optimal rotation matrix R* and optimal translation vector t* are obtained, minimizing the distance between the interior points of the RGB-D image and their corresponding points in the TIR image after transformation. Based on the optimal rotation matrix R* and optimal translation vector t*, all points of the RGB-D image are mapped to the TIR image coordinate system, achieving spatial alignment of the two images. At this point, the two images are in the same coordinate system.

[0044] Removing noise and artifacts introduced during registration, specifically including:

[0045] In this embodiment of the invention, Gaussian filtering is used for noise reduction. Gaussian filtering is a convolution-based smoothing algorithm that effectively removes high-frequency noise from an image by convolving it with a Gaussian kernel. A key feature of Gaussian filtering is its ability to remove subtle noise while preserving overall structural information. Its mathematical formula is as follows:

[0046] ;

[0047] in, It is the weighting function of the Gaussian filter, used to weight the image during the Gaussian filtering process. Define the coordinates of the points in the image. The weights are given by x, which is the horizontal distance between the coordinate point in the image and the center point of the filter, and y, which is the vertical distance between the coordinate point in the image and the center point of the filter. The standard deviation of the Gaussian distribution is used to control the width of the filter. The larger the standard deviation, the wider the weight function of the Gaussian filter, and the stronger the smoothing effect, which is suitable for removing noise over a large area. The smaller the standard deviation, the narrower the weight function of the Gaussian filter, and the more concentrated the filtering effect, which is more localized and suitable for preserving image details and edges.

[0048] Based on Fourier transform, RGB-D and TIR images are converted from the spatial domain to the frequency domain. By analyzing high-frequency components, the image noise of the registered RGB-D and TIR images is obtained. The sliding window size is set, and the sliding window moves step by step on the registered RGB-D and TIR images respectively. The local noise standard deviation of the registered RGB-D and TIR images is calculated, and the local noise standard deviation of the registered RGB-D and TIR images is used as the estimated value of noise intensity.

[0049] The local noise standard deviations of the registered RGB-D and TIR images are input into preset lookup tables in the database. The lookup tables include RGB-D standard deviation lookup tables and TIR standard deviation lookup tables. The Gaussian filter standard deviations corresponding to the local noise standard deviations of the registered RGB-D and TIR images are found and input into the Gaussian filter for application.

[0050] Its calculation expression is as follows:

[0051] ;

[0052] in, For local noise standard deviation, Let be the pixel value of the pixel within the u-th sliding window. Let be the average pixel value within the sliding window, u be the sliding window number, u=1,2,3,...,M, and M be the number of sliding windows.

[0053] It should be noted that noise intensity typically varies across different image regions, especially in RGB-D and TIR images, where significant regional differences in noise distribution can occur due to variations in imaging principles and sensors. Estimating noise intensity using the local noise standard deviation allows for adaptive adjustment of the filter strength based on the noise characteristics of different regions.

[0054] Furthermore, obtaining the standard deviation of the Gaussian filter using a lookup table avoids the need for real-time derivation of the standard deviation for each calculation, thus accelerating the entire denoising process. The lookup table can be generated in advance during the training or preprocessing phase, and the corresponding Gaussian filter standard deviation value can be quickly retrieved by combining it with the local noise standard deviation, saving significant computational resources and ensuring computational efficiency.

[0055] The depth correction module is used to fine-tune the depth correction model. It uses an encoder-decoder to process RGB-D feature maps and TIR feature maps to repair the depth map of transparent objects.

[0056] Bayesian optimization is performed on the deep correction model, and training is carried out by randomly selecting hyperparameter combinations. Initial training results are obtained based on the validation set.

[0057] A surrogate model is constructed using Bayesian methods to estimate the expected output of the objective function under different hyperparameter combinations and obtain the corresponding confidence scores. Based on the surrogate model, Bayesian optimization selects hyperparameter combinations for training through a collection function and obtains advanced training results based on the validation set. The advanced training results are then fed back to the surrogate model to gradually converge and obtain excellent hyperparameter combinations. These excellent hyperparameter combinations are then applied to the deep correction model.

[0058] In Bayesian optimization, a surrogate model is used to estimate the relationship between the objective function (i.e., the model's performance) and hyperparameters. The surrogate model's task is to predict the performance of unknown hyperparameter combinations based on known hyperparameters and preliminary evaluation results. In this embodiment, a Gaussian process model is chosen for implementation. A Gaussian process is a parameter-free probabilistic model that describes the objective function by inputting a set of input-output pairs (hyperparameter-preliminary evaluation result pairs in this embodiment). The objective function includes a mean function and a kernel function. In initial training, the mean function is typically assumed to be 0. As Bayesian optimization progresses, the mean function is updated based on known data to predict the output of the objective function. The kernel function is a covariance function, used to describe the correlation between different input points, i.e., the correlation between hyperparameter combinations. In this embodiment, a radial basis function kernel is used.

[0059] The acquisition function is used to comprehensively consider the expected output and confidence level of hyperparameter combinations. Specifically, it calculates the expected output of each hyperparameter combination, subtracts it from the target output obtained by the objective function to obtain the expected gap, and couples the expected gap of each hyperparameter combination with the confidence level to obtain the expected estimate of each hyperparameter combination. The acquisition function then selects the hyperparameter combination with the largest expected estimate.

[0060] ;

[0061] in, For acquisition functions, Let b be the expected output of the b-th hyperparameter combination. The target output is the output of the objective function. Let be the confidence level of the b-th hyperparameter combination, where b is the hyperparameter combination number, b = 1, 2, 3, ..., B, and B is the total number of hyperparameter combinations.

[0062] Applying excellent hyperparameter combinations to deep correction models also includes performing balance fitting on these combinations, specifically including:

[0063] Balanced fitting refers to eliminating overfitting and underfitting in excellent hyperparameter combinations by adjusting hyperparameters to ensure that the model performs optimally on both training and testing data.

[0064] Overfitting refers to a model that performs very well on the training set but poorly on the test set or unseen data.

[0065] Underfitting refers to a model's inability to fully learn the features of the training data, resulting in poor performance on both the training and test sets.

[0066] The best hyperparameter combination is applied to the deep correction model, and the model is trained again to obtain the validation training results, including training loss and validation loss.

[0067] If the training loss decreases by a rate exceeding a preset threshold while the validation loss increases, the model is considered overfitted. It's important to note that a decrease in training loss means the loss value decreases, while an increase in validation loss means the loss value increases. A decrease in training loss indicates that the model is performing better on the training data, while an increase in validation loss indicates a decrease in the model's generalization ability; that is, the model's capabilities are not limited to the training data but it performs poorly when processing new data (the validation set).

[0068] If the decrease rate of both training loss and validation loss exceeds the preset decrease rate threshold, the model is determined to be underfitting.

[0069] It should be noted that for any loss function, the decrease ratio is obtained by subtracting the loss value of the previous training round from the loss value of the current training round, and then multiplying the ratio of the difference in loss values ​​to the loss value of the previous training round by the percentage.

[0070] Overfitting, specifically the elimination process includes:

[0071] The training loss reduction ratio is extracted and subtracted from a preset reduction ratio threshold to obtain the training loss reduction ratio difference. This difference is then input into a preset mapping set of training loss reduction ratio difference - regularization coefficient adjustment values ​​in the database for mapping matching to obtain the regularization coefficient adjustment value. Regularization coefficients include L2 and L1 regularization coefficients. In machine learning, the regularization coefficient is a hyperparameter used to add a regularization term to the loss function. The role of the regularization term is to penalize the complexity of the model (e.g., to prevent excessive weights), thereby reducing the risk of overfitting. Specifically, the regularization coefficient controls the weight of the regularization term; a larger regularization coefficient increases the model's penalty for complexity, while a smaller regularization coefficient allows the model to fit the data more freely. For example, if the current training loss reduction ratio is 35%, and the preset reduction ratio threshold is 30%, then the training loss reduction ratio difference is 5%, and the corresponding regularization coefficient adjustment value is 0.02. The adjusted regularization coefficient is 0.02 and is applied to the model training process.

[0072] Underfitting, specifically the elimination process includes:

[0073] The process extracts the percentage decrease in training loss and the percentage decrease in validation loss. These are then subtracted from preset percentage decrease thresholds to obtain the difference in loss percentage decrease and the difference in validation loss percentage decrease, respectively. The difference in loss percentage decrease is input into a preset mapping set in the database of the difference in loss percentage decrease minus the increase in network layer count for mapping matching, resulting in the increase in network layer count. Similarly, the difference in validation loss percentage decrease is input into a preset mapping set in the database of the difference in validation loss percentage decrease minus the increase in training epochs for mapping matching, resulting in the increase in training epochs. The increase in network layer count refers to adjusting the model's network depth based on the difference in training loss percentage decrease. Increasing the number of network layers usually makes the model more capable of learning and capturing more complex features, but it can also lead to overfitting. The increase in training epochs refers to determining whether to increase the number of training epochs based on the difference in validation loss percentage decrease (the decrease in validation loss). The number of training epochs represents the number of times the model is trained on the entire training set. Increasing the number of training epochs gives the model more time to adjust and optimize, but too many training epochs may lead to overfitting.

[0074] Repairing the depth map of transparent objects, specifically including:

[0075] The encoder network extracts the encodings of RGB-D and TIR images and feeds them into the decoder. The encodings of the RGB-D and TIR images are processed, and an upsampling operation is used to generate a high-resolution depth map of the transparent object. The depth map is used to represent the position information of the transparent object in three-dimensional space, and each pixel value of the depth map represents the straight-line distance from that position to the camera.

[0076] It's important to note that in the decoder, the encoded low-resolution feature map is gradually restored to a higher-resolution image through upsampling operations (such as deconvolution and bilinear interpolation). The key to this step is to progressively enhance the image's details and, through the network's learning capabilities, accurately fill in the depth information lost due to transparent objects or sensor limitations. Upsampling operations typically preserve the image's spatial structure while gradually restoring the depth value corresponding to each pixel in the depth map.

[0077] The grasping pose generation module is used to input the depth map of the transparent object, output the six-DOF grasping pose of the transparent object, and pass it to the control module.

[0078] Systematic error analysis was performed on the depth map to detect systematic errors caused by the RGB-D camera and the TIR camera. The error regions were calibrated, and a deep learning-based repair network was used to repair the depth map.

[0079] Systematic error refers to a deviation from the true value in each measurement or experiment due to defects or imperfections in the system or method. This deviation exhibits a certain regularity and predictability. Unlike random error, systematic error can be identified, analyzed, and corrected.

[0080] Feature extraction is performed on the depth map to obtain the morphological feature information of the transparent object, including the object contour, surface curvature, volume and surface normal vector. Based on the morphological feature information of the transparent object, multiple candidate grab points are generated. Specifically, the geometric center of the transparent object is calculated by the particle swarm optimization algorithm, and multiple candidate grab points near that location are generated.

[0081] After candidate grasping points are generated, the relative positions of these points and the relative pose of the object are analyzed. By calculating the spatial relationship between the candidate grasping points and the object's center, the pose of the transparent object is obtained. In this embodiment of the invention, the regression network is a deep learning-based pose estimation algorithm, which uses a regression network to obtain the rotation matrix (i.e., rotation angle) and displacement (i.e., position) of the transparent object. Six-degree-of-freedom pose refers to the object's position (translation) and orientation (rotation) in three-dimensional space. This information is fundamental for the robot to understand the object's position and orientation, helping to determine how to accurately grasp the object.

[0082] The control module is used to obtain the robot's grasping path based on the six-degree-of-freedom grasping pose, control the robot's end effector to perform the grasping action, and adjust the grasping strategy based on real-time feedback.

[0083] After obtaining the six-DOF grasping pose of the transparent object, the robot calculates the grasping path from the current state to the target object through inverse kinematics. The grasping path includes the starting position of the robot's end effector, the position of the target object, and the rotation angle. Inverse kinematics is an algorithm that calculates the robot's joint angles given the target position and orientation (including the rotation angle) of the robot's end effector.

[0084] After path planning is completed, the robot determines the opening and closing position of the grippers, the gripping angle, and the gripping force based on the object's shape, pose, and gripping point. For example, consider a transparent object, such as a glass bottle with a diameter of 6cm and a height of 20cm. Based on the bottle's diameter, the gripper opening and closing position is adjusted to 6-7cm. It should be noted that in this example, the grippers are parallel grippers with a bilaterally symmetrical structure. The opening size can be controlled by adjusting the distance between the grippers to ensure stable gripping of the object. The transparent glass bottle has no tilt, and the gripping angle of the grippers remains parallel to the side of the bottle. Since glass is a fragile material, to avoid damage, the maximum force on the transparent glass bottle is analyzed based on the gripping point position and physical mechanics principles, and the gripping force is set below this maximum force.

[0085] Control commands are generated by the robot's motion control system, instructing the end effector to move accurately along the planned path and perform the grasping operation.

[0086] During the gripping process, force sensors monitor whether the gripping action deviates, and automatically adjust the gripper force based on the deviation, including:

[0087] The gripping force is compared with the preset gripping force standard range. If it is within the gripping force standard range, it is considered to be without deviation. If the gripping force is greater than the upper limit of the gripping force standard range, it is considered to need to reduce the force. If the gripping force is less than the lower limit of the gripping force standard range, it is considered to need to increase the force.

[0088] In this embodiment of the invention, the gripper is driven by a servo motor. The output torque of the servo motor is adjusted by controlling the current or voltage. If the result requires increased force, the servo motor's current increase value is obtained by mapping and matching the weak gripping force value and the lower limit of the standard gripping force range into a preset mapping set of weak gripping force value and current increase value in the database, thereby increasing the gripper's closing force.

[0089] If the result is a reduction in the required force, then based on the gripping force excess value and the upper limit of the gripping force standard range, the current reduction value of the servo motor is obtained by mapping and matching the preset mapping set of gripping force excess value and current reduction value in the database, thereby reducing the closing force of the gripper.

[0090] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0091] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementation methods. Clearly, many modifications and variations can be made based on the content of this specification. The selection and detailed description of these embodiments in this specification are intended to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. Any modifications or variations that do not deviate from the structure of the invention or exceed the scope defined by the invention should fall within the protection scope of the invention.

Claims

1. A multimodal depth perception and grasping system based on transparent objects, characterized in that, include: The multispectral sensing module is used to capture RGB-D and TIR images of transparent objects. It performs spatial alignment of the RGB-D and TIR images through hand-eye calibration, and extracts features from the RGB-D and TIR images respectively to obtain RGB-D feature maps and TIR feature maps. The depth correction module is used to fine-tune the depth correction model. It uses an encoder-decoder to process RGB-D feature maps and TIR feature maps to repair the depth map of transparent objects. The fine-tuning depth correction model specifically includes: Bayesian optimization is performed on the deep correction model, and training is carried out by randomly selecting hyperparameter combinations. Initial training results are obtained based on the validation set. A surrogate model is constructed using Bayesian methods to estimate the expected output of the objective function under different hyperparameter combinations and obtain the corresponding confidence scores. Based on the surrogate model, Bayesian optimization selects hyperparameter combinations for training through a collection function and obtains advanced training results based on the validation set. The advanced training results are then fed back to the surrogate model to gradually converge to obtain excellent hyperparameter combinations. These excellent hyperparameter combinations are then applied to the deep correction model. The acquisition function is used to comprehensively consider the expected output and confidence level of the hyperparameter combination. Specifically, it calculates the expected output of each hyperparameter combination, subtracts it from the target output obtained by the objective function to obtain the expected gap, and couples the expected gap of each hyperparameter combination with the confidence level to obtain the expected estimate of each hyperparameter combination. The acquisition function selects the hyperparameter combination with the largest expected estimate from them. The grasping pose generation module is used to take the depth map of the transparent object as input, output the six-DOF grasping pose of the transparent object, and pass it to the control module. The control module is used to obtain the robot's grasping path based on the six-degree-of-freedom grasping pose, control the robot's end effector to perform the grasping action, and adjust the grasping strategy based on real-time feedback.

2. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The specific process for obtaining the RGB-D feature map and the TIR feature map is as follows: An integrated RGB-D camera and a TIR camera are used, wherein the RGB-D camera is used to capture RGB-D images of transparent objects, and the RGB-D images include color information and depth information of the transparent objects; The TIR camera is used to capture TIR images of transparent objects, and the TIR images include thermal radiation information of the transparent objects; The relative positions of the RGB-D camera and the TIR camera are obtained by calibration board. The coordinate systems of the RGB-D camera and the TIR camera are aligned by hand-eye calibration. Based on the results of hand-eye calibration, the RGB-D image and the TIR image are registered. During the registration process, the TIR image is compensated for delay. Multi-level feature extraction is performed on the RGB-D image to obtain the RGB-D features of the RGB-D image, including the color features, edge features, angle features and shape features of transparent objects, forming an RGB-D feature map; Temperature features are extracted from TIR images, including temperature distribution features and temperature gradient features on the surface of transparent objects, to form a TIR feature map.

3. The multimodal depth perception and grasping system based on transparent objects according to claim 2, characterized in that: The delay compensation for the TIR image during registration is performed as follows: Registration is performed by matching feature points in RGB-D and TIR images. After registration, noise and artifacts introduced during the registration process are removed. Feature descriptors are used to match feature points in RGB-D and TIR images. The random sampling consensus algorithm is used to eliminate erroneous matching points. The transformation matrix is ​​estimated by the least squares method to achieve spatial alignment between RGB-D and TIR images. During the latency compensation process, the computing unit load parameters are retrieved from the database to adjust the number of GPU threads, block size, and memory bandwidth. At the same time, the number of CPU threads is adjusted based on the number of CPU cores to enable multi-threaded processing.

4. The multimodal depth perception and grasping system based on transparent objects according to claim 3, characterized in that: The removal of noise and artifacts introduced during the registration process specifically includes: Image noise is acquired for the registered RGB-D and TIR images respectively. The sliding window size is set, and the sliding window is moved step by step on the registered RGB-D and TIR images respectively. The local noise standard deviation of the registered RGB-D and TIR images is calculated, and the local noise standard deviation of the registered RGB-D and TIR images is used as the estimated value of noise intensity. The local noise standard deviations of the registered RGB-D image and TIR image are input into a preset lookup table in the database. The lookup table includes an RGB-D standard deviation lookup table and a TIR standard deviation lookup table. The Gaussian filter standard deviations corresponding to the local noise standard deviations of the registered RGB-D image and TIR image are found and input into the Gaussian filter for application.

5. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The application of excellent hyperparameter combinations to the depth correction model also includes performing balance fitting on the excellent hyperparameter combinations, specifically including: The balanced fitting process refers to eliminating overfitting and underfitting in excellent hyperparameter combinations. The best hyperparameter combination is applied to the deep correction model, and the model is trained again to obtain the validation training results, including training loss and validation loss. If the training loss decreases by more than the preset decrease threshold and the validation loss increases, the model is determined to be overfitting. If the decrease rate of both training loss and validation loss exceeds the preset decrease rate threshold, the model is determined to be underfitting. The overfitting process specifically includes: Extract the training loss decrease ratio, subtract it from the preset decrease ratio threshold to obtain the training loss decrease ratio difference value, input the training loss decrease ratio difference value into the preset training loss decrease ratio difference - regularization coefficient adjustment value mapping set in the database for mapping matching to obtain the regularization coefficient adjustment value. The underfitting process specifically includes: Extract the training loss decrease ratio and the validation loss decrease ratio, respectively, and subtract them from the preset decrease ratio thresholds to obtain the loss decrease ratio difference and the validation decrease ratio difference. Input the loss decrease ratio difference into the preset mapping set of loss decrease ratio difference - network layer increase value in the database for mapping matching to obtain the network layer increase value. Input the validation decrease ratio difference into the preset mapping set of validation decrease ratio difference - training epoch increase value in the database for mapping matching to obtain the training epoch increase value.

6. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The depth map for repairing transparent objects specifically includes: The encoder network extracts the encodings of RGB-D and TIR images and sends them to the decoder. The encodings of the RGB-D and TIR images are processed, and an upsampling operation is used to generate a high-resolution depth map of the transparent object. The depth map is used to represent the position information of the transparent object in three-dimensional space, and each pixel value of the depth map represents the distance from that position to the camera.

7. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The output yields the six-DOF grasping pose of the transparent object, with the specific processing conditions being: Systematic error analysis was performed on the depth map to detect systematic errors caused by the RGB-D camera and the TIR camera, the error areas were calibrated, and a deep learning-based repair network was used to repair the depth map. Feature extraction is performed on the depth map to obtain the morphological feature information of the transparent object. Based on the morphological feature information of the transparent object, multiple candidate grab points are generated. After the candidate grab points are generated, the pose and position of the transparent object are analyzed, and the six-degree-of-freedom pose of the transparent object is obtained using a regression network.

8. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The process of obtaining the robot's grasping path based on the six-degree-of-freedom grasping pose and controlling the robot's end effector to perform the grasping action specifically includes: After obtaining the six-degree-of-freedom grasping pose of the transparent object, the robot calculates the grasping path from the current state to the target object through inverse kinematics. The grasping path includes the starting position of the robot's end effector, the position of the target object, and the rotation angle. After path planning is completed, the robot determines the opening and closing position of the gripper, the gripping angle, and the gripping force based on the object's shape, pose, and gripping point. Control commands are generated by the robot's motion control system, instructing the end effector to move accurately along the planned path and perform the grasping operation.

9. The multimodal depth perception and grasping system based on transparent objects according to claim 1, characterized in that: The method of adjusting the crawling strategy based on real-time feedback specifically includes: During the gripping process, force sensors monitor whether the gripping action deviates, and automatically adjust the gripper force based on the deviation, including: The gripping force is compared with the preset gripping force standard range. If it is within the gripping force standard range, it is considered to be without deviation. If the gripping force is greater than the upper limit of the gripping force standard range, it is considered to need to reduce the force. If the gripping force is less than the lower limit of the gripping force standard range, it is considered to need to increase the force.

Images