Grabbing track generation method and device, equipment, medium and program product

Abstract

The invention provides a grabbing track generation method and device, equipment, a medium and a program product, and relates to the technical field of robotics.The method comprises the steps that scene information of a grabbed target object is obtained, action track data and camera red, green and blue-depth RGB-D data are obtained based on the scene information, and the target object is grabbed according to the action track data. The RGB-D data comprises a first-view-angle RGB image and a second-view-angle RGB image; performing first processing on the first view angle RGB image to obtain a weighted RGB image, and performing second processing on the second view angle RGB image and the weighted RGB image to obtain fusion features; inputting the fusion feature and the noise sample trajectory into a diffusion model to predict the added random noise; and iteratively denoising from the initialized random noise according to a diffusion rule to generate a target grabbing track.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus, device, medium, and program product for generating grasping trajectories. Background Technology

[0002] Most current grasping trajectory generation algorithms use red-green-blue (RGB) or red-green-blue-depth (RGB-D) images as input. Some algorithms also use methods such as extracting the region of interest (ROI) of the target object or masking. However, in the single field of view of the above methods, the detection box may occlude the target object during the movement of the robotic arm, resulting in detection instability and thus low accuracy of grasping trajectory generation. Summary of the Invention

[0003] This application provides a method and apparatus for generating a capture trajectory, which can solve the problem of low efficiency in the use of existing resources.

[0004] To solve the above-mentioned technical problems, this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for generating a grasping trajectory, the method comprising:

[0006] The scene information for grasping the target object is obtained, and motion trajectory data and camera RGB-D data are obtained based on the scene information. The RGB-D data includes a first-view RGB image and a second-view RGB image.

[0007] A first processing step is performed on the first-view RGB image to obtain a weighted RGB image, and a second processing step is performed on the second-view RGB image and the weighted RGB image to obtain fusion features;

[0008] The fused features and noisy sample trajectories are input into the diffusion model to predict the added random noise;

[0009] The target capture trajectory is generated by iteratively denoising the initial random noise according to the diffusion rule.

[0010] Secondly, embodiments of this application provide a grasping trajectory generation device, the device comprising:

[0011] The acquisition module is used to acquire scene information of the target object to be captured, and acquire motion trajectory data and camera red-green-blue-depth RGB-D data based on the scene information. The RGB-D data includes a first-view red-green-blue RGB image and a second-view RGB image.

[0012] The processing module is used to perform a first processing on the first viewpoint RGB image to obtain a weighted RGB image, and to perform a second processing on the second viewpoint RGB image and the weighted RGB image to obtain fusion features;

[0013] An input module is used to input the fused features and noisy sample trajectories into a diffusion model to predict added random noise;

[0014] The generation module is used to iteratively denoise the initial random noise according to the diffusion rules to generate the target capture trajectory.

[0015] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and when the program or instructions are executed by the processor, they implement the steps of the grasping trajectory generation method as described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the capture trajectory generation method as described in the first aspect.

[0017] In this embodiment, scene information of the target object to be grasped is acquired, and motion trajectory data and camera RGB-D data (red-green-blue-depth) are obtained based on the scene information. The RGB-D data includes a first-view RGB image and a second-view RGB image. The first-view RGB image undergoes a first processing step to obtain a weighted RGB image, and the second-view RGB image and the weighted RGB image undergo a second processing step to obtain fusion features. The fusion features and noisy sample trajectories are input into a diffusion model to predict added random noise. Noise is iteratively removed from the initial random noise according to the diffusion rules to generate the target grasping trajectory. Thus, by linking dual-view RGB-D data, highlighting the target with the weighted image while maintaining compatibility with the robotic arm state, fusing gradient details and multi-view features to enhance the target description, and then performing iterative noise correction, background interference is reduced and feature alignment accuracy is improved, thereby enhancing the accuracy of the robot's grasping trajectory generation. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the grasping trajectory generation method provided in this application embodiment;

[0019] Figure 2 This is a schematic diagram of the structure of the grasping trajectory generation device provided in the embodiments of this application;

[0020] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application provides a method, apparatus, device, medium, and program product for generating a capture trajectory. The embodiments of this application will be described in detail below with reference to the accompanying drawings and specific embodiments and application scenarios.

[0023] Please see Figure 1 , Figure 1 A flowchart illustrating a method for generating a capture trajectory, as provided in this application embodiment, is shown in the figure. The method includes:

[0024] Step 110: Obtain scene information for grasping the target object, and obtain motion trajectory data and camera RGB-D data based on the scene information. The RGB-D data includes a first-view RGB image and a second-view RGB image.

[0025] The aforementioned scene information refers to environmental information related to the robot's performance of the grasping task, including the placement of the target object, the distribution of surrounding obstacles, ambient lighting conditions, and the dimensions of the workspace. This information serves as the basis for subsequent data acquisition and trajectory planning. The aforementioned motion trajectory data can be understood as the motion trajectory data corresponding to the robot completing one effective grasping action. It may include key parameters such as the rotation angles of each joint of the robotic arm, movement speed, movement time, and the position coordinates of the end effector, serving as a reference benchmark for model training and trajectory optimization.

[0026] Red-green-blue-depth (RGB-D) data contains both red, green, and blue color information and depth information. This data can provide robots with comprehensive information support on the visual features and spatial position of target objects.

[0027] The aforementioned first-person perspective RGB images refer to color images captured from a third-party, independent perspective (not from a camera mounted on the robotic arm), and are a core component of RGB-D data. Camera RGB-D data collected through hand-held controls or playback of preset trajectories can serve as the basis for target localization and segmentation: the target object detection box is obtained using the Qwen3-VL-32B model, the target segmentation region is obtained and tracked using the SAM2 segmentation model, and then a weighted image is generated by dot product with the mask image. This image highlights the target region while also accommodating the robotic arm's state in non-target areas, providing accurate target localization data for feature extraction.

[0028] The aforementioned second-view RGB image refers to a color image taken from a close-up view (close to the gripping execution end) of a camera mounted on a robotic arm. It can provide detailed feature support for the target object. For example, gradient information (including direction and magnitude) can be extracted using the Sobel algorithm, and scene-adaptive image features can be extracted using the dinov2 model. The gradient information can enhance the description of the target edge and contour. After being fused with features from other perspectives, it can improve the accuracy of subsequent trajectory generation and can solve the problem of insufficient feature alignment in traditional algorithms to a certain extent.

[0029] Step 210: Perform a first processing on the first viewpoint RGB image to obtain a weighted RGB image, and perform a second processing on the second viewpoint RGB image and the weighted RGB image to obtain fusion features;

[0030] The first processing described above can be understood as a series of processing operations performed on the first-view RGB image with the goal of obtaining a weighted RGB image. This may include key steps such as object detection, segmentation, tracking, and weighted calculations, which can highlight the features of the target object and suppress background interference. The aforementioned weighted RGB image refers to the image obtained by performing region weight allocation processing on the original RGB image, that is, assigning higher weights to the region where the target object is located to enhance its feature representation. The second processing described above refers to a series of processing operations performed on the second-view RGB image and the weighted RGB image with the goal of obtaining fused features. This may include steps such as feature extraction, feature fusion, and feature enhancement, which can integrate effective information from multiple source images to form comprehensive features with strong representational capabilities.

[0031] Step 310: Input the fused features and noisy sample trajectories into the diffusion model to predict the added random noise;

[0032] The aforementioned fusion feature refers to a comprehensive feature obtained by integrating features extracted from multiple image sources, including first-view related features, second-view related features, and gradient related features. This feature can comprehensively reflect the visual attributes, spatial relationships, and detailed information of the target object, providing core data support for subsequent rule generation. The aforementioned noise correction model can be understood as a deep learning model with noise prediction and noise correction functions. Its core capability is to learn and output rules that can correct invalid noise and retain valid trajectory information based on the input fusion features, providing a decision basis for trajectory generation.

[0033] The aforementioned noisy sample trajectories can be understood as training samples formed by adding random noise to the original valid motion trajectory data. They can be used to allow the diffusion model to learn the mapping relationship between "noise and valid trajectory". They can also be used as one of the inputs to the diffusion model. In conjunction with the fusion features, they can help the model learn to "identify invalid noise in the trajectory", laying the foundation for subsequent denoising and generating more accurate trajectories.

[0034] The aforementioned diffusion model is a deep learning model based on the logic of "noise prediction-iterative denoising." It can learn the correlation between fused features and effective trajectories and guide trajectory generation. During training, the model parameters can be optimized based on the fused features and the predicted random noise from noisy sample trajectories. During inference, the iterative denoising of the initial random noise can be guided by the fused features, ultimately generating effective trajectories.

[0035] The aforementioned random noise can be a set of random numerical values ​​without any valid trajectory information, such as random vectors that conform to a Gaussian distribution. It can be understood as the "initial input carrier" for the trajectory generated by the diffusion model.

[0036] Step 410: Iteratively remove noise from the initial random noise according to the diffusion rule to generate the target capture trajectory.

[0037] The aforementioned diffusion policy can be understood as the "noise correction and trajectory generation logic" learned by the diffusion model through training. It can include key parameters such as the direction, magnitude, number of iterations, and gradient descent step size for noise removal. In this implementation, the diffusion policy is the "effective trajectory generation rule" learned by the diffusion model from "noisy sample trajectories" and "fusion features," which can be used to guide the initialization of random noise to gradually transform it into a precise grasping trajectory.

[0038] The aforementioned initial random noise refers to randomly generated raw data that does not contain valid trajectory information. Essentially, it is a set of randomly distributed numerical values ​​that serve as the initial input for trajectory generation and are gradually transformed into valid trajectories through subsequent corrections.

[0039] The iterative denoising process described above refers to repeatedly correcting the initial noise according to the initial noise correction rules. Each correction removes invalid components from the noise, enhances valid trajectory information, and gradually reduces noise interference until a precise trajectory that meets the requirements is generated. The target grasping trajectory described above refers to the precise motion trajectory required for the robot to complete the grasping task. This trajectory includes the motion parameters of the robotic arm from its initial position to the target object's position, completing the grasping action, and returning to the designated position, ensuring that the robot can complete the grasping operation efficiently and accurately.

[0040] In the grasping trajectory generation method implemented in this application, the target description is enhanced by dual-view RGB-D data linkage processing, weighted image highlighting and compatibility with the robotic arm state, fusion of gradient details and multi-view features, and then noise correction and iterative denoising, which reduces background interference and improves the accuracy of feature alignment, thereby improving the accuracy of robot grasping trajectory generation.

[0041] Optionally, the first process includes:

[0042] The target object detection box is obtained based on the first viewpoint RGB image, the detection box is input into the segmentation model to obtain the target segmentation region, and the first viewpoint RGB image is continuously segmented by the tracking algorithm of the segmentation model;

[0043] Based on the target segmentation region and the first-view RGB image, a weighted RGB image is obtained.

[0044] In this embodiment, the target object detection box is used to select the rectangular area of ​​the target object in the first-view RGB image. It can accurately identify the position and range of the target object in the image, and can provide a clear positioning basis for subsequent segmentation operations.

[0045] The aforementioned segmentation model refers to a deep learning model with image segmentation capabilities. It can separate target objects from the background in an image and extract specific regions containing only the target object. This model can be focused on single-target segmentation or support multi-target segmentation. The target segmentation region mentioned above is the image region containing only the target object separated from the first-view RGB image after processing by the segmentation model.

[0046] The aforementioned tracking algorithm refers to an algorithm that works in conjunction with a segmentation model to continuously lock onto a target object. It can track based on the visual feature correlation of the target object or based on the positional change pattern of the target object, ensuring that the same target object is always accurately segmented in subsequent consecutive first-view RGB images.

[0047] The aforementioned weighted RGB image refers to the image obtained by assigning and calculating regional weights to the first-view RGB image. This allows for assigning higher weights to the target segmentation region to enhance the expression of target features, and assigning lower weights to non-target segmentation regions to reduce background interference, thereby improving the image's anti-interference performance and target feature recognition.

[0048] In some alternative implementations, the target object detection box can be obtained based on the first-view RGB image. The detection box can be obtained using the Qwen3-VL-32B model or other visual models with high detection accuracy, as long as the target object can be accurately located in the image.

[0049] After obtaining the target object detection box, the detection box can be input into the segmentation model to obtain the target segmentation region. The segmentation model can be the SAM2 segmentation model, or other deep learning models that support detection box-guided segmentation, to ensure that the target object and the background can be accurately separated from the range specified by the detection box.

[0050] To ensure consistency in target segmentation across subsequent images, a tracking algorithm from the segmentation model can be used to continuously segment the first-person perspective RGB images. This tracking algorithm can be feature-matching based, continuously locking onto the target object by comparing visual features of the target object in previous and subsequent frames; or it can be motion-prediction based, continuously segmenting by predicting the target object's trajectory. Regardless of the tracking algorithm used, the goal is to ensure stable output of the segmented target region even under conditions of changing ambient lighting or slight target displacement.

[0051] In the grasping trajectory generation method implemented in this application, the target is accurately located by detection boxes, and continuous segmentation is carried out by segmentation model and tracking algorithm. Weighted RGB image is generated by weighted processing, which highlights the target area, is compatible with the state of the robotic arm, reduces background interference and improves detection stability.

[0052] Optionally, the second process includes:

[0053] First-view features are extracted from the weighted RGB image, gradient information and second-view image features are extracted from the second-view RGB image, gradient features are extracted from the gradient information, and the first-view features, second-view image features and gradient features are fused to obtain fused features.

[0054] The fused features are input into the noise correction model to obtain an initial noise correction rule that matches the fused features.

[0055] The gradient information mentioned above refers to the information extracted from the second-view RGB image that reflects the changes in the edges and details of the object. It can include the gradient direction and gradient magnitude. The gradient direction is the direction of the grayscale change of the image pixels, and the gradient magnitude is the intensity of the change. This information can provide a more accurate description of the target object and help improve the grasping accuracy.

[0056] The aforementioned second-view image features refer to data extracted from second-view RGB images that characterize their core visual attributes. These features encompass key information such as the object's color distribution, texture structure, and shape contours, and are an important component of multi-source feature fusion, supplementing the information blind spots of the first-view image. First-view features refer to feature data extracted from weighted RGB images that enhance the target and weaken interference. These features retain the core visual information of the target object while also incorporating relevant state information from non-target regions, exhibiting strong anti-interference capabilities. Gradient features refer to feature data further extracted from gradient information that possesses strong representational capabilities. They are a condensed expression of gradient information, highlighting the edge details and contour features of the target object, providing precise detail support for feature fusion.

[0057] The aforementioned fusion features refer to comprehensive feature data obtained by integrating first-view features, second-view image features, and gradient features. This data can comprehensively reflect the visual attributes, spatial relationships, edge details, and other multi-dimensional information of the target object, avoiding the limitations of single features and providing a high-quality data foundation for rule generation.

[0058] The aforementioned noise correction model refers to a deep learning model with noise prediction, correction, and rule generation functions. It can learn the grasping patterns of target objects based on the fusion features of the input and output specific rules that can guide trajectory optimization.

[0059] In the grasping trajectory generation method implemented in this application, by extracting and fusing multi-view features and gradient features, a more accurate input is provided to the noise correction model, generating reliable noise correction rules, which helps to improve the anti-interference ability and accuracy of grasping trajectory generation.

[0060] Optionally, extracting the first viewpoint features includes using a visual language model to encode the features of the weighted RGB image and outputting first viewpoint features with uniform dimensions.

[0061] The extraction of second-view image features includes using a self-supervised visual basic model to extract features from the second-view RGB image and outputting second-view image features with scene adaptability.

[0062] The extraction of the gradient features includes using a Transformer network with rotational position encoding to perform feature transformation on the gradient information, enhancing the spatial correlation of the gradient features, and outputting the gradient features.

[0063] The aforementioned visual language model can be understood as a deep learning model that combines visual information processing and language feature encoding capabilities. Its core advantage lies in its ability to transform the visual features of an image into structured, fixed-dimensional feature vectors, while also possessing strong cross-modal representation capabilities and adapting to multi-source feature fusion scenarios. Feature encoding refers to the process of transforming raw image features into structured feature vectors that can be efficiently processed by computers. This process unifies feature dimensions, enhances effective information, and reduces redundant interference, giving features stronger representational capabilities and compatibility.

[0064] The aforementioned self-supervised visual foundation model refers to a deep learning model that does not rely on a large amount of manually labeled data and completes training by autonomously learning the inherent rules and structural features of images. Its core advantages are strong generalization ability and scene adaptability, enabling it to stably extract effective visual features in different grasping scenarios. The aforementioned gradient features refer to features derived from gradient information that enhance spatial correlation. The aforementioned rotational position encoding refers to an encoding method used to improve the representation ability of spatial information. By rotating and transforming the positional information of features, the spatial correlation between features is strengthened, enabling the model to more accurately capture the spatial structure of the target object.

[0065] The aforementioned Transformer network refers to a deep learning network built on a self-attention mechanism. It can efficiently capture long-distance dependencies between features, achieve deep integration and enhancement of information through feature transformation, and adapt to the spatial correlation enhancement needs of gradient features.

[0066] The aforementioned feature transformation refers to the process of transforming the original gradient information into gradient features with strong representational capabilities. This can include operations such as spatial correlation enhancement and dimensionality optimization, so that the gradient information can be transformed from the original data form into a feature form that is suitable for subsequent fusion processing.

[0067] In the grasping trajectory generation method of this application, features are extracted by a visual language model, a self-supervised visual basic model and a Transformer network with rotation position encoding, respectively, so as to achieve unified feature dimensions, scene adaptation and enhanced spatial correlation, providing high-quality input for fusion, thereby improving the anti-interference and accuracy of grasping trajectory generation.

[0068] Optionally, obtaining a weighted RGB image based on the segmented region and the first-view RGB image includes:

[0069] A mask image is generated based on the target segmented region, wherein the weight value of the segmented region is a first preset value, the weight value of the non-segmented region is a second preset value, the first preset value is greater than the second preset value, and the second preset value is greater than 0;

[0070] The mask image is multiplied by the first-view RGB image to produce the weighted RGB image.

[0071] The mask image mentioned above can be understood as an image generated based on the target segmentation region for weight allocation. Its pixel value can correspond to the weight of each region in the original image, and can clearly distinguish the weight difference between the target object region and the non-target region. It is the core intermediate carrier for realizing weighted operation.

[0072] For example, firstly, based on the target segmentation region, the target object region and non-target region in the image are clearly distinguished to ensure the targeted weight allocation. Then, corresponding weight values ​​are assigned to the two types of regions. The weight value for the target segmentation region is a first preset value, and the weight value for the non-segmentation region is a second preset value, which must be greater than 0. The second preset value can be 1.0, which preserves the original visual features of the target object and ensures that the target information is not weakened; or it can be 0.9, which strengthens the target features while moderately balancing computational complexity, as long as it meets the requirement of being greater than the second preset value. The second preset value can also be 0.3, which effectively reduces background interference while fully preserving key information such as the robotic arm state in the non-target region; or it can be 0.2, which further suppresses interference while maintaining basic information in the non-target region, as long as it meets the requirement of being greater than 0 and less than the second preset value. Following the above weight rules, the pixel values ​​of the target segmentation region and the non-target segmentation region can be set to the second preset value to generate a complete mask image.

[0073] The aforementioned second preset value refers to the weight value set for the target segmentation region. It can be used to enhance the feature representation of the target object, ensuring that the core visual information of the target object receives priority in subsequent processing. The value must be greater than the second preset value to reflect the weight priority. The aforementioned second preset value refers to the weight value set for the non-target segmentation region. It can appropriately retain relevant information of the non-target region, be compatible with the state observation of non-target objects such as robotic arms, and at the same time, it can reduce background interference by setting it lower than the second preset value. The value must be greater than 0 to avoid completely losing information of the non-target region.

[0074] The dot product operation mentioned above refers to the operation of multiplying the mask image and the first-view RGB image pixel by pixel. This operation can accurately apply the weights of the mask image to each pixel of the original RGB image, thereby achieving precise implementation of feature enhancement of the target area and interference suppression of the non-target area.

[0075] For example, the generated mask image can be dot-producted with the original first-view RGB image. During the operation, the weight value of each pixel in the mask image is multiplied pixel-by-pixel with the color value of the corresponding pixel in the first-view RGB image. This strengthens the color value of pixels in the target segmentation region by a second preset value, while moderately preserving the color value of pixels in non-target regions. The dot-product operation can be implemented using element-by-element multiplication at the pixel level to ensure accurate weight allocation; alternatively, it can be implemented using batch pixel grouping to improve processing efficiency while maintaining accuracy. Through the dot-product operation, the core visual features of the target object, such as color, texture, and shape, can be effectively highlighted, while reducing interference from the background environment. At the same time, key information such as the state of the robotic arm in non-target regions is fully preserved. The final output is a weighted RGB image that combines anti-interference capability and information integrity, laying a solid foundation for the accurate extraction of subsequent second-view features.

[0076] In the grasping trajectory generation method of this application, a mask image is generated by setting differential weights and then dot productd with the first-view RGB image. This highlights the target area while retaining effective information in the non-target area, reduces background interference, and lays a solid foundation for subsequent accurate feature extraction.

[0077] Optionally, the fusion processing of the first viewpoint features, the second viewpoint image features, and the gradient features to obtain fused features includes:

[0078] The first viewpoint features, the second viewpoint image features, and the gradient features are merged into a feature vector with unified dimensions to obtain the first fused feature.

[0079] The first fused feature is input into the Transformer network for deep feature enhancement processing to obtain the fused feature.

[0080] In this implementation, the merging method can employ a concatenation operation, sequentially connecting the numerical sequences of the three feature types in a preset order to form a feature vector with unified dimensions. This method fully preserves the original information of each feature type, ensuring no information loss. Alternatively, a weighted summation merging method can be used. Different weight coefficients are assigned to each feature type based on its importance, and then the weighted features are summed to achieve preliminary feature optimization and integration. The weight coefficients can be flexibly adjusted according to the accuracy and anti-interference requirements of the actual capture scenario. Regardless of the merging method used, the core objective is to achieve dimensional unification of the three feature types, eliminate dimensional differences between different features, and ensure the smooth progress of subsequent deep processing.

[0081] The aforementioned Transformer network refers to a deep learning network built on a self-attention mechanism. It possesses the ability to efficiently capture long-distance dependencies between features and can achieve deep interaction and optimization of features through nonlinear transformations. The aforementioned deep feature enhancement processing refers to the enhancement processing performed on the initially fused features through the Transformer network. This can include operations such as feature association enhancement, redundant information removal, and effective information condensation, aiming to improve the comprehensive representation ability of features.

[0082] The aforementioned fusion features refer to the final comprehensive features obtained after deep feature enhancement processing. They integrate multi-dimensional information such as visual attributes, spatial relationships, and edge details of the three types of original features, possessing comprehensiveness and accuracy, and can provide high-quality data support for subsequent processing.

[0083] Furthermore, the Transformer network employed can be a standard Transformer network, whose self-attention mechanism can efficiently capture the correlation between various sub-features in the second fused feature, achieving deep feature interaction; or it can be a Transformer network with rotational position encoding, which can further enhance the spatial correlation of features, especially adapting to the feature representation of the spatial structure and edge details of the target object, improving the accuracy of the fused features in guiding the grasping posture. During the deep feature enhancement process, the Transformer network can perform multiple rounds of nonlinear transformation and information interaction on the second fused feature, eliminating redundant interference information, strengthening effective feature correlation, so that the final output fused feature has both comprehensive multi-source information and accurate representational ability, providing high-quality input data for the subsequent noise correction model and helping to generate accurate initial noise correction rules.

[0084] In some alternative implementations, the grasping accuracy and interference resistance of vision-based motion model-based robots can be improved through the following steps:

[0085] Step 1: Given the scenario of grabbing the target object;

[0086] Step 2: Collect trajectory data (traj) and camera RGB-D data by shaking the camera or playing a preset trajectory.

[0087] Step 3: Obtain the target object detection bounding box (bbox) from a third-party perspective using the acquired RGB data via Qwen3-VL-32B;

[0088] Step 4: Feed the detected target bounding box (bbox) into the SAM2 segmentation model to obtain the target object segmentation mask region, and use the SAM2 tracking algorithm to segment the subsequent RGB image from a third-party perspective.

[0089] Step 5: Set the weight of the mask region to 1.0 and the weight of the non-mask region to 0.3, and record it as MaskImg;

[0090] Step 6: Perform a dot product between MaskImg and the third-view RGB image, and record it as MaskRgbImg;

[0091] Step 7: Extract the gradient information of the first-view RGB image using the Sobel algorithm and denot it as GradRgb, where GradRgb includes the direction and magnitude of the gradient.

[0092] In some alternative implementations, the gradient of a color image can be obtained in the following way:

[0093] For example, in a color image, R, G, and B represent the red, green, and blue channel data, respectively. The gradient and direction of the color image can be calculated based on the principle of local maxima of gradients, as shown in the following formula:

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] in, It reflects the intensity of color change in the image in the horizontal direction (x-axis, such as the left-right direction of the image); It reflects the intensity of color change in the vertical direction (y-axis, such as the vertical direction of the image); It reflects the degree of coupling of color changes in the image within the xy plane (i.e., the image plane); k represents a natural number, R represents red channel data, G represents green channel data, and B represents blue channel data.

[0100] Step 8: Extract the third-view features from MaskRgbImg using Flamingo and denote them as MaskRgbImgFeats;

[0101] Step 9: Extract features from the first-view image FirstViewImg using dinov2 and denote them as FirstViewImgFeats. Then, extract features from GradRgb using Transformer and denote them as GradRgbFeats. In the Transformer structure, RoPE can be used when performing position embedding. Experiments have shown that this method is superior to other commonly used position encoding methods.

[0102] Step 10: Combine the features of MaskRgbImgFeats, FirstViewImgFeats, and GradRgbFeats using the concat operation to form a new feature, which can be denoted as ConFeats. Feed ConFeats into the Transformer structure to extract the new feature and denot it as ConFeatsTrans.

[0103] Step 11: Feed ConFeatsTrans into the Diffusion Policy. During training, it can perform gradient descent to learn model parameters by predicting noise. During inference, it can infer the trajectory to be executed from the noise.

[0104] Please see Figure 2 , Figure 2 This is a schematic diagram of a grasping trajectory generation device 200 provided in an embodiment of this application. As shown in the figure, the grasping trajectory generation device 200 includes:

[0105] The acquisition module 210 is used to acquire scene information of the target object to be grasped, and acquire motion trajectory data and camera red-green-blue-depth RGB-D data based on the scene information. The RGB-D data includes a first-view red-green-blue RGB image and a second-view RGB image.

[0106] Processing module 220 is used to perform a first processing on the first viewpoint RGB image to obtain a weighted RGB image, and to perform a second processing on the second viewpoint RGB image and the weighted RGB image to obtain fusion features;

[0107] Input module 230 is used to input the fused features and noisy sample trajectories into the diffusion model to predict the added random noise;

[0108] The generation module 240 is used to iteratively denoise the initial random noise according to the diffusion rule to generate the target capture trajectory.

[0109] Optionally, the processing module 220 can also be used for;

[0110] The target object detection box is obtained based on the first viewpoint RGB image, the detection box is input into the segmentation model to obtain the target segmentation region, and the first viewpoint RGB image is continuously segmented by the tracking algorithm of the segmentation model;

[0111] Based on the target segmentation region and the first-view RGB image, a weighted RGB image is obtained.

[0112] Optionally, the processing module 220 can also be used for;

[0113] First-view features are extracted from the weighted RGB image, gradient information and second-view image features are extracted from the second-view RGB image, gradient features are extracted from the gradient information, and the first-view features, second-view image features and gradient features are fused to obtain fused features.

[0114] The fused features are input into the noise correction model to obtain an initial noise correction rule that matches the fused features.

[0115] Optionally, the processing module 220 can also be used for;

[0116] The weighted RGB image is feature-encoded using a visual language model to output first-view features with uniform dimensions.

[0117] A self-supervised visual basic model is used to extract features from the second-view RGB image, and output second-view image features with scene adaptability.

[0118] A Transformer network with rotational position encoding is used to perform feature transformation on the gradient information, enhance the spatial correlation of gradient features, and output gradient features.

[0119] Optionally, the processing module 220 can also be used for;

[0120] A mask image is generated based on the target segmented region, wherein the weight value of the segmented region is a first preset value, the weight value of the non-segmented region is a second preset value, the first preset value is greater than the second preset value, and the second preset value is greater than 0;

[0121] The mask image is multiplied by the first-view RGB image to produce the weighted RGB image.

[0122] Optionally, the processing module 220 can also be used for;

[0123] The first viewpoint features, the second viewpoint image features, and the gradient features are merged into a feature vector with unified dimensions to obtain the first fused feature.

[0124] The first fused feature is input into the Transformer network for deep feature enhancement processing to obtain the fused feature.

[0125] The grasping trajectory generation device in this application embodiment can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip.

[0126] The grasping trajectory generation device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments shown achieve the same technical effects, and will not be described again here to avoid repetition.

[0127] For details, see Figure 3 As shown in the figure, this application embodiment also provides an electronic device, including a bus 301, a transceiver 302, an antenna 303, a bus interface 304, a processor 305, and a memory 306.

[0128] Processor 305, used for:

[0129] The scene information for grasping the target object is obtained, and motion trajectory data and camera RGB-D data are obtained based on the scene information. The RGB-D data includes a first-view RGB image and a second-view RGB image.

[0130] A first processing step is performed on the first-view RGB image to obtain a weighted RGB image, and a second processing step is performed on the second-view RGB image and the weighted RGB image to obtain fusion features;

[0131] The fused features and noisy sample trajectories are input into the diffusion model to predict the added random noise;

[0132] The target capture trajectory is generated by iteratively denoising the initial random noise according to the diffusion rule.

[0133] exist Figure 3In this document, a bus architecture (represented by bus 301) is used. Bus 301 can include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 305 and memory represented by memory 306. Bus 301 can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 304 provides an interface between bus 301 and transceiver 302. Transceiver 302 can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 305 is transmitted over a wireless medium via antenna 303, which further receives data and transmits it to processor 305.

[0134] Processor 305 manages bus 301 and general processing, and also provides various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. Memory 306 can be used to store data used by processor 305 during operation.

[0135] Alternatively, the processor 305 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD).

[0136] Optionally, the processor 305 is specifically used for:

[0137] The target object detection box is obtained based on the first viewpoint RGB image, the detection box is input into the segmentation model to obtain the target segmentation region, and the first viewpoint RGB image is continuously segmented by the tracking algorithm of the segmentation model;

[0138] Based on the target segmentation region and the first-view RGB image, a weighted RGB image is obtained.

[0139] Optionally, the processor 305 is specifically used for:

[0140] First-view features are extracted from the weighted RGB image, gradient information and second-view image features are extracted from the second-view RGB image, gradient features are extracted from the gradient information, and the first-view features, second-view image features and gradient features are fused to obtain fused features.

[0141] The fused features are input into the noise correction model to obtain an initial noise correction rule that matches the fused features.

[0142] Optionally, the processor 305 is specifically used for:

[0143] The weighted RGB image is feature-encoded using a visual language model to output first-view features with uniform dimensions.

[0144] A self-supervised visual basic model is used to extract features from the second-view RGB image, and output second-view image features with scene adaptability.

[0145] A Transformer network with rotational position encoding is used to perform feature transformation on the gradient information, enhance the spatial correlation of gradient features, and output gradient features.

[0146] Optionally, the processor 305 is specifically used for:

[0147] A mask image is generated based on the target segmented region, wherein the weight value of the segmented region is a first preset value, the weight value of the non-segmented region is a second preset value, the first preset value is greater than the second preset value, and the second preset value is greater than 0;

[0148] The mask image is multiplied by the first-view RGB image to produce the weighted RGB image.

[0149] Optionally, the processor 305 is specifically used for:

[0150] The first viewpoint features, the second viewpoint image features, and the gradient features are merged into a feature vector with unified dimensions to obtain the first fused feature.

[0151] The first fused feature is input into the Transformer network for deep feature enhancement processing to obtain the fused feature.

[0152] It should be noted that the electronic device provided in this application embodiment is a device capable of executing the above-described grasping trajectory generation method. Therefore, all implementation methods in the above-described grasping trajectory generation method embodiments are applicable to this electronic device and can achieve the same or similar beneficial effects. To avoid repetition, this embodiment will not elaborate further.

[0153] This invention also provides an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor. When the program is executed by the processor, it implements the various processes of the above-described capture trajectory generation method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0154] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described capture trajectory generation method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0155] This application also provides a computer program product, including computer instructions. When these computer instructions are executed by a processor, they implement the various processes of the above-described capture trajectory generation method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0158] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for generating a grasping trajectory, characterized in that, The method includes: The scene information for grasping the target object is obtained, and motion trajectory data and camera RGB-D data are obtained based on the scene information. The RGB-D data includes a first-view RGB image and a second-view RGB image. A first processing step is performed on the first-view RGB image to obtain a weighted RGB image, and a second processing step is performed on the second-view RGB image and the weighted RGB image to obtain fusion features; The fused features and noisy sample trajectories are input into the diffusion model to predict the added random noise; The target capture trajectory is generated by iteratively denoising the initial random noise according to the diffusion rule.

2. The method according to claim 1, characterized in that, The first process includes: The target object detection box is obtained based on the first viewpoint RGB image, the detection box is input into the segmentation model to obtain the target segmentation region, and the first viewpoint RGB image is continuously segmented by the tracking algorithm of the segmentation model; Based on the target segmentation region and the first-view RGB image, a weighted RGB image is obtained.

3. The method according to claim 1, characterized in that, The second process includes: First-view features are extracted from the weighted RGB image, gradient information and second-view image features are extracted from the second-view RGB image, gradient features are extracted from the gradient information, and the first-view features, second-view image features and gradient features are fused to obtain fused features. The fused features are input into the noise correction model to obtain an initial noise correction rule that matches the fused features.

4. The method according to claim 3, characterized in that, The extraction of the first perspective features includes using a visual language model to encode the features of the weighted RGB image and outputting first perspective features with uniform dimensions. The extraction of second-view image features includes using a self-supervised visual basic model to extract features from the second-view RGB image and outputting second-view image features with scene adaptability; The extraction of the gradient features includes using a Transformer network with rotational position encoding to perform feature transformation on the gradient information, enhancing the spatial correlation of the gradient features, and outputting the gradient features.

5. The method according to claim 2, characterized in that, The step of obtaining a weighted RGB image based on the target segmentation region and the first-view RGB image includes: A mask image is generated based on the target segmented region, wherein the weight value of the segmented region is a first preset value, the weight value of the non-segmented region is a second preset value, the first preset value is greater than the second preset value, and the second preset value is greater than 0; The mask image is multiplied by the first-view RGB image to produce the weighted RGB image.

6. The method according to claim 3, characterized in that, The process of fusing the first viewpoint features, the second viewpoint image features, and the gradient features to obtain fused features includes: The first viewpoint features, the second viewpoint image features, and the gradient features are merged into a feature vector with unified dimensions to obtain the first fused feature. The first fused feature is input into the Transformer network for deep feature enhancement processing to obtain the fused feature.

7. A grasping trajectory generation device, characterized in that, include: The acquisition module is used to acquire scene information of the target object to be captured, and acquire motion trajectory data and camera red-green-blue-depth RGB-D data based on the scene information. The RGB-D data includes a first-view red-green-blue RGB image and a second-view RGB image. The processing module is configured to perform a first processing on the first-view RGB image to obtain a weighted RGB image, and perform a second processing on the second-view RGB image and the weighted RGB image to obtain fusion features; An input module is used to input the fused features and noisy sample trajectories into a diffusion model to predict added random noise; The generation module is used to iteratively denoise the initial random noise according to the diffusion rules to generate the target capture trajectory.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the grasping trajectory generation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the grasping trajectory generation method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the grasping trajectory generation method as described in any one of claims 1 to 6.

Images