Object grasping pose estimation method based on global-local feature fusion

Abstract

The application belongs to the technical field of robot visual perception and deep learning, and aims to provide a robot grasping operation control method to improve the intelligentization and autonomy of robots and auxiliary functions, and the application is based on a global-local feature fusion object grasping pose estimation method, color images and depth images are used as inputs of a global-local feature extraction network, the global-local feature extraction network comprises an encoder, a decoder and an output layer, the encoder comprises multiple levels, global-local integrated feature representation is obtained and input into the decoder, wherein the global feature extraction module is built by using a shift window Transformer, and the local feature extraction module is built based on a convolutional neural network (CNN); the decoder gradually restores the spatial size of the feature map to the output layer, the output layer finally determines the optimal grasping position, and the grasping pose prediction result of the target object is obtained. The application is mainly applied to robot design and manufacturing occasions.

Description

Technical Field

[0001] This invention belongs to the field of robot vision perception and deep learning technology, specifically involving an object grasping pose estimation method based on global-local feature fusion, and more particularly involving a global-local feature extraction network that integrates a convolutional neural network (CNN) and a transformer architecture, and a corresponding grasping pose prediction method. Background Technology

[0002] Robotic systems have been widely applied in fields such as intelligent manufacturing, aerospace, and domestic services. In these applications, grasping is one of the core capabilities that enables robots to provide intelligent and autonomous assistance. In the early stages, robot grasping was mainly deployed in structured environments, such as factory production lines, where the pose of the target object was usually known in advance. Robots could perform grasping and placement tasks through pre-programmed processes. However, in most unstructured environments, target objects often exhibit significant diversity in shape, scale, texture, and material, and may even be unknown new objects. These variations greatly increase the difficulty of accurately predicting the grasping pose, placing higher demands on the robustness and generalization ability of grasping prediction models.

[0003] With the rapid development of deep learning technology, CNNs (Convolutional Neural Networks), a type of deep learning network that utilizes convolutional operations to extract local spatial features, have been widely applied to various computer vision tasks. By inputting multi-channel images into a neural network, convolutional kernels perform progressive downsampling to extract hierarchical feature representations, thereby supporting various visual tasks such as object detection, semantic segmentation, and pose estimation. In recent years, CNN-based methods have become the mainstream choice for robot grasping pose estimation. However, due to the limited receptive field of convolutional kernels, multiple downsampling layers are usually required to capture broader contextual information. This repeated downsampling process inevitably leads to the loss of fine-grained spatial details in the original image, negatively impacting the accuracy of grasping detection. The Transformer architecture is a deep learning network structure based on a self-attention mechanism, capable of establishing long-distance dependencies between input features, thereby extracting global semantic information from the image. It has achieved significant success in the field of natural language processing, with its main advantage stemming from its powerful self-attention mechanism. This mechanism enables the model to capture global dependencies in the input while exhibiting high parallelism and faster training efficiency. In recent years, the application of Transformer has expanded to the field of computer vision, achieving remarkable performance in tasks such as image classification. However, its ability to express local geometric details and edge structures is relatively insufficient, which can easily lead to a decrease in grasping and positioning accuracy.

[0004] Extensive research has been conducted by scholars both domestically and internationally on the problem of robot grasping detection. Regarding CNN-based grasping detection methods, in 2018, Zhou et al. proposed a fully convolutional grasping detection network, achieving end-to-end grasping pose prediction. In 2021, Ainetter et al. proposed a refined module integrating semantic segmentation and grasping detection, further improving grasping prediction accuracy. In 2023, Gilles et al. proposed a single-shot multi-grasp strategy to accelerate scene understanding, inferring the action sequence required to grasp multiple objects in cluttered environments using only a single image acquisition. Regarding Transformer-based grasping detection methods, in 2022, Wang et al. first applied the Transformer architecture to robot grasping detection, achieving good results in public datasets and real-world scenes using a self-attention mechanism. In summary, scholars both domestically and internationally have conducted fruitful research on the problem of robot grasping detection, yielding rich research results. Although models such as CNN and Transformer have been widely used, a single CNN is difficult to take into account the global context, while a single Transformer is difficult to take into account local details. Current research methods still have insufficient detection accuracy and robustness in complex scenes. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to propose a method for object grasping pose estimation that integrates global and local features. The technical solution adopted by this invention is an object grasping pose estimation method based on global-local feature fusion. Color images and depth images are used as inputs to a Transformer-CNN fused global-local feature extraction network. This network consists of an encoder, a decoder, and an output layer. The encoder contains multiple layers, each with a global feature extraction module and a local feature extraction module. Global semantic features and local detail features are concatenated and fused to obtain a unified global-local feature representation, which is then input into the decoder. The global feature extraction module is constructed using a shifted-window Transformer, and the local feature extraction module is based on a convolutional neural network. A CNN network is constructed; the decoder introduces skip connections to access shallow features from the encoder, and an upsampling decoding module is used to parse the high-level abstract features of the encoder, gradually restoring the spatial dimensions of the feature map to the output layer. The output layer finally outputs a capture quality score heatmap, a capture angle heatmap, and a capture width heatmap. The optimal capture position is determined based on the capture quality score heatmap, and the capture pose prediction result of the target object is obtained by combining the capture angle and capture width of the corresponding pixels. Among them, a ground truth heatmap is generated from the training set data using two-dimensional Gaussian distribution coding. The heatmap predicted by the global-local feature extraction network is compared with the ground truth heatmap pixel by pixel, and the training of the global-local feature extraction network is completed through iteration.

[0006] The global-local feature extraction network structure based on Transformer-CNN fusion is designed as follows:

[0007] The global-local feature extraction network consists of three main parts: an encoder, a decoder, and an output layer. Following the data flow, it includes an input layer, encoder, decoder, and output layer. The input layer receives a four-channel color-depth image after resolution preprocessing. Each layer of the encoder has parallel global and local feature extraction modules, which concatenate and fuse global semantic features and local detail features to obtain a unified global-local feature representation. The decoder introduces skip connections, accessing shallow features from the encoder. An upsampling decoding module parses the high-level abstract features of the encoder and gradually restores the spatial dimensions of the feature map to the output layer. The output layer ultimately outputs a grasping quality score heatmap, a grasping angle heatmap, and a grasping width heatmap. The optimal grasping position is determined based on the grasping quality score heatmap, and the grasping pose prediction result of the target object is obtained by combining the grasping angle and grasping width of the corresponding pixels. The structural designs of the encoder, decoder, and output layer are as follows:

[0008] (1) Encoder

[0009] The encoder core consists of three global feature extraction modules and three local feature extraction modules, used to capture global semantic information and local detail features in the scene, respectively. The encoder receives 4-channel color-depth image features from the input layer as the first layer input. During each encoding stage, the same input feature is simultaneously fed into both the global and local feature extraction modules, with the two branches completing feature extraction in parallel. Subsequently, the two types of features are concatenated and fused along the channel dimension to form the fused feature of the current layer. This fused feature is then used as the input to both branches of the encoder for the next layer. The detailed process is as follows:

[0010] Input features First, the data passes through a global feature extraction module and a local feature extraction module in the first layer, respectively, to obtain global semantic information and local detail features. Then, these two types of features are concatenated and fused to obtain a global-local integrated feature that combines global context and local structural information. This feature serves as the new input feature to the next layer's global and local feature extraction modules, requiring a total of three feature concatenation and fusion operations. The feature fusion module receives the global semantic features from the global feature extraction module and the local detail features from the local feature extraction module, and performs feature concatenation operations along the channel dimension to obtain a fused feature representation. This fused feature serves as the output of the current encoding layer, inputting into the next layer's encoder to continue extracting high-level semantic information, and also being passed to the corresponding scale's decoder through skip connections during the decoding stage.

[0011] The global feature extraction module is built using a shifted window Transformer to extract global semantic features from the input image. It receives the input image from the previous encoding layer. After preprocessing, the input image is first input to the block embedding module for vectorization encoding, and then sent to the window attention module for feature extraction. The final global features output by the module are sent to the feature fusion module to be fused with the output of the local feature extraction module. The specific workflow is as follows:

[0012] Input features First, the feature maps are vectorized and encoded using a block embedding module, mapping them to a high-dimensional feature space. Then, the encoded features are input into a window attention module to perform attention calculations, thereby modeling the global dependencies between features and ultimately generating the module's output features. :

[0013]

[0014] The block embedding operation can be represented as:

[0015]

[0016] in, For the input feature map, For block operations, For flattening operation, This is a linear projection operation, and simultaneously, the input features... The second path also requires a patch merging module to perform a merging operation. This involves first stitching adjacent image patches together, then performing layer normalization and linear projection. The patch merging operation, PatchMerging, is represented as:

[0017]

[0018] in, For splicing operations, For layer normalization, the input features are normalized along the channel dimension, effectively mitigating the internal covariate shift problem, thereby stabilizing and accelerating the network training process and improving the model's convergence performance. The feature maps downsampled by the block merging module are further input into the window attention module to model the global-local dependencies between features;

[0019] The outputs of both the block embedding module and the block merging module serve as inputs to the window attention module. The window attention module is the core computational unit of the global feature extraction module. Its structure consists of layer normalization, multilayer perceptron, window-based multi-head self-attention, and shifted window multi-head self-attention, coupled with residual connections for stable training. The computational flow of this module is as follows:

[0020]

[0021]

[0022]

[0023]

[0024] in, Indicates the first The feature representation output by the layer Transformer module, The output feature of the previous layer is first fed into a window-based multi-head self-attention system for feature modeling. This represents the intermediate feature representation after window self-attention computation. Multi-head self-attention for windows is achieved by performing multi-head self-attention computation within a local window to model the feature relationships within the window. It is a multilayer perceptron, consisting of multiple fully connected layers and nonlinear activation functions, used to perform nonlinear transformation and channel information fusion on the features output by the window attention module, so as to further enhance the feature representation capability; To enable multi-head self-attention in shifted windows, the window division position is shifted to establish information interaction between adjacent windows;

[0025] The local feature extraction module, built on a CNN, extracts fine-grained local features such as image edges, textures, and contours. It receives the same input image as the global feature extraction module and ultimately outputs local detail features, which are then fed into the feature fusion module. It incorporates a residual structure and a batch normalization layer, using Gaussian error linear units (GaUs) as the activation function. By weighting the input probabilities, the GaUs introduces a smoother nonlinear transformation, providing a smoother nonlinear mapping. The specific computational flow of this module is as follows:

[0026] Input features Firstly, through Convolutional layers extract features, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and introduce nonlinearity to enhance the network's expressive power, ultimately yielding the first set of features. Simultaneously, the original input features... through Convolution expands the dimensions to obtain a second set of features. The two sets of features are then added element-wise to obtain the intermediate features. ; Passing by again After convolutional layers, batch normalization layers, and Gaussian error linear unit activation functions, and with the initial intermediate features... Element-wise addition, with residual connections used to mitigate the gradient vanishing problem, yields the final output features of the local feature extraction module. The above process can be represented as follows:

[0027]

[0028]

[0029] in, For convolution operations, For batch normalization operations, The activation function for the Gaussian error linear unit;

[0030] (2) Decoder

[0031] The decoder receives the fused features from the encoder's last stage output as input, and after three stages of upsampling, restores the spatial resolution to match the input image. The decoder consists of three upsampling decoding modules. In each module, the fused feature map is first compressed in channel dimension through convolution, then upsampled using transposed convolution, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and enhance the network's nonlinear expressive power. The decoding process is formally represented as follows:

[0032]

[0033] in, This indicates a transposed convolution operation. During the decoding stage, a feature fusion operation is introduced at the output of the first upsampling decoding module. The global-local fusion features obtained from the second layer of the encoder are introduced to compensate for the details that may be lost during the upsampling process. At the output of the second upsampling decoding module, the first layer features in the encoder are fused with the decoder features through skip connections. This allows the decoder to utilize both high-level global semantic information and the fine spatial structure information contained in the low-level features during the process of recovering the high-resolution feature map.

[0034] (3) Output layer

[0035] The output layer receives the final high-resolution fused features from the decoder. The global-local feature extraction network generates multi-channel heatmaps with the same spatial dimensions as the original input image through the output layer. These heatmaps represent the capture quality score heatmaps. Capture angle heatmap and capture width heatmap It provides pixel-level prediction results for crawling tasks.

[0036] Global-Local Feature Extraction Network Training Method and Capture Pose Inference

[0037] The pose estimation problem is modeled as a pixel-level regression problem. During the training phase, the Smooth L1 loss function is used to construct the total loss function, which is defined as follows:

[0038]

[0039] In the modeling of the grasping angle, an angle representation based on sine and cosine components is adopted to avoid the discontinuity problem caused by the periodicity of the angle. The final grasping angle mapping can be calculated using inverse trigonometric functions. This process avoids explicit candidate generation and filtering steps, making the capture detection process more computationally efficient and real-time. Finally, the global-local feature extraction network is trained by weighted combination of four sub-losses.

[0040]

[0041] in This represents the loss corresponding to the crawling quality map, used to guide the network to focus on areas with a higher crawling success rate; To compensate for the loss in the gripping width prediction, we ensure that the gripper width predicted by the network is consistent with the actual gripping requirements, thereby achieving stable gripping. and These represent the losses calculated from the sine and cosine components of the grasping angle, respectively. Predicting these two components allows for a continuous and relatively accurate estimation of the grasping angle, effectively mitigating the discontinuity problem in the angle representation. The coefficients... , , and These are the weighting coefficients for each loss, used to balance the contribution of different losses to the overall objective function, ensuring that the model achieves a reasonable balance between grasping quality, gripping width, and angle prediction.

[0042] By capturing quality score heatmaps Search for the optimal location to obtain the optimal grab point in the current scene. :

[0043]

[0044] Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. By calibrating the camera's intrinsic and extrinsic parameters, the capture pose in the pixel coordinate system can be determined. Grab pose representation in world coordinate system ,in This represents the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system. This represents the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0045] The global-local feature extraction network based on Transformer-CNN fusion takes a 4-channel color-depth image, fused from RGB and depth images acquired by a sensor, as input, denoted as:

[0046]

[0047] in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. :

[0048]

[0049] in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as:

[0050]

[0051] To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels. However... This only represents the grasping pose in pixel coordinates. A feasible grasping pose is defined in world coordinates, as shown in the attached figure. Figure 1 As shown, it is represented as:

[0052]

[0053] in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawling point.

[0054] The specific steps of the two-dimensional Gaussian distribution coding are as follows:

[0055] A two-dimensional Gaussian distribution is used to encode the crawled and labeled information during the training phase, as shown in the attached figure. Figure 2 As shown. For each labeled grasping rectangle, with its center pixel as the mean center of the Gaussian distribution, the closer the pixel is to the center, the closer the grasping quality score is to 1; the farther the pixel is from the center, the smoother the quality score decreases. The grasping pose parameterized by the two-dimensional Gaussian distribution encoding in the pixel coordinate system is as follows:

[0056]

[0057] in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is:

[0058]

[0059] In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for Gaussian distribution.

[0060] The training process for the global-local feature extraction network based on Transformer-CNN fusion is as follows:

[0061] (1) Training set preprocessing and ground truth label generation: All color-depth map samples in the Cornell dataset and Jacquard dataset were processed respectively: The original color and depth channels were stitched together into a 4-channel image and uniformly scaled to 224×224 resolution; For all manually labeled grabbing rectangles in each image, the ground truth grabbing quality score heatmap was generated using two-dimensional Gaussian distribution encoding with the mean of the center pixel of the grabbing box; Ground truth grabbing width heatmap and ground truth angle component heatmap were generated simultaneously; Random cropping, scaling and random rotation transformation data augmentation were performed on the small-scale Cornell dataset to expand the sample and suppress overfitting;

[0062] (2) Training batch sampling: During the training phase, a random sampling strategy is used to select training samples from the dataset. The default training batch size is set to 8.

[0063] (3) Network forward propagation: Batch color-depth maps are fed into the global-local feature extraction network, which sequentially completes encoder dual-branch feature extraction, three-layer feature fusion, decoder skip connection upsampling, and output layer heatmap prediction, outputting batch prediction capture quality score heatmap. Through By searching for the optimal location, the optimal grab point in the current scene can be obtained. :

[0064]

[0065] Even if Find the pixel position of the maximum value Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. This yields a heatmap of the grab angle and a heatmap of the grab width.

[0066] (4) Loss calculation and backpropagation: The Smooth L1 sub-loss is calculated pixel by pixel between the heatmap predicted by the network and the ground truth heatmap generated by preprocessing, and the total loss is obtained by weighted summation according to the weight coefficients. Backpropagation of gradients based on the chain rule updates all network parameters of the encoder, decoder, and output layer.

[0067] (5) Optimizer and learning rate update: The adaptive moment estimation optimizer with weight decay regularization is selected to perform parameter gradient update, and the initial learning rate is set to 0.001. A segmented learning rate decay strategy is adopted: the initial learning rate is fixed throughout the first 5 training rounds; starting from the 6th training round, the learning rate is gradually reduced by multiplying by the decay coefficient 0.8 after each complete training round, so as to slow down the parameter update amplitude in the later stage and improve the convergence accuracy.

[0068] (6) Set the total number of iterations to 100, and execute steps (2) to (5) in a complete loop until the training limit of 100 rounds is reached; retain the weight model corresponding to the optimal validation set accuracy for subsequent pose inference;

[0069] The complete execution flow of pose reasoning is as follows:

[0070] After the global-local feature extraction network is trained, the optimal weight model is loaded, gradient calculation is disabled, and only forward propagation inference is performed. Preprocessing is performed on color-depth images captured in real-time by the camera or color-depth map samples from the test set in the dataset, scaling the resolution to 224×224 and using this as input to the global-local feature extraction network. The network output is a 224×224 resolution heatmap of the capture quality score, capture angle, and capture width. The capture quality score heatmap is then traversed to search for the pixel coordinates corresponding to the global maximum value. The optimal capture point in the current scene is obtained. The angle and width values ​​of the capture angle heatmap and capture width heatmap at that pixel are read. By calibrating the camera's intrinsic and extrinsic parameters, the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system is obtained. Coordinate transformation matrix from camera coordinate system to world coordinate system This is used to capture the pose in the pixel coordinate system. Grab pose representation in world coordinate system The task is then assigned to the robotic arm to perform the grasping action.

[0071] The features and beneficial effects of this invention are:

[0072] This method is of great significance to the research of robot visual perception and grasping. When faced with object categories not included in the dataset in the robot's field of vision, this method can predict the graspable region of any object in the scene at the pixel level, and simultaneously regress the corresponding grasping angle and gripper closure width, providing complete and unified pose information for subsequent grasping execution. This method has high theoretical and engineering value, and its advantages are summarized as follows:

[0073] (1) This method proposes a parameterized representation of grasping pose, which models the grasping pose estimation problem as a pixel regression problem. By predicting the grasping quality score, grasping angle and gripper width multi-channel heat map, the feasible grasping poses in the scene are uniformly represented, avoiding the explicit grasping candidate generation and screening steps, and making the grasping detection process more computationally efficient and real-time.

[0074] (2) This method designs a two-dimensional Gaussian distribution-based encoding of crawling annotation information, distinguishing the quality weights of different locations within the crawling area, highlighting the optimal crawling center region, and guiding the model to prioritize learning high-confidence crawling locations. Compared with traditional annotation methods that do not distinguish the importance of different pixels within a rectangular region and assign the same crawling quality score to all pixels, this method more reasonably portrays the continuous spatial variation of crawling quality scores, highlights the importance of the crawling center in the prediction process, and helps the model to more accurately focus on the potential optimal crawling region during training, significantly improving the model's position regression accuracy.

[0075] (3) This method designs a global-local feature extraction network based on Transformer-CNN fusion, which effectively solves the problems of insufficient global context modeling ability of pure CNN-based grasping detection methods and weak local detail extraction ability of pure Transformer-based grasping detection methods in traditional robot grasping detection. A Transformer-CNN dual-branch fusion encoder is adopted to integrate local details and global semantics at the same time, which solves the defects of single network feature expression and insufficient expressive ability. It can directly predict the grasping pose of the object from visual information and adapt to complex and messy grasping scenarios. Multi-layer skip connections are built to directly pass the low-level features of the encoder to the decoder, compensate for the spatial details lost in the downsampling process, and improve the accuracy of grasping prediction. Attached Figure Description

[0076] Figure 1 A schematic diagram of the parametric representation of the grab pose in the world coordinate system.

[0077] Figure 2 Comparison of the captured positions using the original encoding and Gaussian distribution encoding formats.

[0078] Figure 3 Overall structure of a global-local feature extraction network model based on Transformer-CNN fusion.

[0079] Figure 4 It consists of a global feature extraction module, a local feature extraction module, and an upsampling decoding module.

[0080] Figure 5 An internal component of the global feature extraction module.

[0081] Figure 6 A visualization of some experimental results from the Cornell dataset.

[0082] Figure 7 A visualization of some experimental results from the Jacquard dataset. Detailed Implementation

[0083] To address current shortcomings, this paper proposes an object grasping pose estimation method based on global-local feature fusion. A robot grasping detection network based on Transformer-CNN fusion is designed. This method simultaneously mines global semantic information and local spatial details, taking into account the expressive power of both global and local features. This enables the prediction of the graspable area of ​​any object in the scene at the pixel level, significantly improving the detection accuracy of robotic arm grasping and greatly increasing the success rate of robotic arm grasping physical objects.

[0084] The technical solution adopted in this invention is as follows:

[0085] A method for object grasping pose estimation based on global-local feature fusion uses color and depth images as inputs to a Transformer-CNN fusion global-local feature extraction network. This network consists of an encoder, a decoder, and an output layer. The encoder comprises multiple layers, each with both global and local feature extraction modules. Global semantic features and local detail features are concatenated and fused to obtain a unified global-local feature representation, which is then input into the decoder. The global feature extraction module is built using a shift-window Transformer, while the local feature extraction module is built using a Convolutional Neural Network (CNN). The encoder introduces skip connections to access shallow features from the encoder. An upsampling decoding module is used to parse the high-level abstract features of the encoder and gradually recover the spatial dimensions of the feature map to the output layer. The output layer finally outputs a grasping quality score heatmap, a grasping angle heatmap, and a grasping width heatmap. The optimal grasping position is determined based on the grasping quality score heatmap, and the grasping pose prediction result of the target object is obtained by combining the grasping angle and grasping width of the corresponding pixels. Among them, a ground truth heatmap is generated from the training set data using two-dimensional Gaussian distribution coding. The heatmap predicted by the global-local feature extraction network is compared with the ground truth heatmap pixel by pixel, and the training of the global-local feature extraction network is completed through iteration.

[0086] The global-local feature extraction network structure based on Transformer-CNN fusion is designed as follows:

[0087] The global-local feature extraction network consists of three main parts: an encoder, a decoder, and an output layer. Following the data flow, it includes an input layer, encoder, decoder, and output layer. The input layer receives a four-channel color-depth image after resolution preprocessing. Each layer of the encoder has parallel global and local feature extraction modules, which concatenate and fuse global semantic features and local detail features to obtain a unified global-local feature representation. The decoder introduces skip connections, accessing shallow features from the encoder. An upsampling decoding module parses the high-level abstract features of the encoder and gradually restores the spatial dimensions of the feature map to the output layer. The output layer ultimately outputs a grasping quality score heatmap, a grasping angle heatmap, and a grasping width heatmap. The optimal grasping position is determined based on the grasping quality score heatmap, and the grasping pose prediction result of the target object is obtained by combining the grasping angle and grasping width of the corresponding pixels. The structural designs of the encoder, decoder, and output layer are as follows:

[0088] (1) Encoder

[0089] The encoder core consists of three global feature extraction modules and three local feature extraction modules, used to capture global semantic information and local detail features in the scene, respectively. The encoder receives 4-channel color-depth image features from the input layer as the first layer input. During each encoding stage, the same input feature is simultaneously fed into both the global and local feature extraction modules, with the two branches completing feature extraction in parallel. Subsequently, the two types of features are concatenated and fused along the channel dimension to form the fused feature of the current layer. This fused feature is then used as the input to both branches of the encoder for the next layer. The detailed process is as follows:

[0090] Input features First, the data passes through a global feature extraction module and a local feature extraction module in the first layer, respectively, to obtain global semantic information and local detail features. Then, these two types of features are concatenated and fused to obtain a global-local integrated feature that combines global context and local structural information. This feature serves as the new input feature to the next layer's global and local feature extraction modules, requiring a total of three feature concatenation and fusion operations. The feature fusion module receives the global semantic features from the global feature extraction module and the local detail features from the local feature extraction module, and performs feature concatenation operations along the channel dimension to obtain a fused feature representation. This fused feature serves as the output of the current encoding layer, inputting into the next layer's encoder to continue extracting high-level semantic information, and also being passed to the corresponding scale's decoder through skip connections during the decoding stage.

[0091] The global feature extraction module is built using a shifted window Transformer to extract global semantic features from the input image. It receives the input image from the previous encoding layer. After preprocessing, the input image is first input to the block embedding module for vectorization encoding, and then sent to the window attention module for feature extraction. The final global features output by the module are sent to the feature fusion module to be fused with the output of the local feature extraction module. The specific workflow is as follows:

[0092] Input features First, the feature maps are vectorized and encoded using a block embedding module, mapping them to a high-dimensional feature space. Then, the encoded features are input into a window attention module to perform attention calculations, thereby modeling the global dependencies between features and ultimately generating the module's output features. :

[0093]

[0094] The block embedding operation can be represented as:

[0095]

[0096] in, For the input feature map, For block operations, For flattening operation, This is a linear projection operation, and simultaneously, the input features... The second path also requires a patch merging module to perform a merging operation. This involves first stitching adjacent image patches together, then performing layer normalization and linear projection. The patch merging operation, PatchMerging, is represented as:

[0097]

[0098] in, For splicing operations, For layer normalization, the input features are normalized along the channel dimension, effectively mitigating the internal covariate shift problem, thereby stabilizing and accelerating the network training process and improving the model's convergence performance. The feature maps downsampled by the block merging module are further input into the window attention module to model the global-local dependencies between features;

[0099] The outputs of both the block embedding module and the block merging module serve as inputs to the window attention module. The window attention module is the core computational unit of the global feature extraction module. Its structure consists of layer normalization, multilayer perceptron, window-based multi-head self-attention, and shifted window multi-head self-attention, coupled with residual connections for stable training. The computational flow of this module is as follows:

[0100]

[0101]

[0102]

[0103]

[0104] in, Indicates the first The feature representation output by the Transformer module of the layer. The output feature of the previous layer is first fed into a window-based multi-head self-attention system for feature modeling. This represents the intermediate feature representation after window self-attention computation. Multi-head self-attention for windows is achieved by performing multi-head self-attention computation within a local window to model the feature relationships within the window. It is a multilayer perceptron, consisting of multiple fully connected layers and nonlinear activation functions, used to perform nonlinear transformation and channel information fusion on the features output by the window attention module, so as to further enhance the feature representation capability; To enable multi-head self-attention in shifted windows, the window division position is shifted to establish information interaction between adjacent windows;

[0105] The local feature extraction module, built on a CNN, extracts fine-grained local features such as image edges, textures, and contours. It receives the same input image as the global feature extraction module and ultimately outputs local detail features, which are then fed into the feature fusion module. It incorporates a residual structure and a batch normalization layer, using Gaussian error linear units (GaUs) as the activation function. By weighting the input probabilities, the GaUs introduces a smoother nonlinear transformation, providing a smoother nonlinear mapping. The specific computational flow of this module is as follows:

[0106] Input features Firstly, through Convolutional layers extract features, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and introduce nonlinearity to enhance the network's expressive power, ultimately yielding the first set of features. Simultaneously, the original input features... through Convolution expands the dimensions to obtain a second set of features. The two sets of features are then added element-wise to obtain the intermediate features. ; Passing by again After convolutional layers, batch normalization layers, and Gaussian error linear unit activation functions, and with the initial intermediate features... Element-wise addition, with residual connections used to mitigate the gradient vanishing problem, yields the final output features of the local feature extraction module. The above process can be represented as follows:

[0107]

[0108]

[0109] in, For convolution operations, For batch normalization operations, The activation function for the Gaussian error linear unit;

[0110] (2) Decoder

[0111] The decoder receives the fused features from the encoder's last stage output as input, and after three stages of upsampling, restores the spatial resolution to match the input image. The decoder consists of three upsampling decoding modules. In each module, the fused feature map is first compressed in channel dimension through convolution, then upsampled using transposed convolution, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and enhance the network's nonlinear expressive power. The decoding process is formally represented as follows:

[0112]

[0113] in, This indicates a transposed convolution operation. During the decoding stage, a feature fusion operation is introduced at the output of the first upsampling decoding module. The global-local fusion features obtained from the second layer of the encoder are introduced to compensate for the details that may be lost during the upsampling process. At the output of the second upsampling decoding module, the first layer features in the encoder are fused with the decoder features through skip connections. This allows the decoder to utilize both high-level global semantic information and the fine spatial structure information contained in the low-level features during the process of recovering the high-resolution feature map.

[0114] (3) Output layer

[0115] The output layer receives the final high-resolution fused features from the decoder. The global-local feature extraction network generates multi-channel heatmaps with the same spatial dimensions as the original input image through the output layer. These heatmaps represent the capture quality score heatmaps. Capture angle heatmap and capture width heatmap It provides pixel-level prediction results for crawling tasks.

[0116] Global-Local Feature Extraction Network Training Method and Capture Pose Inference

[0117] The pose estimation problem is modeled as a pixel-level regression problem. During the training phase, the Smooth L1 loss function is used to construct the total loss function, which is defined as follows:

[0118]

[0119] In the modeling of the grasping angle, an angle representation based on sine and cosine components is adopted to avoid the discontinuity problem caused by the periodicity of the angle. The final grasping angle mapping can be calculated using inverse trigonometric functions. This process avoids explicit candidate generation and filtering steps, making the capture detection process more computationally efficient and real-time. Finally, the global-local feature extraction network is trained by weighted combination of four sub-losses.

[0120]

[0121] in This represents the loss corresponding to the crawling quality map, used to guide the network to focus on areas with a higher crawling success rate; To compensate for the loss in the gripping width prediction, we ensure that the gripper width predicted by the network is consistent with the actual gripping requirements, thereby achieving stable gripping. and These represent the losses calculated from the sine and cosine components of the grasping angle, respectively. Predicting these two components allows for a continuous and relatively accurate estimation of the grasping angle, effectively mitigating the discontinuity problem in the angle representation. The coefficients... , , and These are the weighting coefficients for each loss, used to balance the contribution of different losses to the overall objective function, ensuring that the model achieves a reasonable balance between grasping quality, gripping width, and angle prediction.

[0122] By capturing quality score heatmaps Search for the optimal location to obtain the optimal grab point in the current scene. :

[0123]

[0124] Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. By calibrating the camera's intrinsic and extrinsic parameters, the capture pose in the pixel coordinate system can be determined. Grab pose representation in world coordinate system ,in This represents the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system. This represents the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0125] The global-local feature extraction network based on Transformer-CNN fusion takes a 4-channel color-depth image, fused from RGB and depth images acquired by a sensor, as input, denoted as:

[0126]

[0127] in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. :

[0128]

[0129] in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as:

[0130]

[0131] To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels. However... This only represents the grasping pose in pixel coordinates. A feasible grasping pose is defined in world coordinates, as shown in the attached figure. Figure 1 As shown, it is represented as:

[0132]

[0133] in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawling point.

[0134] The specific steps of the two-dimensional Gaussian distribution coding are as follows:

[0135] A two-dimensional Gaussian distribution is used to encode the crawled and labeled information during the training phase, as shown in the attached figure. Figure 2 As shown. For each labeled grasping rectangle, with its center pixel as the mean center of the Gaussian distribution, the closer the pixel is to the center, the closer the grasping quality score is to 1; the farther the pixel is from the center, the smoother the quality score decreases. The grasping pose parameterized by the two-dimensional Gaussian distribution encoding in the pixel coordinate system is as follows:

[0136]

[0137] in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is:

[0138]

[0139] In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for Gaussian distribution.

[0140] The training process for the global-local feature extraction network based on Transformer-CNN fusion is as follows:

[0141] (1) Training set preprocessing and ground truth label generation: All color-depth map samples in the Cornell dataset and Jacquard dataset were processed respectively: The original color and depth channels were stitched together into a 4-channel image and uniformly scaled to 224×224 resolution; For all manually labeled grabbing rectangles in each image, the ground truth grabbing quality score heatmap was generated using two-dimensional Gaussian distribution encoding with the mean of the center pixel of the grabbing box; Ground truth grabbing width heatmap and ground truth angle component heatmap were generated simultaneously; Random cropping, scaling and random rotation transformation data augmentation were performed on the small-scale Cornell dataset to expand the sample and suppress overfitting;

[0142] (2) Training batch sampling: During the training phase, a random sampling strategy is used to select training samples from the dataset. The default training batch size is set to 8.

[0143] (3) Network forward propagation: Batch color-depth maps are fed into the global-local feature extraction network, which sequentially completes encoder dual-branch feature extraction, three-layer feature fusion, decoder skip connection upsampling, and output layer heatmap prediction, outputting batch prediction capture quality score heatmap. Through By searching for the optimal location, the optimal grab point in the current scene can be obtained. :

[0144]

[0145] Even if Find the pixel position of the maximum value Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. This yields a heatmap of the grab angle and a heatmap of the grab width.

[0146] (4) Loss calculation and backpropagation: The Smooth L1 sub-loss is calculated pixel by pixel between the heatmap predicted by the network and the ground truth heatmap generated by preprocessing, and the total loss is obtained by weighted summation according to the weight coefficients. Backpropagation of gradients based on the chain rule updates all network parameters of the encoder, decoder, and output layer.

[0147] (5) Optimizer and learning rate update: The adaptive moment estimation optimizer with weight decay regularization is selected to perform parameter gradient update, and the initial learning rate is set to 0.001. A segmented learning rate decay strategy is adopted: the initial learning rate is fixed throughout the first 5 training rounds; starting from the 6th training round, the learning rate is gradually reduced by multiplying by the decay coefficient 0.8 after each complete training round, so as to slow down the parameter update amplitude in the later stage and improve the convergence accuracy.

[0148] (6) Set the total number of iterations to 100, and execute steps (2) to (5) in a complete loop until the training limit of 100 rounds is reached; retain the weight model corresponding to the optimal validation set accuracy for subsequent pose inference;

[0149] The complete execution flow of pose reasoning is as follows:

[0150] After the global-local feature extraction network is trained, the optimal weight model is loaded, gradient calculation is disabled, and only forward propagation inference is performed. Preprocessing is performed on color-depth images captured in real-time by the camera or color-depth map samples from the test set in the dataset, scaling the resolution to 224×224 and using this as input to the global-local feature extraction network. The network output is a 224×224 resolution heatmap of the capture quality score, capture angle, and capture width. The capture quality score heatmap is then traversed to search for the pixel coordinates corresponding to the global maximum value. The optimal capture point in the current scene is obtained. The angle and width values ​​of the capture angle heatmap and capture width heatmap at that pixel are read. By calibrating the camera's intrinsic and extrinsic parameters, the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system is obtained. Coordinate transformation matrix from camera coordinate system to world coordinate system This is used to capture the pose in the pixel coordinate system. Grab pose representation in world coordinate system The task is then assigned to the robotic arm to perform the grasping action.

[0151] The technical solution of this invention is briefly described as follows:

[0152] Part 1: Capturing the parametric representation of pose:

[0153] The 4-channel color-depth image, formed by fusing color and depth images acquired by the sensor, is used as input and denoted as:

[0154]

[0155] in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. :

[0156]

[0157] in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as:

[0158]

[0159] To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels. However... This only represents the grasping pose in pixel coordinates. A feasible grasping pose definition in world coordinates is as follows:

[0160]

[0161] in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawl point;

[0162] The second part involves encoding the labeled information using a two-dimensional Gaussian distribution:

[0163] A two-dimensional Gaussian distribution is used to encode the capture annotation information during the training phase. For each annotation's capture rectangle, its center pixel is taken as the mean center of the Gaussian distribution. The closer a pixel is to the center, the closer its capture quality score is to 1; the farther away it is, the lower the quality score becomes smoothly. The capture pose parameterized by the two-dimensional Gaussian distribution encoding in the pixel coordinate system is as follows:

[0164]

[0165] in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is:

[0166]

[0167] In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for the Gaussian distribution;

[0168] The third part involves processing using a global-local feature extraction network based on the fusion of Transformer-CNN:

[0169] The structure of the encoder, decoder, and output layer is as follows:

[0170] (1) Encoder

[0171] The encoder core consists of three global feature extraction modules and three local feature extraction modules, which are used to capture global semantic information and local detail features in the scene, respectively, and input features. First, the features are processed by a global feature extraction module and a local feature extraction module in the first layer to obtain global semantic information and local detail features, respectively. Then, the two types of features are concatenated and fused to obtain a global-local integrated feature that combines global context and local structural information. This feature will be used as a new input feature to enter the next layer's global feature extraction module and local feature extraction module. A total of three feature concatenation and fusion processes are required.

[0172] (2) Decoder

[0173] The decoder consists of three upsampling decoding modules. In each module, the channel dimension of the fused feature map is first compressed through convolution, followed by upsampling of the feature map using transposed convolution. Then, batch normalization layers and Gaussian error linear unit activation functions are introduced sequentially to stabilize the training process and enhance the network's non-linear expressive power. The decoding process is formally represented as follows:

[0174]

[0175] in, This indicates the transpose convolution operation. For batch normalization operations, is the Gaussian error linear unit activation function. During the decoding stage, a feature fusion operation is introduced at the output of the first upsampling decoding module, incorporating the global-local fused features obtained from the second layer of the encoder to compensate for the details lost during upsampling. At the output of the second upsampling decoding module, the first-layer features from the encoder are fused with the decoder features via skip connections.

[0176] (3) Output layer

[0177] The network generates multi-channel heatmaps with the same spatial dimensions as the original input image through the output layer. These heatmaps represent the capture quality score heatmaps. Capture angle heatmap and capture width heatmap This provides pixel-level prediction results for crawling tasks;

[0178] Part Four: Global-Local Feature Extraction Network Training Method and Pose Inference for Grasping

[0179] During the training phase, the total loss function is constructed using Smooth L1 loss. The Smooth L1 loss function is defined as follows:

[0180]

[0181] In the modeling process of the grasping angle, a sinusoidal-based approach is adopted. Sum and cosine components To avoid discontinuities caused by angle periodicity, the final grasping angle mapping is calculated using inverse trigonometric functions. Finally, the global-local feature extraction network is trained by weighted combination of four sub-losses:

[0182]

[0183] in This represents the loss corresponding to the crawling quality map, used to guide the network to focus on areas with a higher crawling success rate; To compensate for the loss in the gripping width prediction, we ensure that the gripper width predicted by the network is consistent with the actual gripping requirements, thereby achieving stable gripping. and These represent the losses calculated using the sine and cosine components of the grasping angle, respectively, with coefficients... , , and These are the weighting coefficients for each type of loss.

[0184] By capturing quality score heatmaps Search for the optimal position, that is, obtain the optimal grab point in the current scene. :

[0185]

[0186] Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. By calibrating the camera's intrinsic and extrinsic parameters, the capture pose in the pixel coordinate system can be determined. Grab pose representation in world coordinate system ,in This represents the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system. This represents the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0187] In Part Three:

[0188] The global feature extraction module is built using a shifted window Transformer to extract global semantic features from the input image. The shifted window Transformer is a visual Transformer structure that employs local window attention and a shifted window mechanism, achieving cross-window feature modeling through information interaction between adjacent windows. After preprocessing, the input image is first input to the block embedding module for vectorization encoding, converting two-dimensional image blocks into one-dimensional feature vectors, and then fed into the window attention module for feature extraction. The specific workflow is as follows:

[0189] Input features First, the feature maps are vectorized and encoded by the block embedding module, mapping them to a high-dimensional feature space. Then, the encoded features are input into the window attention module to perform attention calculations, thereby modeling the global dependencies between features and ultimately generating the module's output features. :

[0190]

[0191] The block embedding operation PatchEmbed is represented as follows:

[0192]

[0193] in, For the input feature map, For block operations, For flattening operation, This is a linear projection operation. Simultaneously, the input features... The second path also requires a patch merging module to perform a merging operation. This involves first stitching adjacent image patches together, then performing layer normalization and linear projection. The patch merging operation, PatchMerging, is represented as:

[0194]

[0195] in, For splicing operations, As a layer normalization operation, the feature maps downsampled by the block merging module will be further input into the window attention module to model the global-local dependencies between features;

[0196] The outputs of both the block embedding module and the block merging module serve as inputs to the window attention module. The window attention module is the core computational unit of the global feature extraction module. Its structure consists of layer normalization, a multilayer perceptron, window-based multi-head self-attention, and shifted window multi-head self-attention, coupled with residual connections for stable training. The computational flow of this module is as follows:

[0197]

[0198]

[0199]

[0200]

[0201] in, Indicates the first The feature representation output by the layer Transformer module, The output feature of the previous layer is first fed into a window-based multi-head self-attention system for feature modeling. This represents the intermediate feature representation after window self-attention computation. Multi-head self-attention for windows is achieved by performing multi-head self-attention computation within a local window to model the feature relationships within the window. It is a multilayer perceptron, consisting of multiple fully connected layers and nonlinear activation functions, used to perform nonlinear transformation and channel information fusion on the features output by the window attention module, so as to further enhance the feature representation capability; To enable multi-head self-attention in shifted windows, the window division position is shifted to allow information interaction between adjacent windows.

[0202] The local feature extraction module is built on CNN and is used to extract fine-grained local features, including image edges, textures, and contours. It incorporates a residual structure and a batch normalization layer, and uses Gaussian error linear units as the activation function. The specific computation process is as follows:

[0203] Input features Firstly, through Convolutional layers extract features, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and introduce nonlinearity to enhance the network's expressive power, ultimately yielding the first set of features; simultaneously, the original input features... through Convolution expands the dimensions to obtain a second set of features. The two sets of features are then added element-wise to obtain the intermediate features. ; Passing by again After convolutional layers, batch normalization layers, and Gaussian error linear unit activation functions, and with the initial intermediate features... Element-wise addition, with residual connections used to mitigate the gradient vanishing problem, yields the final output features of the local feature extraction module. The specific process is as follows:

[0204]

[0205]

[0206] in, For convolution operations, For batch normalization operations, is the activation function for the Gaussian error linear unit.

[0207] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0208] To overcome the shortcomings of existing technologies, this method aims to propose an object grasping pose estimation method that integrates global and local features. To this end, this method adopts the following technical solution: a global-local feature extraction network based on Transformer-CNN fusion, consisting of three main parts: encoder, decoder, and output layer, combining grasping pose parameterization representation with grasping region encoding based on a two-dimensional Gaussian distribution; the network takes a color-depth fused image as input and outputs pixel-level grasping quality score heatmap, grasping angle heatmap, and grasping width heatmap, providing complete and unified pose information for subsequent grasping execution. The specific steps are as follows:

[0209] Part 1: Capturing the parametric representation of pose:

[0210] The system takes a 4-channel color-depth image, fused from the RGB and depth images acquired by the sensor, as input, denoted as...

[0211]

[0212] in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. :

[0213]

[0214] in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as:

[0215]

[0216] To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels. However... This only represents the grasping pose in pixel coordinates. A feasible grasping pose is defined in world coordinates, as shown in the attached figure. Figure 1 As shown, it is represented as:

[0217]

[0218] in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawling point.

[0219] The second part involves encoding the labeled information using a two-dimensional Gaussian distribution:

[0220] A two-dimensional Gaussian distribution is used to encode the crawled and labeled information during the training phase, as shown in the attached figure. Figure 2 As shown. For each labeled grasping rectangle, with its center pixel as the mean center of the Gaussian distribution, the closer the pixel is to the center, the closer the grasping quality score is to 1; the farther the pixel is from the center, the smoother the quality score decreases. The grasping pose parameterized by the two-dimensional Gaussian distribution encoding in the pixel coordinate system is as follows:

[0221]

[0222] in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is:

[0223]

[0224] In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for Gaussian distribution.

[0225] Part Three: Design of a Global-Local Feature Extraction Network Structure Based on Transformer-CNN Fusion:

[0226] The network consists of three main parts: an encoder, a decoder, and an output layer, as shown in the attached diagram. Figure 3 As shown, the global-local feature extraction network proposed in this embodiment includes an input layer, an encoder, a decoder, and an output layer in sequence according to the data flow. The input layer receives the four-channel color-depth image after resolution preprocessing. Each layer of the encoder has a parallel global feature extraction module and a local feature extraction module, which concatenate and fuse global semantic features and local detail features to obtain a unified global-local feature representation. The decoder introduces skip connections to access shallow features from the encoder, uses an upsampling decoding module to parse the high-level abstract features of the encoder, and gradually restores the spatial dimensions of the feature map to the output layer. The output layer ultimately outputs a capture quality score heatmap, a capture angle heatmap, and a capture width heatmap. The optimal capture position is determined based on the capture quality score heatmap, and the capture pose prediction result of the target object is obtained by combining the capture angle and capture width of the corresponding pixels. The structural designs of the encoder, decoder, and output layer are as follows:

[0227] (1) Encoder

[0228] The encoder core consists of three global feature extraction modules and three local feature extraction modules, used to capture global semantic information and local detail features in the scene, respectively. The encoder receives 4-channel color-depth image features from the input layer as the first-layer input. During each encoding stage, the same input features are simultaneously fed into both the global and local feature extraction modules, with the two branches completing feature extraction in parallel. Subsequently, the two types of features are concatenated and fused along the channel dimension to form the fused feature of the current layer, which is then used as the input to both branches of the next encoder layer. Detailed steps are as follows: Input Features First, the data passes through a global feature extraction module and a local feature extraction module in the first layer, respectively, to obtain global semantic information and local detail features. Then, these two types of features are concatenated and fused to obtain a global-local integrated feature that combines global context and local structural information. This feature will then serve as new input features to the next layer's global and local feature extraction modules, as shown in the appendix. Figure 5As shown, a total of three feature concatenation and fusion operations are required. The feature fusion module is responsible for receiving global semantic features from the global feature extraction module and local detail features from the local feature extraction module, and performing feature concatenation operations along the channel dimension to obtain the fused feature representation. The fused feature retains both the global semantic information learned by the Transformer and the local texture structure information extracted by the CNN. This fused feature serves as the output of the current encoding layer, inputting into the next encoder layer to continue extracting high-level semantic information, and also being passed to the decoder at the corresponding scale through skip connections during the decoding stage.

[0229] The global feature extraction module is built using a shifted window Transformer to extract global semantic features from the input image. It receives the input image from the previous encoding layer. After preprocessing, the input image is first input to the block embedding module for vectorization encoding, and then sent to the window attention module for feature extraction, as shown in the attached diagram. Figure 4 As shown. The global features output by the module will be fed into the feature fusion module, where they will be fused together with the output of the local feature extraction module. The specific workflow is as follows:

[0230] Input features First, the feature maps are vectorized and encoded using a block embedding module, mapping them to a high-dimensional feature space. Then, the encoded features are input into a window attention module to perform attention calculations, thereby modeling the global dependencies between features and ultimately generating the module's output features. :

[0231]

[0232] The block embedding operation can be represented as:

[0233]

[0234] in, For the input feature map, For block operations, For flattening operation, This is a linear projection operation. Simultaneously, the input features... The second path also requires a patch merging module to perform a merging operation. First, adjacent image patches are stitched together, then layer normalization and linear projection are performed. The patch merging operation, PatchMerging, can be represented as...

[0235]

[0236] in, For splicing operations, This is a layer normalization operation. By normalizing the input features along the channel dimension, the internal covariate shift problem is effectively mitigated, thereby stabilizing and accelerating the network training process and improving the model's convergence performance. The feature maps downsampled by the block merging module are further input into the window attention module to model the global-local dependencies between features.

[0237] The outputs of both the block embedding module and the block merging module serve as inputs to the window attention module. The window attention module is the core computational unit of the global feature extraction module. Its structure consists of layer normalization, a multilayer perceptron, window-based multi-head self-attention, and shifted window multi-head self-attention, coupled with residual connections for stable training. The computational flow of this module is as follows:

[0238]

[0239]

[0240]

[0241]

[0242] in, Indicates the first The feature representation output by the layer Transformer module, The output feature of the previous layer is first fed into a window-based multi-head self-attention system for feature modeling. This represents the intermediate feature representation after window self-attention computation. For multi-head self-attention of windows, It is a multilayer perceptron. For multi-head self-attention in shifted windows.

[0243] The local feature extraction module is built based on CNN (see attached image). Figure 4 This module is used to extract fine-grained local features such as image edges, textures, and contours. It receives the same input image as the global feature extraction module and ultimately outputs local detail features, which are then fed into the feature fusion module. It incorporates a residual structure and a batch normalization layer, using a Gaussian error linear unit as the activation function. Compared to traditional methods, the Gaussian error linear unit introduces a smoother nonlinear transformation by probability-weighting the input, providing a smoother nonlinear mapping. The specific computational flow of this module is as follows:

[0244] Input features Firstly, through Convolutional layers extract features, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and introduce nonlinearity to enhance the network's expressive power, ultimately yielding the first set of features. Simultaneously, the original input features... through Convolution expands the dimensions to obtain a second set of features. The two sets of features are then added element-wise to obtain the intermediate features. ; Passing by again After convolutional layers, batch normalization layers, and Gaussian error linear unit activation functions, and with the initial intermediate features... Element-wise addition, with residual connections used to mitigate the gradient vanishing problem, yields the final output features of the local feature extraction module. The above process can be represented as follows:

[0245]

[0246]

[0247] in, For convolution operations, For batch normalization operations, is the activation function for the Gaussian error linear unit.

[0248] (2) Decoder

[0249] The decoder receives the fused features from the encoder's last stage output as input, and after three stages of upsampling, restores the image to the same spatial resolution as the input image. The decoder consists of three upsampling decoding modules (see attached). Figure 4 In each upsampling decoding module, the channel dimension of the fused feature map is first compressed through convolution, followed by upsampling of the feature map using transposed convolution. Then, batch normalization layers and Gaussian error linear unit activation functions are introduced sequentially to stabilize the training process and enhance the network's non-linear expressive power. The above decoding process can be formally represented as:

[0250]

[0251] in, This indicates a transposed convolution operation. During the decoding stage, a feature fusion operation is introduced at the output of the first upsampling decoding module. This introduces global-local fused features obtained from the second layer of the encoder to compensate for details that may be lost during upsampling. At the output of the second upsampling decoding module, skip connections are used to fuse the first-layer features from the encoder with the features from the decoder. This allows the decoder to utilize both high-level global semantic information and the fine spatial structure information contained in low-level features during the recovery of the high-resolution feature map.

[0252] (3) Output layer

[0253] The output layer receives the final high-resolution fused features from the decoder. The global-local feature extraction network generates multi-channel heatmaps with the same spatial dimensions as the original input image through the output layer. These heatmaps represent the capture quality score heatmaps. Capture angle heatmap and capture width heatmap It provides pixel-level prediction results for crawling tasks.

[0254] Part Four: Global-Local Feature Extraction Network Training Method and Pose Inference for Grasping

[0255] This method models the pose estimation problem as a pixel-level regression problem. During the training phase, the Smooth L1 loss function is used to construct the total loss function, which is defined as follows:

[0256]

[0257] In the process of modeling the grasping angle, a sinusoidal (…) model is adopted. ) and cosine component ( To avoid discontinuities caused by angle periodicity, the final grasping angle mapping can be calculated using inverse trigonometric functions. This process avoids explicit candidate generation and filtering steps, resulting in higher computational efficiency and real-time performance in the capture detection process. Finally, the global-local feature extraction network is trained by weighted combination of four sub-losses.

[0258]

[0259] in This represents the loss corresponding to the crawling quality map, used to guide the network to focus on areas with a higher crawling success rate; To compensate for the loss in the gripping width prediction, we ensure that the gripper width predicted by the network is consistent with the actual gripping requirements, thereby achieving stable gripping. and These represent the losses calculated from the sine and cosine components of the grasping angle, respectively. Predicting these two components allows for a continuous and relatively accurate estimation of the grasping angle, effectively mitigating the discontinuity problem in angle representation. Coefficients , , and These are the weighting coefficients for each loss, used to balance the contribution of different losses to the overall objective function, ensuring that the model achieves a reasonable balance between grasp quality, grip width, and angle prediction.

[0260] By capturing quality score heatmaps By searching for the optimal location, the optimal grab point in the current scene can be obtained. :

[0261]

[0262] Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. By calibrating the camera's intrinsic and extrinsic parameters, the capture pose in the pixel coordinate system can be determined. Grab pose representation in world coordinate system ,in This represents the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system. This represents the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0263] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0264] The proposed object grasping pose estimation method based on global-local feature fusion mainly consists of grasping pose parameterization, two-dimensional Gaussian distribution coding, network structure design, network training, and pose inference. To verify the effectiveness of the method, the Cornell dataset and Jacquard dataset, which are commonly used in the field of robot grasping, were selected for validation. A desktop computer was used for network model training. The computer hardware environment consisted of a 13th Gen Intel(R) Core(TM) i7-13700KF (3.40 GHz) CPU, an NVIDIA GeForce RTX 4070 GPU with 12GB of video memory, and 32GB of RAM. For the software environment, the computer used Ubuntu, version 20.04 LTS. Robot Operating System (ROS) is the most commonly used development framework in the robotics field, providing unified mechanisms for topics, messages, services, and subscriptions, integrating a large number of function packages, and is modular and easily portable. Since ROS is exclusive to the Ubuntu system, the computer used in this example uses the Ubuntu system. The program is written in Python, a high-level interpreted computer programming language, with Python version 3.8.20; the network model is built using PyTorch, an open-source deep learning framework based on Python, with PyTorch version 2.4.1; and parallel acceleration computing is performed using the Unified Computing Device Architecture (CUDA), with CUDA version 12.1.

[0265] The specific implementation method is as follows:

[0266] Part 1: Capturing the parametric representation of pose:

[0267] The Cornell dataset contains 885 color-depth images at a resolution of 640×480, covering 240 different real-world objects. Each feasible grasp is labeled with multiple bounding boxes to describe the grasp location, orientation, and scale. Due to the relatively small overall size of the Cornell dataset, various data augmentation strategies were employed during training to mitigate overfitting and improve the model's generalization ability. These strategies included random cropping, scaling, and rotation transformations to effectively expand the number of training samples and enhance data diversity. The Jacquard dataset is a large-scale robot grasping dataset, consisting of 54,485 scenes generated from 11,619 different objects, containing a total of 4,967,454 grasp annotations. For each scene, the dataset provides a rendered RGB image, a segmentation mask, two depth maps, and corresponding grasp annotations. The entire dataset contains approximately 1.1 million valid grasp samples. Thanks to its advantages in scale and object diversity, the Jacquard dataset can provide sufficient data support for model training, which is conducive to improving the robustness and generalization performance of the proposed method in complex scenarios.

[0268] The system takes 4-channel color-depth images from the Cornell and Jacquard datasets as input, denoted as...

[0269]

[0270] in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. :

[0271]

[0272] in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as:

[0273]

[0274] To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels. However... This only represents the grasping pose in pixel coordinates. A feasible grasping pose definition needs to be in world coordinates (see appendix). Figure 1 ), represented as:

[0275]

[0276] in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawling point.

[0277] The second part involves encoding the labeled information using a two-dimensional Gaussian distribution:

[0278] A two-dimensional Gaussian distribution is used to encode the capture annotation information during the training phase. For each annotation's capture rectangle, the center pixel is taken as the mean center of the Gaussian distribution. The closer a pixel is to the center, the closer its capture quality score is to 1; the farther away it is, the lower its quality score becomes gradually. The capture pose parameterized by the two-dimensional Gaussian distribution encoding is as follows:

[0279]

[0280] in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is:

[0281]

[0282] In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for Gaussian distribution.

[0283] Part Three: Design of a Global-Local Feature Extraction Network Structure Based on Transformer-CNN Fusion:

[0284] The network consists of three main parts: an encoder, a decoder, and an output layer. The encoder deploys global and local feature extraction modules in parallel and fuses these two types of features to obtain a unified global-local feature representation. The decoder introduces skip connections to access shallow, low-level features from the encoder and uses an upsampling decoding module to parse high-level abstract features, gradually recovering the spatial dimensions of the feature map. The output layer ultimately outputs a capture quality score heatmap, a capture angle heatmap, and a capture width heatmap, providing pixel-level prediction results for capture pose estimation.

[0285] The network takes a 224×224 image as input and outputs three pixel-level heatmaps with the same resolution as the input, used to represent different grasping-related parameters. Each global feature extraction module in the network contains a shifting Transformer block with corresponding attention heads set to 3, 6, and 12, respectively, to model multi-scale feature representations at different levels. The image patch size in the model is set to 2, the window size to 7, and the output feature dimension of the first block embedding module is 96.

[0286] Part Four: Neural Network Training Methods and Grasping Pose Inference

[0287] This method models the pose estimation problem as a pixel-level regression problem. During the training phase, the Smooth L1 loss function is used to construct the total loss function, which is defined as follows:

[0288]

[0289] Training a global-local feature extraction network is achieved by weighted combination of four sub-losses:

[0290]

[0291] The complete training process is as follows:

[0292] (1) Training set preprocessing and ground truth label generation: All color-depth map samples in the Cornell dataset and Jacquard dataset were processed respectively: The original color and depth channels were stitched together into a 4-channel image and uniformly scaled to 224×224 resolution; For all manually labeled capture rectangles in each image, the ground truth capture quality score heatmap was generated using two-dimensional Gaussian distribution encoding with the mean of the center pixel of the capture box; Ground truth capture width heatmap and ground truth angle component heatmap were generated simultaneously; Random cropping, scaling and random rotation transformation data augmentation were performed on the small-scale Cornell dataset to expand the sample and suppress overfitting.

[0293] (2) Training batch sampling: During the training phase, a random sampling strategy is used to select training samples from the dataset. The default training batch size is set to 8.

[0294] (3) Network forward propagation: Batch color-depth maps are fed into the global-local feature extraction network, which sequentially completes encoder dual-branch feature extraction, three-layer feature fusion, decoder skip connection upsampling, and output layer heatmap prediction, outputting batch prediction capture quality score heatmap. Through By searching for the optimal location, the optimal grab point in the current scene can be obtained. :

[0295]

[0296] Even if Find the pixel position of the maximum value Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. This yields a heatmap of the grab angle and a heatmap of the grab width.

[0297] (4) Loss calculation and backpropagation: The Smooth L1 sub-loss is calculated pixel by pixel between the heatmap predicted by the network and the ground truth heatmap generated by preprocessing, and the total loss is obtained by weighted summation according to the weight coefficients. The gradient is backpropagated based on the chain rule to update all network parameters of the encoder, decoder, and output layer.

[0298] (5) Optimizer and learning rate update: The adaptive moment estimation optimizer with weight decay regularization is selected to perform parameter gradient update, and the initial learning rate is set to 0.001. A segmented learning rate decay strategy is adopted: the initial learning rate is fixed throughout the first 5 training rounds; starting from the 6th training round, the learning rate is gradually reduced by multiplying by the decay coefficient 0.8 after each complete training round, so as to slow down the parameter update amplitude in the later stage and improve the convergence accuracy.

[0299] (6) Set the total number of iterations to 100 and execute steps (2) to (5) in a complete loop until the training limit of 100 rounds is reached. Retain the weight model corresponding to the optimal validation set accuracy for subsequent pose inference.

[0300] The complete execution flow of pose reasoning is as follows:

[0301] After the global-local feature extraction network is trained, the optimal weight model is loaded, gradient calculation is disabled, and only forward propagation inference is performed. Color-depth images captured in real-time by the camera or color-depth map samples from the test set in the dataset are preprocessed, scaling the resolution to 224×224 and using this as input to the global-local feature extraction network. The network outputs a 224×224 resolution heatmap of the grasping quality score, grasping angle, and grasping width. The grasping quality score heatmap is then traversed to search for the pixel coordinates corresponding to the global maximum value. This yields the optimal capture point for the current scene. The angle and width values ​​from the capture angle heatmap and capture width heatmap at that pixel are read. By calibrating the camera's intrinsic and extrinsic parameters, the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system is obtained. Coordinate transformation matrix from camera coordinate system to world coordinate system This is used to capture the pose in the pixel coordinate system. Grab pose representation in world coordinate system The task is then assigned to the robotic arm to perform the grasping action.

[0302] Part 5: Experimental Results and Analysis of the Dataset

[0303] The distribution of experimental results for some samples from the Cornell and Jacquard datasets is shown in the appendix. Figure 6 and attached Figure 7 As shown in the attached diagram, when there are multiple high-scoring crawl candidates in the crawl quality graph, the two highest-scoring crawls are selected as the visualization objects, corresponding to the optimal and second-best crawl postures, respectively. Figure 6 The first row of rectangles shows the prediction results for grasping quality, grasping angle, and grasping width, respectively, to visually demonstrate the network's output under different grasping parameters. Experimental results show that this method can generate grasping postures with high grasping quality scores, while maintaining a consistent and reasonable ranking among multiple candidate grasps; furthermore, it also exhibits good performance in predicting grasping angle and gripper width. These advantages enable this method to effectively improve the grasping success rate in complex scenarios and demonstrate good generalization ability and stability.

[0304] The Cornell dataset is relatively small, so five-fold cross-validation was used for evaluation. The algorithm's performance was tested under two data partitioning methods: image-based partitioning and object-based partitioning. In the image-based partitioning, the dataset was randomly divided into training and validation sets, allowing the same object instance to appear in both sets. This setting primarily evaluates the model's ability to predict and grasp objects when their poses change. The object-based partitioning method partitions the dataset according to object instances, ensuring that objects appearing in the validation set were never encountered during training. This setting more realistically reflects the model's robustness and generalization ability when facing unknown objects.

[0305] Our method achieved 99.32% accuracy on the Cornell dataset using image-based segmentation and 99.1% accuracy using target-based segmentation. Comparative experiments with existing crawling detection methods, including TF-Grasp and SE-ResUNet, showed that existing methods achieved accuracies of 97.99% and 98.2% respectively for image-based segmentation and 96.7% and 97.1% respectively for target-based segmentation. The Jacquard dataset, being large enough for direct testing, yielded an accuracy of 96.8%, while TF-Grasp achieved 94.6% and SE-ResUNet 95.7%. These results demonstrate that our method outperforms existing mainstream crawling detection methods in terms of recognition accuracy across both datasets and segmentation rules.

[0306] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for estimating object grasping pose based on global-local feature fusion, characterized in that, Color and depth images are used as inputs to a Transformer-CNN fusion global-local feature extraction network. This network consists of an encoder, a decoder, and an output layer. The encoder comprises multiple layers, each with both a global feature extraction module and a local feature extraction module. Global semantic features and local detail features are concatenated and fused to obtain a unified global-local feature representation, which is then input into the decoder. The global feature extraction module is built using a shift-window Transformer, while the local feature extraction module is built using a Convolutional Neural Network (CNN). The decoder introduces skip connections to access... The shallow features in the encoder are analyzed using an upsampling decoding module to parse the high-level abstract features of the encoder and gradually restore the spatial dimensions of the feature map to the output layer. The output layer finally outputs a grasping quality score heatmap, a grasping angle heatmap, and a grasping width heatmap. The optimal grasping position is determined based on the grasping quality score heatmap, and the grasping pose prediction result of the target object is obtained by combining the grasping angle and grasping width of the corresponding pixels. Among them, a ground truth heatmap is generated from the training set data using two-dimensional Gaussian distribution coding. The heatmap predicted by the global-local feature extraction network is compared with the ground truth heatmap pixel by pixel, and the training of the global-local feature extraction network is completed through iteration.

2. The object grasping pose estimation method based on global-local feature fusion as described in claim 1, characterized in that, The global-local feature extraction network structure based on Transformer-CNN fusion is designed as follows: The global-local feature extraction network consists of three main parts: an encoder, a decoder, and an output layer. Following the data flow, it includes an input layer, encoder, decoder, and output layer. The input layer receives a four-channel color-depth image after resolution preprocessing. Each layer of the encoder has parallel global and local feature extraction modules, which concatenate and fuse global semantic features and local detail features to obtain a unified global-local feature representation. The decoder introduces skip connections, accessing shallow features from the encoder. An upsampling decoding module parses the high-level abstract features of the encoder and gradually restores the spatial dimensions of the feature map to the output layer. The output layer ultimately outputs a grasping quality score heatmap, a grasping angle heatmap, and a grasping width heatmap. The optimal grasping position is determined based on the grasping quality score heatmap, and the grasping pose prediction result of the target object is obtained by combining the grasping angle and grasping width of the corresponding pixels. The structural designs of the encoder, decoder, and output layer are as follows: (1) Encoder The encoder core consists of three global feature extraction modules and three local feature extraction modules, which are used to capture global semantic information and local detail features in the scene, respectively. The encoder receives 4-channel color-depth image features from the input layer as the first layer input. In each encoding process, the same input feature is simultaneously fed into the global feature extraction module and the local feature extraction module. The two branches complete feature extraction in parallel. Subsequently, the two types of features are concatenated and fused along the channel dimension to form the fused feature of the current layer. This fused feature is then used as the input to both branches of the encoder in the next layer. The detailed process is as follows: Input features First, the data passes through a global feature extraction module and a local feature extraction module in the first layer, respectively, to obtain global semantic information and local detail features. Then, these two types of features are concatenated and fused to obtain a global-local integrated feature that combines global context and local structural information. This feature serves as the new input feature to the next layer's global and local feature extraction modules, requiring a total of three feature concatenation and fusion operations. The feature fusion module receives the global semantic features from the global feature extraction module and the local detail features from the local feature extraction module, and performs feature concatenation operations along the channel dimension to obtain a fused feature representation. This fused feature serves as the output of the current encoding layer, inputting into the next layer's encoder to continue extracting high-level semantic information, and also being passed to the corresponding scale's decoder through skip connections during the decoding stage. The global feature extraction module is built using a shifted window Transformer to extract global semantic features from the input image. It receives the input image from the previous encoding layer. After preprocessing, the input image is first input to the block embedding module for vectorization encoding, and then sent to the window attention module for feature extraction. The final global features output by the module are sent to the feature fusion module to be fused with the output of the local feature extraction module. The specific workflow is as follows: Input features First, the feature maps are vectorized and encoded using a block embedding module, mapping them to a high-dimensional feature space. Then, the encoded features are input into a window attention module to perform attention calculations, thereby modeling the global dependencies between features and ultimately generating the module's output features. : The block embedding operation can be represented as: in, For the input feature map, For block operations, For flattening operation, This is a linear projection operation, and simultaneously, the input features... The second path also requires a patch merging module to perform a merging operation. This involves first stitching adjacent image patches together, then performing layer normalization and linear projection. The patch merging operation, PatchMerging, is represented as: in, For splicing operations, As a layer normalization operation, the input features are normalized in the channel dimension, which effectively reduces the internal covariate offset problem, thereby stabilizing and accelerating the training process of the network and improving the convergence performance of the model. The feature map downsampled by the block merging module will be further input into the window attention module to model the global-local dependencies between features. The outputs of both the block embedding module and the block merging module serve as inputs to the window attention module. The window attention module is the core computational unit of the global feature extraction module. Its structure consists of layer normalization, multilayer perceptron, window-based multi-head self-attention, and shifted window multi-head self-attention, coupled with residual connections for stable training. The computational flow of this module is as follows: in, Indicates the first The feature representation output by the Transformer module of the layer. The output feature of the previous layer is first fed into a window-based multi-head self-attention system for feature modeling. This represents the intermediate feature representation after window self-attention computation. Multi-head self-attention for windows is achieved by performing multi-head self-attention computation within a local window to model the feature relationships within the window. It is a multilayer perceptron, consisting of multiple fully connected layers and nonlinear activation functions, used to perform nonlinear transformation and channel information fusion on the features output by the window attention module, so as to further enhance the feature representation capability; To enable multi-head self-attention in shifted windows, the window division position is shifted to establish information interaction between adjacent windows; The local feature extraction module, built on a CNN, is used to extract fine-grained local features such as image edges, textures, and contours. It receives the same input image as the global feature extraction module and ultimately outputs local detail features, which are then fed into the feature fusion module. It incorporates a residual structure and a batch normalization layer, using a Gaussian error linear unit as the activation function. The Gaussian error linear unit introduces a smoother non-linear transformation by probability-weighting the input. The specific computational flow of this module is as follows: Input features Firstly, through Convolutional layers extract features, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and introduce nonlinearity to enhance the network's expressive power. Finally, the first set of features is obtained, along with the original input features. through Convolution expands the dimensions to obtain a second set of features. The two sets of features are then added element-wise to obtain the intermediate features. ; Passing by again After convolutional layers, batch normalization layers, and Gaussian error linear unit activation functions, and with the initial intermediate features... Element-wise addition, with residual connections used to mitigate the gradient vanishing problem, yields the final output features of the local feature extraction module. The above process can be represented as follows: in, For convolution operations, For batch normalization operations, The activation function for the Gaussian error linear unit; (2) Decoder The decoder receives the fused features from the encoder's last stage output as input, and after three stages of upsampling, restores the spatial resolution to match the input image. The decoder consists of three upsampling decoding modules. In each module, the fused feature map is first compressed in channel dimension through convolution, then upsampled using transposed convolution, followed by batch normalization layers and Gaussian error linear unit activation functions to stabilize the training process and enhance the network's nonlinear expressive power. The decoding process is formally represented as follows: in, This indicates a transposed convolution operation. During the decoding stage, a feature fusion operation is introduced at the output of the first upsampling decoding module. The global-local fusion features obtained from the second layer of the encoder are introduced to compensate for the details that may be lost during the upsampling process. At the output of the second upsampling decoding module, the features from the first layer of the encoder are fused with the features from the decoder through a skip connection. (3) Output layer The output layer receives the final high-resolution fused features output by the decoder. The global-local feature extraction network generates a multi-channel heatmap with the same spatial size as the original input image through the output layer. These multi-channel heatmaps represent the capture quality score heatmaps. Capture angle heatmap and capture width heatmap It provides pixel-level prediction results for crawling tasks.

3. The object grasping pose estimation method based on global-local feature fusion as described in claim 2, characterized in that, globally... The training method for the local feature extraction network and the process for capturing pose are as follows: The pose estimation problem is modeled as a pixel-level regression problem. During the training phase, the Smooth L1 loss function is used to construct the total loss function, which is defined as follows: In the modeling of the grasping angle, an angle representation based on sine and cosine components is adopted to avoid the discontinuity problem caused by the periodicity of the angle. The final grasping angle mapping can be calculated using inverse trigonometric functions. This process avoids explicit candidate generation and filtering steps, making the capture detection process more computationally efficient and real-time. Finally, the global-local feature extraction network is trained by weighted combination of four sub-losses. in This represents the loss corresponding to the crawling quality map, used to guide the network to focus on areas with a higher crawling success rate; To compensate for the loss in the gripping width prediction, we ensure that the gripper width predicted by the network is consistent with the actual gripping requirements, thereby achieving stable gripping. and These represent the losses calculated using the sine and cosine components of the grasping angle, respectively, with coefficients... , , and These are the weighting coefficients for each loss, used to balance the contribution of different losses to the overall objective function, ensuring that the model achieves a reasonable balance between grasping quality, gripping width, and angle prediction. By capturing quality score heatmaps Search for the optimal location to obtain the optimal grab point in the current scene. : Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. By calibrating the camera's intrinsic and extrinsic parameters, the capture pose in the pixel coordinate system can be determined. Grab pose representation in world coordinate system ,in This represents the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system. This represents the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

4. The object grasping pose estimation method based on global-local feature fusion as described in claim 2, characterized in that, The global-local feature extraction network based on Transformer-CNN fusion takes a 4-channel color-depth image, fused from RGB and depth images acquired by a sensor, as input, denoted as: in , Given the height and width of the input image, respectively, a global-local feature extraction network is used to uniformly represent the grasping pose estimation as a mapping relationship from the image space to the grasping feature space. : in The output space, composed of multi-channel crawling feature heatmaps, is jointly modeled at the pixel level for crawling-related information, and is represented as: To capture angle heatmaps, To capture the width heatmap, To capture a quality score heatmap, used to characterize the capture feasibility corresponding to different pixels, but This only represents the grasping pose in pixel coordinates. A feasible grasping pose is defined in world coordinates, as shown in Figure 1, and is represented as follows: in This indicates the coordinates of the center point of the gripper tool in the world coordinate system. Indicates grippers surrounding The angle of rotation of the axis. This indicates the closing width of the gripper jaws when gripping an object. This indicates the crawling quality score for that crawling point.

5. The object grasping pose estimation method based on global-local feature fusion as described in claim 1 or 2, characterized in that, The specific steps of the two-dimensional Gaussian distribution coding are as follows: A two-dimensional Gaussian distribution is used to encode the capture annotation information during the training phase, as shown in Figure 2. For each annotation capture rectangle, its center pixel is taken as the mean center of the Gaussian distribution. The closer the pixel is to the center, the closer the capture quality score is to 1; the farther away, the quality score decreases smoothly step by step. The capture pose parameterized by the two-dimensional Gaussian distribution encoding in the pixel coordinate system is as follows: in The crawling quality score heatmap generated by encoding a two-dimensional Gaussian distribution satisfies Two-dimensional Gaussian distribution The calculation formula is: In the formula This represents the point capture in pixel coordinates. To capture the pixel coordinates of the center point of the rectangle, , Preset parameters for Gaussian distribution.

6. The object grasping pose estimation method based on global-local feature fusion as described in claim 2, characterized in that, The training process for the global-local feature extraction network based on Transformer-CNN fusion is as follows: (1) Training set preprocessing and ground truth label generation: All color-depth map samples in the Cornell dataset and Jacquard dataset were processed respectively: The original color and depth channels were stitched together into a 4-channel image and uniformly scaled to 224×224 resolution; For all manually labeled grabbing rectangles in each image, the ground truth grabbing quality score heatmap was generated using two-dimensional Gaussian distribution encoding with the mean of the center pixel of the grabbing box; Ground truth grabbing width heatmap and ground truth angle component heatmap were generated simultaneously; Random cropping, scaling and random rotation transformation data augmentation were performed on the small-scale Cornell dataset to expand the sample and suppress overfitting; (2) Training batch sampling: During the training phase, a random sampling strategy is used to select training samples from the dataset. The default training batch size is set to 8. (3) Network forward propagation: Batch color-depth maps are fed into the global-local feature extraction network, which sequentially completes encoder dual-branch feature extraction, three-layer feature fusion, decoder skip connection upsampling, and output layer heatmap prediction, outputting batch prediction capture quality score heatmap. Through By searching for the optimal location, the optimal grab point in the current scene can be obtained. : Even if Find the pixel position of the maximum value Furthermore, it reads the remaining grasping attributes from the corresponding pixel positions to recover the complete grasping pose estimation. This yields a heatmap of the grab angle and a heatmap of the grab width. (4) Loss calculation and backpropagation: The Smooth L1 sub-loss is calculated pixel by pixel between the heatmap predicted by the network and the ground truth heatmap generated by preprocessing, and the total loss is obtained by weighted summation according to the weight coefficients. Backpropagation of gradients based on the chain rule updates all network parameters of the encoder, decoder, and output layer. (5) Optimizer and learning rate update: The adaptive moment estimation optimizer with weight decay regularization is selected to perform parameter gradient update, and the initial learning rate is set to 0.001; a segmented learning rate decay strategy is adopted: the initial learning rate is fixed throughout the first 5 training rounds. Starting from the 6th training round, the learning rate is gradually reduced by multiplying the learning rate by a decay factor of 0.8 after each complete training round. (6) Set the total number of iterations to 100, and execute steps (2) to (5) in a complete loop until the training limit of 100 rounds is reached; retain the weight model corresponding to the optimal validation set accuracy for subsequent pose inference; The complete execution flow of pose reasoning is as follows: After the global-local feature extraction network is trained, the optimal weight model is loaded, gradient calculation is disabled, and only forward propagation inference is performed. Preprocessing is performed on color-depth images captured in real-time by the camera or color-depth map samples from the test set in the dataset, scaling the resolution to 224×224 and using this as input to the global-local feature extraction network. The network output is a 224×224 resolution heatmap of the capture quality score, capture angle, and capture width. The capture quality score heatmap is then traversed to search for the pixel coordinates corresponding to the global maximum value. The optimal capture point in the current scene is obtained. The angle and width values ​​of the capture angle heatmap and capture width heatmap at that pixel are read. By calibrating the camera's intrinsic and extrinsic parameters, the coordinate transformation matrix from the pixel coordinate system to the camera coordinate system is obtained. Coordinate transformation matrix from camera coordinate system to world coordinate system This is used to capture the pose in the pixel coordinate system. Grab pose representation in world coordinate system The task is then assigned to the robotic arm to perform the grasping action.

Images