Robotic grasping using high efficiency visual transformer

Abstract

A method for performing robotic grasping includes acquiring an image of a scene having one or more objects, and processing the acquired image using a trained grasping neural network. A depth frame and a color frame of the image are each passed through a convolutional block of an encoder for extracting feature maps via downsampling of the respective image frames. The feature maps extracted from the depth frame and the color frame are fused to produce a fused feature map. The fused feature map is spatially divided into image patches and fed as input to a vision transformer to encode an image patch based on information from other image patches in the fused feature map. An output of the vision transformer is fed to a decoder to construct a grasp map of the scene. A best grasp location is estimated from the grasp map.

Description

Technical Field

[0001] This disclosure generally relates to the field of robotics in industrial automation, and more specifically to methods and systems for learning and performing robotic grasping using neural networks. Background Technology

[0002] Robotic bin picking of unknown objects is a key component in advancing warehouse and production line automation. Recent advances in computer vision and deep learning have enabled the large-scale deployment of this technology for picking objects of various geometries and arbitrary arrangements. Camera systems, such as RGB-D cameras, can collect color images and depth maps or point clouds of bins with randomly configured objects. The camera input can then be fed into deep neural networks (“grasping neural networks”), which have been trained to calculate the optimal gripping position or “picking point” based on that input.

[0003] The success of a robotic bin picking solution can be measured by customer-centric KPIs, including operator interventions and successful picks per hour. Minimizing operator resets and maximizing picks per hour are crucial for continuous system operation and maximizing throughput. However, achieving this necessitates ensuring the gripping neural network not only provides effective pick points but also centers them regardless of geometry and places them on completely unobstructed objects (not blocked by any other objects). This can be challenging due to potential issues such as "double picking" (overpicking when the pick point lies on the boundary between two objects) and "pop-out" (throwing another object out of the bin during high-speed picking when the pick point lies on an obstructed object). Furthermore, the object configurations that must be handled can vary almost infinitely, and the gripping neural network is often deployed at the edge of resource-constrained equipment. Finally, the input image frames may be of poor quality. All these factors must be considered when designing an end-to-end gripping neural network that outputs the best possible pick point given the high variability in object type and arrangement. Summary of the Invention

[0004] This disclosure provides methods, systems, and computer program products for solving and overcoming one or more of the aforementioned technical challenges. This disclosure relates to an end-to-end deep learning-based grasping pipeline that can provide robust, centered, and unobstructed picking points on randomly configured unknown objects and can operate efficiently on low-computing platforms such as edge devices.

[0005] A first aspect of this disclosure provides a computer-implemented method for performing robotic grasping. The method includes: acquiring an image of a scene comprising one or more objects via a camera, the image being defined by depth image frames and color image frames. The method further includes: processing the acquired image using a trained grasping neural network. This processing includes: passing the depth image frames and color image frames through convolutional blocks of an encoder to extract feature maps via downsampling of the respective image frames. The processing further includes: fusing the feature maps extracted from the depth image frames and color image frames to produce a fused feature map. The processing further includes: feeding the fused feature map, spatially divided into image patches, as input to a vision transformer to encode the image patches based on information from other image patches in the fused feature map. The processing further includes: feeding the output of the vision transformer to a decoder to construct a grasping map of the scene. The method further includes: estimating an optimal grasping position based on the grasping map.

[0006] A second aspect of this disclosure provides a computer implementation method for learning robot grasping. The method includes: inputting images from a training dataset into a grasping neural network, the grasping neural network including an encoder-decoder architecture, each image including a color image frame and a corresponding depth image frame, these image frames depicting a scene including one or more objects placed in a random configuration. The method further includes: for each image, passing the depth image frame and the color image frame through a convolutional block of the encoder to extract a feature map via downsampling of the corresponding image frame. The method further includes: for the image, fusing the feature maps extracted from the depth image frame and the color image frame to produce a fused feature map. The method further includes: for the image, feeding the fused feature map, spatially partitioned into image patches, as input to a visual transformer to encode the image patches based on information from other image patches in the fused feature map. The method further includes: for the image, feeding the output of the visual transformer to a first decoder to construct a grasping map of the scene. The method includes: training the grasping neural network based on a grasping loss of the first decoder, the grasping loss being calculated based on a benchmark ground-value grasping map of the images in the training dataset.

[0007] Other aspects of this disclosure provide systems and computer program products embodying the described methods.

[0008] Additional technical features and benefits can be achieved through the technology disclosed herein. Embodiments and aspects of this disclosure are described in detail herein and are considered part of the claimed subject matter. For a better understanding, reference is made to the specific embodiments and accompanying drawings. Attached Figure Description

[0009] The foregoing and other aspects of this disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. For ease of identification of any element or action discussed, the highest digit or multiple highest digits in the reference numerals refer to the figure number in which the element or action is first introduced.

[0010] Figure 1 An autonomous system configured to perform robotic grasping for bin picking applications is illustrated schematically according to one or more embodiments.

[0011] Figure 2 An encoder-decoder neural network architecture combining a vision transducer for robotic grasping, according to one or more embodiments, is schematically illustrated.

[0012] Figure 3 The diagram schematically illustrates a method according to one or more embodiments. Figure 1 Further improvements to the neural network architecture shown include a segmentation decoder in addition to the grasping decoder.

[0013] Figure 4 A comparison between conventional computationally intensive attention and lightweight attention computation is shown in a visual transformer according to the disclosed embodiments.

[0014] Figure 5 , Figure 6 and Figure 7 The output of the neural network is captured in different example scenarios according to the disclosed implementation.

[0015] Figure 8 A computing environment in which embodiments of the present disclosure can be implemented is shown. Detailed Implementation

[0016] This paper describes various techniques related to robotic object grasping in industrial applications. Objects may be placed in bins or otherwise arranged, such as on a workbench. Without loss of generality, the proposed methods are described in the context of robotic bin picking applications. As used herein, the term "bin" refers to a container or other structure capable of holding physical objects (e.g., pallets, bags, shipping pallets, cartons, etc.). Robotic bin picking applications typically involve controlling a robot with a robotic arm equipped with an end effector to individually grasp objects from a pile of objects arranged in a bin. Objects may be of the same or various types and may be arranged in the bin, for example, in a random configuration or orientation.

[0017] To perform robotic grasping, a grasping neural network can be used to compute the optimal grasping location (also referred to as the "pickup point" in this paper) from camera images of a picking scene including bins containing objects. The grasping neural network is typically convolutional, allowing the network to output a grasping map given an input image (usually including depth frames). This map assigns a grasping score of some type to each pixel of the input image, indicating grasping confidence. Based on the grasping map, the optimal grasping location, or pickup point, can be computed, typically by applying the "argmax" operator to the grasping map, i.e., selecting the pixel with the highest grasping confidence.

[0018] The proposed method provides an improved deep learning-based grasping pipeline that delivers robust, centered, and occluded picking points on randomly configured unknown objects, and can operate efficiently on low-computation platforms such as edge devices. This is achieved by redesigning the grasping neural network architecture to truly understand the picking scene and the semantic and geometric properties of all objects within it. The pipeline uses depth and color image frames as inputs to the grasping neural network. The grasping neural network features an encoder-decoder architecture and leverages an efficient visual transformer with self-attention to learn the complex relationships between objects in the scene and the full range of object geometry.

[0019] The proposed method is based on a creative mechanism for fusing features extracted from input depth and color image frames, maximizing information sharing between the two modalities to consistently produce desired results, especially when one of the input modalities is in a suboptimal state. This is important because complementary and non-complementary features exist in the depth and color image frames. For example, textures on objects in the color image may not exist in the depth image, while edges in the color image should match the same edges in the depth image.

[0020] Traditionally, the input to a visual transformer is defined by dividing the input image into multiple non-overlapping image patches, which are fed to the transformer as a sequence of input tokens. For example, an 8×8 pixel image can be decomposed into 16 2×2 pixel image patches. The sequence of input tokens then undergoes a self-attention process at the transformer's attention layer. This approach can introduce several problems, particularly in the context of bin picking by autonomous robots. These challenges may include, but are not limited to, computational complexity and the fact that, unlike words in a sentence, images may begin to lose semantic meaning after being segmented into image patches.

[0021] According to the proposed method, input depth and color image frames are first downsampled through convolutional blocks of the encoder to learn a set of features (feature maps), which are then fused to produce a fused feature map. The fused feature map is spatially partitioned into image patches and passed to the visual transformer. By downsampling the depth and color image frames first, a large number of features in the images can be aggregated, allowing the transformer's attention layer to better understand the spatial relationships between image patches in the fused feature map. This results in better understanding of object boundaries, elongated objects, etc., using less training data. Typically, the proposed method enables the visual transformer to better understand picking scenarios with randomly oriented objects, where each image patch is encoded with information about every other image patch in the fused feature map. Furthermore, the prior downsampling step can reduce the computational complexity of the transformer, which may be particularly beneficial during inference time.

[0022] The proposed method architecture can be understood as following a fork-and-bone or Y-shaped structure, where each input (i.e., a depth image and a color image) is first processed through its own set of convolutional blocks, and then processed together with transformer attention. The reasoning behind this method can be summarized as follows: First, each input is used to learn its own set of low-level features (such as edges and contours); then, the feature maps extracted from these inputs are summed to enhance similar features while canceling out dissimilar features; and finally, the visual transformer jointly focuses on the best representation provided by the two inputs. The transformer output is fed into the decoder to construct a grasping map of the picking scene, based on which the optimal grasping position is estimated. This method can provide an end-to-end solution because the output directly yields the desired result.

[0023] According to the proposed method, the grasping neural network can produce highly centered selection of any unknown object that is completely unoccluded. This demonstrates the grasping neural network's high level of understanding of the geometry and spatial relationships between objects. Furthermore, due to the attention mechanism of the transformer and the fusion mechanism of this invention, the grasping neural network can compensate for poor quality in one of the input modalities, resulting in better generalization ability and robustness when the trained grasping neural network is deployed. In addition, this architecture provides high computational efficiency, enabling it to run on edge devices using only a CPU, thereby greatly expanding its application scope. Furthermore, The aspects of the proposed method can be implemented as software that can be executed by a processor. In some embodiments, the aspects of the disclosed method can be suitably integrated into automated software products based on commercial artificial intelligence (AI), such as SIMATIC Robot Pick AI™ developed by Siemens.

[0024] Now turn to the attached image. Figure 1An autonomous system 100 configured to perform robotic bin picking, according to one or more embodiments, is illustrated. In the following description, unless otherwise specified, the term "system" refers to autonomous system 100. System 100 can be implemented in a factory setting. Unlike conventional automation, autonomy empowers each asset on the factory floor with decision-making and self-control capabilities to act independently in the event of localized problems. System 100 may include one or more robots, such as robot 102, which may be controlled by computing system 104 to perform one or more industrial tasks within a physical environment, such as a factory floor. Examples of industrial tasks include assembly, transportation, etc.

[0025] The computing system 104 may include an industrial PC or any other computing device, such as a desktop or laptop computer or an embedded system. The computing system 104 may include one or more processors configured to process information and / or control various operations associated with the robot 102. The processors may include one or more CPUs, GPUs, microprocessors, or any hardware device suitable for executing instructions stored on memory, including machine-readable media. In particular, the one or more processors may be configured to execute applications for operating the robot 102, such as engineering tools.

[0026] To achieve autonomy for system 100, in some implementations, the application can be designed to operate robot 102 to perform tasks within a skill-based programming environment. Compared to conventional automation, where engineers typically participate in programming the entire task from start to finish (usually using low-level code to generate individual commands), in autonomous systems like those described herein, physical devices (such as robot 102) are programmed at a higher level of abstraction using skills rather than individual commands. These skills are derived to achieve higher-level abstract behaviors centered on how the physical device modifies the physical environment. Illustrative examples of skills include skills for grasping or picking up objects, placing objects, opening doors, detecting objects, and so on.

[0027] The application can generate controller code that defines tasks at a high level, for example, using skill functions as described above. This controller code can be transmitted to robot controller 108. From the high-level controller code, robot controller 108 can generate low-level control signals for one or more motors to control the movement of robot 102, such as the angular position of the robot arm, the rotation angle of the robot base, etc., thereby performing the specified task. In other embodiments, the controller code generated by the application can be transmitted to an intermediate control device, such as a programmable logic controller (PLC), which can then generate low-level control commands for the robot 102 to be controlled. Additionally, the application can be configured to directly integrate sensor data from the physical environment in which robot 102 operates. For this purpose, computing system 104 may include a network interface to facilitate the transfer of real-time data between the application and various sensors, such as camera 122.

[0028] Robot 102 may include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The base 112 may include wheels 114 or may otherwise be configured to move within the physical environment 106. Robot 102 may also include an end effector 116 attached to the robotic manipulator 110. The end effector 116 may include a gripper configured to grasp (hold) and pick up an object 118. Examples of end effectors include vacuum-based grippers or suction cups (as shown), opposing grippers (such as fingers or claws), magnetic grippers, etc. The robotic manipulator 110 may be configured to move to change the position of the end effector 116, enabling the picking or moving of an object 118 within the physical environment.

[0029] A robotic bin picking task may involve using an end effector 116 to individually pick objects 118 from a bin 120 containing objects 118. Objects 118 may be positioned within the bin 120 in a random configuration (or orientation). Objects 118 may be of various types or the same type. To accomplish this task, system 100 may include sensors or cameras that enable robot 102 to perceive the physical environment. As shown, these sensors may include, among other things, a camera 122 for capturing images of the picking scene, in this case, the bin 120 containing the objects 118. Camera 122 may include, for example, an RGB-D camera or a cloud sensor. In some embodiments, camera 122 may be positioned to capture images of bin 120 from a top-down perspective. The camera images may be provided as input to a computing system, such as computing system 104, for calculating the picking point where the end effector 116 performs a gripping action. The picking point can be defined by the coordinates of the end effector 116 in the 3D reference coordinate system 130, and by the approach direction typically defined by the normal to the surface of the object where the picking point is located. The picking point can be calculated using a grasping neural network according to the proposed method. The calculated picking point can be output to a controller, such as a robot controller 108, to control the end effector 116 to pick up the object 118. For example, as described above, the picking point can be output as high-level controller code to a controller from which low-level commands can be generated to control the movement of the end effector 116.

[0030] Figure 2 An architecture of a grasping neural network (hereinafter referred to as "GNN") 200 according to one or more embodiments is shown. The various components of the GNN (such as encoders 206, 208, and 210, and decoder 218 including its sub-components) can be implemented by a computing system in various hardware and programming ways. Programming of the components can take the form of processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware can include a processor for executing these instructions. For example, the program can be... Figure 1 The computing system 104 shown in the diagram operates on this system. Furthermore, processing power can be distributed among multiple system components, such as across multiple processors and memories, optionally including multiple distributed processing systems or cloud / network elements.

[0031] To train the GNN 200, a training dataset comprising images, each depicting a picking scene including one or more objects randomly configured in bins or workbenches, is utilized. In various embodiments, the training dataset may include real-world camera images of the picking scene, or synthetic camera images rendered by a simulation engine using 3D representations (e.g., CAD models) of the objects, bins / workbenches, and other physical structures present in the picking scene, or a combination of real-world and synthetic images. A non-limiting example of a commercially available 3D simulation engine suitable for this application is Process Simulate™, developed by Siemens Industry Software. In some embodiments, the synthetic images of the training dataset may be rendered by randomizing a set of environmental and / or sensor-based parameters. Examples of parameters that may be randomized include one or more of the following: the number and categories of objects present in the picking scene, the location and texture of objects in the bins, the texture of the background including the bins, the setup of the simulation camera, the lighting conditions of the scene, etc. In some embodiments, training may include initially training the GNN 200 using a large number of synthetic images generated by randomizing the parameters, and subsequently fine-tuning it using a smaller number of real-world images.

[0032] Each image in the training dataset consists of a color image frame and a depth image frame. The color image frame comprises a two-dimensional representation of the image pixels, where each pixel includes intensity values ​​for multiple color components. An example of a color frame is an RGB color frame, which is an image frame that includes pixel intensity information in the red, green, and blue channels. The depth image frame, or depth map, comprises a two-dimensional representation of the image pixels, with a depth value for each pixel. The depth value corresponds to the distance from the camera viewpoint to the surface of a scene object. The color image frame and its corresponding depth image frame for each image are aligned at the pixel level.

[0033] In some implementations, images in the training dataset can be acquired (in a physical and / or simulated environment) via an RGB-D camera configured to acquire images with red, green, and blue (RGB) color and depth (D) channels. Alternatively, images in the training dataset can also be acquired using a point cloud sensor (in a physical and / or simulated environment). A point cloud can comprise a set of points representing one or more 3D surfaces in a 3D coordinate system, where the location of each point is defined by its Cartesian coordinates in the 3D reference frame and also by the intensity values ​​of its color components (e.g., red, green, and blue). Therefore, a point cloud can comprise a colored 3D representation of all surfaces in a given scene. The point cloud can be transformed into color (RGB) and depth frames by applying a sequence of transformations based on camera intrinsics. Camera intrinsics are parameters that allow mapping between pixel coordinates in a 2D image frame and a 3D reference frame. Typically, camera intrinsics include the coordinates of the principal point or optical center and the focal length along orthogonal axes.

[0034] Continue to refer to Figure 2 The depth frame 202 of the training image can be provided as input to the first encoder 206, and the corresponding color frame 204 can be provided as input to the second encoder 208. The first encoder 206 may include one or more convolutional blocks configured to downsample each depth image frame 202 to learn a first set of features. In some embodiments, as shown, the first encoder 206 may include a series of convolutional blocks 206a, 206b, 206c, which sequentially downsample the input to extract feature maps at each corresponding convolutional block 206a, 206b, 206c. Similarly, the second encoder 208 may include one or more convolutional blocks configured to downsample each color image frame 204 to learn a second set of features. In some embodiments, as shown, the second encoder 208 may correspondingly include a series of convolutional blocks 208a, 208b, 208c, which sequentially downsample the input to extract feature maps at each corresponding convolutional block 208a, 208b, 208c.

[0035] A "convolutional block" is a building block in a convolutional neural network (CNN) used for image recognition. It can consist of one or more convolutional layers that are used to extract features from an input image. Convolutional layers are typically followed by one or more pooling layers, which are used to reduce the spatial size of the feature map while preserving the most important information.

[0036] In some implementations, each of the convolutional blocks 206a, 206b, 206c, 208a, 208b, and 208c may include an inverted residual bottleneck (IRB) module for extracting feature maps. An inverted residual bottleneck (or inverted residual block) is a type of residual block used in image models that employs an inverted structure for efficiency reasons and is therefore lightweight and suitable for deployment on low-computational platforms. Briefly, an IRB comprises a narrow-wide-narrow architecture. For example, an IRB may follow a method where the input is first expanded using a 1×1 convolution, then a 3×3 depthwise convolution (which significantly reduces the number of parameters), and then a 1×1 convolution is used to reduce the number of channels so that the input and output can be added. For a more detailed understanding of the concept of inverted residual blocks, readers can refer to the following publications: Sandler, M.; Howard, A.; Menglong Zhu; Zhmoginov, A.; Liang- Chieh Chen . MobileNetV2: Inverted Residuals and Linear Bottlenecks Bottlenecks. (IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510-4520).

[0037] In some implementations, leveraging the use of IRBs, the convolutional backbones 206a and 208a (i.e., the first convolutional blocks through which the input passes) for depth image frames and color image frames, respectively, can be designed differently from other convolutional blocks: they include additional convolutional layers in addition to the IRB modules, where these additional convolutional layers perform downsampling instead of the IRB modules performing downsampling. This new adjustment can immediately force the network to learn the most relevant features starting from the input itself, as it has been forced to compress the information.

[0038] Furthermore, in some implementations, a leaky ReLU nonlinearity can be used as the activation function each time the feature map is downsampled at a convolutional block. This helps prevent gradient vanishing during the training of GNN 200.

[0039] To fuse features extracted from depth image frame 202 and color image frame 204, the disclosed GNN 200 utilizes a fusion module defined by a third encoder 210. As shown, the third encoder 210 may further include a series of convolutional blocks 210a, 210b, and 210c. Feature maps extracted at corresponding convolutional blocks of the first encoder 206 and the second encoder 208 (such as feature maps extracted at convolutional blocks 206a and 208a, feature maps extracted at convolutional blocks 206b and 208b, and feature maps extracted at convolutional blocks 206c and 208c) can be fused via the corresponding convolutional blocks 210a, 210b, and 210c of the third encoder 210. Features extracted at the last convolutional block of encoders 206, 208, and 210 can be appropriately aggregated by aggregator 210 to generate a fused feature map 213. For example, aggregator 210 may include one or more computation modules, such as concatenation and addition modules and ASPP (Atrous Spatial Pyramid Pooling) modules. The encoder architecture shown is exemplary, and it should be noted that the number of convolutional layers and the computation modules employed in each encoder 206, 208, and 210 are design choices for those skilled in the art. For example, the number of concatenated convolutional blocks may be determined such that the fused feature map 213 to be processed by visual transformer 214 achieves the desired pixel size.

[0040] Before describing the visual transformer 214, some background on transformers may be helpful. Transformers are deep learning models that can be trained on large datasets using the now-familiar concept of self-attention to detect subtle ways in which elements (“lexicals”) in a sequence are related to each other. The ability of these architectures to model long-term relationships through attention mechanisms has driven the latest revolution in generative AI, such as ChatGPT. Attention is directed at text sentences, allowing each word (represented as a “lexical”) to understand how much it is related to every other word in the sentence through the dot product between its weight matrices. Even if the text sentences are very long, the model can compute this set of affinities for each word due to its highly parallel nature. This type of attention is called self-attention and is very common in natural language processing (NLP) deep learning models because it has brought huge performance gains. However, because the model compares the representation of each word to every other word, the complexity is O(log n). O(n^ 2.d) ,in, n It is the number of words, and d This is the dimension of the vector representing the word (which is usually quite large). Since sentences can be very long, this can be a significant bottleneck. This undoubtedly makes computationally inefficient, as the model will have to allocate a large amount of memory for certain matrices.

[0041] Extending the above principle to visual transformers, since the concept of word sequences does not exist in this case, images are typically segmented into multiple image patches to create an input lexical sequence. In the context of visual transformers, a "lexical" refers to a vector representation of an image patch. Input lexicals are generated by converting each image patch into a vector through embedding processing (typically at the initial layer of the transformer). In the proposed method, instead of the input image, the fused feature map 213 is spatially divided into multiple non-overlapping image patches that define the input lexical sequence of the visual transformer 214. The visual transformer 214 may include one or more transformer blocks that can compute patch-based attention to encode each of the image patches based on information from other image patches in the fused feature map 213.

[0042] Patch-based attention can be computed in several ways, such as Figure 4 As shown. Here, block (A) illustrates a first method for attention computation by calculating the attention matrix in a conventional manner, and block (B) illustrates a second method for lightweight attention computation according to one or more embodiments of this disclosure. For illustrative purposes, in Figure 4 In the input, the number of tokens (i.e., n () is displayed as three.

[0043] In reference block (A), in the first method, each input term can be mapped to a query term using a first weight matrix. 1、2、3 Mapping to keyword elements using the second weight matrix 1’、2’、3’ And using a third weight matrix to map to value terms (not shown), and then calculating each query term. 1、2、3 With each keyword element 1’、2’、3’ The attention matrix is ​​calculated using the inner product (dot product). a As shown in the figure, the total number of inner products is... n 2 In this diagram n =3. It should be understood that for high-resolution images, n They are usually very large, making them computationally intensive.

[0044] To address computational complexity, a second approach can be employed. In this approach, the concept of "separable attention" (described in publications such as: Sachin Mehta and Mohammad Rastegari. Separable Self-attention For Mobile Vision Transformers, Machine Journal of Learning and Research, 2023 It can be appropriately modified and adapted to the current problem so that it is not applicable to all. n Instead of calculating the attention score for each input lexical unit, the context score for each lexical unit is calculated for a single latent lexical unit. L middle.

[0045] Reference block (B), in the second method, uses a linear layer defined by a set of weights learned during the training process. L Each input word 1、2、3 They can be mapped to scalar context scores respectively. , , To calculate the context score c s First, we can calculate each input word. 1、2、3 With linear layer L The inner product is thus obtained. n A dimensional vector. This inner product operation calculates... L The affinity between the input words and the input tokens. This can then be used to... n SoftMax operation is applied to dimensional vectors to produce context scores. , and Keyword element 1’、2’、3’ It also uses a learnable weight matrix for generation, which can then be scored from the corresponding context. , , Scaling is then performed. The scaled keyword elements can then be weighted and summed to compute the context vector. c v This vector encoding draws information from all input tokens and is computationally much less costly. The encoding of each image patch can be achieved by using the context vector... c v The computation is performed by multiplying the matrix of value words representing image patches. This method directly reduces the computational complexity of patch-based attention from... O(2^n) Reduce to O(n) This can significantly accelerate reasoning speed while maintaining the advantage of learning complex relationships within images.

[0046] Turn to Figure 2 In some implementations, the visual transformer 214 may include a series of transformer blocks 214a, 214b, 214c, wherein each transformer block includes an attention layer for computing attention based on image blocks to encode image blocks, thereby generating a transformer output 216 of the visual transformer 214.

[0047] Transformer output 216 is fed to decoder 218 to construct grasping map 220. Although not shown, in some embodiments, the input to decoder 218 may also include the dimensions of the gripper or end effector of the robot used for the task. The gripper dimensions may be defined by pixel dimensions calculated based on the actual gripper dimensions (e.g., expressed in distance measurement units) and the depth image frame 202. Decoder 218 may include multiple convolutional layers (not explicitly shown) configured to upsample transformer output 216 and construct grasping map 220, which depicts pixel-level grasping confidence values ​​that identify grasping points of objects within the scene depicted in the image. Decoder 218 may be, for example, a U-Net-style decoder, which includes skip connections from fusion encoder 210 to decoder 218 for better aggregation of information.

[0048] GNN 200 can be trained during supervised learning by backpropagating the grasping loss of decoder 218. The grasping loss can be computed based on a ground truth grasping map of images in the training dataset. The ground truth grasping map can include pixel labels for each pixel of the input image, indicating a grasping confidence value. In various implementations, the ground truth pixel labels can be assigned based on manual labeling of the image (e.g., by a user locating one or more picking points in the image via a GUI), or automatically generated in the case of synthetic images. In some implementations, the grasping loss can be computed as the cross-entropy loss of decoder 218 using the ground truth grasping map. Training can include processing batches of images from the training dataset to compute the grasping loss and repeatedly updating the parameters of GNN 200 (e.g., weights and biases) until the grasping loss is minimized, for example, based on gradient descent methods.

[0049] Variants of the above architecture in Figure 3 As shown in the figure. Similar elements with similar reference numerals will not be described further. As shown in the figure, this architecture is similar to... Figure 2 The difference between the architectures described above and those described below is that the GNN 300 includes two decoding heads instead of one; that is, a second decoder 302 in addition to the first decoder 218. In this case, the transformer output 216 is tapped and fed as input to both decoders 218 and 302. In some implementations, the input to the first decoder 218 may also include the dimensions of the gripper or end effector of the robot used for the task, as described above.

[0050] The second decoder 302 may also include multiple convolutional layers (not explicitly shown) configured to upsample the transformer output 216 and produce a segmentation map 304 as output, which depicts pixel-wise segmentation of objects in the scene represented in the image. The decoder architecture shown is exemplary and includes a U-Net-style decoder that includes skip connections (shown in dashed lines) from the fusion encoder 210 to the two decoders 218 and 302 to better aggregate information.

[0051] The GNN 300 can be trained during supervised learning using a loss function defined by a combination of the grasping loss of the first decoder 218 and the segmentation loss of the second decoder 302. The grasping loss of the first decoder 218 can be computed based on a ground truth grasping map of images in the training dataset, as described above. The segmentation loss of the second decoder 302 can be computed based on a ground truth segmentation map of images in the training dataset. The ground truth segmentation map of the second decoder 302 can include a pixel label for each pixel of the input image, indicating whether the pixel belongs to an object in the scene. In various implementations, the ground truth pixel labels can be assigned based on manual labeling of the image (e.g., by a user locating objects in the scene via a GUI), or automatically generated in the case of synthetic images. In particular, the ground truth segmentation map for the second decoder 302 can be generated by computed a binary segmentation mask for each image, which segments all objects in the scene from the background and further defines the boundaries separating objects that contact each other.

[0052] Logically, if the GNN 300 begins learning the boundaries between objects on the segmentation map output 304, it should also learn to separate grasp points or pick points on the grasp map output 220. Thus, the grasp map output 220 of the first decoder 218 can be guided during training by the segmentation map output 304 of the second decoder 302 to further enable the GNN to understand the semantic and geometric properties of the scene and all objects therein, and to provide robust, centered, and unoccluded pick points on unknown objects. An example of this method is described in the applicant's International Patent Application No. PCT / US2024 / 035234, which is incorporated herein by reference in its entirety.

[0053] After training and validation, GNNs can be deployed in real-time control systems (such as...). Figure 1 The computing system 104 shown is used. In some embodiments, the real-time control system may include an edge device or other computing device running on a resource-constrained hardware platform. A non-limiting example of a hardware platform on which the trained GNN can be appropriately deployed is SIMATIC, developed by Siemens. TM The TM MFP module in the series controller.

[0054] Now for reference Figure 1 This describes an example of robotic grasping execution utilizing a trained GNN. Camera 122 captures images of a picking scene including a bin 120 containing objects 118. The camera images are processed by a trained GNN, which can be deployed on a computing system 104. (Reference) Figure 2 and Figure 3 In the architecture shown, depth frames and color frames of the camera image can be passed through convolutional blocks 206a-c and 208a-c of a trained GNN, respectively, to extract feature maps via downsampling of the corresponding image frames. The feature maps extracted from the depth and color frames can be fused to generate a fused feature map, for example, via a fusion encoder 210. The fused feature map can be spatially partitioned into image patches and fed as input to a visual transformer 214 of the trained GNN, which can encode the image patches based on information from other image patches in the fused feature map. The output of the visual transformer 214 can then be fed to a decoder 218 of the trained GNN to construct a grasping map. The optimal grasping position or picking point is estimated using the grasping map.

[0055] For example, in some implementations, the pick point can be calculated by applying the argmax operator (i.e., determining the pixel in the pick image with the highest pick confidence value) to the grab map output. The calculated pick point can be transformed from the 2D pixel frame of the grab map to the 3D real-world reference frame 130 using a series of known transformations based on the depth frame and camera intrinsics of camera 122. The estimated pick point is output to robot controller 108 to control robot end effector 116 to perform a grab, and the estimated pick point may include... X, Y, Z Coordinates and approach direction.

[0056] Figures 5 to 7 The sample output of a trained GNN according to the disclosed embodiment is shown. It can be seen that the trained GNN has a significant ability to generate picking points that are always located at the center of the object and always on the topmost (unoccluded) object.

[0057] Figure 5 This demonstrates the robustness of the GNN output in an extremely challenging scenario where closely packed objects of the same height are placed in a bin. Here, image 502 depicts the input depth image frame, image 504 depicts the input RGB image frame, image 506 depicts the GNN's grasping map output, and image 508 depicts the ground truth. Grasping map 506 is post-processed to highlight pixels with grasping confidence values ​​above a threshold. As can be seen from grasping map 506, even though depth image frame 502 provides almost no boundary information between very closely packed objects, the GNN has achieved a superior ability to generate centered grasping points on each object. Final picking point PF This indicates the pixel in image 506 that has the highest capture confidence value.

[0058] Figure 6 This demonstrates the robustness of the GNN output in a challenging scenario where a semi-transparent fluid is present above an object in a bin. Here, image 602 depicts the input depth image frame, image 604 depicts the input RGB image frame, image 606 depicts the GNN's grasping map output, and image 508 depicts the segmentation map. Grab image 606 has been post-processed to highlight pixels with grasping confidence values ​​above a threshold. As can be seen from grasp image 606, the GNN still provides the optimal (centered) picking point even though the depth image frame 602 is damaged by the semi-transparent fluid, due to the combination of feature fusion extracted from the RGB and depth images and the visual transformer effect. Final picking point P F This indicates that the pixel with the highest capture confidence value in image 606 has been captured.

[0059] Figure 7 This demonstrates the robustness of the GNN output in scenes prone to salience. Here, image 702 depicts the input depth image frame, image 704 depicts the input RGB image frame, image 706 depicts the GNN's grasping map output, and image 708 depicts the segmentation map. Grasping image 706 has been post-processed to highlight pixels with grasping confidence values ​​above a threshold. As can be seen from grasping image 706, the GNN explicitly produces very low confidence for those grasps it deems occluded or incorrectly oriented, and gives the optimal position on the easiest and most visible one. Final picking point P F This indicates the pixel in image 706 that has the highest capture confidence value.

[0060] Furthermore, particularly in the case of GNN 300, this architecture allows for the handling of ambiguity in model grasp predictions. In some scenarios, the highest grasp confidence value in the grasp map may not reach a threshold. If this occurs, the GNN may fail to generate well-defined pick points on objects in the grasp map. In some implementations, if the highest grasp confidence value is below the threshold, the estimation of pick points can be informed by a combination of the grasp map output and the segmentation map output. In this case, the segmentation map output, which has proven to be very reliable, can help guide the selection because it reveals the boundaries and regions of the objects. The final selection of pick points can be determined, for example, based on user input.

[0061] The performance of the proposed GNN architecture on low-computing platforms has also been validated. For example, when ported using the OPENVino toolkit and run on a TM MFP equipped only with a CPU and without any dedicated GPU acceleration, the GNN 300 achieves an average runtime of approximately 500 ms. In contrast, most state-of-the-art vision transformers cannot even accommodate the memory of this hardware platform. To further illustrate this, the current computing power of this hardware platform is comparable to or lower than that of a typical laptop CPU, which precisely demonstrates the level of efficiency achievable by the proposed GNN architecture in realizing its performance level.

[0062] Figure 8 An exemplary computing environment, including a computing system 802, is shown in which aspects of this disclosure can be implemented. The computing system 802 may be embodied, for example, but not limited to, an industrial PC equipped with a Linux operating system, for performing real-time control of physical equipment such as robots.

[0063] like Figure 8 As shown, the computing system 802 may include communication mechanisms, such as a system bus 804 or other communication mechanisms for transmitting information within the computing system 802. The computing system 802 also includes one or more processors 806 coupled to the system bus 804 for processing information. The processor 806 may include one or more central processing units (CPUs), graphics processing units (GPUs), AI accelerators, or any other processor known in the art.

[0064] The computing system 802 also includes a system memory 808, which is coupled to the system bus 804 for storing information and instructions to be executed by the processor 806. The system memory 808 may include computer-readable storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 810 and / or random access memory (RAM) 812. The system memory RAM 812 may include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). The system memory ROM 810 may include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). Additionally, the system memory 808 may be used to store temporary variables or other intermediate information during the execution of instructions by the processor 806. A basic input / output system 814 (BIOS) containing basic routines may be stored in the system memory ROM 810, which facilitates the transfer of information between elements within the computing system 802, such as during startup. The system memory RAM 812 may contain data and / or program modules that are immediately accessible and / or currently operated by the processor 806. The system memory 808 may additionally include, for example, an operating system 816, an application program 818, other program modules 820, and program data 822.

[0065] The computing system 802 also includes a disk controller 824 coupled to the system bus 804 to control one or more storage devices, such as magnetic hard disks 826 and removable media drives 828 (e.g., floppy disk drives, optical disk drives, tape drives, and / or solid-state drives), for storing information and instructions. Storage devices can be added to the computing system 802 using appropriate device interfaces, such as Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or FireWire.

[0066] The computing system 802 may also include a display controller 830 coupled to a system bus 804 to control a display 832, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to a computer user. The computing system 802 includes a user input interface 834 and one or more input devices (such as a keyboard 836 and a clicking device 838) for interacting with the computer user and providing information to one or more processors 806. For example, the clicking device 838 may be a mouse, light pen, trackball, or pointing stick for transmitting directional information and command selections to one or more processors 806 and for controlling cursor movement on the display 832. The display 832 may provide a touchscreen interface that allows input to supplement or replace the transmission of directional information and command selections by the clicking device 838.

[0067] The computing system 802 also includes an I / O adapter 846, which is coupled to the system bus 804 to connect the computing system 802 to a controllable physical device, such as a robot. Figure 8 In the example shown, I / O adapter 846 is connected to robot controller 848. In some implementations, robot controller 848 may include one or more motors for controlling the linear and / or angular positions of various parts of the robot, such as the arm, base, etc.

[0068] The computing system 802 can perform some or all of the processing steps of embodiments of the present disclosure in response to one or more processors 806 executing one or more sequences of one or more instructions contained in memory (such as system memory 808). Such instructions can be read into system memory 808 from another computer-readable storage medium (such as magnetic hard disk 826 or removable media drive 828). Magnetic hard disk 826 may contain one or more data repositories and data files used by embodiments of the present disclosure. The contents of the data repositories and data files can be encrypted to improve security. The processor 806 can also be employed in a multiprocessing device to execute one or more sequences of instructions contained in system memory 808. In alternative embodiments, hard-wired circuitry can be used instead of or in combination with software instructions. Therefore, the embodiments are not limited to any particular combination of hardware circuitry and software.

[0069] The computing system 802 may include at least one computer-readable storage medium or memory for storing instructions programmed according to embodiments of the present disclosure and for containing data structures, tables, records, or other data described herein. As used herein, the term "computer-readable storage medium" means any medium involved in providing instructions to one or more processors 806 for execution. Computer-readable storage media may take many forms, including but not limited to non-transitory media, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical discs, solid-state drives, magnetic disks, and magneto-optical discs, such as magnetic hard disk 826 or removable media drive 828. Non-limiting examples of volatile media include dynamic memory, such as system memory 808. Non-limiting examples of transmission media include coaxial cables, copper wires, and optical fibers, including wires constituting system bus 804. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0070] The computing environment 800 may also include a computing system 802 that operates in a networked environment using logical connections to one or more remote computers, such as a remote computing device 844. The remote computing device 844 may be a personal computer (laptop or desktop computer), a mobile device, a server, a router, a network PC, a peer-to-peer device, or other common network nodes, and typically includes many or all of the elements described above with respect to the computing system 802. When used in a networked environment, the computing system 802 may include a modem 842 for establishing communication over a network 840, such as the Internet. The modem 842 may be connected to the system bus 804 via a network interface 845 or via another suitable mechanism.

[0071] Network 840 can be any network or system generally known in the art, including the Internet, intranet, local area network (LAN), wide area network (WAN), metropolitan area network (MAN), direct connection or a series of connections, cellular telephone network, or any other network or medium that facilitates communication between computing system 802 and other computers (e.g., remote computing device 844). Network 840 can be wired, wireless, or a combination thereof. Wired connections can be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections can be implemented using Wi-Fi, WiMAX and Bluetooth, infrared, cellular networks, satellite, or any other wireless connection method generally known in the art. Furthermore, several networks can operate independently or communicate with each other to facilitate communication within network 840.

[0072] Embodiments of this disclosure can be implemented using any combination of hardware and software. Additionally, embodiments of this disclosure can be included in an article of manufacture having, for example, a non-transitory computer-readable storage medium (e.g., one or more computer program products). For example, a computer-readable storage medium may contain computer-readable program instructions implemented therein for providing and facilitating embodiments of this disclosure. This article of manufacture may be included as part of a computer system or sold separately.

[0073] Computer-readable storage media may include tangible devices that can retain and store instructions for use by an instruction execution device. Computer-readable storage media may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. The computer-readable program instructions described herein can be downloaded from the computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device.

[0074] The systems and processes shown in the figures are not exclusive. Other systems, processes, and menus can be derived from the principles of this disclosure to achieve the same objective. Although this disclosure has been described with reference to specific embodiments, it should be understood that the embodiments and variations shown and described herein are for illustrative purposes only. Modifications to the present design can be made by those skilled in the art without departing from the scope of this disclosure.

Claims

1. A computer-implemented method for performing robotic grasping, comprising: Images of a scene, including one or more objects, are acquired via a camera; these images are defined by depth image frames and color image frames. The acquired image is processed using a trained grasping neural network, the processing including: The depth image frame and the color image frame are passed through a convolutional block of the encoder to extract feature maps via downsampling of the corresponding image frames. The feature maps extracted from the depth image frame and the color image frame are fused to generate a fused feature map. The fused feature map, spatially divided into image patches, is fed as input to a visual transformer to encode the image patches based on information from other image patches in the fused feature map. The output of the visual transformer is fed into the decoder to construct a capture image of the scene, and The optimal grab position is estimated based on the grab map.

2. The method according to claim 1, wherein, The visual transformer includes a series of transformer blocks, each transformer block including an attention layer for computing attention based on image blocks to encode the image blocks, thereby generating the output of the visual transformer.

3. The method according to any one of claims 1 and 2, wherein, Encoding the image patch by the visual transformer includes calculating patch-based attention in the following manner: Each image patch represented by the input lexical is mapped to a scalar context score using a single latent lexical defined by a set of learned weights; Scaling of keyword units for each input lexical unit using the corresponding context score, and The scaled keyword tokens are summed to compute a context vector that encodes information from all input tokens.

4. The method according to any one of claims 1 to 3, wherein, The depth image frame and the color image frame are downsampled by a first series of convolutional blocks and a second series of convolutional blocks, respectively.

5. The method according to claim 4, wherein, Each of the convolutional blocks used for the depth image frame and the color image frame includes an inverted residual bottleneck (IRB) module.

6. The method according to claim 5, wherein, The first series of convolutional blocks and the second series of convolutional blocks each include convolutional backbones for depth images and color images, respectively. Each convolutional backbone includes an IRB module and an additional convolutional layer, which performs downsampling in place of the IRB module.

7. The method according to any one of claims 1 to 6, wherein, The fused feature map is generated by fusing feature maps extracted from corresponding convolutional blocks used for the depth image frame and the color image frame via a fusion module comprising a series of convolutional blocks.

8. The method according to any one of claims 1 to 7, wherein, The capture graph depicts pixel-level capture confidence values ​​for identifying capture points used to grasp objects in the scene.

9. The method according to claim 8, wherein, The optimal grab position is estimated by determining the pixel with the highest grab confidence value in the grab map constructed by the decoder.

10. The method according to any one of claims 1 to 19, comprising: The estimated optimal gripping position is output to the robot controller to control the robot to perform the gripping action.

11. A computer-implemented method for learning robot grasping, comprising: Images from a training dataset are input into a grasping neural network, which includes an encoder-decoder architecture. Each image includes a color image frame depicting a scene comprising one or more objects placed in a random configuration and a corresponding depth image frame. For each image: The depth image frame and the color image frame are passed through a convolutional block of the encoder to extract feature maps via downsampling of the corresponding image frames. The feature maps extracted from the depth image frame and the color image frame are fused to generate a fused feature map. The fused feature map, spatially divided into multiple image patches, is fed as input to a visual transformer to encode the image patches based on information from other image patches in the fused feature map. The output of the visual transformer is fed into the first decoder to construct a capture image of the scene, and The crawling neural network is trained based on the crawling loss of the first decoder, the crawling loss being calculated based on the ground truth crawling map of the images in the training dataset.

12. The method according to claim 12, wherein, For each image, the output of the visual transformer is also fed into a second decoder to construct a segmentation map. The grasping neural network is trained using a loss function defined by a combination of the grasping loss of the first decoder and the segmentation loss of the second decoder, wherein the segmentation loss is calculated based on ground truth segmentation maps of the images in the training dataset. During the training process, the capture map output of the first decoder is guided by the segmentation map output of the second decoder.

13. A non-transitory computer-readable storage medium comprising instructions that, when processed by one or more processors, configure the one or more processors to perform the method according to any one of claims 1 to 12.

14. A system for performing robotic grasping, comprising: The robot can be controlled by a robot controller. A camera is configured to capture images of a scene that includes one or more objects. Computing systems, including: One or more processors, and A memory that stores instructions executable by the one or more processors to perform the method according to any one of claims 1 to 10, thereby performing the robot's grasping based on the captured image.

Images