Network modularization for learning high-dimensional robotic tasks

Abstract

To provide a method for modularizing high dimensional neural networks into neural networks of lower input dimensions.SOLUTION: A method is suited to generating full-DOF robot grasping actions based on images of parts to be picked. In one example, a first network encodes grasp positional dimensions and a second network encodes rotational dimensions. The first network is trained to predict a position at which a grasp quality is maximized for any value of the grasp rotations. The second network is trained to identify the maximum grasp quality while searching only at the position from the first network. Thus, the two networks collectively identify an optimal grasp, while each network's searching space is reduced. Many grasp positions and rotations can be evaluated in a search quantity of the sum of the evaluated positions and rotations, rather than the product. Dimensions may be separated in any suitable fashion, including three neural networks in some applications.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates more broadly to a method for separating a high-dimensional neural network into two or more neural networks with lower input dimensions, and more particularly to a network modularization method for generating robotic motions for high-dimensional tasks, which decomposes high-degree-of-freedom (DOF) motions into multiple groups, each of which is individually explored by a neural network using specially designed data. [Background technology]

[0002] Industrial robots are widely used to perform manufacturing, assembly, and material handling operations. One such application is a pick-and-place operation, where a robot retrieves individual parts from a bin and places each part on a conveyor or transport container. In this example, molded or machined parts are dropped into a bin and placed in random positions and orientations, and a robot picks up each part and places it on a conveyor in a predefined orientation (pose), which then transports the parts for packaging or further processing. Depending on the type of parts in the bin and other factors, finger-type grippers or suction-type grippers can be used as robotic tools. A vision system (one or more cameras) is typically used to identify the position and orientation of individual parts in the bin. [Overview of the Initiative] [Problems that the invention aims to solve]

[0003] It is well known in the art to use a trained neural network system to calculate gripping commands for parts in a bin. However, existing neural network gripping learning systems have drawbacks that limit their practical use. One known system encodes top-down gripping candidates within an image patch and trains the network to predict the quality of multiple gripping candidates. This system requires a long time to calculate gripping candidates and can only generate top-down (vertical) gripping with parallel jaw grippers. Furthermore, this system is trained only on individual, separated parts / objects rather than on random gathering of parts in a bin, so it cannot predict the effects of interference between parts in a cluttered environment.

[0004] Another well-known system removes the requirement for time-consuming gripping candidate calculation by obtaining the original depth image and training the network to output the quality of each pixel. However, because of the large number of pixels contained in each image, this system cannot accurately predict each pixel. Therefore, this system is not as accurate as the system described above. Furthermore, because the encoding of the gripper's angle / width is ambiguous in this system, it cannot handle the dense and cluttered environments typical of parts in a bin. Additionally, this system can only generate straight top-down gripping solutions. Finally, this system may push the robot's gripper between adjacent parts in the bin without predicting depth, potentially damaging the gripper or the parts.

[0005] Yet another existing system attempts to determine 6-degree-of-freedom gripping with a single network. However, this system cannot handle cluttered gripping environments (such as piles of parts) in the gripping evaluation network and requires a gripping improvement step after gripping evaluation with the neural network.

[0006] In light of the above situation, a method of decomposing a high-dimensional learning neural network into two or more low-dimensional neural networks is desired, and this method is applicable to gripping planning with sufficient degrees of freedom and other applications.

Means for Solving the Problem

[0007] According to the teachings of the present disclosure, a method of separating or modularizing a high-dimensional neural network into two or more low-input-dimensional neural networks is described and illustrated. The disclosed network modularization method is particularly suitable for generating full-degree-of-freedom (full-DOF) robotic grasping operations based on images of parts to be removed in bins. In one example, a first network encodes the grasping position dimension and a second network encodes the grasping rotation dimension. The first network is trained to predict the position at which the grasping quality is maximized for any value of the grasping rotation. The second network is trained to identify the maximum grasping quality while searching only at positions specified in advance from the first network. In this way, the two networks jointly identify the optimal grasp while the dimensional search space of each network is significantly reduced. In particular, a large number of grasping positions and grasping rotations can be evaluated by the total number of searches equal to their sum rather than the product of the evaluated positions and rotations. The separation of dimensions between networks can be designed to be optimal for a particular application and, in some applications, includes three neural networks rather than two.

[0008] Additional features of the disclosed apparatus and method will become apparent from the following description and the claims, in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0009] [Figure 1] It is a block diagram of a robotic part removal system that calculates full-degree-of-freedom grasping using a separated neural network according to an embodiment of the present disclosure.

[0010] [Figure 2] It is a diagram comparing the prior art of calculating high-degree-of-freedom robotic operations using a single large neural network and the technology of the present disclosure in which the degrees of freedom are separated in a segmented neural network.

[0011] [Figure 3] This is a diagram of a multidimensional search space illustrating a method for separating search dimensions using a maximum margin network modularization approach according to an embodiment of the present disclosure, and further finding the maximum condition from the initial search space.

[0012] [Figure 4] This is a block diagram showing the steps of the training phase of a neural network modularization technique according to an embodiment of the present disclosure.

[0013] [Figure 5] This is a graph illustrating how 3D depth image data, according to an embodiment of this disclosure, is processed into a set of depth layers and provided in the training phase of Figure 4.

[0014] [Figure 6] A depth image layer of a specific depth layer in Figure 5, according to an embodiment of the present disclosure, is a diagram of the depth image layer provided in the training phase of Figure 4.

[0015] [Figure 7] This is a block diagram of the steps in the estimation phase of the neural network modularization technology according to an embodiment of the present disclosure.

[0016] [Figure 8] This is a block diagram of a neural network modularization technique applied to a 3-degree-of-freedom grasp search problem according to an embodiment of this disclosure.

[0017] [Figure 9] This is a block diagram of a neural network modularization technique applied to a 5-degree-of-freedom grasp search problem using two neural networks, according to an embodiment of the present disclosure.

[0018] [Figure 10]Figure 9 illustrates a method for training a modularized grasping learning network of the type shown, using a combination of grasping optimization of individual objects and simulation of fluctuations and interferences of clustered objects, according to an embodiment of this disclosure.

[0019] [Figure 11] This is a block diagram of a neural network modularization technique applied to a 7-degree-of-freedom grasp search problem using three neural networks, according to an embodiment of the present disclosure.

[0020] [Figure 12] This is a flowchart illustrating a method for modularizing a high-dimensional robotic task into two or more low-dimensional neural networks connected in series, according to an embodiment of the present disclosure. [Modes for carrying out the invention]

[0021] The following description of embodiments of this disclosure relating to neural network modularization techniques for learning high-dimensional robotic tasks is essentially illustrative and is not intended to limit the disclosed devices and techniques, or their applications or use.

[0022] It is well known that industrial robots are used to retrieve parts from a source and place them at a destination. In one common application, parts are supplied in bins or other containers filled with freshly cast or molded parts. Teaching a robot to recognize and grasp individual parts in a bin filled with parts in real time has always been challenging. Conventional methods involve manually teaching the robot in a structured environment. For high-dimensional tasks in unstructured environments, it is desirable to acquire robust grasping skills through deep learning using neural networks trained for pattern recognition.

[0023] However, learning high-dimensional robotic tasks typically requires encoding high-dimensional states / actions and exploring them in a high-dimensional action space using learning-based methods. For example, to learn a common 6-degree-of-freedom (DOF) bin-picking task, a neural network would need to encode high-dimensional observations and 6-degree-of-freedom grasping actions before exploring them in action space. This can complicate the network and increase its computational load.

[0024] Regarding the challenges of high-dimensional learning, two well-known existing methods reduce the search to four dimensions and restrict the direction of the grasping approach in a top-down manner. Furthermore, these learning-based methods are either not fast enough (due to the time-consuming requirements of grasping candidate computation) or not accurate enough (because they attempt to predict too many dimensions, which is difficult for neural networks). Yet another existing method uses a single neural network for a 6-degree-of-freedom grasping proposal, but this method has high search complexity, requires subsequent grasping improvements, and cannot handle the cluttered object environment typical of parts in a bin.

[0025] To overcome the shortcomings of existing methods and systems, this disclosure describes techniques for modularizing or decoupling large, high-dimensional neural networks into two or three smaller, lower-dimensional networks. This neural network modularization technique allows for significant improvements in network performance and efficiency while maintaining search accuracy. One application of the disclosed neural network modularization technique is robotic part grasping, where all degrees of freedom (DOF) of the grasp are calculated from an image of a bin filled with the part, and the calculated grasp exceeds a quality threshold.

[0026] Figure 1 is a block diagram of a robotic part picking system that uses a separate neural network to compute full-degrees-of-freedom (full-DOF) gripping, according to one embodiment of the present disclosure. The system shown in Figure 1 is provided to illustrate an exemplary application in which the disclosed neural network modularization technique can be used. A robot 100 having a gripper 102 operates within a workspace, and the robot 100 moves a part or object from a first location (bin) to a second location (conveyor).

[0027] The movement of the robot 100 is controlled by the controller 110, which typically communicates with the robot 100 via a cable 112. The controller 110 provides the robot 100 with joint movement commands and receives joint position data from the encoders of the robot 100's joints, as is well known in the art. The controller 110 also provides commands for controlling the movement of the gripper 102, including the rotation angle and width of the gripper, as well as grip / ungrip commands.

[0028] Computer 120 communicates with controller 110. Computer 120 includes a processor and memory / storage configured with a neural network for calculating grasp proposals based on three-dimensional (3D) camera images. In one embodiment, the computer 120 running the neural network in execution mode or estimation mode is the same computer on which the neural network was previously trained. In another embodiment, the neural network is trained on a different computer and provided to computer 120 for use in live robotic grasp operations.

[0029] A pair of 3D cameras 130 and 132 communicate with a computer 120 via a wired or wireless connection to provide images of the workspace. In particular, cameras 130 / 132 provide images of objects 140 within bin 150. Images from cameras 130 / 132 (including depth data) provide point cloud data that defines the position and orientation of objects 140 within bin 150. If there are two 3D cameras 130 and 132 with different viewpoints, it is possible to calculate or project a 3D depth map of objects 140 within bin 150 from any suitable viewpoint. In another embodiment, only one of the 3D cameras (130) is used and, for example, pointed to a direct vertical line of sight.

[0030] Since the position of the bin 150 relative to the robot 100 is known, when the gripping of an object 140 at its location in the bin 150 is calculated, the robot 100 can control the gripper 102 to perform the gripping. The task of the robot 100 is to pick up one of the objects 140 from the bin 150 and move that object to the conveyor 160. In the illustrated example, individual parts 142 gripped by the robot 100's gripper 102 are selected and moved along the path 180 to the conveyor 160.

[0031] For each part picking operation, computer 120 receives one or more images of the object 140 in bin 150 from camera 130 / 132. From the camera images, computer 120 calculates one or more depth maps of the pile of object 140 in bin 150. A neural network running on computer 120 uses the depth maps to determine high-quality, full-degree-of-freedom grasps for individual objects in bin 150. For example, an object at the top of a pile of objects would be a good grasp candidate because a significant portion of its side is exposed and there are no objects around it.

[0032] Once an object (such as object 142) is identified as being in a position for high-quality gripping according to the technique detailed below, the computer 120 provides individual object gripping data to the controller 110, which then commands the robot 100 to grip and move the object. The individual object gripping data provided to the controller 110 by the computer 120 preferably includes the 3D coordinates of the gripping target point, the approach angle that the gripper 102 should follow, and the rotation angle and width of the gripper (or the position of all fingers).

[0033] The controller 110 can use individual object gripping data to calculate robot motion commands that cause the gripper 102 to grip an identified object (e.g., object 142) and move the object to a desired location along a collision-free path (path 180). Instead of the conveyor 160, the destination may be a transport container in which objects are placed into individual compartments, or another surface or device in which objects are further processed in subsequent operations.

[0034] After object 142 is moved to conveyor belt 160, the pile of object 140 changes, so new image data is provided by cameras 130 / 132. Next, computer 120 must use a trained neural network to identify a new target object to grasp based on the new image data. The new target object must be identified very quickly by computer 120, as object identification and path calculation must be performed in real time at the same speed that robot 100 can move one of object 140 to pick up the next object. Efficient search provided by a low-dimensional modularized neural network enables the high-speed grasping calculation required in this grasping application.

[0035] The application described above in the system of Figure 1 preferably uses a 7-degree-of-freedom (DOF) gripping proposal from the computer 120. In one embodiment, the 7 degrees of freedom include 2 degrees of freedom for the approach direction (x and y components combined with the downward z-direction unit of the robot gripper), 3 degrees of freedom for the gripping position (x / y / z coordinates), and 2 or more degrees of freedom for the gripper rotation / spin angle and gripper width. To provide a 7-degree-of-freedom gripping proposal based on an image of a bin filled with parts, the computer 120 can advantageously use the disclosed neural network modularization technique to separate the gripping search dimension into two or three networks. The general concept of the disclosed neural network modularization is described below, followed by an example of network modularization applied to a robotic gripping application.

[0036] Figure 2 is a diagram comparing conventional techniques for computing high-degree-of-freedom robot tasks using a single large-scale neural network with the techniques of this disclosure, which separate the degrees of freedom into neural networks. In some of the figures following Figure 2, the concept of dimensions evaluated by the neural network is explained in terms of position dimension p and rotation dimension r. For example, if the width of the gripper is ignored, the position and orientation of the robot's gripper can be fully described by three positional degrees of freedom (x / y / z) (p) and three rotational degrees of freedom (yaw / pitch / roll, or two angles describing the direction of approach plus a rotation angle, or Euler angles or other appropriate angles) (r).

[0037] Box 210 in Figure 2 shows a conventional approach to compute gripping using a neural network. A single neural network 220 is used to encode all six degrees of freedom, including three position dimensions p and three rotation dimensions r. In one example, the neural network 220 is trained using a depth image of a pile of objects, along with a candidate for a quality grip of a single object manually identified. In the execution phase (also called the deployment phase or estimation phase, where the trained neural network is used to determine the grip in a live robot operation), the input at line 230 is a depth image containing enough environmental data to estimate the quality of a fully-degree-of-freedom gripping, including both the gripper's position and rotation {p,r}. The output at line 232 is the proposed optimal gripper position and rotation (p * ,r * This is the proposed dimension for grasping the quality of individual objects, which includes both of the above.

[0038] The neural network 220 in box 210 encodes the six dimensions of the gripper's pose, defined by the input environment data associated with the grip {p,r}, and in the execution phase, searches the depth image to find the optimal condition for all six dimensions (p * ,r * The goal is to identify the dimensions. A neural network that encodes and searches through these many dimensions becomes extremely complex (many layers), resulting in very slow training and search, and potentially inaccurate or ambiguous search results.

[0039] Box 250 in Figure 2 shows a novel approach to calculating grasp using a neural network. Instead of using a single neural network to calculate all dimensions of the object as in the conventional approach described above, two neural networks (260 and 270) are used. The first neural network 260 uses the input environment data at line 252 to determine the positional dimension p that yields the optimal grasp (maximum grasp quality metric) for the first neural network 260. *Three position dimensions p are encoded so that they can be explored over all values of the three rotational dimensions r. In the second neural network 270, in addition to the original input data from line 252, the optimal position dimension value p from line 262 * is used to encode all position and rotational dimensions {p,r}. However, during the execution phase, since the network 270 has the position value p * that results in an optimal grasp, the second neural network 270 only needs to explore the rotational dimension r. The output at line 272 is the proposed optimal gripper position and rotation (p * ,r * ), which are the dimensions of the proposed quality grasp of the individual object, including both.

[0040] JPEG0007851779000001.jpg69169

[0041] FIG. 3 is a diagram of a multi-dimensional search space showing a way to separate search dimensions and find maximum conditions from an original search space using a maximum margin network modularization approach according to an embodiment of the present disclosure. The combined search space over dimensions p and r is represented by grid 310. Grid 310 visually represents the search space, with columns indicating the values of p (p0, p1,..., p i ), and rows indicating the values of r (r0, r1,..., r j ). The size of the dots within each cell of grid 310 represents the value of the grasp quality Q(r,p) for that particular combination of p and r. The larger the dot, the higher the grasp quality.

[0042] As explained earlier, if p and r represent the position and rotation dimensions of the gripper pose, then each of p and r contains three dimensions or degrees of freedom. Therefore, it is easy to imagine that many different values ​​of p and r need to be explored to find the optimal gripper candidate. In other words, the values ​​of i and j can be at least several hundred. For example, if the x, y, and z dimensions of p are each divided into increments of 10, then the dimensional size of p becomes 10 × 10 × 10 = 1000. If the search of grid 310 is performed by a single neural network (such as network 220 in Figure 2), the search must be performed i·j times (i multiplied by j). This results in a very large search space, and as a result, the neural network becomes very complex and slow. If the search of grid 310 is performed by two low-dimensional neural networks (such as networks 260 and 270 in Figure 2), the search needs to be performed i+j times, but that number is much smaller. This results in a very simple and fast neural network solution, as detailed below.

[0043] As also stated above, this disclosure specifies a technique for separating (modularizing) one large, high-dimensional neural network into two (or more) simpler neural networks. The key to separating one large neural network into two simpler neural networks is to encode the first neural network into a value p that gives the greatest overall gripping quality at one of the corresponding r values. * The goal is to find the optimal position of p, which allows the second neural network to explore the r dimension at the optimal location of p. The following is a description of this method.

[0044] JPEG0007851779000002.jpg59169

[0045] The second neural network in box 320 (shown in 340) is p * =p iIn this case, by searching all values ​​of r and identifying the maximum value of Q, all other values ​​of p are hidden. Therefore, the second neural network finds the value of p that occurs at r0. * =p i Find the maximum gripping quality. Visual inspection of grid 310 is performed (r0,p i This reveals that the gripping quality at ) is not the overall maximum gripping quality. Therefore, the usual margin method shown in box 320 cannot reliably find the maximum value when used in the modularization of neural networks.

[0046] JPEG0007851779000003.jpg56169

[0047] The second neural network in box 350 (shown in 370) is p * When all values ​​of r are searched at =p0 to find the maximum value of Q, all other values ​​of p are hidden. Therefore, the second neural network, r j p that occurs * Find the maximum gripping quality of =p0. Visual inspection of grid 310 is performed (r j This reveals that the gripping quality at p0 is actually the overall maximum gripping quality. Therefore, the maximum margin method shown in box 350 yields the maximum value when used in the modularization of neural networks in one dimension (p * ) can be determined as a target value.

[0048] The second neural network 370 in box 350 is p * Since r is explored only at p0 and all other values ​​of p are hidden, the search of the second neural network is much faster (by factor of j) than a single neural network across all dimensions of grid 310. This significant improvement in the neural network's search performance is crucial in robotic grasping applications where grasping proposals corresponding to images of piles of items need to be computed in real time to support robotic control.

[0049] JPEG0007851779000004.jpg32169

[0050] Figure 4 is a block diagram of the steps in the training phase of a neural network modularization technique according to an embodiment of the present disclosure. Box 400 includes steps used to train a first neural network 410. Box 450 includes steps used to train a second neural network 460. The first (410) and second (460) neural networks in Figure 4 operate similarly to the first and second neural networks in Figure 3 described above.

[0051] As schematically shown at the top of boxes 400 and 450, the first neural network 410 encodes the grouped dimension p, thereby the first neural network, based on the input data describing the environment of {p}, * (The value of p that yields the highest quality at several values ​​of r). Next, the second neural network 460 predicts the previously identified p * Based on the input data defining the environment of all r in the context, all dimensions (p,r) are encoded, and the value (p) where the maximum grasping quality exists is obtained. * ,r * Identify )

[0052] To train the first neural network 410, maximum margin data is first prepared as shown in 420 and 430. Input data I shown in 420 r (p) represents the state that uniquely encodes action p. Since action r is hidden, I r (p) is a function of p. Output data Q shown in 430 r (p) is the quality Q(r,p) projected along the r direction using the maximum margin method (described above with respect to Figure 3). Output data Q at 430 r (p) is provided in the form of a grip quality map, which is further described below. Q r(p) is a function of p only, with r hidden. Quality is the output of the first neural network 410, but Q r (p) The data is provided as input for network training. For the grasping task, as shown in 420 and 430, r (p) represents various pieces of information about p (e.g., depth), Q r (p) is the maximum quality of various r at a specific p. Figures 5 and 6 show the input data I r (p), gripping quality data Q r (p) and the relationships between them are further shown, and these relationships are ultimately encoded within the first neural network 410 through training.

[0053] Figure 5 is a graph 500 showing how 3D depth image data is processed into a set of depth layers according to one embodiment of the present disclosure and provided in the training phase of Figure 4. For a particular 3D depth image (shown in Figure 420), multiple depth image layers are used as input data I for training the first neural network 410. r Provided as (p). In graph 500, the x-axis 510 represents one horizontal dimension of the 3D depth image, and the z-axis 520 represents the vertical dimension of the depth image (for example, the z-axis 520 could be the height of a bin filled with parts). The y-axis 512 of the 3D depth image is "into the page" of graph 500. Curve 530 shows the collective height of the pile of parts in the xz plane. In other words, curve 530 is the top of the cross-section of the pile of parts in the xz plane.

[0054] 3D depth images are each at different heights (z0, z1, ..., z i ,…,z N It can be divided into multiple layers. Height z0, represented by line 540, indicates the highest point of the depth image (e.g., a pile of parts). One depth image layer is provided for height z0, which is from the z0 level (the highest-height item) to z NThis shows a complete depth map including all items down to the level (bottom of the bin). In the z0 depth image layer, the z-coordinate origin of the depth image is set to z0, so everything in the depth image has a negative z-coordinate. Height z1, represented by line 542, represents a level slightly below the top of the depth image. Height z1 is provided with another depth image layer. This also shows the z-coordinate from the z0 level down to the bottom. N This shows a depth map including all items up to the level. However, in the z1 depth image layer, the reference origin of the depth image's z coordinate is set to z1, so everything above z1 in the depth image will have a positive z coordinate, and everything below z1 in the depth image will have a negative z coordinate. Similarly, the depth image layer will have additional level z i Provided (indicated by line 544). Each depth image layer is a complete depth image of the entire pile of parts, but each depth image layer has a different origin in the z direction. By providing multiple depth image layers by slicing at different levels, the z direction is encoded during the training of the neural network 410. The encoding of x and y is naturally performed by the 2D information of each layer. This is shown in Figure 6.

[0055] Curve 530 has shape features 550 and 560. Shape features 550 and 560 are high spots in the depth map, indicating locations where the pile of parts becomes higher due to the presence of one or more parts. Shape features 550 and 560 appear in the shape of the depth image and the gripping quality map, as will be described later.

[0056] Figure 6 is a diagram of a depth image layer 600 as a specific depth layer in Figure 5 according to an embodiment of the present disclosure, provided in the training phase of Figure 4. In the depth image layer 600, the x-axis 510 corresponds to the x-axis 510 in Figure 5, and the y-axis 512 corresponds to the y-axis 512 in Figure 5. In other words, the depth image layer 600 is a top-down depth image, and the z-axis 520 in Figure 5 is the "direction coming out of the page" in Figure 6.

[0057] Depth image layer 600 includes shapes 620 and 630, which correspond to shape features 550 and 560 in Figure 5, respectively. Shapes 620 and 630 are darkest in the center and exhibit the highest z-coordinate, corresponding to the peaks of shape features 550 and 560 in Figure 5. Depth image layer 600 is at height z i In this case, the reference origin of the depth image's z coordinate is set to z1, everything above z1 (line 544) in the depth image becomes a positive z coordinate, and everything below z1 (line 544) in the depth image becomes a negative z coordinate. Similarly, the quality map shown in 430 of Figure 4 is also stacked in layers or slices. Each layer of quality map 430 has a height z i This indicates grab candidates close to the current layer of the depth image, such as within the shaded region 570. For example, height z i The quality map includes the central quality grips of shapes 620 and 630, but with height z N The quality map may not include quality grips because there are no graspable geometric features within the depth image at that level. The grip candidates provided in quality map 430 may be generated manually by conventional methods such as heuristics, or they may be generated automatically using gripper / part optimization calculations followed by physical environment (part pile) simulations.

[0058] To summarize the preparation of maximum margin data for training the first neural network 410 (in box 400 in Figure 4), for a pile of specific items, depth images 420 (in multiple layers) and corresponding quality maps 430 (specifying the gripping quality at various points in each depth image layer) are provided using point cloud or other depth map data (e.g., a 3D camera). The above procedure is performed for piles of various items (preferably thousands). After the maximum margin data is prepared, as shown in 420 and 430 in Figure 4 and further detailed in Figures 5-6, the first neural network 410 is trained. The training steps are shown at the bottom of box 400, where the first neural network 410 is trained on environmental image data input I r (p) Based on grip quality Q rIt is trained to predict (p). Once the data is prepared, training the first neural network is straightforward and fast. The training shown in box 400 is known as supervised learning because the grasping quality (output of the first neural network 410) is provided as training data.

[0059] To train the second neural network 460 (Figure 4), data is prepared to encode all dimensions r and p. First, depth images 420 (in multiple layers) used to train the first neural network 410 are combined with corresponding quality maps 430 (specifying the gripping quality and position in each depth image layer) as shown in 470. Each of the images 420 is encoded at the position of the quality gripping associated with a particular depth image layer. The result is the input data I(r,p) shown in 480, which includes multiple depth image trims (482, 484, 486, ...), each trim being a portion of one of the depth images 420 that has been trimmed and rotated as defined by the quality map 430.

[0060] For grasping applications, in the input data I(r,p) shown in 480 (depth images are cropped in 482, 484, 486, ...), p is encoded by the cropping center (x and y from the position on the depth image layer, and z from the depth origin of that layer), and r is encoded by the cropping angle, both of which are determined from the quality map 430. To provide the best bin picking capability, it is often advantageous to enable non-vertical grasping directions. The grasping approach direction (i.e., non-vertical approach direction) can be encoded in the first and second neural networks (410 / 460) by supplying depth images 420 (used again in 470) of different field of view angles. As shown in Figure 1, depth images of different field of view angles can be calculated from point cloud data acquired from two 3D cameras with different positions and orientations.

[0061] The output data Q(r,p) is represented in 490 and includes quality metric values ​​associated with each depth image trimming 482 / 484 / 486. The quality metrics are also provided by an external training data source for each grasp in the quality map 430 (further described in relation to Figure 10). Thus, the output data Q(r,p) is available as training data for the second neural network 460. For grasps, the output data Q(r,p) is a scalar value representing the quality associated with the grasp operations r and p. The second neural network 460 encodes all operation dimensions r and p, but only needs to explore the r direction at estimation for the p-dimensional encoding performed by the first neural network 410. After the input data for the joints (r and p) is prepared, as shown in 480 and 490 of Figure 4, the second neural network 460 is trained. The training phase is shown at the bottom of box 450, where a second neural network 460 is trained to predict the grasping quality Q(r,p) based on the environmental image data input I(r,p).

[0062] The first neural network 410 may be a fully convolutional network (FCN) best suited for "image-in / image-out" applications. The second neural network 460 may be a convolutional neural network (CNN) best suited for "image-in / scalar-out" applications, where high accuracy is possible due to the low-dimensional content of the output. Both the first and second neural networks 410 and 460 are trained using supervised learning, meaning that desired output data from the networks (quality map 430 from the first neural network 410 and grasping quality metric 490 from the second neural network) are provided as input for training. Following the training procedure shown in boxes 400 and 450 of Figure 4 and described above, the neural networks 410 and 460 are trained and ready for use in the type of live robot operation shown in Figure 1.

[0063] Figure 7 is a block diagram of the steps in the estimation phase of the neural network modularization technique according to an embodiment of the present disclosure. In the estimation phase (also known as the deployment phase or execution phase), neural networks 410 and 460 are used in a live robotic part picking operation to analyze 3D depth images and provide recommendations for quality grasp candidates. Again, it is emphasized that robotic part picking (grasp determination from 3D depth images) is just one example of the modularization of neural networks for separating a high-dimensional task into two or more low-dimensional networks.

[0064] The neural networks 410 and 460, shown in Figure 4 and trained as described above, are used in the estimation phase shown in Figure 7. Although the neural networks 410 and 460 are trained independently, they are connected during estimation to determine their behavior in all dimensions. There are several steps to run the networks in the estimation phase; box 700 has a step involving the first neural network 410, and box 750 has a step involving the second neural network 460.

[0065] In box 700, the first step is to encode the input data I r (p) is to prepare. For gripping applications, I r (p) can be depth images associated with different p. Various input / output types are possible depending on the encoding method and network structure. In the illustrated grasping example, multiple depth image layers are provided for depth encoding in 710, and a fully convolutional network structure is used. Thus, the input is depth images (horizontal slices encoding p with x and y) centered on various heights or layers (encoding p with z).

[0066] JPEG0007851779000005.jpg59169

[0067] JPEG0007851779000006.jpg59169

[0068] JPEG0007851779000007.jpg51169

[0069] Figure 7 illustrates what the modularized neural networks 410 and 460 described above did. Using the input data shown in Figure 710, the first neural network 410 calculated the maximum margin value p * We found this value p * This is the value of the position dimension p that has the highest individual gripping quality across the entire search space. Next, the value p * Using this as input, the second neural network 460 obtains the value r * We found this value r * The position dimension is p * This is the value of the rotational dimension r that has the greatest gripping quality when constrained by the connected dimension (r). * ,p * This defines object grasping by a fully 6-degree-of-freedom robot. The disclosed neural network separation allows the search in p-space to be separated from the search in r-space, resulting in a dramatic reduction in the size of the search space (it becomes a sum of dimensions i and j instead of a product).

[0070] The above description and examples shown in Figure 4-7 illustrate an example of an application for finding high-quality, full-degree-of-freedom gripping of articles for robotic part picking operations. As mentioned above, this is just one example of the network modularization technique disclosed. This technique is generally applicable to modularizing high-dimensional neural networks into two or more low-input-dimensional networks for much faster execution. Several specific gripping applications are described further below.

[0071] Figure 8 is a block diagram of the disclosed neural network modularization technique applied to a 3-degree-of-freedom grasp search problem according to one embodiment of the present disclosure. In the example shown in Figure 8, a relatively simple 3-degree-of-freedom grasp is identified from a 3D depth image of a pile of articles. The three degrees of freedom to be computed are the x, y, and z positions of the grasp point. Rotation is not computed in this example. Rather than computed the three degrees of freedom in a single network, the computation can be modularized into two neural networks. The first neural network computes the x and y coordinates of the highest quality grasp, and the second neural network computes the z coordinate.

[0072] Box 800 contains a first neural network 810 and its associated input and output data. Depth images 820, such as from one or more 3D cameras, are provided as input. In the training phase, the corresponding highest quality grasp positions are also provided for supervised learning. Through training using many depth images 820, the first neural network 810 learns to encode x and y from the shape features of the depth images 820 to correspond to the best grasp quality. In the estimation phase, depth images 820 are provided to the first neural network 810, and a single quality map 830 indicating the highest quality grasp position in x and y dimensions is output. That is, the first neural network 810 determines Q z Encode the maximum margin of (xy). The first neural network 810, as indicated by arrow 840, sends the best gripping position (x) to the second neural network 860 during the estimation phase. * y * It provides the xy dimensions of ).

[0073] JPEG0007851779000008.jpg82169

[0074] Final output gripping position (x * y * z *) includes the best x and y dimensions identified by the first neural network 810 and the best z dimension identified by the second neural network 860. Final output gripping position (x * y * z * The data is provided to the robot control unit, which then provides the robot with commands to grasp the part at the specified coordinates. After the part is grasped, the new depth image is provided to the neural networks 810 and 860, and the coordinates of the new best grasping position are calculated.

[0075] The overall search performance is improved by modularizing a 3-degree-of-freedom grasped search into two networks (one network exploring two dimensions and the other exploring one dimension). For example, consider the case where the x and y dimensions are each divided into a fairly coarse 20x20 grid, and the z dimension is divided into 10 layers. Using the disclosed network modularization technique, if the first neural network 810 explores a space of size 20x20=400 and the second neural network 860 explores a space of size 10, the resulting modularized search space will be 400+10=410. If all three dimensions were explored by a single network, the size of that network's search space would be 20x20x10=4000.

[0076] Figure 9 is a block diagram of a neural network modularization technique applied to a 5-degree-of-freedom grasp search problem using two neural networks, according to an embodiment of the present disclosure. In the example shown in Figure 9, 5 degrees of freedom of a grasp are identified from a 3D depth image of a pile of parts. The 5 degrees of freedom to be computed include the gripper width and gripper rotation angle, as well as the x, y, and z positions of the grasp point. In this example, the approach angle is not computed, and a top-down vertical approach is assumed. Rather than compute the 5 degrees of freedom in a single network, the compute can be modularized into two neural networks using the disclosed method. Here, the first neural network computes the x / y / z coordinates of the highest quality grasp, and the second neural network computes the gripper width (w) and gripper rotation angle (θ) along with the maximized final grasp quality.

[0077] A 3D depth image 910 (for example, showing a pile of items in a bin) is provided to the grasp proposal network 920. The grasp proposal network 920 is a fully convolutional network (FCN) that receives the image as input (depth image 910) and provides the image as output (grasp quality map 930). The grasp quality map 930 is provided to the grasp ranking network 940, which is a convolutional neural network (CNN), and the network 940 receives the image and provides scalar data (gripper width and rotation angle). The gripper width and rotation angle (shown in 950) from the grasp ranking network 940 are combined with the best grasp position (x / y / z) from the grasp quality map 930 created by the grasp proposal network 920. In addition, this provides a 5-degree-of-freedom grasp definition (x / y / z / w / θ) used by the robot control device.

[0078] As previously detailed, the gripping suggestion network 920 and the gripping ranking network 940 are first trained using supervised learning and then operated in estimation mode. During training, the gripping suggestion network 920 has depth images and corresponding gripping quality maps. The gripping ranking network 940 is trained by providing the depth images and corresponding quality maps provided to the gripping suggestion network 920, along with desired outputs of gripper width / rotation angle and final gripping quality. An automated method for performing this training is further described below with reference to Figure 10.

[0079] In estimation mode, the gripping suggestion network 920 has only depth images (and provides a quality map as an output), while the gripping ranking network 940 has depth images and a corresponding quality map as input (and provides a final gripping quality output related to the gripper width / rotation angle and the position of the highest quality grip selected from the quality map).

[0080] The overall search performance is improved by modularizing the 5-degree-of-freedom gripping search in Figure 9 into two networks (one network exploring three dimensions and the other exploring two dimensions). For example, consider the case where the x and y dimensions are each divided into a fairly coarse 20x20 grid, the z dimension is divided into 10 layers, there are positions with a gripper width of 10, and there are positions with a gripper angle of 20. Using the disclosed network modularization technique, if the first neural network 920 explores a space of size 20x20x10=4000 and the second neural network 940 explores a space of size 10x20=200, the resulting modularized search space will be 4000+200=4200. If all five dimensions were explored by a single network, the search space of that network would be 4000x200=800000.

[0081] Figure 10 illustrates a method for training a modularized grasp learning network of the type shown in Figure 9, using a combination of individual item grasp optimization and clustered item variation, along with interference simulation, according to one embodiment of the present disclosure. The steps of the training method shown in Figure 10 can be performed in an "offline" environment using a computer other than the robot control device, which will later be used to control the robot's movements, and provide a highly efficient means of generating grasp data for training the neural networks 920 and 940 in Figure 9.

[0082] The first data preparation step in box 1010 demonstrates the automatic grip search for individual articles in the database. Multiple grips of article 1012 by gripper 1014 are illustrated. A 3D solid or surface model of the part to be analyzed is provided along with gripper data including geometry and operating parameters (e.g., finger joint positions, joint angle ranges). A robust grip candidate is created based on the part's shape and gripper parameters using an iterative optimization method. The steps shown in box 1010 provide multiple quality grip positions and orientations for an individual part (article 1012) by a specific gripper (gripper 1014). These grip poses can be automatically calculated for many different parts using many different grippers.

[0083] In the second data preparation step in box 1020, a robust grasp simulation is performed, taking into account variability and interference. In this step, items are sampled into further randomly dense clusters by simulating a stream of items rolling into a bin and randomly accumulating in a pile of items with various positions, orientations, and entanglements. Since the pose of each item in the simulated pile is known, previously generated grasps (from box 1010) can be tested to determine their effectiveness under simulated real-world conditions (entanglement and interference). The success of each previously generated grasp is thus tested using a 3D depth image of the simulated pile of items along with the previously generated grasps. The steps shown in box 1020 are physical environment simulations performed entirely using mathematical simulations, without the use of actual parts and images. The simulated depth images, grasp position quality maps, grasp poses, and success rates (collectively shown in box 1022) are saved and later used to train the grasp learning network in Figure 9. These are shown in Figure 10 below.

[0084] The grip optimization and simulation described above and shown in boxes 1010 and 1020 are disclosed in U.S. Patent Application No. 17 / 016,731, filed on 10 September 2020, titled "Efficient Data Generation for Grasp Learning with General Grippers," which has the same applicant as this application and is incorporated herein by this reference in its entirety.

[0085] The grasp learning network in Figure 9 is shown in box 1030 in Figure 10. The following describes how to use automatically generated grasp data to train modularized neural networks, which includes separately training the grasp proposal network 920 and the grasp ranking network 940 using data from box 1022.

[0086] Training the grasp proposal network 920 requires depth images 910 as input. Depth images 910 are available from box 1022 and represent a pile of articles from a physical environment simulation. For supervised learning, training the grasp proposal network 920 also requires grasp quality maps 930 that represent grasp quality at different pixel locations. Quality maps 930 are also available from box 1022 and are calculated from the physical environment simulation. The physical environment simulation shown in box 1020 can be run many times (thousands of times), and each random simulation provides a different random pile of articles, resulting in a sufficient quantity and variety of depth images 910 and corresponding quality maps 930 for training the grasp proposal network 920.

[0087] Next, the grip ranking network 940 is trained. This network takes depth image trimmings at various angles (prepared from the quality map 930) as input and outputs gripper rotation angle (θ) and gripper width (w) as shown by line 950, along with the corresponding grip quality, and uses the simulation results from box 1022 as the basis for training (supervised learning). Because gripper rotation angle and gripper width are included, the output of box 960 contains five gripping degrees of freedom. To reiterate for clarification, the grip optimization method in box 1010 generates many different grips for an article using a specific gripper. The physical environment simulation method in box 1020 generates grip quality simulation results for different gripping poses applied to a randomly generated pile of articles. The output of the physical environment simulation is then used to train the grip proposal network 920 and the grip ranking network 940.

[0088] Following the training of two neural networks (920, 940) as shown in Figure 10, the trained neural networks (920, 940) are used to actually operate the robotic grasping system, determining grasping based on depth image input and controlling the robot, as described above with respect to Figure 9. The data generation steps shown in boxes 1010 and 1020 can be applied not only to the 5-degree-of-freedom grasping system / method in Figure 9, but also to modularized grasping networks with fewer degrees of freedom (e.g., 3 degrees of freedom) or more complex ones with more degrees of freedom (e.g., 7 degrees of freedom).

[0089] Figure 11 is a block diagram of a neural network modularization technique applied to a 7-degree-of-freedom grasp search problem using three neural networks, according to one embodiment of the present disclosure. In the system of Figure 11, the seven grasp degrees of freedom include selection of approach direction (2 degrees of freedom), selection of grasp position (3 degrees of freedom), and selection of gripper direction and width (2 degrees of freedom). Similar to the 3-degree-of-freedom and 5-degree-of-freedom neural network systems described above, the neural network shown in Figure 11 is first trained and then used in estimation mode to perform grasp calculations during an actual part grasping operation by a robot as shown in Figure 1.

[0090] Box 1110 is a first-phase step of the method, selecting the best approach direction. For example, given an input scene 1112 which may be part images in a bin from a pair of 3D cameras, the associated point cloud is projected by the direction coding box 1120 in multiple sampled approach directions to generate a composite image 1122. The images in the input scene 1112 contain depth information, which allows the algorithm of the direction coding box 1120 to generate a composite surface image viewed from multiple randomly selected projection viewpoints. In other words, if a robot's gripper approaches from a particular direction, what would the 3D surface image from that direction look like? These composite images are created for many different projection angles that are randomly sampled and provided in the composite image 1122. The projection angles are within the limits of the directions in which the robot can approach.

[0091] The approach direction suggestion network 1130 is used to predict the overall quality of the grasp from each approach direction proposed in the composite image 1122. In other words, in network 1130, the grasp is hidden, and the approach direction containing the best grasp quality is determined using the maximum margin method described above. The approach direction is defined as two vectors v1 and v2, which can be the azimuth and elevation angles in polar coordinates, or other components that define the three-dimensional approach direction vector. The selected approach direction (v1,v2) is saved as the approach direction vector 1138 and later used by the robot control device. The depth image associated with the optimal approach direction is also saved and used in the next phase.

[0092] Box 1140 contains the second phase step of the method, which determines the best gripping position. A depth image 1142 is provided, which is a composite depth image (from image set 1122) associated with the best approach direction 1138 selected above. The depth image 1142 is sliced ​​at different heights in the slicing box 1150 and sent to the gripping position suggestion network 1160. The gripping position suggestion network 1160 generates quality maps of image slices at different heights, as shown in 1164 and as described above with respect to Figure 4. In other words, the network 1160 hides the gripper width and angle and, given a defined approach angle, encodes the gripping quality as a function of position. The quality map 1164 shows the gripping quality that results from selecting a particular position (x,y,z) given the approach direction 1138. The 3D coordinates of the highest quality gripping position are stored in 1168 and provided as input to the next block, which is later used by the robot control unit.

[0093] Box 1170 contains the third and final phase step of the method, which determines the gripping angle (θ) and width (w). A depth image 1172 is provided, which is a depth image associated with the desired approach direction 1138 (v1,v2) and the desired gripping position 1168 (x,y,z) selected above. In box 1180, image patches are cropped at different angles around the highest quality gripping position stored in 1168. These image patches (1182) are sent to the gripping ranking network 1190 to output the quality and width (1194) of each evaluated image patch. The gripping angle (θ) and width (w) corresponding to the highest quality are selected and stored in 1198.

[0094] The desired approach direction (v1, v2, 2 degrees of freedom) stored in 1138, the optimal gripping position (x, y, z, 3 degrees of freedom) stored in 1168, and the best gripping width / angle (w, θ, 2 degrees of freedom) stored in 1198 are transmitted to the robot control unit for execution, as shown in 1199. That is, the robot control unit instructs the robot to grip the part from the bin using robot motion commands calculated from the information of the approach direction, gripping position, and gripping width / angle. The robot then places the part in a predetermined location (such as a conveyor or transport container). The process then returns to box 1110, which is provided with a new image of a new input scene 1112, and the gripping selection step and robot execution are repeated.

[0095] The use of three neural networks for a 7-degree-of-freedom grasp search application, as shown in Figure 11, offers several advantages. The approach direction suggestion network 1130 enables sophisticated grasps by eliminating the constraints of top-down vertical grasps. The grasp position suggestion network 1160 rapidly computes the three degrees of freedom of the grasp position by calculating these dimensions separately from the other degrees of freedom. The grasp ranking network 1190 provides the accuracy inherent in the scalar output of a convolutional neural network. Furthermore, using three neural networks instead of one large neural network in a 7-degree-of-freedom grasp search reduces the size of the search space by several orders of magnitude, and consequently improves the search speed. The system shown in Figure 11 is designed to leverage the advantages of various types of neural networks by using fully convolutional networks (FCNs) for neural networks 1130 and 1160, which acquire images as input and provide images as output, and a convolutional neural network (CNN) for neural network 1190, which provides scalar data output.

[0096] The embodiments described above with respect to Figure 8-11 (3-degree-of-freedom, 5-degree-of-freedom, and 7-degree-of-freedom gripping) demonstrate the capabilities and flexibility provided by the disclosed neural network modularization technique in different gripping applications. It is reiterated that the disclosed neural network modularization technique can be used in applications other than gripping, namely applications requiring neural network encoding and computation of high-dimensional tasks.

[0097] Figure 12 is a flowchart 1200 of a method for solving a multidimensional pattern recognition problem using two or more low-dimensional neural networks, according to one embodiment of the present disclosure. In box 1210, the problem to be solved (a multidimensional search problem suitable for solving with neural network pattern recognition) is defined, and the dimensions of the problem are divided into sets of grouped dimensions. This has been explained several times in the previous description and drawings. In Figure 2-7, a 6-degree-of-freedom grasping problem is divided into a position dimension p(x,y,z) and a rotation dimension r(r1,r2,r3). In other examples, 3-degree-of-freedom and 5-degree-of-freedom grasping searches are divided into two sets of grouped dimensions, and a 7-degree-of-freedom grasping search is divided into three sets of grouped dimensions.

[0098] Box 1220 provides two or more neural networks, with one neural network provided for each grouped set of dimensions from Box 1210. The neural networks are connected in series (for estimation mode) as shown in the previous figure and have been broadly described. The neural networks are run on a computer such as computer 120 in Figure 1. Each neural network encodes a set of grouped dimensions from other neural networks upstream in the series, in addition to its own group of dimensions. For example, in Figure 2-4, the first neural network encodes p dimensions, and the second network encodes both p and r dimensions. In the three-network embodiment in Figure 11, the first network encodes the degrees of freedom for the approach direction, the second network encodes the degrees of freedom for the gripper position related to the approach direction, and the third network encodes the degrees of freedom for the gripper angle and width related to the gripper position and approach direction.

[0099] In box 1230, two or more neural networks are trained individually using supervised learning. The supervised learning method provides each network with a number of training examples, each example containing both the network's input and desired output. For example, in Figure 4, the first neural network 410 is trained by providing depth images 420 (in multiple layers) and corresponding quality maps 430 (specifying the gripping quality at various points in each depth image layer), while the second neural network 460 is trained by providing rotated and cropped depth images 480 and quality metric output data 490. In a two-network system, the first network uses the maximum margin method, which was first described in relation to Figures 3 and 4, while the first network is trained to encode its grouped set of dimensions, while hiding the rest of the grouped set of dimensions. In a three-network system, the first two networks can use the maximum margin method for training. In the maximum margin method, the neural networks find target values ​​for the corresponding set of grouped dimensions that have optimal values ​​for the output variables (quality, etc.). The target value is used as input to other neural networks downstream in the series.

[0100] In box 1240, the neural networks are run in estimation mode, provided with inputs that define the environment of the problem, and each neural network searches only the corresponding grouped set of dimensions to find the target value. The target value output from each neural network is used as input to other neural networks downstream in series. For example, in a two-network system, the input depth image is the only input provided to the first neural network and is used by the first neural network to compute a grasp quality map output with target values ​​for the position coordinates of the high-quality grasp (the first set of grouped dimensions). The input depth image and quality map are then provided as input to the second neural network (further processed by cropping and rotation) and are used by the second neural network to compute target values ​​for the rotation of the high-quality grasp (the second set of grouped dimensions).

[0101] In box 1250, the outputs of two or more neural networks are combined to provide the final output. In the example of two networks described for box 1240, the highest quality metric value is selected, and then the corresponding target values ​​of the second set of grouped dimensions are combined with the target values ​​of the first set of grouped dimensions (from the first neural network) to produce the complete final output. In the case of grasp search using depth images, the final output is a concatenated set of dimensions or degrees of freedom corresponding to the highest quality grasp.

[0102] The disclosed method for modularizing neural networks to learn high-dimensional robotic tasks offers several advantages over known methods. One major advantage of the disclosed method is computationally fast computation due to the avoidance of searching in high-dimensional action spaces (e.g., 5 or more degrees of freedom). As mentioned above, separating the search dimension into two or three neural networks improves computation speed by several orders of magnitude. This reduction in complexity and improvement in speed makes it possible to compute high-dimensional tasks that were not feasible with existing methods.

[0103] By separating the search spaces of different neural networks, it becomes possible to optimally select the type of neural network for each task, such as a fully convolutional network for image input / output computation and a convolutional neural network used for the final scalar output computation. Furthermore, the network design and performance are clear and easy to analyze. In the disclosed technique, each network explores a separate part of the control action space, and each network is trained independently of the others. Therefore, the performance of each network can be analyzed individually without considering the output from other networks. Network isolation or modularization is made possible by encoding a set of grouped dimensions of one network using the maximum margin method, thereby hiding the others.

[0104] Throughout the above description, various computers and control devices have been described and implied. It should be understood that the software applications and modules of these computers and control devices run on one or more computing devices having processors and memory modules. In particular, this includes the processors contained in the robotic control device 110 that controls the robot that performs object grasping, the computer 120, and any other computers used for training and estimation / execution of neural networks. Specifically, the processors in the computers are configured to perform image analysis, neural network training and execution in the manner described throughout the above disclosure for grasping learning or other neural network applications.

[0105] While several exemplary aspects and embodiments of the disclosed modularization techniques for high-dimensional neural networks have been described, those skilled in the art will recognize their modifications, rearrangements, additions, and subcombinations. Accordingly, the appended claims and assertions should be construed to include all such modifications, rearrangements, additions, and subcombinations that are in their true spirit and scope. [Configuration 1] A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps involve dividing a multidimensional problem into two or more mutually exclusive, grouped sets of dimensions, A step of providing two or more neural networks connected in series, which run on a computer having a processor and memory, wherein one of the neural networks is provided for each of the grouped sets of dimensions, and each of the neural networks encodes the corresponding grouped set of dimensions and the grouped set of dimensions of the other neural network in series upstream; A step of training two or more neural networks independently using supervised learning, wherein at least one serial neural network is trained using the maximum margin method to find a target value for a corresponding set of grouped dimensions having the optimal value for the output variable, and A step of running two or more neural networks in estimation mode, wherein an initial input defining the environment of the problem is provided, each neural network searches only a corresponding set of grouped dimensions to find the target value, and the target value from each neural network is used as input by other neural networks in a downstream serial manner. A step of combining the outputs of two or more neural networks to provide a final output that includes the target values ​​for all dimensions in two or more sets of grouped dimensions, Methods that include... [Configuration 2] The method according to configuration 1, wherein the two or more neural networks connected in series include a first neural network in series which is a fully convolutional network that receives image data input and provides image data output, and a last neural network in series which is a convolutional neural network that receives image data input and provides scalar data output. [Configuration 3] The method according to configuration 2, wherein training of the two or more neural networks comprises providing each network with a plurality of training samples, each training sample comprising the image data input and either the image data output or the scalar data output of a particular network, the image data output or the scalar data output being the desired result. [Structure 4] The multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping, as described in configuration 3. [Composition 5] The method according to configuration 4, wherein the training samples are provided by a data generation system, the data generation system calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated grippings. [Composition 6] The method according to configuration 1, wherein the maximum margin method comprises identifying a target value for a corresponding set of grouped dimensions of the first neural network, the set of dimensions having an optimal value for an output variable for any one combination of target values ​​of other sets of grouped dimensions. [Composition 7] The method according to Configuration 1, wherein the step of running the two or more neural networks in estimation mode includes the first neural network in series identifying a maximum margin target value for a corresponding grouped set of dimensions, and the data preparation technique includes providing a depth image layer of initial input to the first neural network in series and providing a rotated and cropped depth image patch to the last neural network in series. [Structure 8] A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps involve dividing a multidimensional problem into two or more mutually exclusive, grouped sets of dimensions, A step of providing two series-connected neural networks that run on a computer having a processor and memory, wherein the first series neural network encodes only the corresponding pair of grouped dimensions, and the second series neural network encodes both pairs of grouped dimensions. A step of individually training the neural networks using supervised learning, wherein the first neural network is trained using the maximum margin method to find a target value for an output variable for any one combination of target values ​​for other sets of grouped dimensions, A step of running the neural network in estimation mode, wherein an initial input defining the environment of the problem is provided, each neural network searches only for the corresponding grouped set of dimensions to find the target value, and the target value from the first neural network is used as input by the second neural network. The steps include combining the outputs of the neural network to provide a final output that includes the target values ​​for all dimensions in both sets of grouped dimensions, Methods that include... [Composition 9] The method according to configuration 8, wherein the first neural network in series is a fully convolutional network that receives image data input and provides image data output, and the second neural network in series is a convolutional neural network that receives image data input and provides scalar data output. [Configuration 10] The method according to configuration 9, wherein training the neural network comprises providing a plurality of training samples, the training sample of the first neural network comprising an initial input as the image data input and a first output as the image data output, and the training sample of the second neural network comprising an initial input as the image data input and a final output as the scalar data output. [Composition 11] The multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping, according to the method of configuration 10. [Composition 12] The method according to configuration 11, wherein the training samples are provided by a data generation system, the data generation system calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated grippings. [Composition 13] The method according to configuration 8, wherein the step of running the neural network in estimation mode includes the serial first neural network identifying a maximum margin target value for a corresponding grouped set of dimensions, and the data preparation technique includes providing a depth image layer of initial input to the serial first neural network and providing a rotated and cropped depth image patch to the serial second neural network. [Composition 14] A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps include dividing the multidimensional problem into a set of mutually exclusive, grouped dimensions: first, second, and third, A step of providing three series-connected neural networks to run on a computer having a processor and memory, wherein the first series neural network encodes only a first set of grouped dimensions, the second series neural network encodes the first and second sets of grouped dimensions, and the third series neural network encodes all three sets of grouped dimensions. A step of individually training the neural networks using supervised learning, wherein the first neural network is trained using the maximum margin method to find the target values ​​of the first group of grouped dimensions, which have an optimal value for the output variable for any one combination of the target values ​​of the other group of grouped dimensions, and the second neural network is trained using the maximum margin method to find the target values ​​of the second group of grouped dimensions, which have an optimal value for the output variable for any one combination of the target values ​​of the third group of grouped dimensions. A step of running the neural network in estimation mode, wherein an initial input defining the environment of the problem is provided, each neural network searches only for a corresponding grouped set of dimensions to find the target value, the target value from the first neural network is used as input by the second neural network, and the target values ​​from the first and second neural networks are used as input by the third neural network. The steps include combining the outputs of the neural network to provide a final output that includes the target values ​​for all dimensions in all three sets of grouped dimensions, Methods that include... [Composition 15] The method according to configuration 14, wherein the first and second neural networks in series are fully convolutional networks that receive image data input and provide image data output, and the third neural network in series is a convolutional neural network that receives image data input and provides scalar data output. [Composition 16] The method according to configuration 15, wherein training the neural network comprises providing a plurality of training samples, the training sample of the first neural network comprising an initial input as the image data input and a first output as the image data output, the training sample of the second neural network comprising an initial input as the image data input and the first output and a second output as the image data output, and the training sample of the third neural network comprising an initial input as the image data input, the first output and the second output and a final output as the scalar data output. [Composition 17] The multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping, as described in configuration 16. [Composition 18] The method according to configuration 17, wherein the training samples are provided by a data generation system, the data generation system calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated gripping. [Composition 19] The method according to configuration 14, wherein the step of running the neural network in estimation mode includes the first series neural network determining the maximum margin target value for the first set of grouped dimensions, and the second series neural network determining the maximum margin target value for the second set of grouped dimensions using the target value for the first set of grouped dimensions. [Configuration 20] The method according to configuration 19, wherein the step of running the neural network in estimation mode includes providing the initial input and the target values ​​of the first set of grouped dimensions as input to the second neural network, and providing the initial input and the target values ​​of the first and second sets of grouped dimensions as input to the third neural network.

Claims

1. A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps involve dividing a multidimensional problem into two or more mutually exclusive, grouped sets of dimensions, A step of providing two or more neural networks connected in series, which run on a computer having a processor and memory, wherein one of the neural networks is provided for each of the grouped sets of dimensions, and each of the neural networks encodes the corresponding grouped set of dimensions and the grouped set of dimensions of the other neural network in series upstream; A step of training two or more neural networks individually using supervised learning, wherein at least one serial neural network is trained using the maximum margin method to find a target value for a corresponding set of grouped dimensions having the optimal value for the output variable, and A step of running two or more neural networks in estimation mode, wherein an input defining the environment of the problem is provided, each neural network searches only a corresponding grouped set of dimensions to find the target value, and the target value from each neural network is used as input by other neural networks in a downstream serial manner. A step of combining the outputs of two or more neural networks to provide a final output that includes target values ​​for all dimensions in two or more sets of grouped dimensions, Methods that include...

2. The method according to claim 1, wherein the two or more neural networks connected in series include a first neural network in series which is a fully convolutional network that receives image data input and provides image data output, and a last neural network in series which is a convolutional neural network that receives image data input and provides scalar data output.

3. The method according to claim 2, wherein the training of the two or more neural networks comprises providing each network with a plurality of training examples, each training example comprising the image data input and either the image data output or the scalar data output of a particular network, the image data output or the scalar data output being a desired result.

4. The method according to claim 3, wherein the multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping.

5. The method according to claim 4, wherein the training example is provided by a data generation system, which calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated grippings.

6. The method according to claim 1, wherein the maximum margin method comprises identifying a target value for a corresponding set of grouped dimensions of the first neural network, the set of dimensions having an optimal value for an output variable for any one combination of target values ​​of other sets of grouped dimensions.

7. The method according to claim 1, wherein the step of running the two or more neural networks in estimation mode includes the first neural network in series identifying a maximum margin target value for a corresponding grouped set of dimensions, and the data preparation method includes providing a depth image layer of inputs to the first neural network in series and providing a rotated and cropped depth image patch to the last neural network in series.

8. A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps involve dividing a multidimensional problem into two or more mutually exclusive, grouped sets of dimensions, A step of providing two series-connected neural networks that run on a computer having a processor and memory, wherein the first series neural network encodes only the corresponding pair of grouped dimensions, and the second series neural network encodes both pairs of grouped dimensions. A step of individually training the neural networks using supervised learning, wherein the first neural network is trained using the maximum margin method to find a target value for an output variable for any one combination of target values ​​for other sets of grouped dimensions, A step of running the neural network in estimation mode, wherein an input defining the environment of the problem is provided, each neural network searches only the corresponding grouped set of dimensions to find the target value, and the target value from the first neural network is used as input by the second neural network. The steps include combining the outputs of the neural network to provide a final output that includes the target values ​​for all dimensions in both sets of grouped dimensions, Methods that include...

9. The method according to claim 8, wherein the first series neural network is a fully convolutional network that receives image data input and provides image data output, and the second series neural network is a convolutional neural network that receives image data input and provides scalar data output.

10. The method according to claim 9, wherein training the neural network comprises providing a plurality of training examples, the first training example of the neural network comprising an input as the image data input and a first output as the image data output, and the second training example of the neural network comprising an input as the image data input and a final output as the scalar data output.

11. The method according to claim 10, wherein the multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping.

12. The method according to claim 11, wherein the training example is provided by a data generation system, which calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated gripping.

13. The method according to claim 8, wherein the step of running the neural network in estimation mode includes the serial first neural network identifying a maximum margin target value for a corresponding grouped set of dimensions, and the data preparation method includes providing a depth image layer of inputs to the serial first neural network and providing a rotated and cropped depth image patch to the serial second neural network.

14. A method for solving high-dimensional pattern recognition problems using a low-input-dimensional neural network, The steps include dividing the multidimensional problem into a set of mutually exclusive, grouped dimensions: first, second, and third, A step of providing three series-connected neural networks to run on a computer having a processor and memory, wherein the first series neural network encodes only a first set of grouped dimensions, the second series neural network encodes the first and second sets of grouped dimensions, and the third series neural network encodes all three sets of grouped dimensions. A step of individually training the neural networks using supervised learning, wherein the first neural network is trained using the maximum margin method to find the target values ​​of the first group of grouped dimensions, which have an optimal value for the output variable for any one combination of the target values ​​of the other group of grouped dimensions, and the second neural network is trained using the maximum margin method to find the target values ​​of the second group of grouped dimensions, which have an optimal value for the output variable for any one combination of the target values ​​of the third group of grouped dimensions. A step of running the neural network in estimation mode, wherein an input defining the environment of the problem is provided, each neural network searches only for a corresponding grouped set of dimensions to find the target value, the target value from the first neural network is used as input by the second neural network, and the target values ​​from the first and second neural networks are used as input by the third neural network. The steps include combining the outputs of the neural network to provide a final output that includes the target values ​​for all dimensions in all three sets of grouped dimensions, Methods that include...

15. The method according to claim 14, wherein the first and second neural networks in series are fully convolutional networks that receive image data input and provide image data output, and the third neural network in series is a convolutional neural network that receives image data input and provides scalar data output.

16. The method according to claim 15, wherein training the neural network comprises providing a plurality of training examples, the first training example of the neural network comprising an input as the image data input and a first output as the image data output, the second training example of the neural network comprising an input as the image data input and the first output and a second output as the image data output, and the third training example of the neural network comprising an input as the image data input, the first output and the second output and a final output as the scalar data output.

17. The method according to claim 16, wherein the multidimensional problem is a robot grasp search problem, and the grouped set of dimensions includes degrees of freedom for robot grasping.

18. The method according to claim 17, wherein the training example is provided by a data generation system, which calculates stable gripping of multiple objects by a gripper, simulates a physical environment including a random pile of the objects, evaluates the stable gripping applied to the random pile of the objects, and outputs depth images of the random pile of objects and the corresponding successful simulated gripping.

19. The method according to claim 14, wherein the step of running the neural network in estimation mode includes: the serial first neural network determining a maximum margin target value for the first set of grouped dimensions; and the serial second neural network determining a maximum margin target value for the second set of grouped dimensions using the target values ​​for the first set of grouped dimensions.

20. The method according to claim 19, wherein the step of running the neural network in estimation mode includes providing the input and the target values ​​of the first set of grouped dimensions as input to the second neural network, and providing the input and the target values ​​of the first and second sets of grouped dimensions as input to the third neural network.

Images