Feature detection using deep learning and vector field estimation

Abstract

To provide a system for and a method of extracting a feature from a 2D image of a target object by using deep layer learning neural network and a vector field estimation process.SOLUTION: A method according to the present invention has a step of extracting a plurality of possible feature points 96, a step of generating a mask image 200 for use in defining pixels in a 2D image 82 in which an object is located, and a step of generating, with respect to each of the extracted feature points, a vector field image 202 with an arrow to the extracted feature point. The method further has a step of generating a vector intersection image 206 by identifying an intersection point xij where arrows regarding all combination of two pixels Pi, Pj in the image are crossed. A score is allocated to each intersection point in accordance with a distance from each pixel of each combination of the two pixels and the intersection, thereby to generate point polling image for identifying feature location from a plurality of collected points.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates generally to systems and methods for extracting features from images, and more particularly to systems and methods for extracting features from images using deep learning neural networks and vector field estimation processes. [Background technology]

[0002] Robots perform many tasks, including pick-and-place operations, in which the robot picks up and transports multiple objects from one location, such as a conveyor belt, to another location, such as a collection bin, with the objects at slightly different positions and orientations, known as their 3D pose. Thus, in order for the robot to efficiently pick up objects, it is often necessary for the robot to know the 3D pose of the object. To this end, various techniques for determining the 3D pose of an object use a 3D camera that acquires a 3D image of the object in real time. In this case, the 3D image identifies distance measurements to many points within the camera's field of view, and these measurements are transmitted to the robot controller. These techniques use structured light to create a 3D map of the scene and can identify the object's 3D features to locate the object in the map. For robotic applications, using 3D cameras to determine the 3D pose of an object has proven effective. However, 3D cameras are expensive devices and often operate slower than required.

[0003] One technique known in the art for acquiring the 3D pose of an object for robotic applications instead uses a two-dimensional (2D) camera. This technique acquires a single 2D color image of the scene and processes the image using a fully-convolutional architecture. An algorithm divides the image into a 2D grid containing SXS cells and identifies objects within the cells. The algorithm projects a bounding box around each object in the image and predicts the 2D positions of the box's eight corners. However, this technique for predicting the bounding box corners has limited accuracy and low robustness to occlusion.

[0004] Some of the above processes for determining the 3D pose of an object use artificial intelligence (AI), which is the part of computer science that uses algorithms to allow software applications to learn from an environment and make decisions from it to achieve a particular result. Machine learning is the part of AI that uses software applications that acquire their own knowledge by repeatedly analyzing large amounts of raw input data, extracting patterns from the data and learning to perform a task even if the software application was not specifically programmed to perform the task. Deep learning is a specific type of machine learning that provides better learning performance by representing a particular real-world environment as a hierarchy of concepts of increasing complexity.

[0005] Deep learning typically uses a software structure that includes multiple layers of neural networks that perform nonlinear processing, where each subsequent layer receives the output from the previous layer. The layers typically include an input layer that receives raw data from sensors, multiple hidden layers that extract abstract features from the data, and an output layer that identifies specific things based on feature extraction from the hidden layers. Neural networks include neurons or nodes, each with a "weight" that can be multiplied by the input to the node to obtain a probability that something is correct or not. More specifically, each node has a weight, which is a floating point number, that is multiplied by the input to the node to generate the output of that node, which is a percentage of the input. The weights are first "trained" or set by having the neural network analyze a set of known data under supervised processing and minimizing a cost function to allow the network to obtain the highest probability of obtaining the correct output.

[0006] Deep learning neural networks are frequently used to perform image feature extraction and transformation for visual detection and classification of objects in images. Videos or streams of images can be analyzed by the network, which can identify and classify objects and, through the process, learn to better recognize objects. Thus, with these types of networks, the system can detect certain objects using the same processing configuration and classify them differently based on how the algorithm has learned to recognize them. Summary of the Invention

[0007] The following description discloses and describes a system and method for detecting features from a two-dimensional image of an object using a deep learning neural network and a vector field estimation process. The method includes extracting a plurality of possible feature points of the object from the two-dimensional image, generating a mask image defining pixels in the two-dimensional image where the object is located, and generating a separate vector field image for each extracted feature point having x and y direction values ​​for each pixel in the two-dimensional image with an arrow pointing toward the extracted feature point in the two-dimensional image, where the extraction of possible feature points, generation of the mask image, and generation of the vector field image are performed with a deep learning neural network. The method also includes generating the vector field extraction image by multiplying pixels in the mask image with corresponding pixels in the vector field image such that arrows not associated with pixels in the two-dimensional image that include a portion of the object are discarded, and generating a vector intersection image by identifying intersection points where the arrows intersect for all combinations of two pixels in the two-dimensional image. The method assigns a score to each intersection for each extracted feature point according to its distance from each pixel for each combination of two pixels and intersections, generating a grid of scores and generating a point voting image that identifies the feature configuration from a set of points.

[0008] Additional features of the present disclosure will become apparent from the following description and claims, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of a robotic system that includes a robot that picks up an object from a conveyor belt and places it in a bin. The system uses a 2D camera to capture 2D images of the object and a controller that uses the 2D images to calculate the 3D pose of the object. [Diagram 2] FIG. 1 is a flow-type diagram showing the process of acquiring a dataset from an object using a 2D camera for training a neural network. [Diagram 3]FIG. 1 is a diagram of a neural network including multiple nodes, each with adjustable weights. [Figure 4] FIG. 1 is a flow-type diagram illustrating a learning-based neural network process that uses a trained neural network to estimate a 3D pose of an object using a 2D image of the object and the neural network. [Diagram 5] FIG. 5 illustrates a perspective n-point (PnP) process for determining a 3D pose estimate of an object in the process shown in FIG. [Figure 6] FIG. 1 is a diagram of a camera system that uses multiple cameras to image an object. [Figure 7] FIG. 1 is a block diagram of a system for estimating the 3D pose of an object using multiple 2D cameras. [Figure 8] FIG. 8 illustrates a PnP process for estimating a 3D pose of an object in the system shown in FIG. [Figure 9] FIG. 1 is a flow-type diagram illustrating a vector field estimation process for predicting feature points on an object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following description of embodiments of the present disclosure directed to systems and methods for extracting image features using deep learning neural networks and vector field estimation processes is merely exemplary in nature and is not intended to limit the invention or its applications or uses. For example, the systems and methods have application in determining the position and orientation of an object grasped by a robot. However, the systems and methods may have other applications.

[0011] FIG. 1 is a diagram of a robotic system 10 including a robot 12 that picks up an object 14 from a conveyor belt 16 and places it in a collection bin 18. The system 10 is intended to represent any type of robotic system that can benefit from the description herein, and the robot 12 can be any robot suitable for that purpose. Each object 14 has a different orientation and position on the conveyor belt 16 relative to the other objects 14 as it moves downstream on the conveyor belt 16. In order for the robot 12 to effectively grasp and pick up the objects 14, the end effector 20 of the robot 12 must be able to be positioned in a proper position and orientation before the end effector 20 grasps the object 14. To accomplish this, the system 10 includes a 2D camera 22 that is positioned at a desired position relative to the object 14 and provides a 2D image to a robot controller 24 that controls the robot 12 to move the end effector 20. As described in more detail below, the robot controller 24 uses algorithms to convert the 2D images from the camera 22 into a 3D pose of the object 14 so that the end effector 20 is properly oriented.

[0012] As will be described below, in order for the controller 24 to accurately determine the 3D pose of the object 14, in one embodiment, a learning-based neural network is used in the algorithms and software executed by the controller 24 to extract features of the object. The neural network must be trained with data acquired from a representative object. FIG. 2 is a flow-type diagram 30 showing a process for acquiring image data from an object 32 identical to the object 14 in the system 10 using a 2D camera 40 used to train the neural network. The object 32 is placed on a marker board 34 having an appropriate number of markers 36 with special detectable designs that are used to identify the ground truth or actual position of the object 32. Only one of the markers 36 on the board 34 is shown, and the offset distance represented by a line 38 between the marker 34 and the object 32 is measured.

[0013] The camera 40 provides one or more 2D images 42, here four images, of the marker board 34 at different angles. The process then performs a projection on the image 42 to determine the rotation and position of one of the markers 36 identified by the coordinate frame 44. The algorithm then adds the previously measured offset between the marker 36 and the object 32 to the rotation and position of the marker 36 to determine the rotation and position of the object 32. Once the true position of the object 32 is determined, the positions of feature points 46, such as corners of the object 32, can be calculated using known feature point extraction processes, where at least four feature points are required. An algorithm, which may use, for example, a Gaussian distribution process, looks at each identified feature point 46 in the image 42 and generates a "heat map" 48 of that point 46, where each heat map 48 determines the likelihood that one of the feature points of the object 32 is present at a particular location, depending on the intensity of the color. Locations in the image 42 where feature points are believed to be present, such as areas 50, are assigned a particular color, such as red. Also, since it is unlikely that a feature point exists away from region 50, the color changes to blue, for example region 52.

[0014] The data provided by the 2D image 42 and the corresponding heat map 48 generated by the process shown in diagram 30 are then used to train a neural network using known AI training schemes and used in the system 10 to identify the 3D pose of the object 14. FIG. 3 is a diagram of a neural network 60 including multiple nodes 62, each with adjustable weights W, which is intended to represent the neural network discussed herein in a schematic manner. The neural network 60 has an input layer 64 that receives individual pixel data from the 2D image, multiple residual block layers 66 and 68 that identify features within the pixel data of the 2D image, and multiple convolutional layers 70, each node 62 of the convolutional layer 70 providing a heat map. It is noted that the convolutional layer 70 and the residual block layers 66 and 68 are standard modules utilized in the field of deep learning. These types of convolutional layers are composed of multiple filter kernels that perform convolution operations on the image input or previous convolutional layers, so that edge, texture, or contour information can be extracted from the input. The residual block layer is a structure that contains two convolutional layers with skip connections, where the output of the second convolutional layer is appended to the input of the first convolutional layer to serve as the block output.

[0015] 4 is a flow-type diagram 80 showing an algorithm running in the controller 24 employing a learning-based neural network 78 that uses a trained neural network to estimate the 3D pose of the object 14 from a 2D image 82 of the object 14 captured by the camera 22. The image 82 is provided to an input layer 84 and multiple successive residual block layers 86 and 88 including a feed-forward loop in the neural network 78 running in the AI ​​software of the controller 24, for feature extraction such as gradients, edges, contours, etc. of possible feature points of the object 14 in the image 82 using a filtering process. The image with the extracted features is provided to multiple successive convolutional layers 90 in the neural network 78, and the possible feature points resulting from the extracted features are defined as a series of heat maps 92, one for each feature point. The heat maps 92 represent the likelihood that the feature point is present on the object 14 based on the color of the heat maps 92. An image 94 is generated using the image 82 of the object 14, which includes feature points 96 for all feature points from all heat maps 92. Each feature point 96 is assigned a confidence value based on the color of that feature point's heat map 92, and feature points 96 that do not have a confidence value above a certain threshold are not used.

[0016] The images 94 are then compared to a nominal or virtual 3D CAD model of the object 14, having the same feature points, in a pose estimation processor 98 to provide an estimated 3D pose 100 of the object 14. One suitable algorithm for comparing the images 94 to the CAD model is known in the art as Perspective n Points (PnP). In general, the PnP process estimates the pose of the object relative to a calibrated camera given a set of n 3D points of the object in the world coordinate frame and the corresponding 2D projections in the images from the camera. The pose includes six degrees of freedom (DOF), consisting of the rotation (roll, pitch, yaw) and 3D translation of the object relative to the coordinate frame of the camera.

[0017] 5 is an illustration 104 showing how the PnP process can be performed in this embodiment to obtain a 3D pose of the object 14. The illustration 104 shows a 3D object 106 representing the object 14 in a ground truth or real-world position. The object 106 is observed by a camera 112 representing the camera 22 and projected as a 2D object image 108 onto a 2D image plane 110. Here, the object image 108 represents the image 94, and the point 102 on the image 108 is a feature point predicted by the neural network 78 and represents the point 96. The illustration 104 also shows a virtual 3D CAD model 114 of the object 14 with a feature point 132 at the same position as the feature point 96, which is randomly placed in front of the camera 112 and projected onto the image plane 110 as a 2D model image 116 that also includes the projected feature point 118. The CAD model 114 is rotated and translated in front of the camera 112, such that the model image 116 is rotated and translated so that the distance between each feature point 118 of the model image 116 and the corresponding feature point 102 of the object image 108 is minimized, i.e., so that the images 116 and 108 are aligned. Once the model image 116 is aligned as closely as possible to the object image 108, the pose of the CAD model 114 relative to the camera 112 becomes the estimated 3D pose 100 of the object 14.

[0018] This analysis is expressed in equation (1) for one of the corresponding feature points between image 108 and image 116, and equation (1) is used for all feature points of images 108 and 116. In equation (1), V i is one of the feature points 132 of the CAD model 114, and v i are the corresponding projected feature points 102 in the model image 116, and a i is one of the feature points 102 in the object image 108, R and T are the rotation and translation of the CAD model 114 relative to the camera 112, respectively, the symbol "'" is the vector transpose, and ∀ refers to any feature point with index i. By solving equation (1) with an optimization solver, the optimal rotation and translation can be calculated and thus the 3D pose 100 of the object 14 is estimated.

[0019]

number

[0020] The above process of using a learning-based neural network to estimate the 3D pose of the object 14 provides a high level of accuracy, but can be improved. For example, a small pixel error in the 2D projection of the object 14 on the image plane 110 in the x-axis or y-axis orthogonal to the optical z-axis of the camera 112 will result in a relatively large error in the z-axis, which creates significant uncertainty in the object's position and reduces the performance of the 3D pose estimation. Therefore, this disclosure also proposes a 3D pose estimation process that reduces the uncertainty in the z-axis by using multiple cameras.

[0021] 6 is a diagram of a camera system 120 including a first camera 122 and a second camera 124 oriented relative to an x, y and z coordinate frame, where the z-axis is along the optical axis of the cameras 122 and 124. An illustrated uncertainty region 126 indicates the limit of accuracy with which the images from the camera 122 can estimate the position of the object 14, and an illustrated uncertainty region 128 indicates the limit of accuracy with which the images from the camera 124 can estimate the position of the object 14. As shown, both cameras 122 and 124 are highly accurate in the x and y axes, but have limited accuracy in the z-axis, with accuracy in all axes depending on the distance between the camera 122 or 124 and the imaged object 14. However, by fusing the 3D pose estimates provided by both cameras 122 and 124, the accuracy of the estimate of the 3D pose of the object 14 in the z-axis is significantly improved, as shown by the uncertainty region 130. In other words, in this illustration, the z-axis of one of the cameras 122 and 124 is the x-axis of the other of the cameras 122 and 124, so the overall uncertainty in the estimation process is the uncertainty in the x-axis direction.

[0022] In this non-limiting embodiment, the cameras 122 and 124 are oriented such that their optical axes are orthogonal to one another and the distance from the cameras 122 or 124 to the object 14 is optimal to provide the smallest uncertainty region 130. However, practical systems may be limited in terms of where the cameras 122 and 124 with their z-axes can be placed relative to one another, the distance between the cameras 122 or 124 and the object 14, and other limiting factors such as lighting and the type of camera used. This may lead to optical axes that are not orthogonal to one another and other limitations. In those systems, it may be desirable to use more than one camera to bring the size of the uncertainty region closer to the minimum uncertainty region 130.

[0023] 7 is a block diagram of a system 140 for estimating the 3D pose of an object 14 using 2D cameras 122 and 124 and a learning-based neural network of the type described above. For each angle of the object 14, camera 122 provides a 2D image 142 and camera 124 provides a 2D image 144. Image 142 is provided to processor 146 and image 144 is provided to processor 148. Processors 146 and 148 generate feature point images 150 and 152, respectively, as well as images 94 from their perspectives. Processors 146 and 148, for example, generate heat maps using input layer 84, multiple successive residual block layers 86 and 88, and multiple successive convolutional layers 90, which are then used to generate images 150 and 152 in the manner described above. Images 150 and 152 are fused in a pose estimation block 154 to generate a more accurately estimated 3D pose 156 of the object 14 than a single camera 22 can provide, the pose estimation process also using the PnP algorithm as described above.

[0024] 8 is an illustration 160 similar to illustration 104, with like elements indicated by the same reference numbers, illustrating the PnP process for image fusion in block 154 described above. In illustration 160, camera 162 represents camera 122 and camera 164 represents camera 124. Illustration 160 shows a 2D model image 166 with feature points 172 that is a projection of CAD model 114 onto the 2D image plane 170 for camera 164, and a 2D object image 168 with feature points 174 that is a projection of 3D object 106 onto the image plane 170 for camera 164. The CAD model 114 is rotated and translated in front of both cameras 162 and 164 to simultaneously obtain an optimal model pose for both cameras 162 and 164, such that feature points 118 of model image 116 on image plane 110 are as close as possible to feature points 102 of object image 108, and feature points 172 of model image 166 on image plane 170 are as close as possible to feature points 174 of object image 168. Once this is achieved, an optimal orientation of the 3D model 114 is obtained, which represents the 3D pose of object 14.

[0025] This analysis is represented by equation (2) for one of the corresponding feature points between the images 108, 116 and 166, 168, and equation (2) is used for all feature points of the images 108, 116 and 166, 168. In equation (2), u i is one of the feature points 172 in the model image 166, and b i are the corresponding feature points 174 in the object image 168.

[0026]

number

[0027] The above-mentioned technique of extracting features from 2D images to estimate the 3D pose of the object 14 using a learning-based neural network is one suitable feature extraction technique, although there are others. In the following description, a feature extraction technique using deep learning and vector field estimation is described that can satisfactorily replace the learning-based technique. As described in more detail below, the vector field estimation process replaces the neural network 78, which includes an input layer 84, multiple successive residual block layers 86 and 88, and multiple successive convolutional layers 90 that generate a heat map 92, with a vector field estimation element. It is emphasized that although a vector field estimation process is described here for obtaining the 3D pose of an object, the process may be applied to any other suitable process that requires feature extraction from an image.

[0028] 9 is a flow-type diagram 180 illustrating a vector field estimation process operating in the controller 24 to generate a 3D pose of the object 14 using a 2D image 82 of the object 14 captured by the camera 22, with like elements designated with the same reference numerals. The image input is a three-channel RGB (Red-Green-Blue) image defined by a three-dimensional matrix N×N×3, where N is the height and width of the image 82. The image 82 is provided to an encoder / decoder neural network 182, which includes an encoder network 184 that extracts possible key features in the image 82, and a decoder network 186 that generates an image from the extracted features and identifies possible feature points of the object 14, as described below. Here, the networks 184 and 186 include multiple neural network layers of the type described above. In particular, in one non-limiting embodiment, the neural network layers of the encoder network 184 include a convolutional layer 188, a batch normalization (BN) layer 190, and a rectified linear unit (ReLu) layer 192, and the neural network layers of the decoder network 186 include a pooling layer 194, an upsampling layer 196, and a softmax layer 198.

[0029] The network 182 processes the three-dimensional matrix of the image 82 and generates from the three channels a one-dimensional (N×N×1) mask image 200 and a two-dimensional (N×N×2) vector field image 202 for each identified feature point. To generate the mask image 200, the network 182 assigns a digital 1 or 0 to each pixel in the image 82 based on whether or not a portion of the object 14 is present at the particular pixel, where pixels in light areas of the mask image 200 have a digital 1 and pixels in dark areas of the mask image 200 have a digital 0. Each vector field image 202 includes a number of arrows, each defined by an x-direction vector value and a y-direction vector value provided to each pixel in the image 82, with the arrows pointing towards one of the feature points of the image 82. Thus, each pixel in the image 82 has the ability to predict the direction to its feature. The values ​​of pixels in the mask image 200 are multiplied by the values ​​of corresponding pixels in the vector field image 202 so that arrows that are not associated with pixels that contain part of the object 14 and do not provide useful directions to feature points are discarded, as shown in vector field extraction image 204.

[0030] Each arrow in the vector field extraction image 204 represents a direction to one of the feature points, but does not indicate the distance to that feature point. Thus, two object pixels p i and p j For all combinations of , the process finds the point x ij Identify the point x ij Vector of v i and v j intersect with each other as shown in the vector intersection image 206. ij In pixel p i and point x ij The distance between pixels p j and point x ij The distance between i and p j Depending on the magnitude of the gradient at, a score is assigned. For example, at pixel p i and point x ij and the distance between pixel p jand point x ij The smaller the distance between pixel p, the higher the score. i and p j The larger the magnitude of the gradient at x, the higher the score. As a result, there are many points x, each with a score ij A grid containing most of the points x ij Regions in the image that contain , identify possible interest point locations, as shown in the point voting image 208.

[0031] The vector field image 202 shows the vector arrows of one possible feature point identified by the network 182. The network 182 also generates a separate vector field image 202 for each possible feature point multiplied by the mask 200. All feature points of all point voting images 208 are combined in a single image, such as image 94 containing feature points 96, with each individual feature point 96 contributed by one of the point voting images 208. The images 94 can then be processed by a PnP process in a pose estimation processor 98 to generate the 3D pose 100 in the manner described above. Additionally, due to the uncertainty along the optical axis of a single camera as described above, multiple cameras can be used, with each of the images from the cameras being processed by the vector field estimation process.

[0032] As mentioned before, the vector field estimation process can be applied to other processes that require the extraction of feature points. For example, one application in robotics is to find the center of an object being picked up by a robot, where the orientation of the object is not required. Another application could be to determine the measurements of an object for quality assurance purposes.

[0033] As will be appreciated by those skilled in the art, some of the various steps and processes discussed herein to describe the present invention may refer to operations performed by a computer, processor, or other electronic computing device that manipulates and / or transforms data using electrical phenomena. These computers and electronic devices may use a variety of volatile and / or non-volatile memories, including non-transitory computer-readable media on which executable programs having various codes or executable instructions executable by the computer or processor are stored. The memory and / or computer-readable media may include memories and other computer-readable media of all forms and types.

[0034] The foregoing description discloses and describes merely exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such description, together with the accompanying drawings and claims, that various changes, modifications, and variations can be made without departing from the spirit and scope of the present disclosure, as defined in the following claims. [Configuration 1] 1. A method for extracting features from an image of an object, comprising: acquiring a two-dimensional image of the object using a two-dimensional camera, the two-dimensional image comprising a plurality of pixels; extracting a plurality of possible feature points of the object from the two-dimensional image; generating a mask image defining pixels in the two-dimensional image where the object is located; generating a separate vector field image for each extracted feature point having x and y direction values ​​for each pixel in the two-dimensional image with a directional arrow pointing towards the extracted feature point in the two-dimensional image; generating a separate vector field extracted image for each extracted feature point by multiplying pixels in the mask image by corresponding pixels in the vector field image such that arrows that are not associated with pixels in the two-dimensional image that include a portion of the object are discarded; generating a vector intersection image for each extracted feature point by identifying intersection points where arrows intersect for every combination of two pixels in the two-dimensional image; assigning a score to each intersection for each extracted feature point according to its distance from each pixel for each combination of two pixels and said intersection to generate a grid of scores for all combinations of pixels; generating a point voting image for each of the extracted feature points, the point voting image identifying a feature configuration from a set of multiple points; The method includes: [Configuration 2] 2. The method of claim 1, wherein the extracting of potential feature points, the generating of the mask image, and the generating of the vector field image are performed with a deep learning neural network. [Configuration 3] 2. The method of claim 1, wherein assigning a score to each intersection point comprises using a gradient magnitude of said pixel. [Configuration 4] 4. The method of claim 3, wherein the smaller the distance between the pixel and the intersection point, the higher the score, and the larger the gradient magnitude at the pixel, the higher the score. [Configuration 5] 2. The method of claim 1, wherein the point voting images of all of the extracted feature points are combined as a combined point voting image. [Configuration 6] 6. The method of claim 5, wherein the connection point voting image is used to estimate a 3D pose of the object. [Configuration 7] 7. The method of claim 6, wherein estimating the 3D pose of the object includes comparing a feature arrangement to a 3D virtual model of the object. [Configuration 8] 8. The method of claim 7, wherein estimating the 3D pose of the real object includes using a perspective n-point algorithm that compares a 2D projection of the 3D virtual model onto an image with a 2D projection of a real object having the feature arrangement onto the image. [Configuration 9] 8. The method of claim 7, wherein the method is performed in a robotic system and the object is picked up by a robot. [Configuration 10] 2. The method of claim 1, wherein the two-dimensional image is a three-channel RGB (Red-Green-Blue) image defined by a three-dimensional matrix. [Configuration 11] 1. A method for extracting features from an image comprising a plurality of pixels of an object using a camera, comprising: extracting a plurality of possible feature points of the object from the image using a deep learning neural network; generating, for each extracted feature point, a separate vector field image using a deep learning neural network, having x and y direction values ​​for each pixel in the image with a directional arrow pointing towards the extracted feature point in the image; identifying intersections of said arrows using a vector field estimation process to identify feature locations for each distinct vector field image; The method includes: [Configuration 12] 12. The method of claim 11, further comprising providing a combined image as a combination of all of the identified feature configurations. [Configuration 13] 12. The method of claim 11, wherein the image is a three-channel RGB (Red-Green-Blue) image defined by a three-dimensional matrix. [Configuration 14] 1. A system for extracting features from an image of an object, comprising: a two-dimensional camera for acquiring a two-dimensional image including a plurality of pixels of the object; means for extracting a plurality of possible feature points of the object from the two-dimensional image; means for generating a mask image defining pixels within the two-dimensional image where the object is located; means for generating a separate vector field image for each extracted feature point, the vector field image having x and y direction values ​​for each pixel in the two-dimensional image, with an arrow pointing towards the extracted feature point in the two-dimensional image; means for generating a separate vector field extracted image for each extracted feature point by multiplying pixels in the mask image by corresponding pixels in the vector field image such that arrows that are not associated with pixels in the two-dimensional image that include a portion of the object are discarded; means for generating a vector intersection image for each extracted feature point by identifying intersection points where arrows intersect for every combination of two pixels in the two-dimensional image; means for assigning a score to each intersection for each extracted feature point according to its distance from each pixel for each combination of two pixels and said intersection to generate a grid of scores for all combinations of pixels; A means for generating a point voting image for each extracted feature point, the point voting image identifying a feature arrangement from a set of a plurality of points; A system having [Configuration 15] 15. The system of claim 14, wherein the means for extracting potential feature points, the means for generating a mask image, and the means for generating a vector field image are deep learning neural networks. [Configuration 16] 15. The system of claim 14, wherein the means for assigning a score to each intersection point uses a gradient magnitude of the pixel. [Configuration 17] 17. The system of claim 16, wherein the smaller the distance between the pixel and the intersection point, the higher the score, and the larger the gradient magnitude at the pixel, the higher the score. [Configuration 18] 15. The system of claim 14, wherein the means for generating a point voting image combines point voting images for a plurality of feature configurations. [Configuration 19] 19. The system of claim 18, wherein the combined point voting image is used to estimate a 3D pose of the object. [Configuration 20] The system of configuration 14, wherein the system is used for robot control and the object is picked up by the robot.

Claims

1. 1. A method for extracting features from an image of an object, comprising: acquiring a two-dimensional image of the object using a two-dimensional camera, the two-dimensional image comprising a plurality of pixels; extracting a plurality of possible feature points of the object from the two-dimensional image; generating a mask image defining pixels in the two-dimensional image where the object is located; generating a separate vector field image for each extracted feature point having x and y direction values ​​for each pixel in the two-dimensional image with a directional arrow pointing towards the extracted feature point in the two-dimensional image; generating a separate vector field extracted image for each extracted feature point by multiplying pixels in the mask image by corresponding pixels in the vector field image such that arrows that are not associated with pixels in the two-dimensional image that include a portion of the object are discarded; generating a vector intersection image for each extracted feature point by identifying intersection points where arrows intersect for every combination of two pixels in the two-dimensional image; assigning a score to each intersection for each extracted feature point according to its distance from each pixel for each combination of two pixels and said intersection to generate a grid of scores for all combinations of pixels; generating a point voting image for each of the extracted feature points, the point voting image identifying a feature configuration from a set of multiple points; The method includes:

2. The method of claim 1 , wherein the extracting the potential feature points, generating the mask image, and generating the vector field image are performed with a deep learning neural network.

3. The method of claim 1 , wherein assigning a score to each intersection point comprises using a gradient magnitude of the pixel.

4. The method of claim 3 , wherein the smaller the distance between the pixel and the intersection point, the higher the score, and the larger the gradient magnitude at the pixel, the higher the score.

5. The method of claim 1 , wherein the point voting images of all of the extracted feature points are combined as a connected point voting image.

6. The method of claim 5 , wherein the connection point voting image is used to estimate the 3D pose of the object.

7. The method of claim 6 , wherein estimating the three-dimensional pose of the object comprises comparing feature configurations to a three-dimensional virtual model of the object.

8. 8. The method of claim 7, wherein estimating the 3D pose of the real object comprises using a perspective n-point algorithm that compares a 2D projection of the 3D virtual model onto an image with a 2D projection of a real object having the feature arrangement onto the image.

9. The method of claim 7 , wherein the method is performed in a robotic system and the object is picked up by a robot.

10. The method of claim 1 , wherein the two-dimensional image is a three-channel RGB (Red-Green-Blue) image defined by a three-dimensional matrix.

11. 1. A system for extracting features from an image of an object, comprising: a two-dimensional camera for acquiring a two-dimensional image comprising a plurality of pixels of the object; means for extracting a plurality of possible feature points of the object from the two-dimensional image; means for generating a mask image defining pixels in the two-dimensional image where the object is located; means for generating a separate vector field image for each extracted feature point, the vector field image having x and y direction values ​​for each pixel in the two-dimensional image, with a directional arrow pointing towards the extracted feature point in the two-dimensional image; means for generating a separate vector field extracted image for each extracted feature point by multiplying pixels in the mask image by corresponding pixels in the vector field image such that arrows that are not associated with pixels in the two-dimensional image that include a portion of the object are discarded; means for generating a vector intersection image for each extracted feature point by identifying intersection points where arrows intersect for every combination of two pixels in the two-dimensional image; means for assigning a score to each intersection for each extracted feature point according to its distance from each pixel for each combination of two pixels and said intersection to generate a grid of scores for all combinations of pixels; A means for generating a point voting image for each extracted feature point, the point voting image identifying a feature arrangement from a set of a plurality of points; A system having

12. 12. The system of claim 11, wherein the means for extracting potential feature points, the means for generating a mask image, and the means for generating a vector field image are deep learning neural networks.

13. The system of claim 11 , wherein the means for assigning a score to each intersection point uses a gradient magnitude of the pixel.

14. The system of claim 13 , wherein a smaller distance between the pixel and the intersection point results in a higher score, and a larger gradient magnitude at the pixel results in a higher score.

15. The system of claim 11 , wherein the means for generating a point voting image combines point voting images for multiple feature configurations.

16. The system of claim 15 , wherein the combined point voting images are used to estimate the 3D pose of the object.

17. The system of claim 11 , wherein the system is used for controlling a robot, and the object is picked up by the robot.

Images