Identification of unknown object for robot device

Abstract

To provide a method for identifying an object.SOLUTION: A method for identifying an object includes generating a feature vector of an unknown object from an image of an environment. The method also includes comparing the feature vector of the unknown object with feature vectors of known objects. The method further includes determining whether similarity between the feature vector of the unknown object and one feature vector of the known objects satisfies a threshold value. The method further includes identifying the unknown object on the basis of the determining.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Certain aspects of the present disclosure relate generally to robotic devices, and more particularly to systems and methods for teaching a robotic device an unknown object through visual recognition and analysis of elements of previously learned objects. [Background technology]

[0002] As robotic assistance technology improves, robots may be programmed to perform a wide variety of tasks performed by people in environments such as the home. Traditionally, classifiers identify objects by training the classifier on a training set containing instances of specific objects. In such cases, when the classifier is faced with an object not included in the training set, it selects the closest known object. However, this selection may be incorrect. It is desirable to improve the classifier to identify unknown objects based on the elements and features of learned / known objects. Summary of the Invention

[0003] A method for identifying an object includes generating a feature vector of an unknown object from an image of an environment. The method also includes comparing the feature vector of the unknown object with feature vectors of known objects. The method further includes determining whether a similarity between the feature vector of the unknown object and one of the feature vectors of the known objects satisfies a threshold. The method further includes identifying the unknown object based on the determination.

[0004] A system for identifying an object is described. The system includes a memory and one or more processors coupled to the memory. The processor is configured to generate a feature vector of an unknown object from an image of an environment. The processor is also configured to compare the feature vector of the unknown object with feature vectors of known objects. The processor is further configured to determine whether a similarity between the feature vector of the unknown object and one of the feature vectors of the known objects satisfies a threshold. The processor is further configured to identify the unknown object based on the determination.

[0005] A non-transitory computer-readable medium having non-transitory program code recorded thereon is described. The program code is used to identify an object. The program code, when executed by a processor, includes program code for generating a feature vector of an unknown object from an image of an environment. The program code also includes program code for comparing the feature vector of the unknown object with feature vectors of known objects. The program code further includes program code for determining whether a similarity between the feature vector of the unknown object and a feature vector of one of the known objects satisfies a threshold. The program code further includes program code for identifying the unknown object based on the determination.

[0006] This has outlined rather broadly the features and technical advantages of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure are described below. Those skilled in the art should appreciate that this disclosure may readily be utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be recognized by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in conjunction with the accompanying figures. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure.

[0007] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a classifier system. [Figure 2] FIG. 1 illustrates an example architecture of a classifier, according to aspects of the present disclosure. [Figure 3] FIG. 1 illustrates an image of a training environment used to train a robot, according to aspects of the present disclosure. [Figure 4] FIG. 1 illustrates an example of an image of an environment captured by a sensor of a robotic device, according to aspects of the present disclosure. [Figure 5] FIG. 1 illustrates an example hardware implementation of a robotic device for recognizing unknown objects, according to aspects of the present disclosure. [Figure 6] FIG. 1 illustrates a method for a robotic device to recognize an unknown object, according to one aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] The detailed description set forth below in connection with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0010] Based on the teachings, one skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented, or a method may be practiced using any number of the described aspects. Furthermore, the scope of the present disclosure is intended to cover such an apparatus or method that is implemented using other structure, function, or structure and function in addition to or other than the various aspects of the present disclosure that are described. It should be understood that any aspect of the present disclosure may be embodied by one or more elements of a claim.

[0011] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0012] While particular aspects are described herein, many variations and permutations of these aspects are within the scope of the present disclosure. While certain benefits and advantages of the preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to particular benefits, applications, or objectives. Rather, aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are shown as examples in the figures of the preferred aspects and in the following description. The detailed description and drawings are merely illustrative of the present disclosure rather than limiting, the scope of the present disclosure being defined by the appended claims and their equivalents.

[0013] A robotic device can identify objects in its environment using one or more sensors. The sensors may include a red-green-blue (RGB) camera, a radio detection and ranging (RADAR) sensor, a light detection and ranging (LiDAR) sensor, or another type of sensor. Traditionally, perception software is used to recognize objects that have been included in a training set. Objects in the training set may be referred to as known objects or old objects. Objects not used in training may be referred to as unknown objects or new objects. When a classifier is faced with an unknown object, it may select the most similar known object. However, this selection may be incorrect.

[0014] Aspects of the present disclosure are directed to systems and methods for training a robotic system to identify new or unknown objects based on elements and characteristics of learned / known objects. The robotic system may also be referred to as a robotic device or robot. The robotic system may be autonomous or semi-autonomous.

[0015] The objects may include man-made objects (e.g., chairs, desks, cars, books, etc.), natural objects (e.g., rocks, fruit, trees, animals, etc.), and humans. In one embodiment, the robotic system predicts an N-dimensional or multidimensional vector for each detected object. For example, the robotic system predicts a 16-dimensional vector. In some embodiments, the multidimensional vector may include any value between 10 and 20. Of course, other values ​​may be used. The multidimensional vector may be predicted for each pixel of multiple pixels associated with each known object. A feature vector representing characteristics of the known object is assigned to each pixel of the known object. Each pixel may be associated with multiple feature vectors based on images acquired from different views / angles.

[0016] Given a training set of objects, the neural network is coded with an N-dimensional vector, or multidimensional vector, of distinct objects. The values ​​of the multidimensional vector can capture feature vectors or object embeddings of shape, color, size, texture, or the like. As a result, unknown objects different from the objects the classifier is trained on can be identified based on the multidimensional vector of distinct objects.

[0017] For example, a neural network may be trained on five different types of fruit and later introduced to an unknown fruit that is different from all the others. In one embodiment, an image of an environment containing unknown fruit may be captured by a sensor of the robotic device. For example, a neural network associated with the robotic device generates an output of the unknown fruit from the image. The output includes a feature vector of the unknown fruit. The feature vector may be a multidimensional vector. The feature vector may be determined for each pixel of the unknown fruit.

[0018] The neural network architecture used in this disclosure includes a network layer (e.g., a convolutional layer) that receives an input, performs several convolutions, and the final output is a tensor. In one aspect, the feature vector associated with the tensor includes the same number of rows and columns as the image. However, instead of having three channels as in a traditional tensor, the tensor may include N-channels (e.g., 16) per pixel (e.g., pixel coordinates). For example, for every pixel in the initial image processed by the network, there may be 16 values ​​per pixel in the output.

[0019] A neural network may use the features (e.g., feature vector) of the unknown fruit to infer the type of the unknown fruit. A neural network associated with the robotic device may compare the feature vector of the unknown fruit with feature vectors of known fruits. Based on the comparison, the neural network may classify the unknown fruit.

[0020] For example, a neural network may determine the color of a fruit based on knowledge of the colors of known fruits. Furthermore, the neural network may determine that the texture is more similar to the texture of known fruits. The texture and color may be defined by one or more values ​​of a multidimensional vector that classifies the unknown fruit. Furthermore, the shape of the fruit may be identified based on the shapes of known fruits. The shape may also be defined as one or more values ​​of a multidimensional vector.

[0021] When a classifier is trained on objects corresponding to different classes, unknown objects that share characteristics with objects in the training classes can be identified based on their similarity to the training characteristics. The neural network determines the differences and similarities between the unknown and known objects. For example, the neural network determines whether the differences and / or similarities between the feature vectors of an unknown fruit and the feature vectors of a known fruit meet a threshold. The selected known fruit may be the known fruit that is most similar to the unknown fruit. For example, the texture and shape of the unknown fruit are similar to a lime. Therefore, in this case, the selected known fruit is a lime.

[0022] The neural network then infers or identifies the unknown fruit based on the threshold decision. For example, if the similarity between the feature vector of the unknown fruit and one or more feature vectors of limes is greater than the similarity threshold, the unknown fruit is identified as a lime. In addition to similarity comparisons, dissimilarity comparisons can also be used to infer the type of unknown fruit. For example, if the dissimilarity between the feature vector of the unknown fruit and one or more feature vectors of limes is less than the dissimilarity threshold, the unknown fruit is identified as a lime.

[0023] In one embodiment, the unknown fruit may not be identified when the similarity between the feature vector of the sixth fruit and one or more feature vectors of the identified known fruit (e.g., lime) is less than a similarity threshold. Alternatively or additionally, the unknown fruit may not be identified when the dissimilarity between the feature vector of the unknown fruit and one or more feature vectors of the identified known fruit is greater than a dissimilarity threshold.

[0024] Thus, aspects of the present disclosure provide a network that can learn unknown objects based on elements of trained objects. Specifically, the network learns (e.g., identifies) unknown objects by determining similarities and differences between the characteristics of the trained objects and the unknown objects.

[0025] 1 illustrates an example of a classifier system 100. The classifier system 100 may be implemented to perform the methods and other functions disclosed herein. While the classifier system 100 is shown as a single, contained system, it should be understood that in various embodiments, the classifier system 100 is a distributed system made up of components that may be provided as a centralized server, a cloud-based service, or the like.

[0026] The classifier system 100 includes a processor 110. The processor 110 may represent a distributed processing resource, an individual local processor (e.g., a CPU, GPU, or application-specific processor), or the classifier system 100 may access the processor 110 via a data bus or another communication path. In one aspect, the classifier system 100 includes a memory 120. The memory 120 may be random-access memory (RAM), read-only memory (ROM), a hard-disk drive, flash memory, a processor cache, or other suitable memory.

[0027] Embodiments of classifier system 100 may be implemented as a cloud-based service (e.g., software as a service), a centralized service, and / or other types of services. Additionally, in one or more embodiments, classifier system 100 includes a database 150 stored in a separate data store, memory 120, or another suitable configuration. Database 150 may include image data of objects.

[0028] In one aspect, the machine learning module includes a machine learning implementation such as a convolutional neural network (CNN) or other suitable deep learning algorithm.

[0029] In a further aspect, the classifier can be provided to the robotic device on which it is implemented such that the classifier is stored in its local memory. Thus, the robotic device can implement the classifier to analyze objects from the camera 190, thereby enabling the robotic device to recognize objects in its environment.

[0030] FIG. 2 provides an example architecture of a classifier 200. As shown, the classifier 200 includes a convolutional neural network (CNN). The classifier 200 may include a transformation module for providing a transformation of an input image. The classifier 200 pairs a machine learning implementation with a transformation implementation that performs preprocessing on input data (e.g., an image). For example, the transformation module accepts an image input, performs preprocessing on the input, and passes the transformed input to a convolutional node layer that forms the CNN. In one configuration, each convolutional layer is followed by a max pooling layer, connected in one or more ways by a rectified linear activation (ReLu) layer. The final convolutional layer feeds into a fully connected ReLu layer.

[0031] FIG. 3 shows an image 301 of a training environment 302 used to train a robotic device 300 according to an embodiment of the present disclosure. In the example of FIG. 3, the robotic device 300 is a humanoid robot and the training environment 302 is a kitchen. Aspects of the present disclosure are not limited to humanoid robots. The robotic device 300 may be any type of autonomous or semi-autonomous device, such as a drone or a vehicle. Furthermore, the robotic device 300 may be in any type of environment. The robotic device 300 may be trained to identify unknown objects according to an embodiment of the present disclosure.

[0032] In one configuration, the robotic device 300 acquires an image 301 of the training environment 302 via one or more sensors of the robotic device 300. The robotic device 300 can detect and locate one or more objects in the image 301. Localization refers to determining the location (e.g., coordinates) of the detected objects in the image 301. In conventional object detection systems, a bounding box can indicate the location of the detected objects in the image 301. The detected objects may be one or more specific classes of objects, such as a table 304, a pushed-in chair 306, a closed window 308, a bottle 310, appliances 320 and 322, a counter 340, a sink 342, a cabinet 330 with a handle 332, or all objects in the image 301. The objects can be detected and identified using an object detection system, such as a pre-trained object detection neural network.

[0033] The 3D camera of the robotic device 300 captures images of the training environment 302 and an object (e.g., a lime or apple on a table) from different views / angles. A 3D model of the environment is generated from the captured images. The 3D model is used to create images from a perspective different from the perspective of the image 301 captured by the 3D camera. For example, during training, the 3D camera captures an image of an object (e.g., an apple) from an angle. A feature vector of the object may be determined for the angle (e.g., view) of the 3D camera. In one aspect, the view from which the object is captured is extrapolated to a circle or some path around the object. The feature vectors from each view of the object are collected as an example of the object. In the context of a neural network, an embedding or feature vector is a low-dimensional, learned continuous vector representation of a discrete variable.

[0034] Aspects of the present disclosure are directed to a system that uses the output (e.g., 16-dimensional output) of a neural network to teach a robotic device to identify unknown objects. For example, given a training set of objects, the neural network is encoded with an N-dimensional vector of various objects. The values ​​of the N-dimensional vector can capture the object's feature vector of shape, color, size, texture, or the like for each selected pixel in the training set of objects. As a result, unknown objects that differ from the objects on which the classifier was trained can be identified based on the N-dimensional vector of the various trained objects. Thus, the robotic device identifies unknown objects after the neural network training phase.

[0035] For example, when a robotic device performs a task such as identifying an unknown object, the robotic device's neural network receives an image (e.g., an RGB image) as input and outputs a feature vector containing a value assigned to each pixel in the image. A feature vector can contain image features encoded into a series of numbers to provide a numerical "fingerprint" for distinguishing one feature from another. The feature vector serves as a feature descriptor that can be established for each pixel correspondence in different images. Because a feature vector can be defined for each pixel, a keyframe (image) can be broad enough to include every pixel in the image or narrow enough to use only identified pixels or pixels within a user-defined mask.

[0036] In one aspect, a neural network (e.g., a dense embedding network) processes an input image and outputs an N-dimensional feature vector for the image's pixels. Each pixel in the input image may correspond to a 16-dimensional feature vector. The user may only select one pixel or descriptor of the image. The feature vector and image may be combined to generate a voxel map. The feature vector may also be input to a matcher (e.g., a keyframe or image matcher) that compares the feature vector of an unknown object's pixel with the feature vector of a known object's pixel (e.g., a known object's feature vector).

[0037] In one embodiment, images of the set of training objects and unknown objects can be acquired from different views / angles. Each set of objects (e.g., apples, bottles, etc.) can be taught from various angles. In one embodiment, the robotic device can extrapolate selected pixels into a three-dimensional representation. Thus, as the robotic device moves around, feature vectors can be matched without user intervention. In this way, the accuracy of the classifier may improve without further interaction from the user. Therefore, user interaction time is reduced. Because user interaction time is reduced, more data may be acquired.

[0038] For example, an apple can be taught from one angle, and then that angle can be extrapolated to a circle or path around the apple. For an example apple, a feature vector can be obtained for each view of the apple. Several examples of apples can be obtained. This allows the unknown object to be compared with feature vectors corresponding to various views / angles of known objects, improving the reliability of identifying the unknown object.

[0039] The features that extrapolate a single pixel into a three-dimensional representation of the feature vector are used to train a robotic device with a different set of training objects. Data from the training or education phase is input into the robotic device and used to recognize unknown objects. For example, consider a soda can with front and back labels, where the front label is different from the back label. The ability to identify both the front and back labels with a single selected pixel is advantageous because it requires less interaction from the user.

[0040] In one aspect, the robotic device uses depth information and 3D features to map pixels of an object to 3D locations. As the robotic device moves around the same object, the robotic device can use the depth information to select a pixel of the object. The selected pixel is a 3D point that is a first projection point corresponding to the initial pixel and the initial viewpoint. As the robot moves, the 3D point can be reprojected to a new camera position or viewpoint representing the object. Feature vectors associated with the various viewpoints are matched to the object's pixels and saved for use in recognizing unknown objects.

[0041] FIG. 4 illustrates an example image 450 of an environment 402 captured by a sensor of a robotic device 400 according to an embodiment of the present disclosure. The sensor may be a three-dimensional camera. The environment 402 is a dining area (e.g., a restaurant dining area) containing an unknown object (e.g., an unknown fruit 410) on a table 404. The environment also includes a chair 406, a window 408, a cabinet 430, and utensils 420 and 422 on the cabinet 430. In one example, a neural network associated with the robotic device 400 is trained on five different types of fruit (e.g., bananas, apples, limes, blueberries, and grapes). The neural network is later introduced to the unknown fruit 410, which is different from all the others.

[0042] A neural network associated with the robotic device 400 can use the feature vector to infer or identify the type of unknown fruit 410. In one embodiment, the neural network compares the feature vector of the unknown fruit 410 with feature vectors of known fruits. For example, the neural network may determine the color of the unknown fruit 410 based on knowledge of the colors of the known fruits. Additionally, the neural network may determine the texture of the unknown fruit 410 based on the texture of the known fruits. The neural network may determine that the round shape of the unknown fruit 410 is identical to the shape of the known fruits.

[0043] The neural network determines the dissimilarity and similarity between the feature vectors of the unknown fruit 410 and the known fruits. For example, the neural network determines whether the dissimilarity and / or similarity between the feature vector of the unknown fruit 410 and the feature vector of a selected one of the known fruits (e.g., lime) meets a threshold. The neural network identifies the unknown fruit 410 based on the determination using the threshold.

[0044] For example, an unknown fruit 410 is identified as a lime when the similarity between the feature vector of the unknown fruit 410 and one or more feature vectors of a lime is greater than a similarity threshold. In addition to similarity comparisons, dissimilarity comparisons can also be used to infer the type of unknown fruit 410. In this example, an unknown fruit 410 is identified as a lime when the dissimilarity between the feature vector of the unknown fruit and one or more feature vectors of a lime is less than a dissimilarity threshold.

[0045] 5 is a diagram illustrating an example hardware implementation of an object identification system 500 according to an embodiment of the present disclosure. The object identification system 500 may be a component of a vehicle, a robotic device, or another device. For example, as shown in FIG. 5, the object identification system 500 is a component of the robotic device 400.

[0046] Aspects of the present disclosure are not limited to object identification system 500 being a component of robotic device 400. Other devices, such as buses, boats, drones, or vehicles, are also contemplated for use with object identification system 500. Robotic device 400 may operate in at least an autonomous mode of operation and a manual mode of operation.

[0047] The object identification system 500 may be implemented with a bus architecture, generally represented by bus 550. The bus 550 may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the object identification system 500. The bus 550 couples together various circuits, including one or more processors and / or hardware modules, represented by processor 520, communications module 522, position module 524, sensor module 502, movement module 526, navigation module 528, and computer-readable medium 530. The bus 550 may also couple various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further.

[0048] The object identification system 500 includes a processor 520, a sensor module 502, an object identification module 510, an object matching module 512, a communication module 522, a location module 524, a movement module 526, a navigation module 528, and a transceiver 540 coupled to a computer-readable medium 530. The transceiver 540 is coupled to an antenna 542. The transceiver 540 communicates with various other devices via a transmission medium. For example, the transceiver 540 can receive commands via transmission from a user or a remote device. As another example, the transceiver 540 can transmit statistics and other information from the object identification module 510 to a server (not shown).

[0049] The object identification system 500 includes a processor 520 coupled to a computer-readable medium 530. The processor 520 performs processes, including executing software stored on the computer-readable medium 530, to provide functionality according to the present disclosure. The software, when executed by the processor 520, causes the object identification system 500 to perform the various functions described for a particular device, such as the robotic device 400 or any of the modules 502, 510, 512, 522, 524, 526, and 528. The computer-readable medium 530 may also be used to store data manipulated by the processor 520 when executing the software.

[0050] The sensor module 502 can acquire measurements via different sensors, such as a first sensor 504 and a second sensor 506. The first sensor 504 may be a vision sensor for capturing 3D images, such as a stereoscopic camera or a red-green-blue (RGB) camera. The second sensor 506 may be a ranging sensor, such as a light detection and ranging (LiDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the aforementioned sensors, as other types of sensors, such as, for example, thermal, sonar, and / or laser, are also contemplated for either the first sensor 504 or the second sensor 506.

[0051] Measurements from the first sensor 504 and the second sensor 506 may be processed by one or more of the processor 520, the sensor module 502, the object identification module 510, the communication module 522, the position module 524, the movement module 526, and the navigation module 528, used in conjunction with a computer-readable medium 530 to implement the functionality described herein. In one configuration, data captured by the first sensor 504 and the second sensor 506 may be transmitted to an external device via a transceiver 540. The first sensor 504 and the second sensor 506 may be coupled to or in communication with the robotic device 400.

[0052] The location module 524 may determine the location of the robotic device 400. For example, the location module 524 may determine the location of the robotic device 400 using a global positioning system (GPS). The communication module 522 may facilitate communication via the transceiver 540. For example, the communication module 522 may be configured to provide communication capabilities via different wireless protocols, such as WiFi, Long Term Evolution (LTE), 5G, etc. Additionally, the communication module 522 may be used to communicate with other components of the robotic device 400 that are not modules of the object identification system 500.

[0053] The locomotion module 526 can facilitate movement of the robotic device 400. As another example, the locomotion module 526 can be in communication with one or more power sources of the robotic device 400, such as motors and / or batteries. Locomotion can be evidenced via wheels, movable limbs, propellers, treads, fins, jet engines, and / or other sources of movement.

[0054] The robotic device 400 may use one or more sensors, such as a first sensor 504 and a second sensor 506, to identify objects in the environment. The sensors may include a red-green-blue (RGB) camera, a radio detection and ranging (RADAR) sensor, a light detection and ranging (LiDAR) sensor, or another type of sensor. In one aspect, an image of the environment is received by an object identification system 500 of the robotic device 400. The object identification system 500 includes an object identification module 510 that generates an output of unknown objects identified in the image of the environment. The output includes a feature vector of the unknown object.

[0055] The object identification module 510 may include an object matching module 512 (e.g., a keyframe or image matcher). For example, a feature vector may be input to the object matching module 512, which compares the feature vector of a pixel of an unknown object with the feature vector of a pixel of a known object. The object identification module 510 determines whether the dissimilarity and / or similarity between the feature vector of the unknown object and the feature vector of a selected one of the known objects meets a threshold. The object identification module 510 then identifies the unknown object based on the determination. The modules may be software modules executing on the processor 520 and residing / stored on the computer-readable medium 530, one or more hardware modules coupled to the processor 520, or some combination thereof.

[0056] The object identification module 510 can be in communication with the sensor module 502, the transceiver 540, the processor 520, the communication module 522, the position module 524, the movement module 526, the navigation module 528, and the computer-readable medium 530. In one configuration, the object identification module 510 receives sensor data from the sensor module 502. The sensor module 502 can receive sensor data from a first sensor 504 and a second sensor 506. According to aspects of the present disclosure, the sensor module 502 can filter the data, remove noise, encode the data, decode the data, combine the data, extract frames, or perform other functions. In an alternative configuration, the object identification module 510 can receive sensor data directly from the first sensor 504 and the second sensor 506.

[0057] In one configuration, the object identification module 510 identifies the detected object based on information from the processor 520, the location module 524, the computer-readable medium 530, the first sensor 504, and / or the second sensor 506.

[0058] 6 illustrates a method 600 for a robotic device to recognize an unknown object according to one embodiment of the present disclosure. In block 602, a feature vector of the unknown object is generated from an image of the environment. In block 604, the feature vector of the unknown object is compared with feature vectors of known objects. In block 606, the robotic device determines whether the similarity between the feature vector of the unknown object and the feature vector of one of the known objects meets a threshold. In block 608, the robotic device identifies the unknown object based on the determination.

[0059] The various operations of the above methods may be performed by any suitable means capable of performing the corresponding functions, which may include various hardware and / or software components and / or modules, including, but not limited to, circuits, application specific integrated circuits (ASICs), or processors. Generally, where there are operations illustrated in figures, those operations may include corresponding corresponding means and functional components that are similarly numbered.

[0060] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, and the like. Furthermore, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Furthermore, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0061] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc.

[0062] The various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed using a processor configured in accordance with this disclosure, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A processor may be a microprocessor, controller, microcontroller, or state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or other special configurations as described herein.

[0063] The steps of a method or algorithm described in connection with this disclosure may be embodied directly in hardware, in a software module executed by a processor, or a combination of the two. A software module may reside in a storage or machine-readable medium, including random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium usable to carry or store program code in the form of instructions, or data structures accessible by a computer. A software module may comprise a single instruction or many instructions, and may be distributed across several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0064] The methods disclosed herein include one or more steps or actions for achieving the described method. The method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless the order of certain steps or actions is specified, the order and / or use of certain steps and / or actions may be changed without departing from the scope of the claims.

[0065] The described functionality may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an exemplary hardware configuration may include a processing system within a device. The processing system may be implemented with a bus architecture. The bus may include any number of interconnecting buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may interconnect various circuits, including processors, machine-readable media, and bus interfaces. The bus interface may connect a network adapter to the processing system via the bus, among other things. The network adapter may implement signal processing functions. In certain aspects, a user interface (e.g., a keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also couple various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, and the like, which are well known in the art and therefore will not be described further.

[0066] The processor may include the execution of software stored on a machine-readable medium and may be responsible for managing the bus and processing. Software shall be taken to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0067] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which may be accessed by the processor via a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as with a cache and / or special register files. The various components described may be described as having a particular location, such as a local component, or may be organized in various ways, such as a particular component configured as part of a distributed computing system.

[0068] The processing system may be comprised of one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all coupled together with other support circuitry via an external bus architecture. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neuron model and model of the nervous system described herein. As another alternative, the processing system may be implemented with an application-specific integrated circuit (ASIC) that includes the processor, bus interface, user interface, support circuitry, and at least a portion of the machine-readable medium integrated into a single chip. Alternatively, the processing system may be implemented with one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how best to implement the functionality described for the processing system depending on the particular application and the overall design constraints imposed on the entire system.

[0069] The machine-readable medium may include several software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside on a single storage device or be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, a processor may load some instructions into a cache to increase access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. Furthermore, it should be understood that aspects of the present disclosure provide improvements to the functionality of a processor, computer, machine, or other system implementing such aspects.

[0070] If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any storage medium that facilitates transfer of a computer program from one place to another. Additionally, all connections are properly referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, or microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, or microwave are included in the definition of media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, which reproduce data optically with lasers, whereas discs typically reproduce data magnetically. Thus, in some aspects computer-readable medium may include non-transitory computer-readable medium (e.g., tangible media). Further, in other aspects computer-readable medium may include transitory computer-readable medium (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media.

[0071] Accordingly, certain aspects may include a computer program product for performing the operations presented herein. For example, such a computer program product may include a computer-readable medium having instructions stored thereon (and / or encoded thereon), the instructions being executable by one or more processors to perform the operations described herein. In certain aspects, the computer program product may include packaging materials.

[0072] Furthermore, it should be understood that modules and / or other suitable means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by a user terminal and / or base station, as appropriate. For example, such devices can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein can be provided via storage means, such that the user terminal and / or base station can obtain the various methods when the storage means is coupled to or provided to the device. Furthermore, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

[0073] It is to be understood that the claims are not limited to the precise configuration and components shown above. Various modifications, changes and variations can be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. 1. A method for identifying an object, comprising: generating feature vectors of unknown objects from images of the environment; comparing the feature vector of the unknown object with the feature vector of a known object; determining whether a similarity between the feature vector of the unknown object and a feature vector of one of the known objects satisfies a threshold; and identifying the unknown object based on said determining; A method comprising:

2. The method of claim 1 , wherein the feature vector of the unknown object and the feature vector of each of the known objects comprise multi-dimensional vectors.

3. The method of claim 1 , wherein the feature vector of the unknown object and each of the feature vectors of the known objects correspond to characteristics including at least one of shape, color, size, and texture.

4. the feature vector of the unknown object contains the same number of rows and columns as the image; The feature vector of the unknown object comprises N channels for each pixel or descriptor of the image. The method of claim 1.

5. each of the known objects comprises a number of feature vectors; Each of the multiple feature vectors is based on an image taken from a different view. The method of claim 1.

6. 2. The method of claim 1, further comprising inferring that the unknown object is the same as one of the known objects when the dissimilarity between their feature vectors is less than the threshold and / or the similarity between their feature vectors is greater than the threshold.

7. 1. A system for identifying an object, comprising: memory, and at least one processor, the at least one processor Generate feature vectors for unknown objects from images of the environment, comparing the feature vector of the unknown object with the feature vector of a known object; determining whether a similarity between the feature vector of the unknown object and a feature vector of one of the known objects satisfies a threshold; and configured to identify the unknown object based on the determination. system.

8. The system of claim 7 , wherein the feature vector of the unknown object and the feature vector of each of the known objects comprise multi-dimensional vectors.

9. The system of claim 7 , wherein the feature vector of the unknown object and each of the feature vectors of the known objects correspond to characteristics including shape, color, size, and / or texture.

10. the feature vector of the unknown object contains the same number of rows and columns as the image; The feature vector of the unknown object comprises N channels for each pixel or descriptor of the image. The system of claim 7.

11. each of the known objects comprises a number of feature vectors; Each of the multiple feature vectors is based on an image taken from a different view. The system of claim 7.

12. 8. The system of claim 7, wherein the at least one processor is further configured to infer that the unknown object is the same as one of the known objects when a dissimilarity between their feature vectors is less than the threshold and / or the similarity between their feature vectors is greater than the threshold.

13. 1. A non-transitory computer-readable medium having program code recorded thereon and executed by a processor for identifying an object, the medium comprising: program code for generating feature vectors of unknown objects from images of the environment; program code for comparing the feature vectors of the unknown object with feature vectors of known objects; program code for determining whether a similarity between a feature vector of the unknown object and a feature vector of one of the known objects satisfies a threshold; and program code for identifying the unknown object based on the determination; Non-transitory computer-readable medium.

14. The non-transitory computer-readable medium of claim 13 , wherein the feature vector of the unknown object and each of the feature vectors of the known objects comprise multi-dimensional vectors.

15. The non-transitory computer-readable medium of claim 13 , wherein the feature vector of the unknown object and each of the feature vectors of the known objects correspond to characteristics including shape, color, size, and / or texture.

16. the feature vector of the unknown object contains the same number of rows and columns as the image; The feature vector of the unknown object comprises N channels for each pixel or descriptor of the image.

14. The non-transitory computer-readable medium of claim 13.

17. each of the known objects comprises a number of feature vectors; Each of the multiple feature vectors is based on an image taken from a different view.

14. The non-transitory computer-readable medium of claim 13.

18. 14. The non-transitory computer-readable medium of claim 13, further comprising program code for inferring that the unknown object is the same as one of the known objects when a dissimilarity between their feature vectors is less than the threshold and / or when the similarity between their feature vectors is greater than the threshold.

Images