Training method of deep network

Abstract

A method for training a deep neural network of a robotic device is described. The method includes constructing a 3D model using images captured via a 3D camera of the robotic device in a training environment. The method also includes generating 3D image pairs from the 3D model using a deep neural network by artificially adjusting parameters of the training environment to form manipulated images. The method further includes processing the 3D image pairs to form reference images that include embedded descriptors of common objects between the 3D image pairs. The method also includes determining correlations using the reference images from the training of the neural network to identify objects detected in future images.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 570,813, filed September 13, 2019, entitled “TRAINING METHODS FOR DEEPNETWORKS”, which claims priority to U.S. Provisional Patent Application No. 62 / 877,792, filed July 23, 2019, entitled “KEYFRAME MATCHER”, U.S. Provisional Patent Application No. 62 / 877,791, filed July 23, 2019, entitled “VISUAL TEACH AND REPEAT FOR MANIPULATION–TEACHING VR”, and U.S. Provisional Patent Application No. 62 / 877,793, filed July 23, 2019, entitled “VISUALIZATION”, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Certain aspects of this disclosure generally relate to object detection training, and more specifically to systems and methods for training deep networks. Background Technology

[0004] Robotic devices can use one or more sensors (e.g., cameras) to identify objects in their environment based on training using real-world images. However, in real-life scenarios, the images encountered may differ from the actual images used to train the robotic device. That is, deformations from the image data used for training, object articulation, changes in viewpoint, and lighting can hinder object detection during real-world operation.

[0005] Traditional systems collect real-world training images under the actual conditions of the desired observation. For example, the training of robotic devices is limited to the actual conditions used to collect training images, including actual lighting levels and the specific angle at which the training images are collected. These traditional systems do not take into account changes in the environment. These variations between training data and real-world objects are particularly problematic when training deep neural networks in robotic devices to perform object detection. Summary of the Invention

[0006] A method for training a deep neural network for a robotic device is described. The method includes constructing a 3D model in a training environment using images captured by a 3D camera on the robotic device. The method also includes generating 3D image pairs from the 3D model using the deep neural network by manually adjusting parameters of the training environment to form manipulated images. The method further includes processing the 3D image pairs to form a reference image, which includes embedding descriptors of common objects between the 3D image pairs. The method also includes using the reference image from the neural network training to determine correlations to identify objects detected in future images.

[0007] A method for controlling a robotic device based on the recognition of detected objects in an unknown environment is described. The method includes detecting objects in the unknown environment. The method also includes selecting a corresponding reference image, which includes an embedding descriptor corresponding to a trained object manipulated according to parameters of the image capture environment that are manually adjusted. The method further includes recognizing the detected object based on the embedding descriptor of the corresponding reference image.

[0008] A system for controlling a robotic device based on the recognition of detected objects in an unknown environment is described. The system includes a pre-trained object recognition module. The object recognition module is configured to select a corresponding reference image to recognize the detected objects in a captured image. The corresponding reference image includes an embedded descriptor of the trained object manipulated based on manually adjusted parameters according to the image capture environment. The system also includes a controller configured to select autonomous behaviors of the robotic device based on the identity of the detected object.

[0009] This has provided a fairly broad overview of the features and technical advantages of this disclosure in order to better understand the detailed embodiments described below. Additional features and advantages of this disclosure will now be described. Those skilled in the art will understand that this disclosure can be readily used as the basis for modifying or designing other structures for achieving the same purposes as this disclosure. Those skilled in the art will also recognize that such equivalent constructions do not depart from the teachings of this disclosure as set forth in the appended claims. The novel features, as well as further objects and advantages considered characteristic of this disclosure in both its organization and manner of operation, will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be clearly understood that each drawing is provided for illustrative and descriptive purposes only and is not intended to be a limitation of this disclosure. Attached Figure Description

[0010] The features, nature, and advantages of this disclosure will become more apparent from the detailed description of the embodiments set forth below when used in conjunction with the accompanying drawings, in which the same reference numerals are indicated accordingly throughout the text.

[0011] Figure 1Original images illustrating the environment used for training robots according to various aspects of this disclosure are provided.

[0012] Figure 2 Examples of manipulation images created using 3D models to train a robot in a training environment, according to various aspects of this disclosure, are illustrated.

[0013] Figure 3A and 3B The illustration shows paired images of a training environment for training a robot, generated according to various aspects of this disclosure.

[0014] Figure 4A and 4B Captured images of a real-world environment captured by a robot, according to various aspects of this disclosure.

[0015] Figure 5 The diagram illustrates an example of a hardware implementation for an object recognition system according to various aspects of this disclosure.

[0016] Figure 6 This is a flowchart illustrating a method for training a deep neural network for a robotic device according to various aspects of this disclosure. Detailed Implementation

[0017] The specific embodiments described below with reference to the accompanying drawings are intended as descriptions of various configurations and are not intended to represent the only configuration in which the concepts described herein can be practiced. The specific embodiments include particular details to provide a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some cases, well-known structures and components are shown in block diagram form to avoid obscuring these concepts.

[0018] Robotic devices can use one or more sensors to identify objects in their environment. These sensors can include red-green-blue (RGB) cameras, radio detection and ranging (RADAR) sensors, light detection and ranging (LiDAR) sensors, or other types of sensors. In the images captured by the sensors, the robotic device identifies one or more objects based on training a deep neural network, performing object detection. However, in real-world scenarios, the images encountered may differ from the actual images used to train the robotic device. That is, deformations within the image data used for training, variations in object representation, viewpoint, and lighting can hinder object detection during real-world operation.

[0019] Traditional systems collect real-world training images under the actual conditions of the desired observation. For example, the actual conditions for collecting training images include the actual lighting level and the specific angle at which the images are collected. These traditional systems do not consider variations in the environment. These discrepancies between training data and real-world objects are particularly problematic when training deep neural networks for robotic devices to perform object detection.

[0020] This disclosure aims to provide data for training deep networks by taking into account changes in the environment. These changes include, for example, object deformation, object representation, viewpoint changes, and / or lighting changes.

[0021] In this disclosure, for simplicity, robotic devices may be referred to as robots. Furthermore, objects may include both stationary and dynamic objects in the environment. Objects may include man-made objects (e.g., chairs, tables, cars, books, etc.), natural objects (e.g., rocks, trees, animals, etc.), and humans.

[0022] Figure 1 Original images 101 illustrate the training environment 102 for training robot 100 according to various aspects of this disclosure. Figure 1 In the example, robot 100 is a humanoid robot and training environment 102 is a kitchen. The various aspects of this disclosure are not limited to humanoid robots. Robot 100 can be any type of autonomous or semi-autonomous device (such as a drone or vehicle). Furthermore, robot 100 can be in any type of environment.

[0023] In one configuration, robot 100 acquires a raw image 101 of training environment 102 via one or more sensors of robot 100. Robot 100 can detect and localize one or more objects in the raw image 101. Localization refers to determining the location (e.g., coordinates) of the detected objects within the raw image 101. In conventional object detection systems, bounding boxes can be used to indicate the location of objects detected in the raw image 101. The detected objects can be one or more objects of a specific category (such as table 104, push-back chair 106, closed window 108, bottle 110, cutlery 120 and 122, countertop 140, sink 142, cabinet 130 with handle 132, or all objects in the raw image 101). Object detection systems (such as pre-trained object detection neural networks) can be used to detect and identify objects.

[0024] In one configuration, a 3D camera on robot 100 captures images of training environment 102 from different viewpoints / angles. A 3D model of training environment 102 is generated from the captured images. The 3D model is used to create images from viewpoints different from the viewpoints of the original images 101 captured by the 3D camera. The 3D model is also used to modify lighting conditions (e.g., adjust light levels) in the created images. Furthermore, the 3D model can create images with manipulated objects. For example, the 3D model can create scenes of opening or closing drawers / windows. Additionally, the system links images with common features. The captured images and the images created by the 3D model (e.g., training data) are used to train a deep network object detection system.

[0025] Figure 2 Examples of manipulation images 200 created from 3D models to train robot 100 in training environment 202, according to various aspects of this disclosure, are illustrated. Figure 2 In the example, training environment 202 is Figure 1 The kitchen is generated by flipping elements horizontally to provide different perspectives. Robot 100 can use... Figure 1 The original image 101 and the manipulated image 200 are used to detect and identify objects in each image via an object detection system (such as a pre-trained object detection neural network).

[0026] In this configuration, the system generates manipulation image 200 and compares manipulation image 200 with... Figure 1 The training environment 102 is paired with the original image 101. According to various aspects of this disclosure, link elements are identified between the original image 101 and the manipulated image 200. That is, each element of the training environment 202 can be assigned pixel coordinates. Overlapping pixel coordinates indicate overlapping portions (e.g., link elements). As an example, the pulled-out chair 206 is a link element between the manipulated image 200 and the pushed-in chair 106 in the original image 101. Links indicate the same element depicted in different expressions. The link portion can be defined by the point correspondence within the original image 101 and the manipulated image 200 (e.g., having the same viewpoint as the original image 101).

[0027] In this example, the closed window 108 of the original image 101 is paired with the open window 208 of the manipulated image 200. For example, the panes of the open window 208 are linked between the original image 101 and the manipulated image 200. Furthermore, the table 204 in the manipulated image 200 is also linked to the table 104 in the original image 101. Similarly, the bottle 210 in the manipulated image 200 is linked to the bottle 110 in the original image 101. The bottle 110 is placed on a countertop 240 linked to the countertop 140 in the original image 101. The sink 242 is also linked between the manipulated image 200 and the original image 101. Additionally, the cabinet 230 and handle 232 of the manipulated image 200 are also linked to the cabinet 130 and handle 132 of the original image 101.

[0028] The robot 100 is trained to detect the pulled-out chair 206 from... Figure 1 The chair 106 shown is pushed in and flipped horizontally. Similarly, the robot 100 is trained to track the bottle 210 that is moved from the countertop 240 and placed on the table 204. Furthermore, the robot 100 is trained to track the areas of cutlery 220 and 222 that move from the table 204 to the countertop 240. Although the original image 101 and the manipulated image 200 are shown, it should be appreciated that various aspects of this disclosure are possible, including generating additional manipulated images with varying lighting conditions, viewpoints, deformations, etc.

[0029] According to various aspects of this disclosure, paired images of a 3D environment are processed by an image-to-image neural network. The network receives RGB images as input and outputs an embedding or descriptor image, including a value assigned to each pixel. The embedding / descriptor image can encode information into a series of numbers to provide a numerical "fingerprint" for distinguishing one feature from another. Ideally, this information is invariant to image transformations. Unfortunately, conventional feature descriptors are not invariant to image transformations because conventional systems are typically trained without considering changes in the environment.

[0030] In this aspect of the disclosure, the embedded / descriptor image determines the relevance to future images (e.g., images captured in real time while robot 100 is operating), defining objects and points in the environment. That is, after training, when placed in a new environment, the robot can identify locations in the new environment that can be manipulated, such as chairs, windows, bottles, cutlery (e.g., spoons), cabinets, etc. Robot 100 can also recognize various elements regardless of deformation, object representation, angle, and lighting. For example, based on the linked elements of the descriptor image formed from paired images (e.g., original image 101 and manipulated image 200), objects detected in poses different from those in original image 101 can be readily identified.

[0031] Figure 3A and 3B Paired images of a training environment 302 generated for training robot 100 according to various aspects of this disclosure are illustrated. For example... Figure 3A and 3B As shown, the training system automatically generates image pairs, where identical elements in the varied images are linked. For example, Figure 3A The original image 300 illustrates the training environment 302. The original image 300 further illustrates the countertop 340, sink 342, and cabinet 330 including handles 332. In this example, cabinet 330 is closed.

[0032] Figure 3B The manipulation of the image 350 in the training environment 302 according to various aspects of this disclosure is illustrated. In this example, in a scenario where the cabinet 330 is closed (e.g., Figure 3A In this image, the handle 332 of cabinet 330 is paired with a scene where cabinet 330 is open. Furthermore, cutlery 320 and 322 are paired between the original image 300 (e.g., inside cabinet 330) and the manipulated image 350 (e.g., showing cabinet 330 when open). This pairing of the original image 300 and the manipulated image 350 would be objects with the same elements but depicted in different ways linked together. The linked portion is defined by the point correspondence within the manipulated image 350 and the original image 300. The corresponding elements between paired images can be determined by identifying the overlapping portions of the training environment 302 captured in each image (i.e., scene).

[0033] The image pairs are then processed by an image-to-image neural network, which takes an RGB image as input and outputs an embedding or descriptor image consisting of the value assigned to each pixel of the image. According to various aspects of this disclosure, the embedding is used to determine the relevance to future images (e.g., images captured in real time while a robot is operating). For example, the embedding can define objects and points in the environment to identify relevant objects. In other words, the system can quickly determine the location of objects in the environment to identify them by the relevance of the embedding to a real-time image, e.g., as... Figure 4A and 4B As shown.

[0034] Figure 4A and 4B This describes captured images of the unknown environment 402 captured by robot 100 according to various aspects of this disclosure. Figure 4A and 4B In the example, the unknown environment 402 is a restaurant, including a table 404, pull-out chairs 406, an open window 408, bottles 410, cutlery 420 and 422, and a cabinet 430. In one configuration, robot 100 uses a training environment (such as...) Figure 1 , Figure 2 , Figure 3A and Figure 3B The training environment of the kitchen shown is illustrated with a paired reference image of the original image and the manipulated image. Using the reference image, robot 100 uses a pre-trained object detection neural network to detect the pulled-out chair 406. Additionally, the reference image enables robot 100 to detect the open window 408.

[0035] Figure 4A This describes captured images 400 of an unknown environment 402 captured by a 3D camera of robot 100, according to various aspects of this disclosure. Figure 4A In the example, the unknown environment 402 is a restaurant, including tables 404, pull-out chairs 406, and open windows 408. In one configuration, robot 100 uses a training environment-based approach (such as...). Figure 1 , Figure 2 , Figure 3A and Figure 3B The image is paired with a reference image (shown as the training environment of the kitchen). Using the reference image, robot 100 uses a pre-trained object detection neural network to localize the pulled-out chair 406. Furthermore, the reference image enables robot 100 to recognize the open window 408.

[0036] like Figure 4A As further shown, the reference images enable robot 100 to detect cutlery 420 and 422 on table 404. Additionally, the reference images enable robot 100 to detect bottle 410 on cabinet 430. Detection is not limited by position and / or object orientation in the environment. According to various aspects of this disclosure, robot 100 is trained to track the movement of objects over time. For simplicity, kitchen items are used as examples of the detected objects. Moreover, various aspects of this disclosure are not limited to detecting kitchen items and are envisioned for use with other objects.

[0037] Figure 4B Images 450 of an unknown environment 402 captured by a 3D camera of robot 100, according to various aspects of this disclosure, are illustrated. Figure 4B In the example, the unknown environment 402 is also a restaurant, including tables 404, pull-out chairs 406, open windows 408, and cabinets 430. In one configuration, robot 100 uses reference images to track cutlery 420 and 422 and bottle 410. Using a pre-trained object detection neural network, robot 100 is able to track the movement of cutlery 420 and 422 and bottle 410. That is, bottle 410 is... Figure 4A and Figure 4B The items are moved from cabinet 430 to table 404. Similarly, cutlery 420 and 422 are... Figure 4A and Figure 4B Move from table 404 to cabinet 430.

[0038] According to various aspects of this disclosure, pre-trained object detection neural networks use embeddings (e.g., object descriptors) to determine relevance to future images (e.g., images captured in real-time while robot 100 is operating), which define objects and points in the environment. In other words, the system can quickly determine the location of objects in an unknown environment by the relevance of embeddings to real-time images. This disclosure provides a method for generating and training deep networks by collecting training images using a 3D camera, manually adjusting illumination levels, and automatically creating image pairs linked together by common features. Therefore, object detection in unknown environments is not limited by the pose or position of objects in the unknown environment.

[0039] Figure 5 This is a diagram illustrating an example of a hardware implementation for an object recognition system 500 according to various aspects of this disclosure. The object recognition system 500 may be a component of a vehicle, robotic device, or another device. For example, as... Figure 5 As shown, the object recognition system 500 is a component of the robot 100 (e.g., a robotic device).

[0040] The various aspects of this disclosure are not limited to the object recognition system 500, which is a component of robot 100. Other devices (such as buses, ships, drones, or vehicles) using the object recognition system 500 are also contemplated. Robot 100 can operate in at least an autonomous operation mode and a manual operation mode.

[0041] The object recognition system 500 can be implemented using a bus architecture, typically represented by a bus 550. Depending on the specific application and overall design constraints of the object recognition system 500, the bus 550 may include any number of interconnect buses and bridges. The bus 550 links together various circuits including one or more processors and / or hardware modules (represented by processor 520, communication module 522, position module 524, sensor module 502, motion module 526, navigation module 528, and computer-readable medium 530). The bus 550 may also link various other circuits well-known in the art (such as timing sources, peripheral devices, voltage regulators, and power management circuits), which will not be described further.

[0042] The object recognition system 500 includes a transceiver 540 coupled to a processor 520, a sensor module 502, an object recognition module 510, a communication module 522, a position module 524, a motion module 526, a navigation module 528, and a computer-readable medium 530. The transceiver 540 is coupled to an antenna 542. The transceiver 540 communicates with various other devices via a transmission medium. For example, the transceiver 540 can receive commands via transmission from a user or remote device. As another example, the transceiver 540 can transmit statistical information and other information from the object recognition module 510 to a server (not shown).

[0043] The object recognition system 500 includes a processor 520 coupled to a computer-readable medium 530. The processor 520 performs processing, including the execution of software stored on the computer-readable medium 530, to provide functionality according to this disclosure. When executed by the processor 520, the software causes the object recognition system 500 to perform various functions described for a particular device (such as robot 100, or any of modules 502, 510, 512, 514, 516, 522, 524, 526, and 528). The computer-readable medium 530 can also be used to store data manipulated by the processor 520 during software execution.

[0044] Sensor module 502 can be used to acquire measurements via various sensors, such as first sensor 504 and second sensor 506. First sensor 504 may be a vision sensor (such as a stereo camera or red-green-blue (RGB) camera) for capturing 3D images. Second sensor 506 may be a ranging sensor (such as a LiDAR sensor or a RADAR sensor). Of course, the aspects of this disclosure are not limited to the sensors described above, as other types of sensors (such as thermal sensors, sonar sensors, and / or laser sensors) are also contemplated for either first sensor 504 or second sensor 506.

[0045] The measurements taken by the first sensor 504 and the second sensor 506 can be processed by one or more of the following components in conjunction with a computer-readable medium 530 to achieve the functions described herein: processor 520, sensor module 502, object recognition module 510, communication module 522, position module 524, motion module 526, and navigation module 528. In one configuration, the data captured by the first sensor 504 and the second sensor 506 can be transmitted to an external device via a transceiver 540. The first sensor 504 and the second sensor 506 can be coupled to or communicate with the robot 100.

[0046] The location module 524 can be used to determine the position of the robot 100. For example, the location module 524 can use a Global Positioning System (GPS) to determine the position of the robot 100. The communication module 522 can be used to facilitate communication via transceiver 540. For example, the communication module 522 can be configured to provide communication capabilities via different wireless protocols such as Wi-Fi, LTE, 5G, etc. The communication module 522 can also be used to communicate with other components of the robot 100 that are not modules of the object recognition system 500.

[0047] Motion module 526 can be used to facilitate the movement of robot 100. As another example, motion module 526 can communicate with one or more power sources of robot 100, such as electric motors and / or batteries. Movement can be demonstrated via wheels, movable limbs, propellers, pedals, fins, jet engines, and / or other motion sources.

[0048] The object recognition system 500 also includes a navigation module 528 for planning routes or controlling the movement of the robot 100 via the motion module 526. Routes can be planned based on data provided via the object recognition module 510. The module can be a software module running in the processor 520, residing in / stored in a computer-readable medium 530, one or more hardware modules coupled to the processor 520, or some combination thereof.

[0049] The object recognition module 510 can communicate with the sensor module 502, transceiver 540, processor 520, communication module 522, position module 524, motion module 526, navigation module 528, and computer-readable medium 530. In one configuration, the object recognition module 510 receives sensor data from the sensor module 502. The sensor module 502 can receive sensor data from the first sensor 504 and the second sensor 506. According to various aspects of this disclosure, the sensor module 502 can filter data to remove noise, encode data, decode data, merge data, extract frames, or perform other functions. In an alternative configuration, the object recognition module 510 can receive sensor data directly from the first sensor 504 and the second sensor 506.

[0050] In one configuration, the object recognition module 510 identifies detected objects based on information from the processor 520, the position module 524, the computer-readable medium 530, the first sensor 504, and / or the second sensor 506. The identification of detected objects from the object detection module 512 can be performed using the embedded correlation module 514. Based on the identified objects, the object recognition module 510 can control one or more actions of the robot 100 via the motion module 516.

[0051] For example, actions could include tracking moving objects among various images of a scene captured by robot 100 and performing safety actions (such as contacting safety services). The object recognition module 510 can perform actions via processor 520, position module 524, communication module 522, computer-readable medium 530, motion module 526, and / or navigation module 528.

[0052] In this aspect of the disclosure, the embedding relevance module 514 uses embedded / descriptor images from training to determine relevance to future images, defining objects and points in the unknown environment. That is, after training, when placed in a new environment, the robot 100 can identify locations in the new environment that can be manipulated (such as chairs, windows, bottles, cutlery (e.g., spoons), cabinets, etc.). The robot 100 can also recognize various elements regardless of deformation, object representation, angle, and lighting.

[0053] Figure 6 A flowchart illustrating a method for training a deep neural network for a robotic device according to various aspects of this disclosure is provided. For simplicity, the robotic device will be referred to as a robot.

[0054] like Figure 6 As shown, method 600 begins at block 602, where a 3D model is constructed in a training environment using images captured via a 3D camera of a robotic device. For example, as... Figure 1 As shown, robot 100 captures raw images 101 of training environment 102. Objects can be captured via one or more sensors of robot 100 (such as LiDAR, RADAR, and / or RGB cameras). Objects can be observed over a period of time (such as hours, days, etc.).

[0055] At block 604, 3D image pairs are generated from the 3D model by artificially adjusting the parameters of the training environment using a neural network to form manipulated images. For example, Figure 3B This illustrates an image 350 showing the manipulation of the handle 332 of cabinet 330 in a scene where cabinet 330 is open, paired with a scene where cabinet 330 is closed (e.g., Figure 3A This pairing of the original image 300 and the manipulated image 350 would be objects with the same elements but depicted in different (e.g., artificial) ways linked together.

[0056] At block 606, the 3D image pairs are processed to form a reference image, which includes embedding descriptors of common objects between the 3D image pairs. For example, Figure 4A and Figure 4B This describes the captured images of the unknown environment 402. In one configuration, robot 100 uses images based on the training environment (such as...). Figure 1 , Figure 2 , Figure 3A and Figure 3B The training environment of the kitchen shown is illustrated with a pair of original and manipulated reference images. At block 608, the reference images trained from the neural network are used to determine their relevance to future images. For example, as... Figure 4A and 4BAs shown, using a reference image, robot 100 uses a pre-trained object detection neural network to detect the pulled-out chair 406. Additionally, the reference image enables robot 100 to detect the open window 408.

[0057] According to various aspects of this disclosure, a method for controlling a robotic device based on the recognition of objects detected in an unknown environment is described. The method includes detecting objects in the unknown environment. For example, such as... Figure 4A As shown, robot 100 detects cutlery 420 and 422 on table 404. This detection can be performed by selecting a corresponding reference image, which includes an embedding descriptor corresponding to an object trained to be manipulated according to manually adjusted parameters based on the image capture environment.

[0058] The method also includes identifying the detected objects based on the embedded descriptors of the corresponding reference images. For example, using a pre-trained object detection neural network, robot 100 is able to track the movement of cutlery 420 and 422 and bottle 410. That is, bottle 410... Figure 4A and Figure 4B The items are moved from cabinet 430 to table 404. Similarly, cutlery 420 and 422 are... Figure 4A and Figure 4B Move from table 404 to cabinet 430.

[0059] Based on the teachings described above, those skilled in the art will understand that the scope of this disclosure is intended to cover any aspect of this disclosure, whether implemented independently of or in combination with any other aspect of this disclosure. For example, an apparatus may be implemented using any number of the described aspects, or a method may be practiced. Furthermore, the scope of this disclosure is intended to cover such an apparatus or method practiced using structures, functions, or structures and functions other than those described in this disclosure. It should be understood that any aspect of this disclosure may be embodied by one or more elements of the claims.

[0060] The word “exemplary” is used in this document to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” in this document is not necessarily to be construed as superior to or better than the others.

[0061] While specific aspects are described herein, numerous variations and arrangements of these aspects fall within the scope of this disclosure. Although some benefits and advantages of preferred aspects have been mentioned, the scope of this disclosure is not intended to be limited to specific benefits, uses, or objectives. Rather, various aspects of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the accompanying drawings and the following description of preferred aspects. The detailed description and accompanying drawings are merely illustrative and not limiting of this disclosure, the scope of which is defined by the appended claims and their equivalents.

[0062] As used herein, the term "determining" encompasses a variety of actions. For example, "determining" can include calculation, computation, processing, derivation, investigation, searching (e.g., looking in a table, database, or other data structure), ascertaining, etc. Additionally, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Furthermore, "determining" can include parsing, selecting, choosing, establishing, etc.

[0063] As used herein, the phrase “at least one” in the list of items refers to any combination of those items, including a single member. For example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc.

[0064] The various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented or executed by a processor specifically configured to perform the functions discussed in this disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neuron and nervous system models described herein. The processor may be a microprocessor, a controller, a microcontroller, or a state machine specifically configured as described herein. As described herein, the processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or such other special configurations).

[0065] The steps of the methods or algorithms described in this disclosure can be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module can reside in a storage device or machine-readable medium, including random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store required program code or data structures in the form of instructions and is accessible by a computer. The software module can include a single instruction or multiple instructions and can be distributed across several different code segments, different programs, and across multiple storage media. The storage medium can be coupled to the processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor.

[0066] The methods disclosed herein include one or more steps or actions for implementing the methods. The method steps and / or actions may be interchanged with each other without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of a particular step and / or action may be modified without departing from the scope of the claims.

[0067] The described functionality can be implemented in hardware, software, firmware, or any combination thereof. If implemented in hardware, an example hardware configuration may include a processing system within the device. The processing system can be implemented using a bus architecture. Depending on the specific application and overall design constraints of the processing system, the bus may include any number of interconnect buses and bridges. The bus can link together various circuits, including processors, machine-readable media, and bus interfaces. The bus interface can be used to connect network adapters, etc., to the processing system via the bus. The network adapter can be used to implement signal processing functions. For certain aspects, a user interface (e.g., keyboard, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits well known in the art (such as timing sources, peripherals, voltage regulators, power management circuits, etc.), and therefore will not be described further.

[0068] The processor can be responsible for managing the bus and processing, including the execution of software stored on machine-readable media. Whether referring to software, firmware, middleware, microcode, hardware description languages, or others, software should be interpreted as representing instructions, data, or any combination thereof.

[0069] In hardware implementations, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily understand, machine-readable media or any portion thereof can be external to the processing system. By way of example, machine-readable media may include transmission lines, carrier waves modulated by data, and / or computer products separate from the device, all of which can be accessed by the processor via a bus interface. Alternatively or additionally, machine-readable media or any portion thereof may be integrated into the processor (e.g., in cases where it may have caches and / or dedicated register files). Although the various components discussed may be described as having a specific location (e.g., local components), they can also be configured in various ways (e.g., specific components are configured as part of a distributed computing system).

[0070] Machine-readable media may include multiple software modules. Software modules may include transmission modules and receiving modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, a software module may be loaded from a hard disk drive into RAM when a triggering event occurs. During the execution of a software module, the processor may load certain instructions into a cache to improve access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When the functionality of a software module is referred to below, it will be understood that this functionality is implemented by the processor when executing instructions from that software module. Furthermore, it should be understood that various aspects of this disclosure lead to improvements in the functionality of processors, computers, machines, or other systems implementing these aspects.

[0071] If implemented in software, these functions can be stored on or transmitted thereon as one or more instructions or code. Computer-readable media include both computer storage media and communication media, including any storage medium that facilitates the transfer of a computer program from one place to another.

[0072] Furthermore, it should be understood that modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by the user terminal and / or base station where applicable. For example, such a device may be coupled to a server to facilitate the transmission of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via a storage device, such that the user terminal and / or base station can obtain the various methods when the storage device is coupled to or provided to the device. Furthermore, any other suitable techniques for providing the methods and techniques described herein to the device may be utilized.

[0073] It should be understood that the claims are not limited to the precise configuration and components described above. Various modifications, alterations, and variations may be made to the arrangement, operation, and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. A method for training a deep neural network for a robotic device, comprising: 3D models are built in a training environment using images captured by a 3D camera on a robotic device. Using deep neural networks, 3D image pairs are generated from a 3D model by manually adjusting the parameters of the training environment to form manipulated images; Process 3D image pairs to form a reference image, the reference image including an embedding descriptor of a common object between the 3D image pairs; and The following steps are used to determine relevance to identify objects detected in future images using reference images trained from a neural network: Overlay the corresponding reference image onto the captured image of the scene; and The identity of the detected object is determined based on the point correspondence between the corresponding reference image and the captured image, as well as the embedding descriptor of the corresponding reference image.

2. The method as described in claim 1, wherein, Generating 3D image pairs includes: Pair 3D images with link elements; and Manipulate the linking elements between 3D image pairs to create scenes with different object representations.

3. The method as described in claim 1, wherein, Manual parameter adjustments include: Change the way objects are represented between the original 3D image and the manipulated 3D image.

4. The method of claim 3, wherein, Changing the way objects are represented includes: Change the lighting between the original 3D image and manipulate the 3D image.

5. The method of claim 3, wherein, Changing the way objects are represented includes: Change the perspective between the original 3D image and the manipulated 3D image.

6. The method of claim 1, further comprising: Identify manipulateable objects in unknown environments, regardless of their deformation, representation, angle, and lighting. and Manipulate the identified objects.

7. A method for controlling a robotic device based on the recognition of objects detected in an unknown environment, comprising: Detecting objects in unknown environments; Select the corresponding reference image, including the embedding descriptor corresponding to the trained object manipulated according to the parameters of the image capture environment adjusted manually; and The detected objects are identified based on the embedding descriptor of the corresponding reference image through the following operations: Overlay the corresponding reference image onto the captured image of the scene; and The identity of the detected object is determined based on the point correspondence between the corresponding reference image and the captured image, as well as the embedding descriptor of the corresponding reference image.

8. The method of claim 7, further comprising tracking the identified object over a period of time.

9. The method of claim 7, further comprising: It can be determined that the identified object can be manipulated; and Manipulate the identified objects.

10. A non-transitory computer-readable medium having program code recorded thereon for training a deep neural network for a robotic device, the program code being executed by a processor and comprising: The program code that uses deep neural networks to generate 3D image pairs from a 3D model by manually adjusting the parameters of the training environment to form manipulated images; Program code that processes 3D image pairs to form a reference image, the reference image including an embedding descriptor of a common object between the 3D image pairs; and The following is program code that uses reference images trained from a neural network to determine relevance in order to identify objects detected in future images: The program code that overlays the corresponding reference image onto the captured image of the scene; and The program code that determines the identity of the detected object based on the point correspondence between the corresponding reference image and the captured image, as well as the embedding descriptor of the corresponding reference image.

11. The non-transitory computer-readable medium of claim 10, wherein the program code for generating 3D image pairs comprises: Program code that pairs 3D images with linked elements; and Program code that manipulates linking elements between 3D image pairs to create scenes with different object representations.

12. The non-transitory computer-readable medium of claim 10, wherein, The program code used to generate 3D image pairs includes: Program code that alters the way objects are represented between the original 3D image and the manipulated 3D image.

13. The non-transitory computer-readable medium of claim 12, wherein, Program code used to change the way objects are represented includes: Program code that changes the lighting between the original 3D image and the manipulated 3D image.

14. The non-transitory computer-readable medium of claim 12, wherein, Program code used to change the way objects are represented includes: Program code that changes the perspective between the original 3D image and the manipulated 3D image.

15. A system for controlling a robotic device based on the recognition of objects detected in an unknown environment, the system comprising: The pre-trained object recognition module is configured to select a corresponding reference image to recognize the detected object in the captured image. The corresponding reference image includes an embedding descriptor of the trained object manipulated based on manually adjusted parameters according to the image capture environment. The pre-trained object recognition module is configured to overlay the corresponding reference image onto the captured image of the scene and determine the identity of the detected object based on the point correspondence between the corresponding reference image and the captured image and the embedding descriptor of the corresponding reference image. and The controller is configured to select autonomous behavior for the robotic device based on the identity of the detected object.

16. The system of claim 15, wherein, The pre-trained object recognition module is configured to track the recognized objects over a period of time.

17. The system of claim 15, wherein, The controller is also configured to manipulate the identified object.

18. The system of claim 15, wherein, The pre-trained object recognition module is configured to detect common objects between corresponding reference images and captured images based on correlation, so as to identify the detected objects in future images.

Images