Object recognition using depth and multispectral cameras

Abstract

A camera is configured to output a test depth + multi-spectral image comprising a plurality of pixels. Each pixel corresponds to one of a plurality of sensors of a sensor array of the camera and comprises at least a depth value and a spectral value for each spectral photon band of a plurality of spectral illuminators of the camera. An object recognition machine has been previously trained with a set of labeled training depth + multi-spectral images having the same structure as the test depth + multi-spectral image. The object recognition machine is configured to output a confidence value indicating the likelihood that the test depth + multi-spectral image includes a specified object.

Description

Background Art

[0001] Object recognition is a computer vision technique with many potential applications. However, when conditions are not favorable for a particular camera, accurately identifying an object can be extremely difficult or even impossible. Summary of the invention

[0002] This Summary is provided to introduce in simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages mentioned in any part of this disclosure.

[0003] A camera is configured to output a test depth + multi-spectral image comprising a plurality of pixels. Each pixel corresponds to one of a plurality of sensors of a sensor array of the camera and comprises at least a depth value and a spectral value for each spectral photon band of a plurality of spectral illuminators of the camera. An object recognition machine has been previously trained with a set of labeled training depth + multi-spectral images having the same structure as the test depth + multi-spectral image. The object recognition machine is configured to output a confidence value indicating the likelihood that the test depth + multi-spectral image includes a specified object. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figures 1A-1C is an exploded schematic diagram showing aspects of an example camera.

[0006] Figure 2A-2C A differential sensor of a sensor array is schematically shown.

[0007] Figure 3 Schematically illustrated are aspects of an example object recognition machine configured to recognize specified objects in depth + multispectral images using a convolutional neural network.

[0008] Figure 4 Schematically illustrated are aspects of an example object recognition machine configured to recognize specified objects in a depth + multispectral image using multiple channel-specific models.

[0009] Figure 5-7 Different example outputs of an example object recognition machine are schematically shown.

[0010] Figure 8 An example object recognition method is shown.

[0011] Fig. 9 An example computing system is shown.

[0012] Detailed Description

[0013] The present description relates to an object recognition method that uses an environmentally invariant, depth, and multispectral camera for object recognition. The camera is configured to acquire both a depth image and multiple spectral light images in different spectral light sub-bands on the same sensor array. By accurately calculating the backscattering (reflectivity) coefficient of the imaged subject in each sub-band of spectral light, the camera accurately determines the subject's true spectral signature with minimal or no deviation from ambient light. As a result, limitations of conventional object recognition techniques, including time-related factors (e.g., posture) and environmental factors (e.g., ambient color temperature or intensity), can be overcome to improve the accuracy of object recognition.

[0014] Figures 1A-1C Aspects of an example camera 100 are shown. The term "camera" refers herein to any imaging assembly having at least one optical aperture and a sensor array configured to image a scene or subject 102. The camera 100 includes a sensor array 104 of individually addressable sensors 106. In some implementations, the sensors may be complementary metal oxide semiconductor (CMOS) elements, but other suitable architectures are also contemplated. Each sensor responds to light over a wide wavelength band. For example, for silicon-based sensors, the wavelength response may range from 300 to 1100 nm. For simplicity, a sensor array 104 having only twenty-five sensors 106 is schematically illustrated, but there is no theoretical limit to the number of sensors 106.

[0015] In some implementations, the sensors 106 of the sensor array 104 may be differential sensors. Each differential sensor may include different regions that are excited according to two different clock signals. In one example, to measure modulated active illumination, the two clock signals may be substantially complementary (e.g., both clock signals have a 50% duty cycle and their phases differ by 180 degrees). In other examples, the two different clock signals may have different relationships, such as for measuring ambient illumination or non-modulated active illumination. When different regions are activated based on corresponding clock signals, an electric field is generated that attracts and collects photoelectric charges in different regions. These different regions may be separated by p-type doped regions that create a potential barrier that ensures that charges collected in one region are not transferred to an adjacent region, even if one region is at a higher potential.

[0016] Such operation allows the same sensor array to be used to measure active light across a wide spectrum including UV, visible, NIR, and IR. In addition, differential sensors can reduce system noise because only one read operation is required to perform a differential measurement of active spectrum light and ambient light. Specifically, each pixel readout / reset operation may increase system noise each time it occurs. Thus, rather than performing two separate measurements (i.e., active, passive) that require two readouts and then adding (or subtracting) each result to memory, a single differential measurement can be performed within the pixel and the single differential measurement can be read out to memory.

[0017] In other camera implementations that do not include a sensor array of differential sensors, additional clock cycles may be required to perform differential measurements.While differential sensors provide the advantages described herein, it will be appreciated that any suitable type of sensor array, including non-differential sensor arrays, may be implemented.

[0018] Optionally, a microlens array 108 may be disposed directly over the sensor array 104. The microlens array 108 includes a plurality of microlens elements 110. Each microlens element 110 of the microlens array 108 may be registered to a differential sensor 106 of the sensor array 104. When included, the microlens array 108 may provide a larger effective fill factor at each sensor to improve collection efficiency and reduce crosstalk between pixels.

[0019] Optionally, an optical filter 112 may be disposed above the sensor array 104 so as to optically cover the sensor array. When included, the optical filter 112 is electronically switchable between different filter states. In each filter state, the optical filter 112 transmits light in a particular sub-band and blocks light outside the sub-band from reaching the sensor 106 of the sensor array 104. Depending on the implementation, the blocked light may be absorbed, reflected, and / or scattered by the optical filter 112. The optical filter 112 may improve the signal-to-noise ratio of the IR image and the spectral light image collected by the sensor array 104. The optical filter 112 may include two or more filter states. In one filter state, the optical filter 112 may transmit IR light and block light outside the IR band (e.g., visible light). In another filter state, the optical filter 112 may transmit spectral light and block light outside the spectral light sub-band (e.g., IR light). In some implementations, the optical filter 112 may be configured to switch between a plurality of filter states, each corresponding to a different spectral photon band. In each filter state, the optical filter 112 may be configured to transmit light in the spectral photon band and block light outside the spectral photon band (e.g., spectral light in other spectral sub-bands). The optical filter 112 may switch between any suitable number of different filter states to transmit any suitable photon band(s) while blocking light outside the sub-band(s). Example sub-bands corresponding to different filter states of the optical filter include deep blue (460 nm), blue (470 nm), true green (528 nm), yellow (587 nm), amber (617 nm), red (625 nm), super red (645 nm), far red (730 nm), and near IR (810 nm).

[0020] Optical filter 112 may include any suitable type of filter that transmits narrowband light without significantly reducing the strength of the in-band signal received by the sensor array. In one example, the optical filter may include multiple liquid crystal layers.

[0021] IR illuminator 114 is configured to emit active IR light to illuminate subject 102. In one example, IR illuminator 114 includes an IR laser configured to emit IR light. In some implementations, IR illuminator 114 may optionally include a diffuser covering the illumination field of IR illuminator 114.

[0022] In some implementations, one or more spectral illuminators 116 may be optionally included to provide active spectral illumination. When included, multiple spectral illuminators 116 (e.g., 116A, 116B, 116C, 116D, 116E, 116F) may be configured to emit active spectral light to illuminate the body 102 with multiple different spectral photon bands. Each spectral illuminator may be individually controllable—for example, a single spectral illuminator may be activated while other spectral illuminators remain deactivated. Multiple spectral illuminators 116 may take any suitable form. In one example, the spectral illuminator 116 includes a light emitting diode configured to emit spectral light. There is neither a theoretical limit to the number of spectral illuminators that can be used nor a theoretical limit to the spectral photon bands that each spectral illuminator can be configured to emit.

[0023] In one example implementation, in addition to the IR source, the camera may include six spectral illuminators, which are respectively configured to emit deep blue (460nm), blue (470nm), true green (528nm), yellow (587nm), amber (617nm), and red (625nm). In an example implementation, each spectral illuminator may have a full width at half maximum (FWHM) of 20nm and a field of illumination (FOI) of 80 degrees. Although not required, the camera 100 typically includes more than three spectral illuminators. In some implementations, the spectral illuminators may emit light in other sub-bands, such as super red, near IR, or IR.

[0024] In other implementations, the camera may include a broadband illumination source, such as a white light source. A broadband illumination source may be employed in place of multiple spectrum illuminators. In some implementations, the broadband illumination source may be modulated. In other implementations, the broadband illumination source may not be modulated.

[0025] The electronic controller 118 may include a logic machine and an associated storage machine. The storage machine may hold instructions that cause the logic machine to perform any operation, algorithm, calculation, or transformation disclosed herein. In some implementations, the logic machine may take the form of an application specific integrated circuit (ASIC) or a system on a chip (SoC), in which some or all of the instructions are hardware-encoded or firmware-encoded. The electronic controller 118 includes a depth controller machine 120, a spectrum controller machine 122, and an output machine 124. The machines 120, 122, 124 may be implemented as separate physical hardware and / or firmware components, or incorporated into a single hardware and / or firmware component.

[0026] The depth controller machine 120 is configured to activate the IR illuminator 114 and synchronously address the sensors 106 of the sensor array 104 to acquire an IR image. When the optical filter 112 is included, the depth controller machine 120 is further configured to switch the optical filter 112 synchronously with the activation of the IR illuminator 114 and the addressing of the sensor array 104 to block spectral light outside the IR photon band. Figure 1A In the example shown, at time T0, the depth controller machine 120 adjusts the optical filter 112, activates the IR illuminator 114 to illuminate the subject 102 with active IR light 126, and addresses the sensor 106 of the sensor array 104 simultaneously. The IR light 126' is reflected from the subject 102 back to the camera 100, and is transmitted through the optical filter 112 and incident on the sensor 106 of the sensor array 104 for IR image acquisition. It should be noted that the depth controller machine 120 can acquire multiple IR images in the process of acquiring a depth map.

[0027] Figure 1B The camera 100 is shown measuring the spectral photon band emitted by the first spectral illuminator 116A at time T1. The spectral controller machine 122 is configured to sequentially activate the plurality of spectral illuminators 116 one at a time, adjust the optical filter 112, and synchronously address the sensors 106 of the sensor array 104. In some implementations, the spectral controller machine 122 may be configured to modulate the frequency of the spectral illuminator 116 in 100 kilohertz (Khz) to the low megahertz (MHz) domain for each spectral image acquisition in order to differentially measure the active spectral light and the ambient light. Such operation may allow the acquisition of an accurate spectral signature of the subject being imaged.

[0028] exist Figure 1B In the example shown, the spectral controller machine 122 activates the first spectral illuminator 116A to illuminate the subject 102 with spectral light 128 in a sub-band (e.g., blue) of the spectral illuminator 116A, synchronously adjusts the optical filter 112 to block spectral light outside of the spectral light sub-band of the first spectral illuminator 116A, and addresses the sensor 106 of the sensor array 104. The spectral light 128' is reflected from the subject 102 back to the camera 100, is transmitted through the optical filter 112, and is incident on the sensor 106 that is exactly the same as the sensor used to measure the depth from the object 102. Figure 1B In FIG. 1 , the same sensor is used for spectral light image acquisition in the spectral photon band (eg, blue) of spectral illuminator 116A.

[0029] Further, if Figure 1CAs shown, at time T2, the spectral controller machine 122 activates the second spectral illuminator 116B to illuminate the subject 102 with spectral light 130 in a sub-band (e.g., green) of the spectral illuminator 116B, synchronously switches the optical filter 112 to block spectral light outside the spectral light sub-band of the second spectral illuminator 116B, and addresses the identical sensor 106 of the sensor array 104. The spectral light 130' is reflected from the subject 102 back to the camera 100, is transmitted through the switchable filter 112, and is incident on the differential sensor 106 of the sensor array for spectral light image acquisition in the sub-band (e.g., green) of the spectral illuminator 116B. The spectral controller machine 122 can sequentially activate multiple spectral illuminators 116 one at a time in synchronization with adjusting the optical filter 112 and addressing the sensor array 104 to acquire spectral light images in each of the multiple sub-bands of the spectral illuminator 116.

[0030] In some implementations, the sensors 106 of the sensor array 104 may be differential sensors. Figure 2A , 2B 2C are cross-sectional views illustrating aspects of an example differential sensor 106 of a sensor array 104 that may be implemented in an ambient light invariant, depth + multispectral camera (such as the camera 100 shown in FIG. 1 ). The differential sensor 106 includes first and second polysilicon gates (PG A, PG B) 202A, 202B. The polysilicon gate 202A may be excited according to a clock signal Clk_A, and the polysilicon gate 202B may be excited according to a clock signal Clk_B. The two clock signals may be controlled differently to control the sensor array to measure different types of illumination. For example, to measure active modulated illumination, the two clock signals may be substantially complementary (e.g., Clk_A and Clk_B have a 50% duty cycle and are 180 degrees out of phase). In other examples, the two clock signals may have a different relationship, such as for measuring ambient illumination or non-modulated active illumination.

[0031] When activated based on the clock signal, the polysilicon gates 202A, 202B create electric fields that attract and collect photoelectric charges in different corresponding regions 204A, 204B of the differential sensor 106 corresponding to different polysilicon gates based on the arrival time under the polysilicon gate oxide in the regions 204A, 204B. Specifically, the collected photoelectric charges remain under the gate where the photoelectric charges are initially collected, and eventually slowly diffuse to the collection node by self-repulsion under each polysilicon gate. The p-type doped region 206 between the different regions creates an electric potential barrier that ensures that the charge collected by one polysilicon gate will not be transferred to an adjacent polysilicon gate, even if one polysilicon gate is at a higher potential.

[0032] exist Figure 2A In , both clock signals Clk_A and Clk_B are biased to low voltages, so that the polysilicon gates 202A, 202B collect little or no photoelectric charge in regions 204A, 204B. Figure 2B In FIG. 2 , clock signal Clk_A is biased to a high voltage, while Clk_B is biased to a low voltage. When Clk_A goes high, polysilicon gate 202A becomes energized, and photoelectric charge is collected in region 204A, as indicated by the electric field lines. The electric field lines are tangential to the boundary of region 204A, so charge is not transferred from region 204A to region 204B. Additionally, since Clk_B is biased low, polysilicon gate 202B is not energized, and thus minimal photoelectric charge is collected in region 204B. ... Figure 2C , clock signal Clk_B is biased to a high voltage, while Clk_A is biased to a low voltage. When Clk_B goes high, polysilicon gate 202B becomes energized, and photoelectric charge is collected in region 204B, as indicated by the electric field lines. The electric field lines are tangential to the boundary of region 204B, so charge is not transferred from region 204B to region 204A. Additionally, since Clk_A is biased low, polysilicon gate 202A is not energized, and thus minimal photoelectric charge is collected in region 204A.

[0033] The differential sensor 106 is configured to collect and assign photoelectric charge synchronously with the modulation of the clock signals Clk_A and Clk_B. The photoelectric charge assignment (classified as PGA or B) in the differential sensor 106 occurs simultaneously with the charge collection under the polysilicon gates 202A, 202B, and therefore does not require an additional step of shifting the charge from one gate to the other. Thus, differential measurements can be performed within a single integration / readout cycle. In one example, when the polysilicon gate 202A is excited to collect photoelectric charge from active illumination in region 204A, differential spectral measurements can be performed by synchronously activating a designated spectrum illuminator for 50% of the time period. Additionally, in the case where the designated spectrum illuminator is deactivated to collect photoelectric charge from ambient light in region 204B, ambient light measurements can be performed by stimulating the polysilicon gate 202B. The photoelectric charge collected by region 204B (i.e., the amount of ambient light) is subtracted from the photoelectric charge collected by region 204A (i.e., the amount of active light and ambient light) to determine a differential measurement in which the ambient light bias is significantly reduced or removed from the measurement of active illumination. It is noted that this example is applicable to scenarios in which modulated active illumination and ambient illumination are measured differentially.

[0034] Such operation allows the same sensor array to be used to measure active light across a wide spectrum including UV, visible, NIR, and IR. In addition, differential sensors can reduce system noise because only one read operation is required to perform a differential measurement of active spectrum light and ambient light. Specifically, each pixel readout / reset operation may increase system noise each time it occurs. Thus, rather than performing two separate measurements (i.e., active, passive) that require two readouts and then adding (or subtracting) each result to memory, a single differential measurement can be performed within the pixel and the single differential measurement can be read out to memory.

[0035] In other camera implementations that do not include a sensor array of differential sensors, additional clock cycles may be required to perform differential measurements.While differential sensors provide the advantages described herein, it will be appreciated that any suitable type of sensor array, including non-differential sensor arrays, may be implemented.

[0036] The term "addressing" as applied to sensors 106 of sensor array 104 may have slightly different meanings depending on the imaging mode being described. For planar imaging - for spectral light including both visible and IR light - addressing sensors 106 may include integrating the intensity of active light in a sub-band emitted from a specified spectral (or IR) illuminator and received at each sensor 106 and associating the integrated intensity of the active light in the sub-band with the portion of the image corresponding to that sensor.

[0037] For depth imaging, the sensors 106 may be addressed differently. Here, addressing the sensors may include resolving a phase offset from each sensor relative to the modulated intensity of the IR light. The phase offset (optionally converted into the depth domain) may be associated with the portion of the image corresponding to the addressed sensor. In other words, the depth controller machine 120 may be configured to determine a depth value for each sensor 106 of the sensor array 104. In some implementations, a series of IR image acquisitions (e.g., 6-9) in rapid succession may be used to obtain the phase offset. In combined applications of depth imaging and planar imaging, the above two addressing modes may be used in a multiplexed manner.

[0038] The phase-resolved time-of-flight (ToF) approach described above is one of several depth imaging techniques covered by the present disclosure. In general, a depth imaging camera can be configured to acquire one or more depth maps of a scene or subject. The term "depth map" refers to a region (X) that is registered to the imaged scene. i ,Y i ) pixel array, where the depth value (Z i) indicates the depth of the corresponding area for each pixel. "Depth" is defined as a coordinate parallel to the optical axis of the camera that increases with increasing distance from the camera. The term "depth video" refers to a time-resolved sequence of depth maps in this article. In a ToF implementation, an IR illumination source can project pulsed IR illumination or otherwise modulated IR illumination toward a subject. The sensor array of a depth imaging camera can be configured to detect a phase offset between the illumination reflected back from the subject and the modulated emission. In some implementations, the phase offset of each sensor can be converted into a pixel-resolved flight time of the pulsed illumination, from the illumination source to the subject and then back to the array. The ToF data can then be converted into depth data.

[0039] The term "spectral light image" refers to a region (X) that is registered to the imaged scene. i ,Y i ) pixel matrix, where the spectral value (SV i ) indicates, for each pixel, a spectral signature of a corresponding region in a particular spectral light sub-band. To acquire a spectral light image in each sub-band (e.g., for a multi-spectral image), the spectral controller machine 122 is configured to determine a spectral value of each of the differential sensors based on the depth value and a differential measurement of the active spectral light and the ambient light of the differential sensor.

[0040] The depth and multispectral data collected by the camera 100 can be used to fit the neighborhood of the pixel to a regular surface (e.g., a Lambertian plane, a quadratic surface) and solve the backscatter (reflectivity) coefficient for each sub-band in order to calculate a spectral signature of the surface that is invariant to ambient light and can be robustly classified. In such calculations, the depth data can be used to account for the reduction in light intensity due to the optical transmission of light from the light source (IR illuminator 114 or spectral illuminator 116) to the subject 102. In some implementations, the spectral controller machine 122 can be configured to calculate a surface normal for each of the multiple pixels of the sensor array based on the depth data. The surface normal set of the multiple pixels can represent the surface of the imaging subject modeled by an analytical function (e.g., planar or quadratic). In some implementations, the spectral controller machine 122 can be configured to calculate a curvature for each of the multiple pixels of the sensor array based on the depth data. The curvature set of the multiple pixels can represent the surface of the imaging subject modeled by an analytical function (e.g., planar or quadratic). The spectral controller machine 122 may be configured to normalize spectral light measurements in different spectral photon bands based on one or more of the measured depth and surface normal. This provides a position-invariant and light-invariant spectral signature of the imaged subject.

[0041] The output engine 124 is configured to output a pixel matrix 132 referred to herein as a test depth + multi-spectral image. Each pixel in the matrix includes a depth value (Z) for each spectral photon band of the plurality of spectral illuminators 116 of the camera 100. i ) and spectral values ​​(e.g., SV1 i ,SV2 i ,…SV6 i ). In some implementations, the output engine 124 may be further configured to output a surface normal and / or curvature for each pixel. The output engine 124 may be configured to output the pixel matrix 132 in any suitable form. In some examples, the output engine 124 may output the pixel matrix 132 as a data structure, wherein each element of the matrix corresponds to a different pixel, and each element of the matrix includes an array of values ​​corresponding to a depth value and a spectral value for the pixel. Note that the spectral value of the IR photon sub-band may be determined for each differential sensor in the process of determining the depth value, and thus each pixel in the matrix may include a spectral value for the IR photon sub-band. Further, it should be noted that the output engine 124 may be configured to output the pixel matrix 132 (and / or any other suitable parameter values) to any suitable recipient inside or outside the camera 100. In one example, the output engine 124 may be configured to output the pixel matrix 132 to another processing component for additional image processing (e.g., filtering, computer vision). In some examples, the processing component may be incorporated into the camera 100. In some examples, the processing component may be incorporated into a remote computing device that communicates with the camera 100. In another example, output engine 124 may be configured to output pixel matrix 132 to an external display device for visual presentation as an image.

[0042] Compared to separate depth and / or multispectral images generated by existing approaches, the test depth + multispectral image 132 generated by the camera 100 can provide richer and more accurate information. For example, the test depth + multispectral image 132 provides a spectral signature that is invariant to time-related factors and environmental factors. Such a depth + multispectral image can be utilized to perform image analysis for visual object recognition. Specifically, the rich data of the depth + multispectral image can be used to identify different types of object features or landmarks, which can be used in combination to provide a more accurate assessment of the possibility that the depth + multispectral image includes a specified object.

[0043] like Figure 3As shown, the camera 100 can be configured to output the test depth + multi-spectral image 132 to the object recognition engine 300, which is configured to output a confidence value 302 indicating the possibility that the test depth + multi-spectral image 132 includes a specified object. The object recognition engine 300 can be configured to recognize any suitable type of specified object in the test depth + multi-spectral image. In some implementations, the object recognition engine 300 can be configured to recognize multiple different types of objects in the test depth + multi-spectral image. For example, the object recognition engine 300 can be configured to recognize people, animals, plants, buildings, vehicles, locations, or any other suitable specified objects or entities. Moreover, the object recognition engine 300 can be configured to recognize different parts or features of the specified object. For example, the object recognition engine 300 can be configured to recognize different human body parts, such as the head, face, arms or legs. The object recognition engine 300 can be configured to recognize any suitable part or feature of the specified object.

[0044] The object recognition machine 300 may employ any suitable combination of existing and / or future machine learning (ML) and / or artificial intelligence (AI) techniques. Non-limiting examples of techniques that may be incorporated into the implementation of the object recognition machine 300 include: support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos), recursive neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering) and / or graphical models (e.g., Markov models, conditional random fields, and / or AI knowledge bases).

[0045] In some examples, methods and processes utilized by object recognition engine 300 may be implemented using one or more differentiable functions, wherein the gradient of the differentiable function may be calculated and / or estimated with respect to the input and / or output of the differentiable function (e.g., with respect to training data, and / or with respect to the target function). Such methods and processes may be determined, at least in part, by a set of trainable parameters. Accordingly, the trainable parameters may be adjusted by any suitable training procedure to continually improve the operation of object recognition engine 300.

[0046] Non-limiting examples of training procedures for the object recognition engine 300 include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods. In some examples, multiple components of the object recognition engine 300 may be trained simultaneously with respect to an objective function that measures the performance of the collective operation of the multiple components (e.g., with respect to reinforcement feedback and / or with respect to labeled training data) in order to improve such collective operation. In some examples, one or more components of the object recognition engine 300 may be trained independently of other components (e.g., offline training on historical data). For example, the object recognition engine 300 may be trained on labeled training data via supervised training, the labeled training data comprising a set of depth+multispectral images having the same structure as the test depth+multispectral image 132. In other words, the training data comprises images of the same type as the images generated by the camera 100. The training data set may be labeled with labels indicating whether the specified object exists in the corresponding depth + multispectral image. In some implementations, the label may indicate the location of any specified object present in such an image. In some implementations, each pixel in each depth + multispectral image of the training data set may be labeled with a different specified object or descriptive category. Labeling the training data set in this way enables the object recognition machine 300 to be trained to perform semantic segmentation of the test depth + multispectral image. In other words, the object recognition machine 300 may be configured to output multiple confidence values ​​corresponding to multiple different types of objects for each pixel of the test depth + multispectral image. Each confidence value may indicate the possibility that the pixel is included in the object of that type. The object recognition machine 300 may be trained on the training data set via supervised training with respect to an objective function that measures the accuracy, precision and / or recall of locating the specified object by the object recognition machine 300 compared to the actual location of the specified object(s) indicated in the labeled training data.

[0047] In the illustrated example, the object recognition engine 300 uses a convolutional neural network 304 to determine the confidence value 302. The convolutional neural network 304 includes a plurality of layers, including an input layer 306, one or more hidden layers 308, and an output layer 310. The input layer 306 includes a plurality of input nodes 312. Each input node 312 is configured to receive a different pixel (P) from the plurality of pixels of the test depth + multispectral image 132. i ) corresponding to the pixel value array 314. The pixel value array 314 may include a plurality of pixel parameter values, the plurality of pixel parameter values ​​including the depth value (Zi ) and multiple multispectral values ​​(SV1 i ,…SV6 i ). In some implementations, the pixel value array 314 may further include a surface normal calculated for the pixel. In some implementations, the pixel value array 314 may further include a curvature calculated for the pixel.

[0048] The convolutional neural network 304 is configured to convolve the input vector with one or more predefined, randomized, and / or learned convolution kernels included in the hidden layer(s) 308. By convolving the convolution kernel with the input vector (e.g., representing the depth+multispectral image 132), the convolutional neural network 304 can detect features associated with the convolution kernel. For example, the convolution kernel can be convolved with the test depth+multispectral image to detect low-level visual features such as lines, edges, corners, etc. based on various convolution operations using multiple different convolution kernels. The convolved outputs of the various convolution operations can be processed by a pooling layer (e.g., max pooling), which can detect one or more most salient features of the input image and / or aggregate the salient features of the input image so as to detect salient features of the input image at specific locations in the input image. The pooled outputs of the pooling layer can be further processed by other convolution layers.

[0049] The convolution kernels of other convolution layers can recognize higher-level two-dimensional (2D) visual features (e.g., shapes and patterns), and more generally can recognize the spatial arrangement of lower-level visual features. Some layers of the convolutional neural network can correspondingly recognize and / or locate the 2D visual features of the specified object that the object recognition machine 300 is trained to recognize. For example, if the object recognition machine 300 is trained to recognize a bicycle, the convolution kernels of some convolutional layers can recognize and / or locate the 2D visual features of the bicycle (e.g., wheels, seats, front forks). The convolution kernels of some other convolutional layers can recognize three-dimensional (3D) visual features, such as curvature maps. Returning to the above example, some layers of the convolutional neural network can correspondingly recognize and / or locate the 3D visual features of the bicycle (e.g., pedals, handlebars). Some layers of the convolutional neural network can correspondingly recognize and / or locate different textures. The convolution kernels of some other convolutional layers can recognize spectral features within and outside the visible spectrum (e.g., ultraviolet spectrum, infrared spectrum). Some layers of the convolutional neural network can accordingly identify and / or locate the spectral features (e.g., paint) of a given object. The spectral features can be invariant to temporal factors (e.g., posture) and environmental factors (e.g., ambient lighting). In addition, some layers of the convolutional neural network can identify combined spectral-spatial features based on 2D, 3D, and spectral information.

[0050] The pooled outputs of the hidden layer(s) 308 may be output to an output layer 310. The output layer 310 may be configured to output a confidence value 302 indicating a likelihood that the test depth+multispectral image includes a specified object based on the processing performed by the different hidden layers 308. Accordingly, the convolutional neural network 304 may identify a specified object in a test image input to the convolutional neural network.

[0051] Although the foregoing examples are described with respect to convolutional neural networks, other neural network techniques may be capable of detecting and / or locating specified objects and other salient features based on detecting low-level 2D visual features, 2D higher-level visual features, 3D visual features, spectral features, and / or spatial arrangements of visual features.

[0052] In some implementations, the object recognition engine 300 may optionally be configured to process different pixel parameter values ​​of the test depth + multispectral image separately on a channel-by-channel basis. Figure 4 As shown, the object recognition engine 300 can use multiple models including channel-specific models 404 to determine a confidence value 302 indicating the likelihood that the test depth + multi-spectral image 132 includes a specified object. For example, the object recognition engine 300 can be trained on labeled training data via supervised training, and the labeled training data includes a collection of depth + multi-spectral images having the same structure as the test depth + multi-spectral image 132. The training data set can be labeled with labels indicating whether the specified object is present in such an image. In some implementations, the labels can indicate the location of any specified object present within such an image.

[0053] Multiple channel-specific models (e.g., a depth channel model, an SV1 channel model, ..., an SV6 channel model) 404 may each be configured to process different pixel parameters for multiple pixels of the test depth + multi-spectral image 132. For example, the depth value of the test depth + multi-spectral image 132 may be processed by the depth channel model. As another example, the spectral value of the spectral light in the first spectral photon band corresponding to the first spectral illuminator 116A of the camera 100 may be processed by the SV1 channel model. Each channel-specific model 404 includes multiple input nodes, and for each channel-specific model, each input node is configured to receive a pixel parameter value of a different pixel in the multiple pixels of the test depth + multi-spectral image 132. For example, each input node of the depth channel model may be configured to receive a depth value [Z] of a different pixel in the multiple pixels of the test depth + multi-spectral image 132. i]. In other words, the depth channel model may receive as input a 2D input vector of depth values ​​corresponding to a pixel matrix. As another example, each input node of the SV1 channel model may be configured to receive a spectral value of a different pixel in the plurality of pixels of the test depth+multispectral image 132 [SV1 i ]. In other words, the SV1 channel model may receive as input a 2D input vector of spectral values ​​in the first spectral photon band corresponding to the pixel matrix.

[0054] Multiple channel-specific models 404 may process different types of pixel parameter values ​​in a similar manner as described above. For example, 2D visual features and spectral features may be identified by different channel-specific models that process spectral values ​​(e.g., SV1-SV6), and 3D visual features and textures may be identified by channel-specific models that process depth values. It will be appreciated that the object recognition engine may determine confidence values ​​302 using any suitable type of models arranged in any suitable manner.

[0055] In some implementations, the object recognition machine 300 may optionally be configured to determine the confidence value 302 using a statistical model. For example, the statistical model may include a support vector machine that may be configured to analyze the test depth + multi-spectral image by applying a series of different binary classifiers to determine whether the test depth + multi-spectral image includes a specified object. The support vector machine may be trained using a set of training depth + multi-spectral images, each of which is labeled as either including a specified object or not including a specified object. In another example, the object recognition machine 300 may optionally be configured to determine the confidence value 302 using a statistical model including a nearest neighbor algorithm.

[0056] In some implementations, the object recognition engine 300 may optionally be configured to recognize multiple different types of specified objects in the test depth + multi-spectral image. In such implementations, the object recognition engine 300 may be trained on a training data set including depth + multi-spectral images, which are labeled with labels indicating whether multiple different types of specified objects are present in the corresponding depth + multi-spectral images. In some implementations, the object recognition engine 300 may include two or more different sub-machines, each sub-machine being trained to recognize different types of objects. In such implementations, the test depth + multi-spectral image may be analyzed by different sub-machines (in series or in parallel) to recognize different types of objects within the test depth + multi-spectral image.

[0057] exist Figure 5In the example shown, the object recognition machine 300 receives the test depth + multi-spectral image 500 as input. The object recognition machine 300 can analyze the test depth + multi-spectral image 500 to determine whether the test depth + multi-spectral image includes different types of specified objects that the object recognition machine is trained to recognize. The object recognition machine 300 outputs a plurality of confidence values ​​302 (e.g., 302A, 302B, 302C, 302D, 30E). Each confidence value 302 corresponds to a different type of specified object that the object recognition machine 300 is trained to recognize. Each confidence value 302 indicates the possibility that the test depth + multi-spectral image 500 includes the specified object of that type. Specifically, in the illustrated example, the object recognition engine 300 outputs a first confidence value 302A indicating that the test depth + multi-spectral image 500 has a 95% probability of including a bicycle, a second confidence value 302B indicating that the test depth + multi-spectral image 500 has a 85% probability of including a tree, a third confidence value 302C indicating that the test depth + multi-spectral image 500 has a 90% probability of including a person, a fourth confidence value 302D indicating that the test depth + multi-spectral image 500 has a 5% probability of including a car, and a fifth confidence value 302E indicating that the test depth + multi-spectral image 500 has a 5% probability of including a building. It should be noted that these different types of objects are provided as examples, and the object recognition engine 300 can be configured to recognize any suitable number and / or type of different specified objects.

[0058] In some examples, the object recognition engine 300 may optionally be configured to output a label indicating that the test depth + multi-spectral image 500 includes the object based on the confidence value of various specified objects being greater than a threshold value (e.g., 75%). Such labels may be used for image classification of different test depth + multi-spectral images analyzed by the object recognition engine 300.

[0059] In some implementations, the object recognition engine 300 may optionally be configured to output the location of each identified object on the test depth + multispectral image. In other words, the object recognition engine 300 may optionally be configured to perform object detection in the test depth + multispectral image. In one example, Figure 6As shown, the object recognition machine 300 receives a test depth + multi-spectral image 500 as input. The object recognition machine 300 can analyze the test depth + multi-spectral image 500 to determine whether the test depth + multi-spectral image includes different types of specified objects that the object recognition machine is trained to recognize. The object recognition machine 300 outputs multiple locations on the test depth + multi-spectral image 500 for each recognized object. For example, the object recognition machine 300 may output multiple bounding boxes 600 (e.g., 600A, 600B, 600C) surrounding multiple recognized objects. Specifically, the object recognition machine 300 outputs a first bounding box 600A surrounding the recognized tree, a second bounding box 600B surrounding the recognized person, and a third bounding box 600C surrounding the recognized bicycle. The sizes of the different bounding boxes can be determined in any suitable manner. In some examples, the object recognition machine 300 may optionally output labels indicating the objects recognized in the bounding boxes 600, such as 'tree', 'person', and 'bicycle'. Although Figure 6 Bounding boxes drawn on the image are shown, but this is not required. Object recognition engine 300 may optionally output the coordinates of the bounding box (e.g., the coordinates of the upper left corner and the lower right corner). In some examples, object recognition engine 300 may optionally output the size of each recognized object. In some examples, object recognition engine 300 may optionally output other forms of location information indicating the location of the recognized objects.

[0060] In some implementations, the object recognition engine 300 may optionally be configured to output, for each pixel of the test depth + multispectral image, a plurality of confidence values ​​corresponding to a plurality of different types of objects that the object recognition engine 300 is trained to recognize. Each confidence value may indicate the likelihood that the pixel is included in a recognized object of that type. In other words, the object recognition engine 300 may optionally be configured to perform semantic segmentation of the test depth + multispectral image. Figure 7In the example shown, the object recognition machine 300 receives the test depth + multi-spectral image 500 as input. The object recognition machine 300 can analyze the test depth + multi-spectral image 500 to determine whether the test depth + multi-spectral image includes different types of specified objects that the object recognition machine is trained to recognize. The object recognition machine 300 outputs a plurality of confidence values ​​702 for each pixel 700 of the test depth + multi-spectral image 500. In the illustrated example, four representative pixels 702A, 702B, 702C, 702D of a representative area 704 from the test depth + multi-spectral image 500 are enlarged for ease of viewing. Pixels 702A, 702B, 702C, 702D can represent any pixel of the test depth + multi-spectral image 500. Specifically, the first plurality of confidence values ​​702A for the first pixel 700A indicates that there is a high probability that the first pixel is included in the tree, the second plurality of confidence values ​​702B for the second pixel 700B indicates that there is a high probability that the second pixel is included in the bicycle, the third plurality of confidence values ​​702C for the third pixel 700C indicates that there is a high probability that the third pixel is included in the tree, and the fourth plurality of confidence values ​​702D for the fourth pixel 700D indicates that there is a high probability that the fourth pixel is included in the bicycle. It should be noted that any one of these confidence values ​​702 for each pixel may correspond to Figure 3 's confidence value 302. In other words, in some implementations, the object recognition engine 300 can be configured to output a confidence value for each pixel of the test depth + multi-spectral image 500 without having to output a separate confidence value for the entire test depth + multi-spectral image. In some examples, the object recognition engine 300 can optionally be configured to output a label indicating that the pixel is included in the object based on the confidence value of the identified object being greater than a threshold value (e.g., 75%) for each pixel. The object recognition engine 300 can be configured to output confidence values ​​pixel by pixel for any suitable number and / or type of different objects.

[0061] In some implementations, the object recognition engine 300 may optionally be configured to output information related to the test depth + multispectral image in which the specified object is identified, in addition to the confidence value 302. In some implementations, the object recognition engine 300 may optionally be configured to output the location of one or more identified 2D features of the identified object on the test depth + multispectral image. For example, (all) 2D features may include low-level features (e.g., corners, fragments) and / or high-level features (e.g., wheels, front forks, seats of bicycles) determined based on spectral image data. The object recognition engine 300 may optionally be configured to output the location of one or more identified 3D features of the identified object on the test depth + multispectral image. (all) 3D features may include low-level features (e.g., curves) and / or high-level features (e.g., pedals, handlebars of bicycles) determined based on depth data. The object recognition engine 300 may optionally be configured to output the location of one or more identified spectral features of the identified object on the test depth + multispectral image. The spectral feature(s) may be identified based on spectral signatures determined based on a plurality of spectral light values ​​in a plurality of spectral photon bands.

[0062] The object recognition engine 300 can be configured to include image labels indicating recognized objects included in the test depth + multispectral image, locations of recognized objects (e.g., bounding boxes), pixel-specific labels, and / or any other information and / or another suitable data structure generated as a result of analyzing the test depth + multispectral image in the pixel matrix 132 output by the output engine 124 of FIG. 1.

[0063] The object recognition engine 300 may be implemented as any suitable physical hardware and / or firmware component. In some implementations, the camera 100 and the object recognition engine 300 may be combined in a shared housing. In some implementations, the controller 118 of the camera 100 and various aspects of the object recognition engine 300 may be integrated together as one or more hardware logic components. In some implementations, various aspects of the object recognition engine 300 may be implemented on a computing device located remote from the camera 100. For example, various aspects of the object recognition engine 300 may be implemented via a service computing system that communicates with the camera 100 over a network.

[0064] Note that the object recognition machine 300 can be configured to output the object recognition data to any suitable recipient inside or outside the object recognition machine 300. For example, the object recognition machine 300 can be configured to output the object recognition data to another processing component for additional image processing. In some examples, the processing component can be incorporated into the object recognition machine 300. In some examples, the processing component can be incorporated into the camera 100. In some examples, the processing component can be incorporated into a remote computing device that communicates with the object recognition machine 300 and / or the camera 100. In another example, the object recognition machine 300 can be configured to output the object recognition data to an external display device for visual presentation of the object recognition data.

[0065] Figure 8 An example object recognition method 800 using a depth + multispectral camera (such as camera 100 of FIG. 1 ) is shown. For example, method 800 may be performed by Figure 3-7 The object recognition machine 300 is used to perform.

[0066] At 802 of method 800, an object is actively illuminated with active IR light. At 804 of method 800, a depth profile of the object is measured using multiple sensors of a camera while the object is actively illuminated with active IR light. At 806, the object is actively illuminated with active spectral light in each of a plurality of different spectral photon bands. In other words, the object may be illuminated with active spectral light in one of a plurality of different spectral photon bands at a time until the object has been successively illuminated with active spectral light in each of a plurality of different spectral photon bands. At 808 of method 800, a spectral signature of the object in each of a plurality of different spectral photon bands is measured using multiple sensors of a camera while the object is actively illuminated with active spectral light in each of a plurality of different spectral photon bands. In other words, the same sensor of the same camera is used to measure the depth profile of the object and the spectral signature of the object in a plurality of different spectral photon bands. At 810 of method 800, computer recognition of the object is performed based on the measured depth profile and the measured spectral signature. Computer recognition of objects may be performed in any suitable manner. For example, computer recognition of objects may be performed using one or more previously trained artificial neural networks or statistical models as discussed above.

[0067] The techniques discussed herein may be broadly applicable to object recognition to provide performance benefits of improved recognition accuracy by at least reducing or removing ambient light interference from captured images used to train and test such machine learning object recognition machines.

[0068] In some implementations, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0069] Fig. 9 A non-limiting implementation of a computing system 900 that can perform one or more of the above methods and processes is schematically shown. The computing system 900 is shown in a simplified form. For example, the computing system 900 can be implemented using the camera 100 or the electronic controller 118 of FIG. 1 and Figure 3-7 The object recognition machine 300 is in the form of.

[0070] The computing system 900 includes a logic machine 902 and a storage machine 904. The computing system 900 may optionally include a display subsystem 906, an input subsystem 908, a communication subsystem 910, and / or Fig. 9 Other components not shown.

[0071] The logic machine 902 includes one or more physical devices configured to execute instructions. For example, the logic machine 902 may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise obtain desired results.

[0072] The logic machine 902 may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic machine 902 may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processor of the logic machine 902 may be single-core or multi-core, and the instructions executed thereon may be configured for serial, parallel and / or distributed processing. The individual components of the logic machine may optionally be distributed among two or more separate devices, which may be located remotely and / or configured for collaborative processing. Various aspects of the logic machine 902 may be virtualized and executed by a remotely accessible, networked computing device configured in a cloud computing configuration.

[0073] Storage machine 904 includes one or more physical devices configured to hold instructions executable by logic machine 902 to implement the methods and processes described herein. When implementing these methods and processes, the state of storage machine 904 may be transformed—for example, to hold different data.

[0074] The storage 904 may include semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), etc. The storage 904 may include volatile, non-volatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices.

[0075] It will be appreciated that storage 904 comprises one or more physical devices. However, aspects of the instructions described herein may alternatively be propagated over a communication medium (eg, electromagnetic signals, optical signals, etc.) that is not retained by a physical device for a finite duration.

[0076] Aspects of the logic machine 902 and the storage machine 904 may be integrated together into one or more hardware logic components. Such hardware logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0077] When display subsystem 906 is included, display subsystem 906 can be used to present a visual representation of the data maintained by storage machine 904. This visual representation can take the form of a display image, converting pixel matrix 132 into a visual format perceptible to humans. Since the methods and processes described herein change the data maintained by the storage machine, and thus transform the state of the storage machine, the state of display subsystem 906 can also be transformed to visually represent the changes in the underlying data. Display subsystem 906 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined in a shared package with logic machine 902 and / or storage machine 904, or such display devices may be peripheral display devices.

[0078] When an input subsystem 908 is included, the input subsystem 908 may include or interface with one or more user input devices such as a keyboard, mouse, touch screen, or game controller. In some embodiments, the input subsystem may include selected natural user input (NUI) components or interface with the above-mentioned natural user input (NUI) components. Such components can be integrated or peripheral, and the transduction and / or processing of input actions can be handled on or off the board. Example NUI components may include a microphone for voice and / or speech recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for evaluating brain activity.

[0079] When the communication subsystem 910 is included, the communication subsystem 910 can be configured to communicatively couple the computing system 900 with one or more other computing devices. The communication subsystem 910 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem 910 can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem 910 can allow the computing system 900 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0080] In one example, a camera includes: a sensor array including a plurality of sensors; an IR illuminator configured to emit active IR light in an infrared (IR) photon band; a plurality of spectral illuminators, each spectral illuminator being configured to emit active spectral light in a different spectral photon band; a depth controller machine configured to determine a depth value for each of the plurality of sensors based on the active IR light; a spectral controller machine configured to determine a spectral value for each spectral photon band of the plurality of spectral illuminators for each of the plurality of sensors; and an output machine configured to output a test depth + multi-spectral image including a plurality of pixels, each pixel corresponding to one of the plurality of sensors of the sensor array and including at least a depth value and a spectral value for each spectral photon band of the plurality of spectral illuminators; an object recognition machine previously trained with a set of labeled training depth + multi-spectral images having the same structure as the test depth + multi-spectral image, the object recognition machine being configured to output a confidence value indicating the likelihood that the test depth + multi-spectral image includes a specified object. In this example and / or other examples, the object recognition machine may also be configured to output a confidence value corresponding to each of a plurality of different types of objects, each confidence value indicating the possibility that the test depth + multi-spectral image includes an object of that type. In this example and / or other examples, the object recognition machine may also be configured to output the position of each identified object on the test depth + multi-spectral image. In this example and / or other examples, the object recognition machine may also be configured to output a plurality of confidence values ​​corresponding to a plurality of different types of objects for each pixel of the test depth + multi-spectral image, each confidence value indicating the possibility that the pixel is included in the object of that type. In this example and / or other examples, each spectral value may be determined based on a depth value determined by a sensor corresponding to the pixel. In this example and / or other examples, the object recognition machine may be configured to determine the confidence value using a convolutional neural network. In this example and / or other examples, the object recognition machine may include a plurality of input nodes, each input node may be configured to receive a pixel value array corresponding to a different pixel in a plurality of pixels of the test depth + multi-spectral image, and the pixel value array may include a depth value and a plurality of multi-spectral values ​​of the pixel. In this example and / or other examples, the plurality of multispectral values ​​of a pixel may include more than three spectral values. In this example and / or other examples, the output engine may be configured to output a surface normal for each pixel of the test depth+multispectral image, and the pixel value array may include the surface normal. In this example and / or other examples, the output engine may be configured to output a curvature for each pixel of the test depth+multispectral image, and the pixel value array may include the curvature.In this example and / or other examples, the object recognition machine may be configured to determine the confidence value using multiple models, the multiple models may include multiple channel-specific models, each channel-specific model may be configured to process different pixel parameters for multiple pixels of the test depth + multispectral image, each channel-specific model may include multiple input nodes, and for each channel-specific model, each input node may be configured to receive a pixel parameter value of a different pixel in the multiple pixels of the test depth + multispectral image. In this example and / or other examples, the object recognition machine may be configured to determine the confidence value using a statistical model. In this example and / or other examples, the statistical model may include a nearest neighbor algorithm. In this example and / or other examples, the statistical model may include a support vector machine. In this example and / or other examples, the object recognition machine is also configured to output the position of the bounding box surrounding the specified object on the test depth + multispectral image. In this example and / or other examples, the object recognition machine may also be configured to output a confidence value indicating the possibility that the pixel is included in the specified object for each pixel of the test depth + multispectral image. In this example and / or other examples, the object recognition machine may also be configured to output one or more of the following: the position of the identified two-dimensional (2D) feature of the specified object on the test depth + multi-spectral image, the position of the identified three-dimensional (3D) feature of the specified object on the test depth + multi-spectral image, and the position of the identified spectral feature of the specified object on the test depth + multi-spectral image. In this example and / or other examples, multiple sensors of the sensor array may be differential sensors, and each spectral value may be determined based on the depth value and differential measurement of the differential sensor.

[0081] In one example, a method includes actively illuminating an object with active IR light; measuring a depth profile of the object using multiple sensors of a camera while the object is actively illuminated with the active IR light; actively illuminating the object with active spectral light in each of a plurality of different spectral photon bands; measuring a spectral signature of the object in the spectral photon bands using multiple sensors of a camera used to measure the depth profile of the object while the object is actively illuminated with active spectral light in each of a plurality of different spectral photon bands; and identifying the object by a computer based on the measured depth profile and the measured spectral signature.

[0082] In one example, a camera includes: a sensor array including a plurality of sensors; an IR illuminator configured to emit active IR light in an infrared (IR) photon band; a plurality of spectral illuminators, each spectral illuminator being configured to emit active spectral light in a different spectral photon band; a depth controller machine configured to determine a depth value for each of the plurality of sensors based on the active IR light; a spectral controller machine configured to determine, for each of the plurality of sensors, a spectral value for each spectral photon band of the plurality of spectral illuminators, wherein each spectral value is calculated based on a depth value determined for the sensor corresponding to the pixel; and an output machine configured to output a test depth + multi-spectral image including a plurality of pixels, each pixel corresponding to one of the plurality of sensors of the sensor array and including at least a depth value and a spectral value for each spectral photon band of the plurality of spectral illuminators; an object recognition machine including a convolutional neural network that has been previously trained with a set of labeled training depth + multi-spectral images having the same structure as the test depth + multi-spectral image, the object recognition machine configured to output a confidence value indicating the likelihood that the test depth + multi-spectral image includes a specified object.

[0083] It should be understood that the configurations and / or approaches described herein are exemplary in nature, and these specific embodiments or examples should not be considered to have limiting significance, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in other orders, in parallel, or omitted. Likewise, the order of the processes described above may be changed.

[0084] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A camera, comprising: a sensor array, the sensor array comprising a plurality of sensors; an infrared (IR) illuminator configured to emit IR light in an IR photonic band; a plurality of spectral illuminators, each spectral illuminator configured to emit spectral light in a different spectral photon band; a depth controller machine configured to determine a depth value for each of the plurality of sensors based on the emitted IR light, a spectral controller machine configured to determine, for each of the plurality of sensors, a spectral value for each spectral photon band of the plurality of spectral illuminators, wherein the plurality of sensors of the sensor array are differential sensors, and wherein each spectral value is determined based on a depth value of the differential sensor and a differential measurement of the spectral light and the ambient light; and an output machine configured to output a test depth+multispectral image comprising a plurality of pixels, each pixel corresponding to one of a plurality of sensors of the sensor array and comprising at least: Depth value, and a spectral value of each spectral photon band of the plurality of spectral illuminators; An object recognition machine that has been previously trained using a set of labeled training depth+multispectral images having the same structure as the test depth+multispectral image, the object recognition machine being configured to output a confidence value indicating a likelihood that the test depth+multispectral image includes a specified object.

2. The camera according to claim 1, wherein: The object recognition engine is further configured to output a confidence value corresponding to each of a plurality of different types of objects, each confidence value indicating a likelihood that the test depth+multispectral image includes an object of that type.

3. The camera according to claim 2, wherein: The object recognition engine is also configured to output the location of each recognized object on the test depth+multispectral image.

4. The camera according to claim 2, wherein: The object recognition engine is further configured to output, for each pixel of the test depth+multispectral image, a plurality of confidence values ​​corresponding to the plurality of different types of objects, each confidence value indicating a likelihood that the pixel is included in an object of that type.

5. The camera according to claim 1, wherein: Each spectral value is determined based on a depth value determined for the sensor corresponding to the pixel.

6. The camera according to claim 1, wherein: The object recognition engine is configured to determine the confidence value using a convolutional neural network.

7. The camera according to claim 6, wherein: The object recognition machine comprises a plurality of input nodes, wherein each input node is configured to receive an array of pixel values ​​corresponding to a different pixel among a plurality of pixels of the test depth+multispectral image, and wherein the array of pixel values ​​comprises the depth value and the plurality of multispectral values ​​of the pixel.

8. The camera according to claim 7, wherein: The plurality of multispectral values ​​of the pixel includes more than three spectral values.

9. The camera according to claim 7, wherein: The output engine is configured to output a surface normal for each pixel of the test depth+multispectral image, and wherein the array of pixel values ​​comprises the surface normal.

10. The camera according to claim 7, wherein: The output engine is configured to output a curvature for each pixel of the test depth+multispectral image, and wherein the array of pixel values ​​comprises the curvature.

11. The camera according to claim 6, wherein: The object recognition machine is configured to determine the confidence value using multiple models, wherein the multiple models include multiple channel-specific models, wherein each channel-specific model is configured to process different pixel parameters for multiple pixels of the test depth+multispectral image, wherein each channel-specific model includes multiple input nodes, and wherein for each channel-specific model, each input node is configured to receive a pixel parameter value of a different pixel among the multiple pixels of the test depth+multispectral image.

12. The camera of claim 1, wherein: The object recognition engine is configured to determine the confidence value using a statistical model.

13. The camera of claim 12, wherein: The statistical model includes a nearest neighbor algorithm.

14. The camera of claim 12, wherein: The statistical model includes a support vector machine.

15. The camera of claim 1, wherein: The object recognition engine is further configured to output a location of a bounding box surrounding the designated object on the test depth+multispectral image.

16. The camera of claim 1, wherein: The object recognition engine is further configured to output, for each pixel of the test depth+multispectral image, a confidence value indicating a likelihood that the pixel is included in the specified object.

17. The camera of claim 1, wherein: The object recognition machine is also configured to output one or more of the following: the position of the identified two-dimensional (2D) features of the specified object on the test depth + multi-spectral image, the position of the identified three-dimensional (3D) features of the specified object on the test depth + multi-spectral image, and the position of the identified spectral features of the specified object on the test depth + multi-spectral image.

18. A method for identifying an object, comprising: actively illuminating the object with IR light; measuring a depth profile of the object using a plurality of sensors of a camera while actively illuminating the object with the IR light; actively illuminating the object with spectral light in each of a plurality of different spectral photon bands; measuring a spectral signature of the object in each of the plurality of different spectral photon bands using a plurality of sensors of the camera used to measure a depth profile of the object while the object is actively illuminated with spectral light in the spectral photon bands, wherein the plurality of sensors are differential sensors, the spectral signature being determined based on depth values ​​of the differential sensors and differential measurements of the spectral light and ambient light; and A computer identifies the object based on the measured depth profile and the measured spectral signature.

19. A camera comprising: a sensor array, the sensor array comprising a plurality of sensors; an infrared (IR) illuminator configured to emit IR light in an IR photonic band; a plurality of spectral illuminators, each spectral illuminator configured to emit spectral light in a different spectral photon band; a depth controller machine configured to determine a depth value for each of the plurality of sensors based on the emitted IR light, a spectral controller machine configured to determine, for each of the plurality of sensors, a spectral value for each spectral photon band of the plurality of spectral illuminators, wherein the plurality of sensors of the sensor array are differential sensors, and wherein each spectral value is determined based on a depth value of the differential sensor and a differential measurement of the spectral light and the ambient light; and an output machine configured to output a test depth+multispectral image comprising a plurality of pixels, each pixel corresponding to one of a plurality of sensors of the sensor array and comprising at least: Depth value, and a spectral value of each spectral photon band of the plurality of spectral illuminators; as well as An object recognition machine comprising a convolutional neural network that has been previously trained using a set of labeled training depth + multispectral images having the same structure as the test depth + multispectral image, the object recognition machine being configured to output a confidence value indicating a likelihood that the test depth + multispectral image includes a specified object.

20. A computer-readable storage medium having instructions which, when executed, cause a machine to perform the method of claim 18.

21. A computer system comprising means for executing the method of claim 18.

Images