Deep learning system

Abstract

To provide a machine learning system, a method, a computer program, and an apparatus for enhancing various aspects of a machine learning model.SOLUTION: A system accesses a three-dimensional (3D) graphical model 1410 having substantially photorealistic resolution of an object and generates from the 3D model a set of training images, a point group, a depth map or other training data 1425 for a 3D graphical mode. A training set generator 1420 automatically renders or generates images or other training data from the 3D model in a manner that intentionally downgrades the resolution and quality of the image obtained as a result. The training data is used by a machine learning system 1430 to train one or more machine learning models 1435.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] [Related Applications] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 675,601, filed May 23, 2018, the entirety of which is incorporated herein by reference.

[0002] The present disclosure relates generally to the field of computer systems, and more specifically, to machine learning systems. [Background technology]

[0003] The worlds of computer vision and graphics are rapidly converging with the emergence of Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) products, such as MagicLeap™, Microsoft™ HoloLens™, Oculus™ Rift™, and other VR systems such as those from Valve™ and HTC™. The current approach in such systems is to use separate graphics processing units (GPUs) and computer vision subsystems that run in parallel. These parallel systems can be assembled from pre-existing GPUs in parallel with a computer vision pipeline implemented in software running on an array of processors and / or programmable hardware accelerators. [Brief description of the drawings]

[0004] Various objects, features, and advantages of the disclosed subject matter can be more fully understood with reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements. The accompanying drawings are schematic and are not intended to be drawn to scale. For the sake of clarity, no attempt has been made to label every component in every figure. Nor has every component of each embodiment of the disclosed subject matter been shown unless the figures are necessary to enable one of ordinary skill in the art to understand the disclosed subject matter.

[0005] [Figure 1] 1 illustrates a conventional augmented or mixed reality rendering system.

[0006] [Diagram 2] 1 illustrates a voxel-based augmented or mixed reality rendering system according to some embodiments.

[0007] [Diagram 3] 1 illustrates the difference between dense and sparse volumetric representations according to some embodiments.

[0008] [Figure 4] 1 illustrates a composite view of a scene according to some embodiments.

[0009] [Diagram 5] 4 illustrates an example element tree structure level of detail according to some embodiments.

[0010] [Figure 6] 1 illustrates applications in which the data structures and voxel data of the present application can be utilized, according to some embodiments.

[0011] [Figure 7] 1 illustrates an exemplary network used to recognize 3D digits, according to some embodiments.

[0012] [Figure 8] 1 illustrates multiple classifications performed on the same data structure using an implicit level of detail, according to some embodiments.

[0013] [Figure 9] 1 illustrates computation elimination with a 2D convolutional neural network according to some embodiments.

[0014] [Figure 10] 1 illustrates experimental results obtained from an analysis of exemplary test images, according to some embodiments.

[0015] [Figure 11] 1 illustrates hardware for culling operations according to some embodiments.

[0016] [Figure 12] 1 illustrates hardware improvements for culling operations according to some embodiments.

[0017] [Figure 13] 1 illustrates hardware according to some embodiments.

[0018] [Figure 14] 1 illustrates an example system employing an example training set generator according to at least some embodiments.

[0019] [Figure 15] 1 illustrates an example generation of synthetic training data according to at least some embodiments.

[0020] [Figure 16] 1 illustrates an example Siamese network in accordance with at least some embodiments.

[0021] [Figure 17]1 illustrates an exemplary use of a Siamese network to perform an autonomous comparison, according to at least some embodiments.

[0022] [Figure 18] 1 illustrates an example voxelization of a point cloud in accordance with at least some embodiments.

[0023] [Figure 19] 1 is a simplified block diagram of an example machine learning model in accordance with at least some embodiments.

[0024] [Figure 20] 1 is a simplified block diagram illustrating aspects of an exemplary training of a model, according to at least some embodiments.

[0025] [Figure 21] 1 illustrates an example robot that uses neural networks to generate a 3D map for navigation, according to at least some embodiments.

[0026] [Figure 22] 1 is a block diagram illustrating an example machine learning model for use with inertial measurement data, according to at least some embodiments.

[0027] [Figure 23] 1 is a block diagram illustrating an example machine learning model for use with image data, according to at least some embodiments.

[0028] [Figure 24] FIG. 24 is a block diagram illustrating an exemplary machine learning model that combines aspects of the models in the examples of FIGS. 22 and 23.

[0029] [Figure 25A] 25 is a graph illustrating the results of a machine learning model similar to the example machine learning model of FIG. 24. [Figure 25B] 25 is a graph illustrating the results of a machine learning model similar to the example machine learning model of FIG. 24.

[0030] [Figure 26] 1 illustrates an example system including an example neural network optimizer according to at least some embodiments.

[0031] [Figure 27] 1 is a block diagram illustrating an example optimization of a neural network model according to at least some embodiments.

[0032] [Figure 28] 1 is a table illustrating exemplary results generated and used during optimization of an exemplary neural network model.

[0033] [Figure 29] 1 shows graphs illustrating results generated and used during optimization of an exemplary neural network model.

[0034] [Figure 30A] 1 is a simplified block diagram illustrating an example of hybrid neural network pruning according to at least some embodiments.

[0035] [Figure 30B] FIG. 1 is a simplified flow diagram illustrating an example pruning of a neural network, according to at least some embodiments.

[0036] [Diagram 31] 1 is a simplified block diagram illustrating an example weight quantization performed in connection with pruning a neural network, according to at least some embodiments.

[0037] [Diagram 32] 1 is a table comparing results of exemplary neural network pruning techniques.

[0038] [Figure 33A] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33B] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33C] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33D] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33E] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33F] FIG. 1 is a simplified flow diagram of an example computer-implemented technique associated with machine learning, according to at least some embodiments.

[0039] [Diagram 34] 1 illustrates an example multi-slot vector processor according to some embodiments.

[0040] [Diagram 35] 1 illustrates an exemplary volumetric acceleration hardware according to some embodiments.

[0041] [Diagram 36] 1 illustrates an organization of voxel cubes according to some embodiments.

[0042] [Figure 37] 1 illustrates a two-level sparse voxel tree according to some embodiments.

[0043] [Figure 38] 1 illustrates a two-level sparse voxel tree according to some embodiments.

[0044] [Figure 39] 1 illustrates an example voxel data storage according to some embodiments.

[0045] [Diagram 40] 1 illustrates an insertion of a voxel into an exemplary volumetric data structure according to some embodiments.

[0046] [Diagram 41] 1 illustrates a projection of an exemplary 3D volumetric object according to some embodiments.

[0047] [Figure 42A] The positional relationship between Figures 42B and 42C is shown. [Figure 42B] 1 illustrates an exemplary operation involving an exemplary volumetric data structure. [Figure 42C] 1 illustrates an exemplary operation involving an exemplary volumetric data structure.

[0048] [Diagram 43] 1 illustrates the generation of a simplified map using projection, according to some embodiments.

[0049] [Diagram 44] 1 illustrates an exemplary aggregation of exemplary volumetric 3D measurements and / or simple 2D measurements from an embedded device according to some embodiments.

[0050] [Figure 45A] The positional relationship between Figures 45B and 45C is shown. [Figure 45B] 1 illustrates an exemplary acceleration of 2D pathfinding on a 2D 2×2 bitmap according to some embodiments. [Figure 45C] 1 illustrates an exemplary acceleration of 2D pathfinding on a 2D 2×2 bitmap according to some embodiments.

[0051] [Diagram 46] 1 illustrates an example acceleration of collision detection using an example volumetric data structure according to some embodiments.

[0052] [Figure 47] 1 is a simplified block diagram of an example network having devices in accordance with at least some embodiments.

[0053] [Figure 48] 1 is a simplified block diagram of an example fog or cloud computing network in accordance with at least some embodiments.

[0054] [Figure 49] 1 is a simplified block diagram of a system including an example device according to at least some embodiments.

[0055] [Figure 50] 1 is a simplified block diagram of an example processing device in accordance with at least some embodiments.

[0056] [Figure 51] 1 is a block diagram of an example processor in accordance with at least some embodiments.

[0057] [Figure 52] 1 is a block diagram of an exemplary computing system according to at least some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0058] In the following description, numerous specific details are provided regarding the systems and methods of the disclosed subject matter, as well as the environments in which such systems and methods may operate, to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features that are well known in the art have not been described in detail to avoid complicating the disclosed subject matter. Furthermore, it will be appreciated that the embodiments provided below are exemplary, and that other systems and methods are contemplated that fall within the scope of the disclosed subject matter.

[0059] Various technologies based on and incorporating augmented reality, virtual reality, mixed reality, autonomous devices, and robots are emerging that may use data models that represent volumes of three-dimensional space and geometry. Descriptions of various real and virtual environments using such 3D or volumetric data traditionally include large data sets that some computing systems have struggled to process in a desirable manner. Furthermore, as devices such as drones, wearable devices, virtual reality systems, etc., become smaller, the memory and processing resources of such devices may also be constrained. As an example, AR / VR / MR applications may require high frame rates for graphical representations generated using supporting hardware. However, in some applications, the GPU and computer vision subsystems of such hardware may need to process data (e.g., 3D data) at high speeds, such as up to 130 fps (7 ms), to generate desirable results (e.g., to generate reliable graphical scenes at frame rates that generate reliable results and prevent users from getting motion sickness due to excessive latency, among other exemplary goals). Further applications may be challenged similarly to satisfactorily process data describing large volumes while meeting the constraints of processing, memory, power, and application requirements of the corresponding systems, among other exemplary problems.

[0060] In some implementations, a computing system may include logic for generating and / or using sparse volumetric data defined according to a format. For example, the defined volumetric data structure may be provided for integrating computer vision and 3D rendering in various systems and applications. The volumetric representation of the object may be imaged using an optical sensor, such as a stereoscopic camera or a depth camera. The volumetric representation of the object may include multiple voxels. An improved volumetric data structure may be defined to allow the corresponding volumetric representation to be recursively refined to obtain a target resolution of the object. During the refinement, free space of the volumetric representation that may be contained in one or more of these voxels may be culled from the volumetric representation (and supporting operations). The free space may be a region of the volumetric representation that does not include a geometric characteristic of the object.

[0061] Thus, in an improved volumetric data structure, individual voxels in a corresponding volume may be tagged as "occupied" (due to some geometry present in the corresponding volumetric space) or "empty" (representing that the corresponding volume is composed of free space). Such tags may be further interpreted as specifying that one or more of the corresponding subvolumes are also occupied (e.g., if a parent or higher level voxel is tagged as occupied) or that all of the subvolumes are free space (i.e., if a parent or higher level voxel is tagged as empty). In some implementations, tagging a voxel as empty may effectively remove the voxel and / or voxels of the corresponding subvolume from the operations used to generate the corresponding volumetric representation. The volumetric data structure may follow a sparse tree structure, for example, a Sparse Sexaquaternary Tree (SST) format. Moreover, such approaches to sparse volumetric data structures may utilize relatively less storage space than traditionally used to store volumetric representations of objects, and compression of volumetric data may increase the feasibility of transmitting such representations and enable faster processing of such representations, among other exemplary benefits.

[0062] Volumetric data structures can be hardware accelerated to allow 3D renderers to be updated quickly, eliminating delays that may occur in separate computer vision and graphics systems. Such delays can introduce latency and, when used in AR, VR, MR, and other applications, can cause motion sickness for the user, among other drawbacks. The ability to rapidly test voxels for occupancy of geometric properties within the accelerated data structures enables the construction of low-latency AR, VR, MR, or other systems that can be updated in real time.

[0063] In some embodiments, the functionality of the volumetric data structure may provide in-frame warnings, such as in AR, VR, MR, and other applications, when a user is likely to collide with a real or synthetic object in the imaged scene, or in computer vision applications for drones or robotics, when such a device is likely to collide with a real or synthetic object in the imaged scene, allowing for warnings of impending collisions with the processing speed provided by the volumetric data structure.

[0064] An embodiment of the present disclosure may relate to storage and processing of volumetric data in applications such as head-mounted displays for robots, augmented and mixed reality headsets, and phones and tablets. In an embodiment of the present disclosure, each volumetric element (e.g., voxel) in a group of voxels, and optionally physical quantities related to the geometry of the voxel, are represented as a single bit. Further parameters related to the group of 64 voxels, such as corresponding red-green-blue (RGB) or other color encoding, transparency, truncated signed distance function (TSDF) information, etc., may be associated with the voxel and stored in an associated optional 64-bit data structure (e.g., such that two or more bits are used to represent each voxel). Such a representation scheme may achieve minimal memory requirements. Furthermore, representing a voxel with a single bit allows many simplified calculations to be performed to logically or mathematically combine elements of the volumetric representation. Combining elements of the volumetric representation can include, for example, ORing planes in the volume to create a 2D projection of the 3D volumetric data, and calculating surface area by counting the number of occupied voxels in a 2.5D manifold, among other things. Comparisons can include comparing 64-bit subvolumes (e.g., 4^3 subvolumes) using XOR logic, and volumes can be inverted. Here, objects can be merged and ORed together to create hybrid objects, among other examples.

[0065] FIG. 1 illustrates a conventional augmented or mixed reality system comprised of a parallel graphics rendering subsystem and a computer vision subsystem with a post-rendering connection device to account for changes due to rapid head movements and environmental changes that may produce occlusions and shadows in the rendered graphics. In one exemplary implementation, the system may include a host processor 100 supported by a host memory 124 to control the execution of a graphics pipeline, a computer vision pipeline, and a post-rendering correction device by interconnection via a bus 101, an on-chip network-on-chip, or other interconnection. The interconnection allows the host processor 100, running appropriate software, to control the execution of a graphics processing unit (GPU) 106, an associated graphics memory 111, a computer vision pipeline 116, and an associated computer vision memory 124. In one example, rendering of graphics using the GPU 106 via an OpenGL graphics shader 107 (e.g., operating on a triangle list 105) may occur at a slower rate than the computer vision pipeline. As a result, post-rendering corrections via the warp engine 108 and display / occlusion processor 109 can be performed to account for head pose and occlusion scene geometry changes that may have occurred since the graphics were rendered by the GPU 106. The output of the GPU 106 is time stamped so that it can be used in conjunction with correct control signals 121 from the head pose pipeline 120 and correct control signals 123 from the occlusion pipeline 122, among other examples, to generate the correct graphics output to account for any changes in head pose 119 and occlusion geometry 113.

[0066] In parallel with the GPU 106, multiple sensors and cameras (e.g., including active and passive stereo cameras for depth and vision processing 117) may be connected to the computer vision pipeline 116. The computer vision pipeline 116 may include one or more of at least three stages, each of which may include multiple stages of lower-level processing. In one example, the stages of the computer vision pipeline 116 may be an image signal processing (ISP) pipeline 118, a head pose pipeline 120, and an occlusion pipeline 122. The ISP pipeline 118 may receive the outputs of the input camera sensors 117 and condition them so that they may be used for subsequent head pose and occlusion processing. The head pose pipeline 120 may receive the output of the ISP pipeline 118 and use it, together with the output 119 of an inertial measurement unit (IMU) in the headset 110, to calculate the change in head pose since the corresponding output graphics frame was rendered by the GPU 106. The output 121 of the head pose pipeline (HPP) 120 may be applied to the warp engine 108 along with a user-specified mesh to warp the GPU output 102 to match the updated head pose position 119. The occlusion pipeline 122 may receive the output of the head pose pipeline 121 and look for new objects in the field of view, such as a hand 113 (or other exemplary object) that enters the field of view, which should generate a corresponding shadow 114 on the scene geometry. The output 123 of the occlusion pipeline 122 may be used by the display and occlusion processor 109 to accurately overlay the field of view on the output 103 of the warp engine 108.Among other example use cases and features, the display and occlusion processor 109 can use the calculated head pose 119 to generate a shadow mask for a synthetic shadow 114, and the display and occlusion processor 109 can composite occlusion geometry of the hand 113 onto the shadow mask to generate a graphical shadow 114 on top of the output 103 of the warp engine 108 to generate a final output frame 104 for display on the augmented / mixed reality headset 110.

[0067] FIG. 2 illustrates a voxel-based augmented or mixed reality rendering system according to some embodiments of the present disclosure. The apparatus illustrated in FIG. 2 may include a host system configured on a host CPU 200 and associated host memory 201. Such a system may communicate via a bus 204, an on-chip network, or other communication mechanism with an integrated computer vision and graphics pipeline 223 and associated integrated computer vision and graphics memory 213 that includes actual and synthetic voxels that are rendered in a final scene for display on a head-mounted augmented or mixed reality display 211. The AR / MR display 211 may include multiple active and passive image sensors 214 and an inertial measurement unit (IMU) 212 used to measure changes in head pose 222 orientation.

[0068] In a combined rendering pipeline, the composite geometry may be generated starting from a triangle list 204 that is processed by an OpenGL JiT (Just-in-Time) translator 205 to generate the composite voxel geometry 202. The composite voxel geometry may be generated, for example, by selecting a principal plane of triangles from the triangle list. Then, 2D rasterization of each triangle in the selected plane may be performed (e.g., in the X and Z directions). A third coordinate (e.g., Y) may be created as an attribute that is interpolated across the triangle. Each pixel of the rasterized triangle may result in the definition of a corresponding voxel. This process may be performed by a CPU or a GPU. Once each rasterized triangle is performed by the GPU, it may be read back from the GPU to create a voxel where the GPU has rendered the pixel, among other example implementations. For example, the composite voxels may be generated using a 2D buffer of the list, where each entry of the list stores depth information for the polygon rendered at that pixel. For example, the model may be rendered using an orthographic perspective (e.g., top-down). For example, every (x,y) provided in the exemplary buffer can represent a column of (x,y) within the corresponding voxel volume (e.g., from (x,y,0) to (x,y,4095)). Each column can then be rendered from the information as a 3D scanline using the information in each list.

[0069] Continuing with the example of FIG. 2, in some implementations, the synthetic voxel geometry 202 may be combined with measured geometry voxels 227 constructed using a simultaneous localization and mapping (SLAM) pipeline 217. The SLAM pipeline may generate output 225 using active and / or passive image sensors 214 (e.g., 214.1 and 214.2) that are first processed using an image signal processing (ISP) pipeline 215, which may be converted to a depth image 226 by a depth pipeline 216. The active or passive image sensors 214 (214.1 and 214.2) may include active or passive stereo sensors, structured light sensors, and time-of-flight sensors, among other examples. For example, the depth pipeline 216 may process depth data from a structured light or time-of-flight sensor 214.1, or alternatively, a passive stereo sensor 214.2. In one exemplary implementation, the stereo sensor 214.2 may include a pair of passive stereo sensors, among other exemplary implementations.

[0070] The depth images generated by the depth pipeline 215 can be processed by a dense SLAM pipeline 217 using a SLAM algorithm (e.g., Kinect Fusion) to generate a voxelized model of the measured geometry voxels 227. The measured geometry voxels 227 (e.g., actual voxel geometry) can be combined with the synthetic voxel geometry 202 to provide a ray tracing accelerator 206 that can generate a 2D rendering of the scene for output to a display device (e.g., a head-mounted display 211 in a VR or AR application) via a display processor 210. In such an implementation, a complete scene model can be constructed from the measured geometry voxels 227 and the actual voxels of the synthetic geometry 202. As a result, there is no need to warp the 2D rendered geometry (e.g., as in FIG. 1). Such an implementation can combine a head pose tracking sensor and corresponding logic to accurately align the actual and measured geometries. For example, an exemplary head pose pipeline 221 can process head pose measurements 232 from an IMU 212 attached to a head-mounted display 212, and the output 231 of the head pose measurement pipeline can be taken into account during rendering via the display processor 210.

[0071] In some examples, a unified rendering pipeline may use measured geometry voxels 227 (e.g., actual voxel model) and synthetic geometry 202 (e.g., synthetic voxel model) to render an audio reverberation model and model the physics of a real-world, virtual, or mixed reality scene. As an example, the physics pipeline 218 can receive the measured geometry voxels 227 and the synthetic voxel geometry 202 and use the ray casting accelerator 206 to calculate output audio samples for the left and right earphones in the head mounted display (HMD) 211 to calculate output samples 230 using acoustic reflection coefficients embedded in the voxel data structure. Similarly, the unified voxel model composed of 202 and 227 can also be used to determine physical updates of synthetic objects in a synthetic AR / MR scene. The physics pipeline 218 receives the synthetic scene geometry as input and uses the ray casting accelerator 206 to calculate collisions before calculating updates 228 to the synthetic geometry 202 for rendering and as a basis for future iterations of the physics model.

[0072] In some implementations, a system such as that shown in FIG. 2 may further comprise one or more hardware accelerators for implementing and / or utilizing a convolutional neural network (CNN) that can process either RGB video / image input from the output of the ISP pipeline 215, or volumetric scene data from the output of the SLAM pipeline 217, among other examples. A neural network classifier can be executed using the hardware (HW) convolutional neural network (CNN) accelerator 207 exclusively, or in combination with a processor and the HW CNN accelerator 207, to generate the output classification 237. Availability of the HW CNN accelerator 207 for inferring a volumetric representation may enable labeling a group of voxels in the measured geometry voxels 227 as belonging to a particular object class, among other example uses.

[0073] Labeling the voxels (e.g., using CNN and supporting hardware acceleration) may enable the system to recognize the objects to which they belong as corresponding to known objects. The source voxels may be removed from the measured geometry voxels 227 and replaced with a bounding box corresponding to the object and / or information about the object's origin, object pose, object descriptors, among other example information. This may result in a much more semantically meaningful description of the scene that may be used, for example, as input by a robot, drone, or other computing system that interacts with objects in the scene, or as input by an audio system that looks up the sound absorption coefficients of objects in the scene and feeds them into an acoustic model of the scene, among other example uses.

[0074] One or more processor devices and hardware accelerators may be provided to implement the pipelines of the example system shown and described in Figure 2. In some implementations, among other example implementations, all of the hardware and software elements of the combined rendering pipeline may share access to a DRAM controller 209, which in turn enables data to be stored in a shared DDR memory device 208.

[0075] FIG. 3 is presented to illustrate the difference between dense and sparse volumetric representations according to some embodiments. As shown in the example of FIG. 3, a real-world or synthetic object 300 (e.g., a rabbit statue) may be represented in terms of voxels in a dense manner as shown at 302, or in a sparse manner as shown at 304. The advantage of a dense representation such as 302 is uniform access speed to all voxels in the volume, but the disadvantage is the amount of storage that may be required. For example, a dense representation such as a 512^3 element volume (e.g., corresponding to 5 m at 1 cm resolution for a volume scanned using a Kinect sensor) requires 512 megabytes to store a relatively small volume with a truncated signed distance function (TSDF) of 4 bytes per voxel. On the other hand, an octree representation 304 embodying a sparse representation may reduce the amount of data required to store the same volume by storing only those voxels for which actual geometry exists in the real-world scene.

[0076] Referring to FIG. 4, an exemplary synthetic view of a scene according to some embodiments is shown. In particular, FIG. 4 illustrates how a synthetic view of a scene 404 may be maintained, displayed, or subjected to further processing using parallel data structures to represent synthetic voxels 401 and measured real-world voxels 403 within equivalent bounding boxes 400 and 402 for synthetic and real-world voxel data, respectively. FIG. 5 illustrates the level of detail of a uniform 4^3 element tree structure according to some embodiments. In some implementations, as little as one bit may be utilized to describe each voxel in a volume using an octree representation such as that depicted in the example of FIG. 5. However, a drawback of octree-based techniques may be the number of indirect memory accesses utilized to access a particular voxel in the octree. In the case of sparse voxel octrees, among other exemplary advantages, the same geometry can be advantageously represented implicitly at multiple levels of detail, enabling operations such as ray casting, game physics, CNNs, and other techniques, and culling empty parts of a scene from further computation, resulting in an overall reduction not only in terms of required storage, but also in terms of power consumption and computational load.

[0077] In one implementation, an improved voxel descriptor (also referred to herein as a "volumetric data structure") may be provided to organize the volumetric information as 4^3 (or 64-bit) unsigned integers as shown at 501 with a memory requirement of 1 bit per voxel. In this example, 1 bit per voxel is insufficient to store the truncated signed distance function value (compared to the TSDF in SLAMbench / KFusion, which utilizes 64 bits). In this example, an additional (e.g., 64-bit) field 500 may be included in the voxel descriptor. This example may be further enhanced such that the TSDF in the 64-bit field 500 is 16 bits, while an additional 2 bits of fractional resolution in x, y, and z may be implicitly provided in the voxel descriptor 501 to make the combination of the voxel TSDF in the 64-bit field 500 and the voxel position 501 comparable to a much higher resolution TSDF as used in SLAMbench / KFusion or other examples. For example, the further data of the 64-bit field 500 (voxel descriptor) may be used to store subsampled RGB color information (e.g., from the scene via a passive RGB sensor), each of 1 byte, an 8-bit transparency value alpha, and two 1-byte reserved fields R1 and R2. These reserved fields may be application specific and may be used, for example, to store acoustic reflectivity for audio applications, stiffness for physics applications, type of object material, among other examples.

[0078] As shown in FIG. 5, the voxel descriptor 501 may be logically grouped into four 2D planes. Each of these 2D planes includes 16 voxels 502. These 2D planes (or voxel planes) may describe each level of an octree-style structure based on successive decompositions in ascending powers of four, as depicted in FIG. 5. In this exemplary implementation, a 64-bit voxel descriptor is selected because it is suitable for the 64-bit bus infrastructure used in the corresponding system implementation (although other voxel descriptor sizes and formats may be provided in other system implementations and sized according to the bus or other infrastructure of the system). In some implementations, the voxel descriptor may be sized to reduce the number of memory accesses used to retrieve a voxel. For example, among other exemplary considerations and implementations, the use of a 64-bit voxel descriptor may reduce the number of memory accesses required to access a voxel at any level in the octree by a factor of two, compared to a traditional octree that operates with 2^3 elements.

[0079] In one example, an octree may be written starting with a 4^3 root volume 503 and each non-zero entry indicating the presence of geometry in the underlying layers 504, 505, and 506 in an exemplary 256^3 volume. In this particular example, four memory accesses may be used to access the lowest level in the octree. If such overhead is too high, an alternative approach may be adopted to encode the top level of the octree as a larger volume, such as 64^3, as shown in 507. In this case, each non-zero entry in 507 may indicate the presence of an underlying 4^3 octree in the underlying 256^3 volume 508. As a result of this alternative organization, only two memory accesses are required to access any voxel in the 256^3 volume 508, compared to the alternative formulations shown in 503, 504, and 505. This latter approach is advantageous when the device hosting the octree structure has a larger amount of built-in memory, thus permitting only the smaller, less frequently accessed portions of the voxel octree 508 in external memory. This approach may be more costly in terms of storage, for example, when a full, larger (e.g., 64^3) volume is stored in on-chip memory, but the trade-off may allow for faster memory access (e.g., 2x) and significantly reduced power consumption, among other example advantages.

[0080] Referring to FIG. 6, a block diagram illustrating exemplary applications in which the data structures and voxel data of the present application may be utilized, according to some embodiments. In one example, such as that shown in FIG. 5, further information may be provided via an exemplary voxel descriptor 500. Although the voxel descriptor may increase the overall memory utilized to 2 bits per voxel, the voxel descriptor may enable a wide range of applications in which the voxel data as depicted in FIG. 6 may be utilized. For example, among other implementations, a shared volumetric representation 602, such as that generated using a dense SLAM system 601 (e.g., SLAMbench), may be utilized in rendering a scene using graphics ray casting or ray tracing 603 used in audio ray casting 604. In yet another example, the volumetric representation 602 may be utilized in convolutional neural network (CNN) inference 605, and may be backed by a cloud infrastructure 607. In some cases, the cloud infrastructure 607 may include detailed volumetric descriptors of objects, such as trees, furniture, or other objects (e.g., 606), that may be accessed via inference. Based on inference or otherwise identification of the object, a corresponding detailed descriptor can be returned to the device, so that the voxels of the volumetric representation 602 can be replaced with a bounding box representation having pose information and a descriptor that includes properties of the object, among other example features.

[0081] In yet other embodiments, the voxel model described above may be additionally or alternatively utilized in some systems to build 2D maps of the exemplary environment 608 using 3D-to-2D projections from the volumetric representation 602. These 2D maps may again be shared over a cloud infrastructure and / or other network-based resources 607 via communication mechanisms and aggregated (e.g., using the same cloud infrastructure) to build higher quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices. In yet a further example, the 2D maps may be refined for ultra-low bandwidth applications using projections followed by piecewise simplification 609 (e.g., assuming the width and height of the vehicle or robot are fixed). The simplified path may then have only a single X,Y coordinate pair for each piecewise linear segment of the path, thereby reducing the amount of bandwidth required to communicate the path of the vehicle 609 to the cloud infrastructure 607 and aggregated to that same cloud infrastructure 607 to build higher quality maps using crowdsourcing techniques. These maps can be shared by the cloud infrastructure 607 to connected machines and devices.

[0082] To enable these different applications, some implementations may provide common functionality, such as through shared software libraries, and some embodiments may accelerate common functionality using hardware accelerators or processor instruction set architecture (ISA) extensions, among other examples. For example, such functionality may include inserting a voxel into a descriptor, deleting a voxel, or searching for a voxel 610. In some implementations, collision detection functionality 620 may be supported, as well as deleting points / voxels from a volume 630, among other examples. As introduced above, the system may include functionality for quickly generating 2D projections 640 in the X, Y, and Z directions from a corresponding volumetric representation 602 (a 3D volume) (e.g., this may serve as a basis for path or collision determination). In some cases, it may also be advantageous to be able to generate triangle lists from the volumetric representation 602 using histogram pyramids 650. Additionally, the system may include functionality for fast determination of free paths 660 in the 2D and 3D representations of the volumetric space 602. Such functionality may be useful in a variety of applications: Further functionality may be provided such as refining the number of voxels in a volume, counting the number of 1-bits in masked regions of the volumetric representation 602 using a population counter to determine the surface of an object, among other examples.

[0083] Referring to the simplified block diagram of FIG. 7, an exemplary network is shown that includes a system capable of recognizing 3D numbers according to at least some embodiments. For example, one of the applications shown in FIG. 6 is a volumetric CNN application 605, which is described in more detail in FIG. 7. In FIG. 7, the exemplary network is used to recognize 3D numbers 700 generated from a dataset such as the mixed National Institute of Standards and Technology (MNIST) dataset. The numbers in such a dataset can be used to train a CNN-based convolutional network classifier 710 by applying appropriate rotations and translations in X, Y, and Z to the numbers before training. The trained network 710, when used for inference on an embedded device, can be used to accurately classify 3D numbers 720 in a scene even when the numbers are subject to rotations and translations in X, Y, and Z, among other examples. In some implementations, the operation of the CNN classifier can be accelerated by the HW CNN accelerator 207 shown in FIG. 2. The first layer of the neural network performs multiplications using voxels of the volumetric representation 602. Therefore, multiplication by 0 is always 0, and multiplication by 1 (a voxel) by a data value A is equal to A, so these arithmetic operations can be omitted.

[0084] FIG. 8 illustrates multiple classifications performed on the same data structure using implicit levels of detail. A further improvement of CNN classification using volumetric representation 602 may be that since the octree representation implicitly includes multiple levels of detail in the octree structure as shown in FIG. 5, multiple classifications on the same data structure may be performed using the implicit levels of detail 800, 810, and 820 in parallel and using a single classifier 830 or multiple classifiers in parallel as shown in FIG. 8. In conventional systems, an equivalent parallel classification may be slower due to the need to resize images between classification passes. Such resizing may be omitted in implementations that apply the voxel structure described herein since the same octree may contain the same information at multiple levels of detail. In fact, a single training dataset based on a volumetric model may cover all of these levels of detail, rather than a resized training dataset as is required for conventional CNN networks.

[0085] With reference to the example of FIG. 9, an exemplary computational elimination by a 2D CNN according to some embodiments is shown. Computational elimination can be used not only in 2D CNN as shown in FIG. 9 but also in 3D volumetric CNN. For example, in FIG. 9, in the first layer, a bitmap mask 900 can be used to describe the expected "shape" of the input 910 and can be applied to the incoming video stream 920. In one example, computational elimination can be used not only in 3D volumetric CNN but also in 2D volumetric CNN. For example, in the 2D CNN of the example of FIG. 9, a bitmap mask 900 can be applied to the first layer of the CNN to describe the expected "shape" of the input 910 and can be applied to the input data of the CNN, such as the incoming video stream 820. As an example, FIG. 9 shows the effect of applying a bitmap mask to an image of a pedestrian for training or inference in a CNN network. Here, 901 represents the original image of the pedestrian 901 and 903 represents the corresponding version with the bitmap mask applied. Similarly, an image without pedestrians is shown at 902, and the corresponding bitmap mask version at 904. The same method can be applied to any kind of 2D or 3D object to reduce the number of operations required for training or inference of a CNN through knowledge of the expected 2D or 3D geometry predicted by the detector. An example of a 3D volumetric bitmap is shown at 911. The use of 2D bitmaps for inference in real scenes is shown at 920.

[0086] In the exemplary implementation of Figure 9, a conceptual bitmap is shown (at 900), and the actual bitmap is generated by averaging a series of training images for a particular class of object 910. Although the example shown is two-dimensional, similar bitmap masks for 3D objects can also be generated with the proposed volumetric data format of one bit per voxel. Indeed, the method may potentially be extended to use additional bits per voxel / pixel to specify the expected color range or other characteristics of a 2D or 3D object, among other exemplary implementations.

[0087] FIG. 10 is a table showing the results of an exemplary experiment involving the analysis of 10,000 CIFAR-10 test images, according to some embodiments. In some implementations, operation elimination can be used to eliminate intermediate calculations in 1D, 2D, and 3D CNNs due to rectified linear unit (ReLU) operations that are frequent in CNN networks, such as LeNet1000 shown in FIG. 10. As shown in FIG. 10, in an experiment using 10,000 CIFAR-10 test images, the percentage of data-dependent zeros generated by the ReLU unit can reach up to 85%. That is, a system can be provided that recognizes zeros when they are zero, and accordingly does not fetch corresponding data or perform corresponding multiplication operations. In this example, 85% represents the percentage of ReLU dynamic zeros generated from a modified National Institute of Standards and Technology (MNIST) test dataset. The corresponding operation eliminations corresponding to these zeros can help reduce power consumption and memory bandwidth requirements, among other exemplary benefits.

[0088] Trivial operations may be culled based on a bitmap. For example, the use of such a bitmap may be in accordance with the principles and embodiments described and illustrated in U.S. Patent No. 8,713,080, entitled "Circuit for compressing data and a processor employing the same," which is incorporated herein by reference in its entirety. Some implementations may provide hardware that can use such bitmaps, such as the systems, circuits, and other implementations described and illustrated in U.S. Patent No. 9,104,633, entitled "Hardware for performing arithmetic operations," which is also incorporated herein by reference in its entirety.

[0089] FIG. 11 illustrates hardware that may be incorporated into a system to provide functionality for culling trivial operations based on a bitmap, according to some embodiments. In this example, a multi-layer neural network is provided that includes repeated convolution layers. The hardware may include one or more processors, one or more microprocessors, one or more circuits, one or more computers, and the like. In this particular example, the neural network includes an initial convolution processing layer 1100, followed by a pooling process 1110, and finally an activation function process, such as a rectified linear unit (ReLU) function 1120. The output of the ReLU unit 1120, which provides a ReLU output vector 1131, may be connected (e.g., possibly via a delay 1132) to a next convolution processing layer 1180 that receives the ReLU output vector 1131. In one exemplary implementation, in parallel with the connection of the ReLU unit 1120 to the next convolution unit 1180, a ReLU bitmap 1130 may be generated, which represents which elements in the ReLU output vector 1131 are 0 and which elements are not 0.

[0090] In one implementation, a bitmap (e.g., 1130) may be generated or otherwise provided to inform available hardware of opportunities to eliminate operations involved in the computation of the neural network. For example, bits in the ReLU bitmap 1130 may be interpreted by the bitmap scheduler 1160, which instructs the multipliers in the next convolution unit 1180 to omit 0 entries in the ReLU output vector 1131 for which a corresponding binary 0 exists in the ReLU bitmap 1130, assuming that multiplication by 0 always produces 0 as an output. In parallel, memory fetches from the address generator 1140 for data / weights corresponding to 0 in the ReLU bitmap 1130 may also be omitted since there is little value in fetching weights that will be omitted by the next convolution unit 1180. If the weights are fetched from the attached DDR DRAM storage device 1170 via the DDR controller 1150, the latency may be so high that only some on-chip bandwidth and associated power consumption can be saved. On the other hand, if the weights are fetched from on-chip RAM 1180 storage, it may be possible to avoid / omit the weight fetch operation altogether, especially if a delay corresponding to the RAM / DDR fetch delay 1132 is added at the input to the next convolution unit 1180.

[0091] Referring to FIG. 12, a simplified block diagram is presented to illustrate an exemplary hardware improvement with circuits and other logic for culling trivial operations (or performing operation elimination) according to some embodiments. As shown in the example of FIG. 12, further hardware logic may be provided to predict in advance the sign of the ReLU unit 1220 input from the immediately preceding max pooling unit 1210 or convolution unit 1200. Adding sign prediction and ReLU bitmap generation to the max pooling unit 1210 may enable the ReLU bitmap information to be predicted earlier in terms of timing to cover the delay that may occur through the address generator 1240 via the external DDR controller 1250 and DDR storage 1270 or the internal RAM storage 1271. If the delay is small enough, the ReLU bitmap may be interpreted in the address generator 1240, since it may be determined that the result of the fetch from memory will never be used, and the memory fetch associated with ReLU bitmap 0 may be omitted entirely. This modification to the scheme of FIG. 11 may provide further power savings, among other example features and functionality, and may also allow for the elimination of a delay stage (e.g., 1132, 1232) at the input to the next convolution unit 1280 if the delay through the DDR access path (e.g., 1240 to 1250, 1250 to 1270) or the RAM access path (e.g., 1240 to 1271) is small enough to not warrant the delay stage 1232.

[0092] FIG. 13 is another simplified block diagram illustrating an example hardware according to some embodiments. For example, a CNN ReLU layer can generate a large number of outputs 0 corresponding to negative inputs. In fact, negative ReLU inputs can be predictively determined by looking at the sign input to the previous layer (e.g., the pooling layer in the example of FIG. 13). Floating point and integer operations can be explicitly signed in terms of the most significant bit (MSB). Therefore, a simple bitwise exclusive OR (XOR) operation over the vector of inputs multiplied in the convolutional layer can predict which multiplications will generate an output 0 as shown in FIG. 13. The resulting sign-predicted ReLU bitmap vector can be used as a basis to determine and eliminate a subset of multiplications and associated coefficient reads from memory, such as in the manner described in other examples above.

[0093] Providing for the generation of a ReLU bitmap back to a previous pooling or convolution stage (i.e., a stage before the corresponding ReLU stage) may result in additional power. For example, a sign prediction logic may be provided to disable a multiplier when generating a negative output that is ultimately set to 0 by the ReLU activation logic. For example, this is shown when two sign bits 1310 and 1315 of multiplier 1314 inputs 1301 and 1302 are logically combined by an XOR gate to form PreReLU bitmap bit 1303. This same signal may be used to disable the operation of multiplier 1314. Otherwise, energy would be unnecessarily consumed generating a negative output that would be set to 0 by the ReLU logic before being input for multiplication in the next convolution stage 1390, among other examples.

[0094] Note that the representations of 1300, 1301, 1302, and 1303 (notation A) show a high-level view of what is shown in the representation given in FIG. 13B. In this example, the input to block 1302 may include two floating-point operands. Input 1301 may include an explicit sign bit 1310, an exponent 1311 including multiple bits, and a mantissa 1312 also including multiple bits. Similarly, input 1302 may include a sign 1315, a mantissa 1317, and an exponent 1316. In some implementations, the precision of the mantissa and exponent may vary, since the sign of result 1303 depends only on the signs of 1301 and 1302, or 1310 and 1315, respectively. In fact, neither 1301 nor 1302 need be floating-point numbers, but may be in any integer or fixed-point format, so long as they are signed numbers and the most significant bit (MSB) is in fact, explicitly or implicitly, the sign bit (e.g., if the numbers are one's or two's complement).

[0095] 13, the two sign inputs 1310 and 1315 may be combined using an XOR (alternatively referred to herein as ExOR or EXOR) gate to generate a bitmap bit 1303, which may then be processed using hardware to identify downstream multiplications that may be omitted in the next convolution block (e.g., 1390). The same XOR output 1303 may also be used to disable multiplier 1314 if the two input numbers 1313 (e.g., corresponding to 1301) and 1318 (e.g., corresponding to 1302) have opposite signs and would generate a negative output 1304 that would be set to 0 by ReLU block 1319, resulting in a 0 value in the ReLU output vector 13191 input to the next convolution stage 1390. Thus, in some implementations, the PreReLU bitmap 1320 may be transmitted in parallel to a bitmap scheduler 1360, which may schedule the multiplications to be performed (and / or omitted) on the convolution unit 1390. For example, for every 0 in the bitmap 1320, the corresponding convolution operation may be omitted in the convolution unit 1390. In parallel, the bitmap 1320 may be consumed by an exemplary address generator 1330, which controls the fetching of weights for use in the convolution unit 1390. A list of addresses corresponding to 1's in the bitmap 1320 may be compiled in the address generator 1330, which controls the routing via the DDR controller 1350 to the DDR storage 1370 or otherwise to the on-chip RAM 1380. In either case, among other examples, weights corresponding to weights in the PreReLU bitmap 1320 may be fetched and presented to the convolution block 1390 (e.g., after some latency in terms of clock cycles to the weight input 1371), but fetching weights corresponding to 0 may be omitted.

[0096] As mentioned above, in some implementations, a delay (e.g., 1361) may be inserted between the bitmap scheduler 1360 and the convolution unit 1390 to balance the delay through the address generator 1330, the DDR controller 1350, and the DDR 1350, or the path through the address generator 1330 and the internal RAM 1380. The delay may allow the convolutions driven by the bitmap scheduler to be correctly aligned with the corresponding weights for the convolution calculation in the convolution unit 1390. In fact, from a timing perspective, it may be possible to gain additional time by generating the ReLU bitmap earlier than the output of the ReLU block 1319. This additional time may be used to intercept reads to memory (e.g., RAM 1380 or DDR 1370) before they are generated by the address generator 1330, so that some of the reads (e.g., corresponding to zeros) may be omitted. Because memory reads can be much more expensive than logical operations on a chip, eliminating such memory fetches can lead to very significant energy savings, among other example benefits.

[0097] In some implementations, if the savings in terms of clock cycles are still insufficient to cover the DRAM access time, a block-oriented technique can be used to read groups of sign bits (e.g., 1301) from the DDR in advance. These groups of sign bits can be used with blocks of signs from the input image or intermediate convolution layers 1302 to generate blocks of PreReLU bitmaps using a set of (multiple) XOR gates 1300 (e.g., to calculate the difference between sign bits in 2D or 3D convolutions between 2D or 3D arrays / matrices, among other examples). In such implementations, an additional bit of storage in the DDR or on-chip RAM can be provided to store the sign of each weight, which may make it possible to cover many latency cycles to avoid reading weights from the DDR or RAM that will be multiplied by zero from the ReLU stage. In some implementations, the additional bit of storage per weight in the DDR or on-chip RAM can be avoided because the signs are stored so that they can be addressed independently from the exponent and mantissa, among other example considerations and implementations.

[0098] In some implementations, it may be particularly difficult to access readily available training sets for training machine learning models, including models such as those described above. Indeed, in some cases, training sets may not exist for a particular machine learning application or may not correspond to the type of sensors that generate the inputs for the model being trained, among other example problems. In some implementations, synthetic training sets may be developed and utilized to train neural networks or other deep reinforcement learning models. For example, rather than acquiring or imaging a training dataset consisting of hundreds or thousands of images of a particular person, animal, object, product, etc., a synthetic 3D representation of the subject may be generated manually (e.g., using a graphic design or 3D photo editing tool) or automatically (e.g., using a 3D scanner), and the resulting 3D model may be used as a basis for automatically generating training data related to the subject of the 3D model. This training data may be combined with other training data to form a training dataset consisting at least in part of the synthetic training data, which may be utilized to train one or more machine learning models.

[0099] As an example, a deep reinforcement learning model or other machine learning model as introduced herein may be used to enable an autonomous machine to scan shelves in a store, warehouse, or another business to assess the inventory of a particular product in the store. Thus, the machine learning model may be trained to enable the autonomous machine to detect individual products. In some cases, the machine learning model may not only identify the products on the shelf, but also identify the number of products on the shelf (e.g., using a depth model). Rather than training the machine learning model with a series of real-world images (e.g., from the same store or different stores), a synthetic 3D model of each product (or at least some of these products) may be generated (e.g., by the product provider, the machine learning model provider, or another source) for each and every product that a store may carry and each and every configuration of the product (e.g., various displays, each pose or view of the product (all and partial) in various lighting, various orientations of the product packaging, etc.). The 3D model may be of photorealistic quality or near photorealistic quality in its detail and resolution. 3D models may be provided for consumption along with other 3D models to generate a variety of different views of a given object (e.g., a product), or even a collection of different objects (e.g., a collection of products on a store shelf, with various combinations of products placed next to each other in different lighting, at different orientations, etc.), among other example applications, to generate a set of synthetic training data images.

[0100] 14, a simplified block diagram 1400 of an exemplary computing system (e.g., 1415) implementing a training set generator 1420 for generating synthetic training data used by a machine learning system 1430 to train one or more machine learning models (e.g., 1435) (such as deep reinforcement learning models, Siamese neural networks, convolutional neural networks, and other artificial neural networks). For example, a 3D scanner 1405 or other tool can be used to generate a set of 3D models 1410 (e.g., of people, interior and / or exterior architecture, landscape elements, products, furniture, transportation elements (e.g., road signs, automobiles, traffic hazards, etc.), and other examples), which can be consumed by providing the training set generator 1420 as input. In some implementations, the training data generator 1420 can automatically render a set of training images, point clouds, depth maps, or other training data 1425 from the 3D model 1410. For example, the training set generator 1420 can be programmatically configured to automatically tilt, rotate, and zoom the 3D model to fill a “complete” and diverse collection of images for imaging the object by capturing several different images, and capture a collection of images of the 3D model at different resulting orientations and poses, and in different (e.g., computer-simulated) lighting, with all or part of the entire object of the 3D model captured in the images, etc.

[0101] In some implementations, synthetic training images generated from a 3D model may have photorealistic resolution comparable to the real-world objects on which they are based. In some cases, the training set generator 1420 may be configured to automatically render or generate images or other training data from a 3D model in a manner that intentionally downgrades the resolution and quality of the resulting images (compared to the high-resolution 3D model). For example, the quality of the images may be degraded by adding noise, applying filters (e.g., Gaussian filters), and adjusting one or more rendering parameters to introduce noise, reduce contrast, reduce resolution, change brightness levels, among other adjustments, to bring the images to a quality level comparable to that which may be generated by a sensor (e.g., a 3D scanner, a camera, etc.) that is expected to provide input to the machine learning model being trained.

[0102] A number of different conditions or rules may be defined and considered by the training set generator system, especially when constructing a dataset for training a deep neural network. For example, CNNs traditionally require a large amount of data for training to generate accurate results. Synthetic data can avoid the case where the available training dataset is too small. Thus, for a particular machine learning model, a target number of training data samples may be identified, and the training set generator may be based on the amount and type of training samples to be generated to meet the desired amount of training samples. Furthermore, conditions for generating sets in which the variability of samples exceeds a threshold may be designed and considered by the training set generator. This is to minimize overfitting of the machine learning model and provide the generalization required to perform well in a large number of highly diverse scenarios. Such variability may be achieved through adjustable parameters applied by the training set generator, such as the camera angle, camera height, field of view, lighting conditions, etc., used to generate individual samples from the 3D model, among other examples.

[0103] In some implementations, a sensor model (e.g., 1440) may be provided that defines aspects of a particular type or model of sensor (e.g., a particular 2D or 3D camera, LIDAR sensor, etc.). The model 1440 defines filters and other modifications to be made to raw images, point clouds, or other training data (e.g., generated from the 3D model) to simulate the data generated by the modeled sensor (e.g., resolution, vulnerability to glare, sensitivity to light and dark, vulnerability to noise, etc.). In such cases, the training set generator may artificially degrade samples generated from the 3D model to mimic equivalent images or samples generated by the modeled sensor. In this manner, samples of synthetic training data (e.g., as generated by a real-world version of the sensor) may be generated that are comparable in quality to the data input to the trained machine learning model.

[0104] Referring to FIG. 15, a block diagram 1500 illustrating an example generation of synthetic training data is shown. For example, a 3D model 1410 of a particular object may be generated. The 3D model may be a lifelike or photorealistic representation of the object. In the example of FIG. 15, the model 1410 represents a cardboard package containing a set of glass bottles. A collection of images (e.g., 1505) may be generated based on the 3D model 1410, capturing various views of the 3D model 1410, including views of the 3D model in various lighting, environments, and conditions (e.g., in use / at rest, open / closed, damaged, etc.). The collection of images 1505 may be processed using a sensor filter (e.g., defined in the sensor model) of an example training data generator. Processing of the images 1505 may modify the images 1505 to degrade the images 1505 to generate "lifelike" images 1425 that mimic the quality and characteristics of images captured using a real sensor.

[0105] In some implementations, to aid in generating degraded versions of synthetic training data samples, the model (e.g., 1410) may include metadata to indicate materials and other characteristics of the model's objects. In such implementations, the characteristics of the objects defined in the model may be considered by the training data generator (e.g., in combination with the sensor model) to determine how likely an actual image (or point cloud) would be generated by a particular sensor, taking into account lighting, the position of the sensor relative to the modeled objects, and the characteristics (e.g., materials) of the objects, among other considerations. For example, the model 1410 in the particular example of FIG. 15 that models a bottle package may include metadata that defines which portions of the 3D model (e.g., which pixels or polygons) correspond to the glass material (the bottle) and which portions correspond to the cardboard (the package). Thus, when the training data generator generates synthetic images and applies sensor filtering 1510 to the images 1505 (e.g., to model the way light reflects off various surfaces of the 3D model), the modeled sensor's response to these characteristics can be more realistically applied to generate reliable training data that is more consistent with what would be produced by the actual sensor if used to generate the training data. For example, it may be possible to generate training data images that model the fragility of the sensor depending on the material modeled in the 3D model, producing images with glare, noise, or other imperfections, e.g., corresponding to reflections from the glass surface of the bottle in the example of FIG. 15, but less noise or glare corresponding to less reflective cardboard surfaces. Similarly, the type of material, temperature, and other characteristics modeled in the 3D model representation of the subject may have different effects on different sensors (e.g., camera sensors vs. LIDAR sensors).Thus, an exemplary test data generation system can consider both the 3D model metadata and the particular sensor model when automatically determining filters or processing to apply to the image 1505 to generate a degraded version of the image (e.g., 1425) that simulates a version of the image that would likely be produced by a real-world sensor.

[0106] Additionally, in some implementations, further post-processing of the image 1505 may include adjusting the depth of field. In some 3D rendering programs, the virtual camera used in the software is perfect and can capture both nearby and distant objects in perfect focus. However, this may not be the case for real-world cameras or sensors (and may be defined as such within the attributes of the corresponding sensor model used by the training set generator). Thus, in some implementations, a depth of field effect on the image may be applied during post-processing (e.g., using the training set generator to create examples of flawed but more photorealistic images (e.g., 1425) by automatically identifying and selecting points where the camera should appear to be focused on the background and out of focus on the modeled subject features). Further post-processing may include adding noise on the image to simulate noisy artifacts present in photography. For example, the training set generator may include adding noise by limiting the number of light bounces that the ray tracing algorithm calculates on the object, among other example techniques. Additionally, slight pixelation may be applied on top of the rendered model in an effort to remove any overly or unrealistically smooth edges or surfaces that arise as a result of the synthesis process. For example, an optical blur layer may be added to average out "blocks" of pixels, which, when combined with other post-processing operations (e.g., based on corresponding sensor models), may result in more realistic synthetic training samples.

[0107] As shown in FIG. 15, when generating training data samples (e.g., 1425) from the exemplary 3D model 1410, the samples (e.g., images, point clouds, etc.) may be added to or included with other real or synthetically generated training samples to build a training dataset for a deep learning model (e.g., 1435). Similarly, as shown in the example of FIG. 14, a model trainer 1455 (e.g., of the exemplary machine learning system 1430) may be used to train one or more machine learning models 1435. In some cases, synthetic depth images may be generated from the 3D model. The trained machine learning models 1435 may then be used by autonomous machines to perform various tasks such as object recognition, automated inventory processing, navigation, among other examples.

[0108] In the example of Figure 14, a computing system 1415 implementing the exemplary training set generator 1420 can include one or more data processing devices 1445, one or more computer-readable memory elements 1450, and logic implemented in hardware and / or software to implement the training set generator 1420. Although the example of Figure 14 illustrates the computing system 1415 (and its components) as separate from the machine learning system 1430 used to train and / or execute the machine learning model (e.g., 1435), it should be understood that in some implementations, a single computing system can be used to implement the combined functions of two or more of the model generator 1405, the training set generator 1420, and the machine learning system 1430, among other alternative implementations and exemplary architectures.

[0109] In some implementations, a computing system may be provided that allows one-time learning using synthetic training data. Such a system may allow objects to be classified without the need to train hundreds of thousands of images. One-time learning allows classification from very few training images, even a single training image in some cases. This saves time and resources in developing a training set for training a particular machine learning model. In some implementations, such as the examples shown in Figures 16 and 17, the machine learning model may be a neural network that learns to distinguish between two inputs, rather than a model that learns to classify inputs. The output of such a machine learning model may identify the similarity of two inputs provided to the model.

[0110] In some implementations, a machine learning system may be provided that can simulate the ability to classify object categories from a few training examples, such as in the example of FIG. 14. Such a system may also eliminate the need to create large datasets of multiple classes to effectively train the corresponding machine learning model. Similarly, a machine learning model that does not require training on multiple classes may be selected. A machine learning model may be used to recognize an object (e.g., a product, human, animal, or other object) by feeding the model a single image of the object along with a comparison image to the system. If the comparison photo is not recognized by the system, the object is determined to be unmatched using a machine learning model (e.g., a Siamese network).

[0111] In some implementations, a Siamese network can be utilized as a machine learning model trained using synthetic training data such as those introduced in the examples above. For example, FIG. 16 shows a simplified block diagram 1600 illustrating an exemplary Siamese network composed of two identical neural networks 1605a, 1605b. Each of these networks has the same weights after training. A comparison block (e.g., 1620) can be provided to evaluate the similarity of the outputs of the two identical networks and compare the determined similarity to a threshold. If the similarity is within a threshold range (e.g., below or above a given threshold), the output of the Siamese network (composed of the neural networks 1605a, 1605b and the comparison block 1620) can indicate (e.g., at 1625) whether the two inputs refer to a common subject. For example, two samples 1610, 1615 (e.g., images, point clouds, depth images, etc.) can be provided as respective inputs to each of the two identical neural networks 1605a, 1605b. In one example, the neural networks 205a, 1605b may be implemented as ResNet-based networks (e.g., ResNet50 or another variant), and the output of each network may be a feature vector input to the comparison block 1620. In some implementations, the comparison block may generate a similarity vector from the two feature vector inputs to indicate how similar the two inputs 1610, 1615 are. In some implementations, the inputs (e.g., 1610, 1615) to such a Siamese network implementation may constitute two images representing the agent's current observations (e.g., images or depth maps generated by sensors of an autonomous machine) and a target object. A deep Siamese network is a type of two-stream neural network model for discriminative embedded learning, and may enable one-time learning using synthetic training data. For example, at least one of the two inputs (e.g., 1610, 1615) may be a synthetically generated training image or reference image as described above.

[0112] In some implementations, the execution of the Siamese network or other machine learning model trained using synthetic data can utilize dedicated machine learning hardware, such as a machine learning accelerator (e.g., Intel Movidius Neural Compute Stick (NCS)), which can interface with a general-purpose microcomputer, among other example implementations. This system can be used in a variety of applications. For example, the network can be used in security or authentication applications, such as applications in which a human, animal, or vehicle is recognized before an actuator that allows access to the human, animal, or vehicle can be triggered. As a specific example, a smart door can be equipped with an image sensor to recognize a human or animal approaching the door, and access can be granted (using a machine learning model) only to those that match one of a set of authorized users. Such machine learning models (e.g., trained on synthetic data) can also be used in industrial or commercial applications, such as product verification, inventory, and other applications in which product recognition in a store is used to determine whether (or how many) a product is present or whether (or how many) a product is present in a particular location (e.g., on the appropriate shelf), among other examples. For example, as shown in the example illustrated by simplified block diagram 1700 in FIG. 17, two sample images 1705, 1710 relating to consumer products may be provided as inputs to a Siamese network having threshold decision logic 1720. The Siamese network model 1720 may determine (at 1715) whether the two sample images 1705, 1710 are likely to be images of the same product. Indeed, in some implementations, such a Siamese network model 1720 may enable identification of products at various rotations and occlusions. In some cases, 3D models may be utilized to generate multiple reference images for more complex products and objects to increase the level of validation, among other example considerations and features.

[0113] In some implementations, the computing system may comprise logic and hardware adapted to perform machine learning tasks to perform point cloud registration or merging of two or more separate point clouds. Performing point cloud merging requires finding a transformation that aligns the contents of the point clouds. Such problems are common in applications involving autonomous machines, such as robotic perception applications, building maps of unknown environments, among other use cases.

[0114] In some implementations, the use of convolutional networks as a solution for finding relative poses between 2D images can provide results comparable to traditional feature-based approaches. Advances in 3D scanning technology can further create multiple data sets with 3D data useful for training neural networks. In some implementations, a machine learning model can be provided that can accept two or more different streams of inputs. Each of the two or more data inputs embodies a respective three-dimensional (3D) point cloud. The two 3D point clouds can be representations of the same physical space or object (or a virtualized version of the physical space or object) measured from two different respective poses. The machine learning model can accept these two 3D point cloud inputs and generate as output a measure of the relative or absolute pose between the two 3D point cloud sources. The relative pose information can then be used to generate a global 3D point cloud representation of the environment from multiple snapshots (of the 3D point clouds) of the environment (from one or more different sensors and devices (e.g., multiple drones or the same drone moving to scan the environment). Relative pose can also be used to determine the relative position of a particular machine within an environment by comparing 3D point cloud input measured by a particular machine with a previously generated global 3D point cloud representation of the environment, among other example uses.

[0115] In one example, a voxelized point cloud processing technique is used that creates a 3D grid for sorting points. Here, convolutional layers may be applied as shown in the example of FIG. 18. In some implementations, the 3D grid or point cloud may be embodied or represented as a voxel-based data structure as described herein. For example, in the example of FIG. 18, a voxel-based data structure (represented by 1810) may be generated from a point cloud 1805 generated from an RGB-D camera or LIDAR scan of an exemplary 3D environment. In some implementations, two point cloud inputs that may be provided to a comparative machine learning model, such as one employing a Siamese network, may be a pair of 3D voxel grids (e.g., 1810). In some cases, the two inputs may first be voxelized (converted to a voxel-based data structure (e.g., 1810)) at any one of multiple potential voxel resolutions as described above.

[0116] In one example depicted in the simplified block diagram 1900 of FIG. 19, the machine learning model may be composed of a representation part 1920 and a regression part 1925. As described above, the Siamese network-based machine learning model may be configured to directly estimate the relative camera pose from a pair of 3D voxel grid inputs (e.g., 1905, 1910) or other point cloud data. In some implementations, the 3D voxel grid may be advantageously used in a neural network with a conventional convolutional layer, given the organized structure of the voxel grid data. The relative camera pose determined using the network may be used to merge the corresponding point clouds of the voxel grid inputs.

[0117] In some implementations, the representation portion 1920 of the exemplary network may include a Siamese network with shared weights and biases. Each branch (or channel of the Siamese network) is formed by successive convolutional layers to extract a feature vector for each input 1905, 1910. Furthermore, in some implementations, a rectified linear unit (ReLU) may be provided as an activation function after each convolutional layer. In some cases, the pooling layer may be omitted to ensure that the spatial information of the data is preserved. The feature vectors output from the representation portion 1920 of the network may be combined and enter the regression portion 1925. The regression portion 1925 includes a set of fully connected layers that can generate an output 1930 representing the relative pose between the two input point clouds 1905, 1910. In some implementations, the regression portion 1925 may be composed of two sets of fully connected layers, one responsible for generating rotation values ​​for the pose estimate and a second set of layers responsible for generating translation values ​​for the pose. In some implementations, the fully connected layers of the regression portion 1925 may be followed by a ReLU activation function (except for the final layer, since the output may have negative values), among other example features and implementations.

[0118] In some implementations, such as in the example of FIG. 19 above, self-supervised learning may be performed on the machine learning model in a training phase. For example, since the goal of the network (shown in the example of FIG. 19) is to solve a regression problem, a loss function may be provided to guide the network to achieve that solution. A training phase may be provided to derive a loss function, such as a loss function based on labels or by quantifying the alignment of two point clouds. In one example, an exemplary training phase 2020 may be implemented as shown in simplified block diagram 2000 of FIG. 20. Here, input 2005 is provided to an iterative closest point (ICP) based method 2015 that is used (e.g., in conjunction with a corresponding CNN 2010) to obtain a database ground truth that is used to predict y poses, where a loss function 2025 is used to compare the network prediction. In such an example, it is not necessary to have a ground truth labeled dataset.

[0119] Trained Siamese network-based models such as those described in the examples of Figures 19 and 20 can be utilized in applications such as 3D map generation and navigation and localization. For example, as shown in the example of Figure 21, such a network can be utilized (in conjunction with machine learning hardware (e.g., NCS device 2110)) by a mobile robot (e.g., 2105) or other autonomous machine to assist in navigating within an environment. The network can also be used to generate a 3D map of the environment 2115 (e.g., which can then be used by the robot or autonomous machine), among other simultaneous localization and mapping (SLAM) applications.

[0120] In some implementations, edge-to-edge machine learning can be utilized to perform sensor fusion within applications. Such solutions can be applied to regress robot motion over time by fusing data from multiple different sensors. This is a well-studied problem, but current solutions drift over time or are computationally expensive. In some examples, machine learning approaches can be utilized in computer vision tasks while being less susceptible to noise in the data, changes in lighting, and subject blur, among other exemplary advantages. For example, convolutional neural networks (CNNs) can be used for object recognition and optical flow computation. System hardware running CNN-based models can employ hardware components and subsystems such as long short-term memory (LSTM) blocks, among other examples, to realize further efficiencies, such as better results for signal regression.

[0121] In one example, a system may be provided that utilizes a machine learning model that can accept input from multiple sources of different types of data (e.g., RGB data and IMU data, etc.) to independently overcome the weaknesses of each source (e.g., monocular RGB: lack of scale, IMU: drift over time, etc.). The machine learning module may include a respective neural network (or other machine learning model) that is tailored to the analysis of each type of data source. Each type of data source may be coupled and fed into a stage of a fully connected layer to generate results (e.g., poses) from the multiple data streams. Such a system may be used, for example, in a computing system intended to enable autonomous navigation of machines such as robots, drones, or vehicles, among other example applications.

[0122] For example, as shown in the example of FIG. 22, IMU data may be provided as input to a network tuned for IMU data. IMU data can provide a way to track a subject's movement by measuring acceleration and orientation. In some cases, however, using IMU data alone in a machine learning application may result in drift over time. In some implementations, using an LSTM to track the relationships of this data over time can help reduce drift. In one example shown in the simplified block diagram 2200 of FIG. 22, a subsequence of n raw accelerometer and gyroscope data elements 2205 (e.g., where each data element consists of six values ​​(three axes each from the accelerometer and gyroscope of the IMU) is used as input to an example LSTM 2210. In another example, the input 2205 may be n (e.g., 10) relative poses between image frames (e.g., frame f i and f i+1 n IMU relative attitudes between

number

[0123] Referring to the example of Figure 23, a simplified block diagram 2300 of a network that can process a data stream of image data, such as monocular RGB data, is shown. Thus, an RGB CNN portion can be provided that is trained to compute optical flow and can feature dimensionality reduction for pose estimation. For example, several fully connected layers can be provided to reduce dimensionality, and / or feature vectors can be reshaped as matrices, and a set of four LSTMs can be used to find correspondences between features and reduce dimensionality, among other examples.

[0124] In the example of FIG. 23, a pre-trained optical flow CNN (e.g., 2310), such as FlowNetSimple, FlowNetCorr, Guided Optical Flow, VINet, or other optical network, may be provided to accept a pair of consecutive RGB images as input 2305. The model may be further constructed to extract a feature vector from the image pair 2305 through the optical flow CNN 2310 and then reduce the vector to obtain a pose vector corresponding to the input 2305. For example, the output of the optical network portion 2310 may be provided to a set of one or more further convolutional layers 2315 (e.g., utilized to reduce the dimensionality from the output of the optical network portion 2310 and / or to remove information used for estimating the flow vector but not required for pose estimation), and the output of these convolutional layers 2315 may be a matrix that may be flattened (at 2320) into a corresponding vector. The vector may be provided to a fully connected layer 2325 to perform a dimensionality reduction of the flattened vector (e.g., from 1536 to 512). This reduced vector may be fed back to a reshape block 2330 to convert or reshape the vector into a matrix. A set of four LSTMs 2335, one for each direction of the reshaped matrix (e.g., left-to-right / top-to-bottom, right-to-left / bottom-to-top, top-to-bottom / left-to-right, and bottom-to-top / right-to-left), may then be used to track feature correspondences over time and reduce dimensionality. The output of the set of LSTMs 2335 may then be provided to a fully connected layer for rotation 2340 to generate rotation values ​​2350 based on the pair of images 2305, and to a fully connected layer for translation 2345 to generate translation values ​​2355 based on the pair of images 2305, among other example implementations.

[0125] Referring to FIG. 24, a simplified block diagram 2400 of a sensor fusion network 2405 is shown. The sensor fusion network 2405 concatenates the results of an IMU neural network portion 2410 (e.g., as shown in the example of FIG. 22) and an RGB neural network portion 2415 (e.g., as shown in the example of FIG. 23). Such a machine learning model 2405 may further enable sensor fusion by choosing the best from each sensor type. For example, the machine learning model may combine CNN and LSTM to provide more robust results (e.g., CNN may extract features from a pair of consecutive images, and LSTM may obtain information about the gradual movement of the sensor). In this respect, the outputs of both CNN and LSTM are complementary, giving the machine an accurate estimate of the difference (and its relative transformation) between two consecutive frames and its representation in real-world units.

[0126] In the example of FIG. 24, among other examples, the results of each sensor-specific portion (e.g., 2405, 2410) can be concatenated and provided to a fully connected layer of a combined machine learning model 2405 (or sensor fusion network) to generate a pose result incorporating both rotational and translational pose. While IMU data and monocular RGB individually may not appear to provide enough information for a reliable solution to the regression problem, combining these data inputs as illustrated and described herein may result in more robust and reliable results (e.g., as shown in the example results shown in graphs 2500a, b of FIG. 25A and FIG. 25B). Such a network 2405 leverages the useful information of both sensor types (e.g., RGB and IMU). For example, in this particular example, the RGB CNN portion 2415 of the network 2405 can extract information about the relative translation between successive images, while the IMU LSTM-based portion 2410 provides the scale to the translation. The respective feature vectors output by each portion 2410, 2415 may be fed to a concatenation block 2420 to concatenate the vectors based on a combination of the RGB image 2305 and the IMU data 2205 and feed the results to a core fully connected layer 2425, which then feeds the results to both a fully connected rotation layer 2430 to generate rotation values ​​2440 and a fully connected translation layer 2435 to generate translation values ​​2445. While the examples of Figures 22-24 show a fusion of RGB and IMU data, it should be understood that other data types may be substituted (e.g., replacing and supplementing IMU data with GPS data) and combined in machine learning models according to the principles described herein. Indeed, more than two data streams (and corresponding neural network portions fed to the concatenation) may be provided in other exemplary implementations to enable a more robust solution, among other exemplary modifications and alternatives.

[0127] In some implementations, a neural network optimizer may be provided that can identify to a user or system one or more neural networks recommended for a particular application and hardware platform that uses a neural network to perform machine learning tasks. For example, as shown in FIG. 26, a computing system 2605 can be provided that includes a microprocessor 2610 and computer memory 2615. The computing system 2605 can implement a neural network optimizer 2620. The neural network optimizer 2620 can include an execution engine that causes a set of machine learning tasks (e.g., 2635) to be performed using machine learning hardware (e.g., 2625) as described herein, among other example hardware. Neural network optimizer 2620 may further include one or more probes (e.g., 2630) for monitoring the execution of a machine learning task as it is performed by machine learning hardware 2625 using one of a set of neural networks selected by neural network optimizer 2620 for execution on and monitored by neural network optimizer 2620. Probes 2630 may measure attributes such as the power consumed by machine learning hardware 2625 during execution of the task, the temperature of the machine learning hardware while it is running, the speed at which or the time elapsed to complete a task using a particular neural network, the accuracy of the task results using a particular neural network, the amount of memory utilized (e.g., to store the neural network being used), among other example parameters.

[0128] In some implementations, the computing system 2605 can interface with a neural network generation system (e.g., 2640). In some implementations, the computing system (e.g., 2605) that evaluates the neural network and the neural network generation system 2640 can be implemented on the same computing system. The neural network generation system 2640 can allow a user to manually design neural network models (e.g., CNNs) for various tasks and solutions. In some implementations, the neural network generation system 2640 can further include a repository 2645 of previously generated neural networks. In one example, the neural network generation system 2640 (e.g., a system such as CAFFE, TensorFlow, etc.) can generate a set of neural networks 2650. The set can be randomly generated, new neural networks can be generated from scratch (e.g., based on some generalized parameters suitable for a given application or according to a type or class of common neural networks) and / or neural networks can be randomly selected from the repository 2645.

[0129] In some implementations, a set of neural networks 2650 may be generated by the neural network generation system 2640 and provided to the neural network optimizer 2620. The neural network optimizer 2620 may use each of the set of neural networks 2650 to cause a particular machine learning hardware (e.g., 2625) to perform a standardized set of one or more machine learning tasks. The neural network optimizer 2620 may monitor the performance of the tasks in association with each of the use of the set of neural networks 2650 by the hardware 2625. The neural network optimizer 2620 may further accept data as input to identify which parameters or characteristics, as measured by the neural network optimizer's probes (e.g., 2630), are most highly weighted or prioritized by the neural network optimizer in determining the "best" of the set of neural networks. Based on these criteria and the neural network optimizer's observations during use of each of a set of (e.g., randomly generated) neural networks, the neural network optimizer 2620 can identify and provide the best performing neural network to a particular machine learning hardware (e.g., 2625) based on the provided criteria. In some implementations, the neural network optimizer can automatically provide this best performing neural network to the hardware for further use, training, etc.

[0130] In some implementations, the neural network optimizer can employ evolutionary search to iteratively improve results identified from an initial (e.g., randomly generated) set of neural networks evaluated by the neural network optimizer (e.g., 2620). For example, the neural network optimizer can identify characteristics of one or more best performing neural networks from the initial set evaluated by the neural network optimizer. The neural network optimizer can then send a request to the neural network generator (e.g., 2640) to generate another diverse set of neural networks having similar characteristics as identified in the best performing neural network for the particular hardware (e.g., 2625). The neural network optimizer 2620 can then repeat its evaluation using a next set or generation of neural networks generated by the neural network generator based on the best performing neural networks from the initial batch evaluated by the neural network optimizer. As before, the neural network optimizer 2620 can identify which of the second generation neural networks performed best according to the provided criteria, and can again determine the characteristics of the best performing neural network in the second generation as a basis for sending a request to the neural network generator to generate a third generation neural network for evaluation. Thus, using the neural network optimizer 2620, neural networks are iteratively evaluated as they evolve (and theoretically improve) from one generation to the next. As in the previous example, the neural network optimizer 2620 can provide an indication or copy of the best performing neural network of the latest generation for use by the machine learning hardware (e.g., 2625), among other example implementations.

[0131] As a specific example shown in block diagram 2700 of FIG. 27, machine learning hardware 2625 such as Movidius NCS may be utilized with a neural network optimizer that acts as a design space exploration tool, using a neural network generator 2640 or a provider such as CAFFE to find the network with the highest accuracy, subject to the hardware constraints. Such a design space exploration (DSX) tool may be provided to leverage a complete API including network graphs and bandwidth measurements. Furthermore, several extensions may be added or provided to the machine learning hardware API to derive additional parameters useful for design space exploration, such as temperature measurements, inference time measurements, among other examples.

[0132] To illustrate the power of the DSX concept, an example is provided that explores the neural network design space of a small, always-on face detector, such as those implemented in modern mobile phones that wake up on face detection. Various neural networks may be provided to the machine learning hardware, and performance may be monitored for each use of the neural network, such as the power usage of the trained network in the inference stage. The DSX tool (or neural network optimizer) may generate different neural networks for a given classification task. Data may be transferred to the hardware (e.g., in the case of the NCS, it may be transferred to and from the NCS using the NCS API via USB). In the implementation described above, rather than manually editing, copying, and pasting any files, the design space exploration may result in finding the best model for multiple different purposes. As an illustration, FIG. 28 shows a table 2800 illustrating exemplary results of the DSX tool's evaluation of multiple different randomly generated neural networks. The results include performance characteristics of the machine learning hardware (e.g., a general-purpose microprocessor connected to the NCS) during the execution of the machine learning task using each one of the neural networks (e.g., showing accuracy, execution time, temperature, size of the neural network in memory, and measured power, among other example parameters that may be measured by a DSX tool). FIG. 29 shows the results of comparing validation accuracy to execution time (2900) and validation accuracy to size (2905). These relationships and ratios, among other examples, may be considered by the NCS when determining which of the evaluated neural networks is "best" for a particular machine learning platform.

[0133] Deep neural networks (DNNs) provide state-of-the-art accuracy on a variety of computer vision tasks, such as image classification and object detection. However, the success of DNNs is often achieved by a significant increase in computation and memory, making them difficult to deploy on resource-constrained inference edge devices. In some implementations, network compression techniques such as pruning and quantization can reduce computation and memory demands. This can also help prevent overfitting with little or no loss in accuracy, especially in the case of transfer learning on small custom datasets.

[0134] In some implementations, a neural network optimizer (e.g., 2620) or other tools may also be provided to dynamically and automatically reduce the size of the neural network used by a particular machine learning hardware. For example, the neural network optimizer may perform fine-grained pruning (e.g., pruning of connections or weights) and coarse-grained pruning (e.g., pruning of kernels, neurons, or channels) to reduce the size of the neural network stored and operated by a given machine learning hardware. In some implementations, the machine learning hardware (e.g., 2625) may include computational circuitry capable of performing sparse matrix multiplication so that the hardware can efficiently process the weight-pruned neural network.

[0135] In one implementation, the neural network optimizer 2620 or other tool may perform hybrid pruning of the neural network (e.g., as shown in block diagram 3000a of FIG. 30A) to prune at both the kernel level and the weight level. For example, one or more algorithms, rules, or parameters may be considered by the neural network optimizer 2620 or other tool to automatically identify (at 3005) a set of kernels or channels that may be pruned 3010 from a given neural network. After this first channel pruning phase is completed (at 3015), weight pruning 3020 may be performed on the remaining channels 3015, as shown in the example diagram of FIG. 30A. For example, rules may govern the pruning of weights such that a threshold is set such that weights below the threshold are pruned (at 3025) (e.g., reassigned a weight of "0"). It may then be possible to run or iterate on the hybrid pruned network to recover the accuracy of the network from pruning, resulting in a compact version of the network without detrimentally reducing the accuracy of the model. Additionally, in some implementations, the weights remaining after pruning may be quantized to further reduce the amount of memory required to store the pruned model weights. For example, as shown in block diagram 3100 of FIG. 31, a log-scale quantization 3110 may be performed such that floating-point weight values ​​(3105) are replaced with their nearest base-2 counterparts (3115). In this manner, 32-bit floating-point values ​​may be replaced with 4-bit base-2 values, dramatically reducing the amount of memory required to store the network weights while only minimally losing accuracy of the compact neural network (e.g., as shown in the example results shown in table 3200 of FIG. 32), among other example quantizations and features. Indeed, in the specific example of FIG. 32, a study of the application of hybrid pruning is shown as applied to an exemplary neural network such as ResNet50.Additionally, we present the application of weight quantization to a pruned sparse and thin ResNet50 to further reduce the model size and make it more hardware friendly.

[0136] As shown in simplified block diagram 3000b of FIG. 30B, in one example, hybrid pruning of an example neural network may be performed by accessing an initial or baseline neural network model 3035 and (optionally) training the model with regularization (L1, L2, or L0) 3040. The importance of individual neurons (or connections) in the network may be evaluated 3045, and neurons determined to be less important may be pruned (at 3050) from the network. The pruned network may be fine-tuned 3055, and a final compact (or sparse) network may be generated 3060 from this pruning. As shown in the example of FIG. 30B, in some cases, the network may be iteratively pruned by performing further training and pruning (e.g., 3040-3050) followed by fine-tuning 3055 of the pruned network, among other example implementations. In some implementations, determining the importance of neurons 3045 may be performed utilizing hybrid pruning techniques as described above. For example, fine-grained weight pruning / sparsification may be performed (e.g., by global gradual pruning with (mean+std*factor)). Coarse-grained channel pruning may be performed layer-by-layer (e.g., weight sum pruning) based on sensitivity tests and / or some target MAC. Coarse pruning may be performed before sparse pruning. For example, weight quantization may be performed to set constraints on non-zero weights to 0 or powers of 2 and / or to use 1 bit for 0 and 4 bits to represent weights. In some cases, low-precision (e.g., weights and activations) quantization may be performed, among other example techniques. Pruning techniques such as those described above may provide various example benefits. For example, compact matrices may reduce the size of stored network parameters, and decompression of weights in run-time may reduce DDR bandwidth. Accelerated computation may be provided, among other example advantages.

[0137] FIG. 33A is a simplified flow diagram 3300a of an exemplary technique for generating a training dataset including synthetic training data samples (e.g., synthetically generated images or synthetically generated point clouds). For example, a digital 3D model can be accessed 3302 from a computer memory, and multiple training samples can be generated 3304 from various views of the digital 3D model. These training samples can be modified 3306 to add defects to the training samples to simulate training samples generated by one or more real-world samples. A training dataset is generated 3308 to include the modified synthetically generated training samples. The generated training dataset can be used to train 3310 one or more neural networks.

[0138] FIG. 33B is a simplified flow diagram 3300b of an exemplary technique for performing one-time classification using a Siamese neural network model. A subject input can be provided 3312 as an input to a first part of a Siamese neural network model, and a reference input can be provided 3314 as an input to a second part of the Siamese neural network model. The first and second parts of the model can be identical and can have identical weights. An output of the Siamese network can be generated 3316 from the output of the first and second parts based on the subject input and the reference input, such as a difference vector. For example, based on a similarity threshold for the output of the Siamese neural network model, it can be determined 3318 that the output indicates whether the subject input is appropriately similar to the reference input (e.g., to indicate that the subject of the subject input is the same as the subject of the reference input).

[0139] 33C is a simplified flow diagram 3300c of an exemplary technique for determining a relative pose using an exemplary Siamese neural network model. For example, a first input can be received 3320 as an input to a first portion of a Siamese neural network model, the first input representing a view of a 3D space (e.g., point cloud data, depth map data, etc.) from a first pose (e.g., of an autonomous machine). A second input can be received 3322 as an input to a second portion of the Siamese neural network model, the second input representing a view of the 3D space from a second pose. An output of the Siamese network can be generated 3324 based on the first input and the second input, the output representing a relative pose between the first pose and the second pose. Based on the determined relative pose, a position of the machine associated with the first pose and the second pose and / or a 3D map in which the machine is present can be determined 3326.

[0140] FIG. 33D is a simplified flow diagram 3300d of an example technique including a sensor fusion machine learning model. The sensor fusion machine learning model combines at least a portion of two or more machine learning models that are adapted for use with corresponding ones of two or more different data types. A first sensor data of a first type can be received 3330 as an input in a first one of the two or more machine learning models in the sensor fusion machine learning model. A second sensor data of a second type (e.g., generated simultaneously with the first sensor data (e.g., by a sensor on the same or different machine)) can be received 3332 as an input to a second one of the two or more machine learning models. The outputs of the first machine learning model and the second machine learning model can be concatenated 3334, and the concatenated output is provided 3336 to a set of fully connected layers of the sensor fusion machine learning model. The sensor fusion machine learning model can generate an output based on the first sensor data and the second sensor data to define 3338 a pose of a device (e.g., a machine on which the sensors generating the first sensor data and the second sensor data are located).

[0141] FIG. 33E is a simplified flow diagram 3300e of an exemplary technique for generating improved or optimized neural networks tailored to specific machine learning hardware according to an evolutionary algorithm. For example, a set of neural networks can be accessed 3340 or generated (e.g., automatically according to randomly selected attributes). A particular piece of hardware can perform machine learning tasks using the set of neural networks 3342, and attributes of the performance of the particular piece of hardware for these tasks can be monitored 3344. Based on the results of this monitoring, one or more best performing neural networks in the set can be identified 3346. Characteristics of the best performing neural networks (for the particular piece of hardware) can be determined 3348, and another set of neural networks can be generated 3350 that includes such characteristics. In some cases, this new set of neural networks can be tested (e.g., through steps 3342-3348) to iteratively improve the set of neural networks being considered for use with the hardware until one or more sufficiently well-performing or optimized neural networks are identified for the particular piece of hardware.

[0142] 33F is a simplified flow diagram 3300f of an exemplary technique for pruning a neural network. For example, a neural network can be identified 3352, and a subset of the kernels of the neural network can be determined 3354 as less important or otherwise good pruning candidates. This subset of kernels can be pruned 3356 to generate a pruned version of the neural network. The remaining kernels can then be further pruned 3358 to prune a subset of weights from these remaining kernels to further prune the neural network at both the coarse-grained and fine-grained levels.

[0143] 34 is a simplified block diagram of an exemplary multi-slot vector processor (e.g., a very long instruction word (VLIW) vector processor) according to some embodiments. In this example, the vector processor may include multiple (e.g., nine) functional units (e.g., 3403-3411), which may be fed by a multi-port memory system 3400 backed by a vector register file (VRF) 3401 and a general purpose register file (GRF) 3402. The processor includes an instruction decoder (IDEC) 3412 that decodes instructions and generates control signals to control the functional units 3403-3411. The functional units 3403-3411 are a predicated execution unit (PEU) 3403, a branch and repeat unit (BRU) 3404, a load store port unit (e.g., LSU0 3405 and LSU1 3406), a vector arithmetic unit (VAU) 3407, a scalar arithmetic unit (SAU) 3410, a compare and move unit (CMU) 3408, an integer arithmetic unit (IAU) 3411, and a volumetric acceleration unit (VXU) 3409. In this particular implementation, the VXU 3409 can accelerate operations on volumetric data, including both arithmetic, logical, and storage / retrieval operations. Although the example of FIG. 34 shows the VXU circuit 3409 as a single component, it should be understood that the functionality of the VXU (and the other functional units 3403-3411) may be distributed across multiple circuits. Further, in some implementations, the functionality of VXU 3409 may be distributed within one or more of the other functional units of the processor (e.g., 3403-3408, 3410, 3411), among other example implementations.

[0144] 35 is a simplified block diagram illustrating an example implementation of a VXU 3500 according to some embodiments. For example, the VXU 3500 may provide at least one 64-bit input port 3501 to accept input from a vector register file 3401 or a general purpose register file 3402. This input may be connected to multiple functional units including register file 3503, address generator 3504, point addressing logic 3505, point insertion logic 3506, point deletion logic 3507, 3D to 2D projection logic in X dimension 3508, 3D to 2D projection logic in Y dimension 3509, 3D to 2D projection logic in Z dimension 3510, 2D histogram pyramid generator 3511, 3D histopyramid generator 3512, population counter 3513, 2D pathfinding logic 3514, 3D pathfinding logic 3515, and possibly further functional units for operating on 64-bit unsigned integer volumetric bitmaps. Output from block 3502 may be written back to the vector register file VRF 3401 or general purpose register file GRF 3402 register file.

[0145] Referring to the example of FIG. 36, a representation of the organization of a 4^3 voxel cube 3600 is shown. A second voxel cube 3601 is also shown. In this example, the voxel cube can be defined in the data as a 64-bit integer 3602, where each and every voxel in the cube is represented by a single corresponding bit of the 64-bit integer. For example, voxel 3602 at address {x, y, z}={3, 0, 3} can be set to "1" to indicate that geometry exists at that coordinate in the volumetric space represented by voxel cube 3601. Furthermore, in this example, all other voxels (other than voxel 3602) may correspond to "empty" space and can be set to "0" to indicate, among other examples, that there is no physical geometry at those coordinates. Referring to FIG. 37, an exemplary two-level sparse voxel tree 3700 according to some embodiments is shown. In this example, only a single "occupied" voxel is contained within the volume (e.g., at location {15, 0, 15}). The top level -0 3701 of the tree in this case contains a single voxel entry {3, 0, 3}. That voxel then points to the next level 3702 of the tree, which contains a single voxel in element {3, 0, 3}. The entry in the data structure corresponding to level 0 of the sparse voxel tree is a 64-bit integer 3703 with one voxel set as occupied. Setting the voxel means that an array of 64-bit integers is then assigned to level 1 of the tree, which corresponds to the voxel volume set in 3703. In the level 1 subarray 3704, only one of the voxels is set as occupied, and all other voxels are set as unoccupied. Since the tree is a two-level tree in this example, level 1 represents the bottom of the tree, and so the hierarchy ends here.

[0146] FIG. 38 illustrates a two-level sparse voxel tree 3800 according to some embodiments, including occupied voxels at locations {15, 0, 3} and {15, 0, 15} for a particular volume. The top level-0 of the tree 3801 in this case (subdividing the particular volume into 64 top level-0 voxels) includes two voxel entries {3, 0, 0} and {3, 0, 3}, and the corresponding data 3804 indicates that the two voxels are set (or occupied). The next level of the sparse voxel tree (SVT) is provided as an array of 64-bit integers including two subcubes 3802 and 3803, one for each voxel set at level 0. In the level 1 subarray 3805, two voxels v15 and v63 are set as occupied, and all other voxels are set as unoccupied and tree. This format is flexible. Because the 64 entries in the next level of the tree are always allocated corresponding to each configured voxel in the upper layer of the tree, this flexibility may allow dynamically changing scene geometry to be inserted into existing volumetric data structures in a flexible manner (i.e., not in a fixed order such as random) as long as the corresponding voxels in the upper layer are configured. Otherwise, a table of pointers would have to be maintained, resulting in high memory requirements, or the tree would have to be at least partially restructured to insert unexpected geometry.

[0147] FIG. 39 illustrates an alternative technique for storing voxels from FIG. 38, according to some embodiments. In this example, the entire volume 3900 includes two voxels stored at global coordinates {15, 0, 3} and {15, 0, 15} as in FIG. 23. In this approach, rather than allocating a 64-entry array to represent all of the level 1 subcubes below level 0, only those level 1 elements that actually contain geometry (e.g., as indicated by whether the corresponding level 0 voxel is occupied) are allocated as corresponding 64-bit level 1 records. As a result, level 1 in this example only has two 64-bit entries (i.e., for each of the 64 level-1 voxels, whether occupied or empty), rather than 64 64-bit entries. Thus, in this example, the first level 0 3904 is equivalent to 3804 in FIG. 38, but the next level 3905 has 62 times smaller memory requirements than its corresponding 3805 in FIG. 38. In some implementations, when inserting new geometry into level 0 that has not had space allocated in level 1, the tree must be copied and rearranged.

[0148] In the example of FIG. 39, the subvolumes can be derived by counting occupied voxels in layers above the current layer. In this way, the system can determine where in the voxel data one higher layer ends and the next lower layer begins. For example, if three layer 0 voxels are occupied, the system can predict that three corresponding layer 1 entries follow in the voxel data, and that the next entry (after these three) corresponds to the first entry in layer 2, etc. Such optimal compression can be very useful when certain parts of a scene do not change over time, or when an application requires remote transmission of volumetric data, for example from a space probe scanning the surface of Pluto, where every bit is costly and time consuming to transmit.

[0149] FIG. 40 illustrates how voxels may be inserted into a 4^3 cube represented as a 64-bit integer volumetric data structure entry to reflect changes to geometry within the corresponding volume, according to some embodiments. In one example, as shown in 4000, each voxel cube may be organized as four logical 16-bit planes within a 64-bit integer. Among other example organization, each of the planes corresponds to a Z value from 0 to 3, within each plane, each y value encodes four logical 4-bit displacements from 0 to 3, and finally, within each 4-bit y plane, each bit encodes four possible x values ​​from 0 to 3. Thus, in this example, to insert a voxel into the 4^3 volume, first one bit may be shifted by the x value 0 to 3, then that value may be shifted by 0 / 4 / 8 / 12 bits to encode the y value, and finally, the z value may be represented by a shift of 0 / 16 / 32 / 48 bits, as shown in the C code expression of 4001. Finally, because each 64-bit integer may be a combination of up to 64 voxels, and each of these voxels is written separately, the new bitmap must be logically combined with the old 64-bit value read from the sparse voxel tree by ORing the old and new bitmap values, as shown at 4002.

[0150] Referring to FIG. 41, a representation is shown to illustrate how a 3D volumetric object stored in a 64-bit integer 4100 may be projected by logically ORing in the X direction to generate a 2D pattern 4101, in the Y direction to generate a 2D output 4102, and finally in the Z direction to generate a pattern shown at 4103, according to some embodiments. FIG. 42 illustrates how bits from an input 64-bit integer may be logically ORed to generate an output projection in X, Y, and Z, according to some embodiments. In this example, table 4201 indicates, column by column, which element indexes from input vector 4200 are ORed to generate X-projected output vector 4202. Table 4203 indicates, column by column, which element indexes from input vector 4200 are ORed to generate Y-projected output vector 4204. Finally, 4205 indicates, column by column, which element indexes from input vector 4200 are ORed to generate Z-projected output vector 4206.

[0151] The X projection logically ORs bits 0, 1, 2, and 3 from the input data 4200 to generate bit 0 of the X projection 4201. For example, bit 1 of 4201 may be generated by ORing together bits 4, 5, 6, and 7, etc. from 4200. Similarly, bit 0 of the Y projection 4204 may be generated by ORing together bits 0, 4, 8, and 12 of 4200. Bit 1 of 4204 may be generated by ORing together bits 1, 5, 9, and 13, etc. of 4200. Finally, bit 0 of the Z projection 4206 may be generated by ORing together bits 0, 16, 32, and 48 of 4200. Bit 1 of 4206 may be generated by ORing together bits 1, 17, 33, and 49, etc. of 4200.

[0152] FIG. 43 shows an example of how a simplified map may be generated using projections, according to some embodiments. The goal in this scenario may be to generate a compact 2D map of a path taken by a vehicle 4300 of height h 4310 and width w 4301 from a voxel volume 4302. Here, Y projection logic may be used to generate an initial coarse 2D map 4303 from the voxel volume 4302. In some implementations, the map may be processed to check whether a particular vehicle of particular dimensions (e.g., a car (or autonomous vehicle), drone, etc.) can pass through the width constraints 4301 and height constraints 4310 of the path. This may be done to ensure that the path is traversable by performing a projection in Z to check the width constraint 4301, and the projection in Y may be masked to limit the calculation to the vehicle height 4310. With further post-processing (e.g., in software), it can be seen that for a route that is traversable and satisfies width and height constraints, only the X and Z coordinates of points A4304, B4305, C4306, D4307, E4308, and F4309 along the route can be stored or transmitted over the network to completely reconstruct a legal route that a vehicle can travel. Assuming that the route can be decomposed into such piecewise segments, a route can be completely described with only one or two bytes per piecewise linear section of the route. This can be useful for rapid transmission and processing of such route data (e.g., by autonomous vehicles), among other examples.

[0153] FIG. 44 illustrates how volumetric 3D measurements or simple 2D measurements from embedded devices may be aggregated according to some embodiments by mathematical means to generate a crowdsourced high-quality map instead of using LIDAR or other expensive means to make accurate measurements. In the proposed system, multiple embedded devices 4400, 4401, etc. may be equipped with various sensors capable of making measurements that may be transmitted to a central server 4410. Software running on the server performs the aggregation 4402 of all of the measurements and performs a numerical solution by a nonlinear solver 4403 of the resulting matrix to generate a high-precision map. This map may then be redistributed to the embedded devices. In fact, the data aggregation may also include high-precision survey data from satellites 4420, airborne LIDAR surveys 4421, and terrestrial LIDAR measurements 4422 to increase the accuracy of the resulting map when these high-fidelity datasets are available. In some implementations, the maps and / or recorded measurements may be generated in, converted to, or otherwise represented using a sparse voxel data structure in a format as described herein, among other example implementations.

[0154] FIG. 45 illustrates how 2D pathfinding on a 2D 2×2 bitmap may be accelerated, according to some embodiments. The principle of operation is that for there to be connectivity between points on the map in the same grid cell, the values ​​of the x or y cells, or consecutive runs of x and y cells, must all be set to 1. Thus, a logical AND of the bits drawn from those cells can be instantiated to test the bitmap in the grid to see if a valid path exists, and a different AND gate can be instantiated for each valid path through the N×N grid. In some cases, this approach can even result in 2D paths being found on an 8×8 2D grid. 64-1 valid paths. Hence, in some improved implementations, the grid may be reduced to 2x2 or 4x4 tiles where connectivity testing can be done hierarchically. 2x2 bitmap 4500 contains 4 bits labeled b0, b1, b2, and b3. These 4 bits can take values ​​0000 to 1111 with corresponding labels 4501 to 4517. Each of these bit patterns represents different levels of connectivity between the faces of the 2x2 grid labeled 4521 through 4530. For example, if 2x2 grid 4500 contains bitmaps 1010 (7112), 1011 (7113), 1110 (7116), or 1111 (7117), then there will be 4521 or v0 representing vertical connectivity between x0 and y0 of 4500. The two-input logical AND or b0 and b3 of 4500, as shown in row 1 of table 4518, produces a connectivity map v0 that can be used in higher level hardware or software to determine global connectivity through a global grid subdivided into 2x2 subgrids. If the global map contains an odd number of grid points on the x-axis or y-axis, the top level grid needs to be padded to the next higher even grid point (e.g., resulting in the need to add one additional row of zeros to the x-axis and / or y-axis on the global grid). FIG. 45 further shows an exemplary 7x7 grid 4550. This grid shows how to pad to 8x8 by adding an additional row 4532 and column 4534 filled with zeros. In this example, the NxN map 4550 can be progressively subsampled down to a 2x2 map to speed up pathfinding compared to other techniques (e.g., depth-first search, breadth-first search or Dijkstra's algorithm, or other graph-based approaches). For example, in this example, cell W of 4540 is populated by ORing the contents of cells A, B, C, and D, etc., of 4550. Similarly, the bits of the 2×2 cells of 4540 are ORed together to populate the cell of 4542. For pathfinding, the algorithm starts with the smallest 2×2 representation of grid 4542 and tests each of the bits.Only the portion of the 4×4 grid of 4540 (composed of four 2×2 grids) that corresponds to a 1 bit in the 2×2 grid 4542 needs to be tested for connectivity, since it can be seen that a 0 bit means that there is no corresponding 2×2 grid cell in 4540. This approach can also be used to search the 8×8 grid of 4520. For example, if cell W of 4540 contains a 0, it can be seen that there is no path in ABCD, such as 4520. This approach prunes branches from the graph search algorithm used, whether it be A*, Dijkstra, DFS, BFS, or variations thereof. In addition to this, the use of a 2×2 organization of hardware elementary path finder 4518 can further limit the computations involved. In fact, the 4×4 elementary hardware element may be constructed using five 2×2 hardware blocks in the same arrangement as 4540 and 4542, further constraining the amount of graph searching that needs to be performed. Furthermore, an 8x8 hardware based search engine can be constructed with 21 2x2 HW blocks (7118) having the same arrangement as the 4542, 4540, 4500, etc., potentially in any NxN topology.

[0155] 46 is a simplified block diagram showing how the proposed volumetric data structure may be used to accelerate collision detection, according to some embodiments. A 3D NxNxN map of geometry may be subsampled into a pyramid consisting of the lowest level of detail (LoD) 2x2x2 volume 4602, the next higher 4x4x4 volume 4601, 8x8x8 volume 4600, etc., up to NxNxN. If the location of the drone, vehicle, or robot 4605 in 3D space is known, either via a positioning means such as GPS, or via re-localization from the 3D map, that location may be quickly used to test whether geometry exists in the relevant quadrant of the 2x2x2 sub-volume by appropriately scaling the drone / robot's x, y, and z positions (dividing them by the relevant number of times 2) and querying 4602 for the presence of geometry (e.g., checking whether the corresponding bitmap bit is indicative of a possible collision). If a potential collision exists (e.g. a "1" is found), further checks on volumes 4601, 4600, etc. can be performed to see if the drone / robot can move there. However, if the voxel in 4602 is empty (e.g. it is a "0") then the robot / drone can interpret it the same as free space and operate the directional controls to move freely through most of the map.

[0156] Although some of the systems and solutions described and illustrated herein are described as including or associated with multiple elements, not all elements explicitly illustrated or described may be utilized in every alternative implementation of the disclosure. Furthermore, one or more of the elements described herein may be located external to the system, while in other examples, a particular element may be included within or as part of one or more of the other elements described and other elements not described in the illustrated implementation. Furthermore, a particular element may be combined with other components and used for alternative or additional purposes in addition to those described herein.

[0157] Further, it should be understood that the examples presented above are merely non-limiting examples provided for the purpose of illustrating certain principles and features, and do not necessarily limit or restrict potential implementations of the concepts described herein. For example, a variety of different embodiments can be realized using various combinations of the features and components described herein, including combinations realized through various implementations of the components described herein. Other implementations, features, and details should be apparent from the teachings of this specification.

[0158] 47-52 are block diagrams of exemplary computer architectures that may be used in accordance with the embodiments disclosed herein. Indeed, the computing devices, processors, and other logic and circuitry of the systems described herein may incorporate all or a portion of the functionality and supporting software and / or hardware circuitry for implementing such functionality. Moreover, other computer architecture designs known in the art for processors and computing systems may also be used beyond the examples shown here. In general, computer architectures suitable for the embodiments disclosed herein may include, but are not limited to, the configurations shown in FIGS. 47-52.

[0159] FIG. 47 illustrates an example domain topology of respective Internet of Things (IoT) networks coupled via links to respective gateways. Internet of Things (IoT) is a concept where a large number of computing devices are interconnected with each other and to the Internet to provide functionality and data acquisition at a very low level. Thus, as used herein, IoT devices can include, among other things, semi-autonomous devices that communicate with other IoT devices and a broader network such as the Internet to perform functions such as sensing or control. Such IoT devices can be equipped with logic and memory to implement and use hash tables as introduced above.

[0160] IoT devices often have limited memory, size, or functionality so that more can be deployed at a similar cost to fewer, larger devices. However, IoT devices may be smartphones, laptops, tablets, or PCs, or other larger devices. Additionally, IoT devices may be virtual devices, such as applications on a smartphone or other computing device. IoT devices may include IoT gateways, used for data storage, process control, and the like, to couple IoT devices to other IoT devices and cloud applications.

[0161] A network of IoT devices can include water distribution systems, power distribution systems, pipeline control systems, plant control systems, commercial and home automation devices such as light switches, thermostats, locks, cameras, alarms, and motion sensors, etc. IoT devices can be accessed through remote computers, servers, and other systems, for example to control the systems or access data.

[0162] The future growth of the Internet and similar networks may involve a very large number of IoT devices. Thus, in the context of the technology described herein, several innovations for such future networking address the need for all these layers to grow unhindered, to support the ability to discover and create accessible connected resources, and to hide and compartmentalize connected resources. Any number of network protocols and communication standards can be used, with each protocol and standard designed to address a specific purpose. Furthermore, protocols are part of the mechanisms that support services accessible by humans that operate regardless of location, time, or space. Innovations include service provisioning and associated infrastructure such as hardware and software, enhanced security, and the provision of services based on service levels and quality of service (QoS) conditions specified in service provision agreements. As will be appreciated, the use of IoT devices and networks such as those introduced in Figures 47 and 48 presents several new challenges in heterogeneous networks of connectivity that include a combination of wired and wireless technologies.

[0163] Specifically, FIG. 47 provides a simplified diagram of a domain topology that may be used for several Internet of Things (IoT) networks including an IoT device 4704. The IoT networks 4756, 4758, 4760, 4762 are coupled to respective gateways 4754 via backbone links 4702. For example, several IoT devices 4704 may communicate with the gateways 4754 and with each other via the gateways 4754. To simplify the drawing, not all IoT devices 4704 or communication links (e.g., links 4716, 4722, 4728, or 4732) have been labeled. The backbone links 4702 may include any number of wired or wireless technologies, including optical networks, and may be part of a local area network (LAN), a wide area network (WAN), or the Internet. Furthermore, such communication links facilitate optical signal paths between both the IoT devices 4704 and the gateways 4754, including the use of MUX / deMUX components to facilitate interconnection of various devices.

[0164] The network topology may include any number of types of IoT networks, such as a mesh network comprising a network 4756 using Bluetooth® Low Energy (BLE) links 4722. Other types of IoT networks that may be present include a wireless local area network (WLAN) network 4758 used to communicate with IoT devices 4704 via IEEE 802.11 (Wi-Fi®) links 4728, a cellular network 4760 used to communicate with IoT devices 4704 via an LTE / LTE-A (4G) or 5G cellular network, and a low power wide area (LPWA) network 4762, such as an LPWA network compatible with the LoRaWan specification promulgated by the LoRa Alliance, or an IPv6 over Low Power Wide-Area Network (LPWAN) network compatible with specifications promulgated by the Internet Engineering Task Force (IETF). Additionally, each IoT network may communicate with external network providers (e.g., Tier 2 or Tier 3 providers) using any number of communication links, such as LTE cellular links, LPWA links, or links based on the IEEE 802.15.4 standard, such as Zigbee. Each IoT network may also operate using a variety of network and Internet application protocols, such as the Constrained Application Protocol (CoAP). Each IoT network may also be integrated with a coordinator device that provides a chain of links that form a cluster tree of linked devices and networks.

[0165] Each of these IoT networks can provide opportunities for new technical features such as those described herein. Improved technologies and networks can enable exponential growth of devices and networks, including using IoT networks as fog devices or systems. As the use of such improved technologies increases, IoT networks can be developed for self-management, functionality evolution, and collaboration without the need for direct human intervention. Improved technologies may even enable IoT networks to function without a centralized system. Thus, the improved technologies described herein can be used to automate and enhance network management and operational capabilities well beyond current implementations.

[0166] In one example, communication between IoT devices 4704, such as over backbone links 4702, may be secured by a distributed system for authentication, authorization, and accounting (AAA). In a distributed AAA system, distributed payment, credit, audit, authorization, and authentication systems may be implemented across interconnected heterogeneous network infrastructures. This may move systems and networks to autonomous operations. In these types of autonomous operations, machines may contract human resources and even negotiate partnerships with other machine networks. This may enable not only the realization of mutual objectives and balanced service delivery against outlined and planned service level agreements, but also solutions that provide metering, measurement, traceability, and traceability. Creating new supply chain structures and methods may enable the creation of numerous services, mining value, and disruption without any human involvement whatsoever.

[0167] Such IoT networks can be further enhanced by integrating sensing technologies such as sound, light, electronic traffic, face and pattern recognition, smell, vibration, etc. into autonomous organization among IoT devices. The integration of sensory systems can enable systematic and autonomous communication and service delivery coordination, orchestration, and Quality of Service (QoS) based resource swarming and fusion for contracted service objectives. Some of the individual examples of network-based resource processing include:

[0168] For example, the mesh network 4756 may be augmented with systems that perform in-line data-to-information conversion. A self-forming chain of processing resources, including, for example, a multi-link network, can decentralize the efficient conversion of raw data to information, as well as the ability to distinguish between assets and resources and the associated management of each. Furthermore, appropriate components of infrastructure and resource-based trust and service indexes can be inserted to improve data integrity, quality, and authenticity, and provide metrics of data reliability.

[0169] For example, the WLAN network 4758 may provide multi-standard connectivity using a system that performs standards conversion, allowing IoT devices 4704 using different protocols to communicate. Further systems can provide seamless interconnectivity across a multi-standard infrastructure, including visible and hidden Internet resources.

[0170] For example, communications in the cellular network 4760 may be enhanced by systems that offload data, extend communications to more remote devices, or both. The LPWA network 4762 may include systems that perform non-Internet Protocol (IP) to IP interconnection, addressing, and routing. Additionally, each of the IoT devices 4704 may include a transceiver suitable for wide area communications with that device. Additionally, each IoT device 4704 may include other transceivers for communications using additional protocols and frequencies. This is further described in connection with the communications environment and hardware of the IoT processing device shown in Figures 49 and 50.

[0171] Finally, a cluster of IoT devices may be equipped to communicate with other IoT devices as well as with a cloud network. This may enable the IoT devices to form ad-hoc networks between themselves and to function as a single device, which may be referred to as a fog device. This configuration is further described in connection with FIG. 48 below.

[0172] Figure 48 illustrates a cloud computing network in communication with a mesh network of IoT devices (devices 4802) operating as fog devices at the edge of the cloud computing network. The mesh network of IoT devices may be referred to as fog 4820 operating at the edge of the cloud 4800. To simplify the illustration, not all of the IoT devices 4802 are labeled.

[0173] The fog 4820 may be considered to be a large-scale interconnected network in which several IoT devices 4802 communicate with each other, for example, by wireless links 4822. By way of example, this interconnected network may be facilitated using the interconnection specifications released by the Open Connectivity Foundation™ (OCF). This standard allows devices to discover each other and establish communications for interconnection. Other interconnection protocols may also be used, including, for example, the Optimized Link State Routing (OLSR) protocol, the Better Approach to Mobile Ad-Hoc Networking (BATMAN) routing protocol, or the OMA Lightweight M2M (LWM2M) protocol, among others.

[0174] In this example, three types of IoT devices 4802 are shown: a gateway 4804, a data aggregator 4826, and a sensor 4828, however, any combination of IoT devices 4802 and functions may be used. The gateway 4804 may be an edge device providing communication between the cloud 4800 and the fog 4820 and may also provide back-end processing functions for data obtained from the sensors 4828, such as operational data, flow data, and temperature data. The data aggregator 4826 may collect data from any number of sensors 4828 and perform back-end processing functions for analysis. Results, raw data, or both may be passed to the cloud 4800 via the gateway 4804. The sensor 4828 may be a full IoT device 4802 that can both collect data and process data, for example. In some cases, the sensor 4828 may be more limited in functionality, for example, collecting data and allowing the data to be processed by the data aggregator 4826 or the gateway 4804.

[0175] Communications from any IoT device 4802 can be passed along a convenient path (e.g., the most convenient path) between any of the IoT devices 4802 to reach the gateway 4804. In these networks, the number of interconnections provides substantial redundancy, allowing communications to be maintained even if some IoT devices 4802 are lost. Furthermore, the use of mesh networks may enable the use of IoT devices 4802 that are very low power or located far from the infrastructure, since the range of connecting to another IoT device 4802 may be much smaller than the range of connecting to a gateway 4804.

[0176] The fog 4820 provided from these IoT devices 4802 may be presented to devices in the cloud 4800, such as server 4806, as a single device, e.g., a fog device, located at the edge of the cloud 4800. In this example, alerts coming from the fog device may be sent without being identified as coming from a particular IoT device 4802 in the fog 4820. In this manner, the fog 4820 may be considered a distributed platform that provides computing and storage resources for performing processing or data-intensive tasks, such as data analysis, data aggregation, and machine learning, among others.

[0177] In some examples, the IoT devices 4802 may be configured using an imperative programming style, e.g., each IoT device 4802 has specific capabilities and communication partners. However, the IoT devices 4802 forming fog devices may be configured in a declarative programming style, allowing the IoT devices 4802 to reconfigure their operation and communication, e.g., determine required resources in response to conditions, queries, and device failures. As an example, a query from a user located at the server 4806 regarding the operation of a subset of equipment monitored by the IoT devices 4802 may cause the fog 4820 device to select IoT devices 4802, such as specific sensors 4828, required to respond to the query. Data from these sensors 4828 may then be aggregated and analyzed by any combination of sensors 4828, data aggregators 4826, or gateways 4804 before being transmitted by the fog 4820 device to the server 4806 to respond to the query. In this example, the IoT devices 4802 in the fog 4820 can select the sensors 4828 to be used based on the query, e.g., add data from a flow sensor or a temperature sensor. Furthermore, if some of the IoT devices 4802 are not operational, other IoT devices 4802 in the fog 4820 device can provide similar data if available.

[0178] In another example, the above operations and functions may be embodied by an IoT device machine in the exemplary form of an electronic processing system, in which a set or sequence of instructions may be executed to cause the electronic processing system to perform any one of the methods described herein, according to an exemplary embodiment. The machine may be an IoT device or IoT gateway, including a machine embodied in the form of a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile phone or a smart phone, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine may be depicted and referred to in the above examples, such a machine shall also be considered to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methods described herein. Furthermore, these and similar examples for processor-based systems shall also be considered to include any set of one or more machines controlled or operated by a processor (e.g., a computer) to individually or jointly execute instructions to perform any one or more of the methods described herein. In some implementations, one or more devices may work in concert to implement functions and perform tasks described herein. In some cases, one or more host devices may supply data, provide instructions, aggregate results, or otherwise facilitate joint operations and functions provided by multiple devices. When implemented by a single device, functions may be considered to be functions local to the device, but in implementations of multiple devices operating as a single machine, functions may be considered collectively local to the devices, and this collection of devices may provide or consume results provided by other remote machines (implemented as a single device or a collection of devices), among other example implementations.

[0179] For example, FIG. 49 shows a diagram of a cloud computing network or cloud 4900 in communication with several Internet of Things (IoT) devices. The cloud 4900 may represent the Internet or may be a wide area network (WAN) or local area network (LAN), such as a corporate proprietary network. The IoT devices may include any number of different types of devices grouped in various combinations. For example, a traffic control group 4906 may include IoT devices along city streets. These IoT devices may include stoplights, traffic flow monitors, cameras, weather sensors, and the like. The traffic control group 4906 or other subgroups may be in communication with the cloud 4900 via wired or wireless links 4908, such as LPWA links and optical links. Additionally, wired or wireless sub-networks 4912 may enable the IoT devices to communicate with each other, such as, for example, via local area networks and wireless local area networks. The IoT devices may communicate with a remote location, such as the cloud 4900, using another device, such as a gateway 4910 or 4928. The IoT devices may also use one or more servers 4930 to facilitate communication with the cloud 4900 or gateway 4910. For example, the one or more servers 4930 may operate as intermediate network nodes to support local edge cloud or fog implementations between local area networks. Additionally, the illustrated gateway 4928 may operate in a cloud-to-gateway-to-many edge device configuration, for example, with the various IoT devices 4914, 4920, 4924 being constrained or dynamic with respect to the allocation and usage of resources in the cloud 4900.

[0180] Other exemplary groups of IoT devices may include, among others, a remote weather station 4914, a local information terminal 4916, an alarm system 4918, a cash dispenser 4920, an alarm panel 4922, or a mobile vehicle such as an emergency vehicle 4924 or other vehicle 4926. Each of these IoT devices may be in communication with other IoT devices, a server 4904, another IoT fog device or system (not shown, but shown in FIG. 48), or a combination thereof. Groups of IoT devices may be deployed in a variety of residential, commercial, and industrial environments, including both private and public environments.

[0181] As can be seen in FIG. 49, a large number of IoT devices may be communicating through the cloud 4900. This may enable multiple different IoT devices to autonomously request or provide information to other devices. For example, a group of IoT devices (e.g., traffic control group 4906) may request a current weather forecast from a group of remote weather stations 4914, which may provide the forecast without human intervention. Additionally, an emergency vehicle 4924 may be alerted by a cash dispenser 4920 that a robbery is in progress. As the emergency vehicle 4924 proceeds toward the cash dispenser 4920, it may access the traffic control group 4906 to request clearance to the location. This clearance may be, for example, by lights turning red to block cross traffic at the intersection in time to allow the emergency vehicle 4924 unhindered access to the intersection.

[0182] A cluster of IoT devices, such as a remote weather station 4914 or a traffic control group 4906, may be equipped to communicate with other IoT devices as well as the cloud 4900. This may enable the IoT devices to form ad-hoc networks between themselves and function as a single device that may be referred to as a fog device or system (e.g., as described above with reference to FIG. 48).

[0183] FIG. 50 is a block diagram of example components that may be present in an IoT device 5050 to implement the techniques described herein. The IoT device 5050 may include any combination of components shown in the example or mentioned in the disclosure above. The components may be implemented as an IC, a portion thereof, a discrete electronic device, or other module, logic, hardware, software, firmware, or combination thereof that fits into the IoT device 5050, or as a component that is otherwise incorporated within the chassis of a larger system. Additionally, the block diagram of FIG. 50 is intended to illustrate a high level view of the components of the IoT device 5050. However, in other implementations, some of the components shown may be omitted, additional components may be present, and different arrangements of the components shown may be made.

[0184] The IoT device 5050 may include a processor 5052. The processor 5052 may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. The processor 5052 may be part of a system-on-a-chip (SoC) in which the processor 5052 and other components are formed within a single integrated circuit or within a single package, such as Intel's Edison™ or Galileo™ SoC substrate. By way of example, the processor 5052 may include an Intel® Architecture Core™ based processor, such as a Quark™, ​​Atom™, i3, i5, i7, or MCU class processor, or another such processor available from Intel® Corporation of Santa Clara, California. However, any number of other processors may be used, such as those available from Advanced Micro Devices, Inc. (AMD) of Sunnyvale, Calif., MIPS-based designs from MIPS Technologies, Inc. of Sunnyvale, Calif., ARM-based designs licensed from ARM Holdings, Ltd. or its customers or its licensees or adopters. Processors may include units such as the A5-A10 processors from Apple® Inc., the Snapdragon™ processors from Qualcomm® Technologies, Inc., or the OMAP™ processors from Texas Instruments, Inc.

[0185] The processor 5052 can communicate with the system memory 5054 via an interconnect 5056 (e.g., a bus). Any number of memory devices can be used to provide a given amount of system memory. By way of example, the memory can be random access memory (RAM) following a Joint Electron Devices Committee (JEDEC) design, such as DDR or mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In various implementations, the individual memory devices can be in any number of different package types, such as single die packages (SDP), dual die packages (DDP), or quad die packages (Q17P). In some examples, these devices can be soldered directly onto a motherboard to provide a low profile solution, while in other examples, the devices are configured as one or more memory modules that also couple to the motherboard at a given connector. Any number of other memory implementations can be used, such as other types of memory modules, such as different kinds of dual in-line memory modules (DIMMs), including, but not limited to, microDIMMs or MiniDIMMs.

[0186] Storage 5058 may also be coupled to processor 5052 via interconnect 5056 to provide persistent storage of information such as data, applications, and operating systems. In one example, storage 5058 may be implemented via a solid-state disk drive (SSDD). Other devices that may be used for storage 5058 include flash memory cards, such as SD cards, microSD cards, and xD picture cards, and USB flash drives. In low-power implementations, storage 5058 may be on-die memory or registers associated with processor 5052. However, in some examples, storage 5058 may be implemented using a micro hard disk drive (HDD). Furthermore, any number of new technologies may be used for storage 5058 in addition to or instead of the technologies described, such as resistive memory, phase change memory, holographic memory, or chemical memory, among others.

[0187] The components can communicate via an interconnect 5056. The interconnect 5056 can include any number of technologies including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies. The interconnect 5056 can be, for example, a proprietary bus used in SoC-based systems. Other bus systems can be included, such as an I2C interface, an SPI interface, a point-to-point interface, and a power bus, among others.

[0188] An interconnect 5056 can couple the processor 5052 to a mesh transceiver 5062 for communication with other mesh devices 5064. The mesh transceiver 5062 can use any number of frequencies and protocols, such as 2.4 Gigahertz (GHz) transmissions based on the IEEE 802.15.4 standard, using the Bluetooth® Low Energy (BLE) standard defined by the Bluetooth® Special Interest Group, or the ZigBee® standard, among others. Any number of radios configured for a particular wireless communication protocol can be used to connect to the mesh devices 5064. For example, a WLAN unit can be used to implement Wi-Fi™ communications according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, wireless wide area communications, for example via cellular or other wireless wide area protocols, can occur via the WWAN unit.

[0189] The mesh transceiver 5062 can communicate using multiple standards or radios for communication at different ranges. For example, the IoT device 5050 can save power by communicating with nearby devices, e.g., within about 10 meters, using a local transceiver based on BLE or another low power radio. More distant mesh devices 5064, e.g., within about 50 meters, can be reached via ZigBee or other medium power radio. Both communication techniques may be done via a single radio at different power levels or via separate transceivers, e.g., a local transceiver using BLE and a separate mesh transceiver using ZigBee.

[0190] A wireless network transceiver 5066 may be included to communicate with devices or services in the cloud 5000 via a local area network or wide area network protocol. The wireless network transceiver 5066 may be an LPWA transceiver according to the IEEE 802.15.4 or IEEE 802.15.4g standard, among others. The IoT device 5050 may communicate over a wide area using LoRaWAN™ (Long Range Wide Area Network) developed by Semtech and the LoRa Alliance. The techniques described herein may be used with any number of other cloud transceivers implementing long range, low bandwidth communications, such as, but not limited to, Sigfox, and other techniques. Additionally, other communication techniques may be used, such as time slotted channel hopping as described in the IEEE 802.15.4e specification.

[0191] As described herein, in addition to the systems mentioned for the mesh transceiver 5062 and the wireless network transceiver 5066, any number of other wireless communications and protocols may be used. For example, the wireless transceivers 5062 and 5066 may include LTE or other cellular transceivers that use spread spectrum (SPA / SAS) communications to implement high speed communications. Additionally, any number of other protocols may be used, such as a Wi-Fi network for providing network communications and medium speed communications.

[0192] The wireless transceivers 5062 and 5066 may include radios compatible with any number of 3GPP (3rd Generation Partnership Project) specifications, in particular Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), and Long Term Evolution Advanced Pro (LTE-A Pro). However, radios compatible with any number of other fixed, mobile, or satellite communication technologies and standards may be selected. These may include, for example, any cellular wide area wireless communication technology. The cellular wide area wireless communication technology may include, for example, the fifth generation (5G) communication system, the Global System for Mobile Communications (GSM) wireless communication technology, the General Packet Radio Service (GPRS) wireless communication technology, or the Enhanced Data Rates for GSM Evolution (EDGE) wireless communication technology, the Universal Mobile Telecommunications System (UMTS) communication technology. In addition to the above standards, any number of satellite uplink technologies may be used for the wireless network transceiver 5066. Satellite uplink technologies include, for example, radios conforming to standards issued by the ITU (International Telecommunications Union) or ETSI (European Telecommunications Standards Institute), among others, and thus the examples provided herein are understood to be applicable to a variety of other communication technologies, both existing and yet to be formulated.

[0193] A network interface controller (NIC) 5068 may be included to provide wired communications to the cloud 5000 or other devices, such as the mesh device 5064. The wired communications may provide an Ethernet connection or may be based on other types of networks, such as a controller area network (CAN), local interconnect network (LIN), DeviceNet, ControlNet, Data Highway+, PROFIBUS, or PROFINET, among others. Further NICs 5068 may be included to allow connection to a second network, e.g., a NIC 5068 providing communications to the cloud via Ethernet and a second NIC 5068 providing communications to other devices via another type of network.

[0194] The interconnect 5056 can couple the processor 5052 to an external interface 5070 used to connect external devices or subsystems. The external devices can include sensors 5072 such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, Global Positioning System (GPS) sensors, pressure sensors, barometric pressure sensors, etc. The external interface 5070 can further be used to connect the IoT device 5050 to actuators 5074 such as power switches, valve actuators, audible sound generators, visual warning devices, etc.

[0195] In some example options, various input / output (I / O) devices may be present in or connected to the IoT device 5050. For example, a display or other output device 5084 may be included to show information such as sensor readings or actuator positions. Input devices 5086 such as a touch screen or keypad may be included to accept input. The output devices 5084 may include any number of forms of auditory or visual indications. The indications may include simple visual outputs such as binary status indicators (e.g., LEDs) and multi-character visual outputs, or more complex outputs such as a display screen (e.g., LCD screen) where output such as text, graphics, and multimedia objects are generated or produced from the operation of the IoT device 5050.

[0196] The battery 5076 can provide power to the IoT device 5050, or in instances where the IoT device 5050 is mounted in a fixed location, can include a power supply that is coupled to a power grid. The battery 5076 can be a lithium-ion battery or a metal-air battery, such as a zinc-air battery, an aluminum-air battery, and a lithium-air battery.

[0197] The IoT device 5050 may include a battery monitor / charger 5078 to track the state of charge (SoCh) of the battery 5076. The battery monitor / charger 5078 may be used to monitor other parameters of the battery 5076 to provide failure predictions such as the state of health (SoH) and state of function (SoF) of the battery 5076. The battery monitor / charger 5078 may include a battery monitoring integrated circuit such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx family of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 5078 may communicate information on the battery 5076 to the processor 5052 via the interconnect 5056. The battery monitor / charger 5078 may also include an analog-to-digital (ADC) converter that allows the processor 5052 to directly monitor the voltage of the battery 5076 or the current drawn from the battery 5076. The battery parameters may be used to determine the actions that the IoT device 5050 can take, such as transmission frequency, mesh network operation, and sensing frequency.

[0198] To charge the battery 5076, a power generation block 5080, or other power supply device coupled to a power grid, may be coupled to a battery monitor / charger 5078. In some examples, the power generation block 5080 may be replaced with a wireless power receiver to obtain power wirelessly, for example, via a loop antenna in the IoT device 5050. The battery monitor / charger 5078 may include wireless battery charging circuitry, such as the LTC4020 chip from Linear Technologies of Milpitas, Calif., among others. The particular charging circuitry selected depends on the size of the battery 5076 and thus the current required. Charging may be performed using, among others, the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Alliance for Wireless Power.

[0199] Storage 5058 may include instructions 5082 in the form of software, firmware, or hardware commands for implementing the techniques described herein. Although such instructions 5082 are illustrated as code blocks contained in memory 5054 and storage 5058, it will be understood that any of the code blocks may be replaced by hardwired circuitry, such as incorporated into an application specific integrated circuit (ASIC).

[0200] In one example, the instructions 5082 provided via the memory 5054, storage 5058, or processor 5052 may be embodied as a non-transitory machine-readable medium 5060. The non-transitory machine-readable medium 5060 includes code that instructs the processor 5052 to perform electronic operations at the IoT device 5050. The processor 5052 can access the non-transitory machine-readable medium 5060 via the interconnect 5056. For example, the non-transitory machine-readable medium 5060 may be embodied by a device described in storage 5058 of FIG. 50 or may include a specific storage unit, such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 5060 may include instructions to direct the processor 5052 to perform a particular flow or sequence of actions, for example, as described in connection with the operational and functional flowcharts and block diagrams above.

[0201] FIG. 51 is an exemplary diagram of a processor according to an embodiment. The processor 5100 is an example of a type of hardware device that may be used in connection with the above implementations. The processor 5100 may be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or other device for executing code. Although only one processor 5100 is shown in FIG. 51, a processing element may alternatively include two or more of the processors 5100 shown in FIG. 51. The processor 5100 may be a single-threaded core. Alternatively, in at least one embodiment, the processor 5100 may be multi-threaded in that it may include two or more hardware thread contexts (or "logical processors") per core.

[0202] 51 also illustrates memory 5102 coupled to the processor 5100 according to an embodiment. The memory 5102 may be any of a wide variety of memories (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. Such memory elements may include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0203] The processor 5100 may execute any type of instruction associated with an algorithm, process, or operation described herein. Generally, the processor 5100 may transform elements or items (e.g., data) from one state or condition to another state or condition.

[0204] Code 5104, which may be one or more instructions executed by the processor 5100, may be stored in memory 5102, software, hardware, firmware, or any suitable combination thereof, or any other internal or external component, device, element, or object as needed based on specific needs. In one example, the processor 5100 may follow a program sequence of instructions indicated by code 5104. Each instruction enters the front-end logic 5106 and is processed by one or more decoders 5108. The decoder may generate as its output micro-operations, such as fixed-width micro-operations of a predefined format, or other instructions, micro-instructions, or control signals that reflect the original code instruction. The front-end logic 5106 also includes register renaming logic 5110 and scheduling logic 5112, which generally allocate resources and queue operations corresponding to the instructions for execution.

[0205] The processor 5100 may also include execution logic 5114 having a set of execution units 5116a, 5116b, 5116n, etc. Some embodiments may include several execution units specialized for a particular function or set of functions. Other embodiments may include only one execution unit, or one execution unit capable of performing a particular function. The execution logic 5114 performs operations specified by code instructions.

[0206] After completing execution of the operation specified by the code instruction, the back-end logic 5118 can retire the instruction of the code 5104. In one embodiment, the processor 5100 allows out-of-order execution but requires in-order retirement of instructions. The retirement logic 5120 can take a variety of known forms (e.g., a reorder buffer, etc.). In this manner, the processor 5100 is transformed during execution of the code 5104 with respect to at least the outputs generated by the decoder, the hardware registers and tables utilized by the register renaming logic 5110, and any registers (not shown) modified by the execution logic 5114.

[0207] Although not shown in Figure 51, a processing element may include other elements on a chip with the processor 5100. For example, a processing element may include memory control logic along with the processor 5100. A processing element may include I / O control logic and / or I / O control logic integrated with the memory control logic. A processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on a chip with the processor 5100.

[0208] FIG. 52 illustrates a computing system 5200 arranged in a point-to-point (PtP) configuration, according to an embodiment. In particular, FIG. 52 illustrates a system in which a processor, memory, and input / output devices are interconnected by several point-to-point interfaces. In general, one or more of the computing systems described herein may be configured in the same or similar manner as computing system 5200.

[0209] Processors 5270 and 5280 may also include integrated memory controller logic (MC) 5272 and 5282, respectively, to communicate with memory elements 5232 and 5234. In alternative embodiments, memory controller logic 5272 and 5282 may be discrete logic separate from processors 5270 and 5280. Memory elements 5232 and / or 5234 may store various data used by processors 5270 and 5280 in implementing the operations and functions outlined herein.

[0210] Processors 5270 and 5280 may be any type of processor, such as those described in connection with other figures. Processors 5270 and 5280 may exchange data over point-to-point interface 5250 using point-to-point (PtP) interface circuits 5278 and 5288, respectively. Processors 5270 and 5280 may exchange data with chipset 5290 over respective point-to-point interfaces 5252 and 5254 using point-to-point interface circuits 5276, 5286, 5294 and 5298, respectively. Chipset 5290 may also exchange data with high performance graphics circuit 5238 over high performance graphics interface 5239 using interface circuit 5292, which may be a PtP interface circuit. In alternative embodiments, any or all of the PtP links shown in FIG. 52 may be implemented as multi-drop buses rather than PtP links.

[0211] The chipset 5290 may be in communication with the bus 5220 via an interface circuit 5296. The bus 5220 may have one or more devices communicating therethrough, such as a bus bridge 5218 and an I / O device 5216. The bus bridge 5218 may be in communication with other devices via the bus 5210, such as a user interface 5212 (such as a keyboard, mouse, touch screen, or other input device), a communication device 5226 (such as a modem, network interface device, or other type of communication device that may communicate over a computer network 5260), an audio I / O device 5214, and / or a data storage device 5228. The data storage device 5228 may store code 5230 that may be executed by the processor 5270 and / or 5280. In alternative embodiments, any portion of the bus architecture may be implemented with one or more PtP links.

[0212] The computer system illustrated in Figure 52 is a schematic diagram of an embodiment of a computing system that may be utilized to implement various embodiments described herein. It will be appreciated that the various components of the system illustrated in Figure 52 may be combined in a system-on-chip (SoC) architecture or in any other suitable configuration capable of implementing the functionality and features of the examples and implementations provided herein.

[0213] In a further example, a machine-readable medium may include any tangible medium capable of storing, encoding, or carrying instructions for execution by a machine and causing a machine to perform any one or more of the methods of the present disclosure, or capable of storing, encoding, or carrying data structures used or associated with such instructions. Thus, a "machine-readable medium" may include, but is not limited to, solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media include, by way of example and without limitation, non-volatile memory including semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Instructions embodied by a machine-readable medium may further be transmitted or received over a communications network using a transmission medium via a network interface device utilizing any one of a number of transfer protocols (e.g., HTTP).

[0214] It should be understood that functional units or functions described herein may be referred to or labeled as components or modules to more specifically emphasize their implementation independence. Such components may be embodied in any number of software or hardware forms. For example, a component or module may be implemented as a hardware circuit including custom Very Large Scale Integration (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A component or module may be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, or programmable logic device. A component or module may be implemented in software for execution by various types of processors. An identified component or module of executable code may include, for example, one or more physical or logical blocks of computer instructions, which may be organized as, for example, an object, procedure, or function. Nevertheless, executable files of an identified component or module need not be physically located together, but may include entirely different instructions stored in different locations that, when logically coupled together, include the component or module and achieve the stated purpose of the component or module.

[0215] Indeed, an executable code component or module may be a single instruction or many instructions, and may even be distributed across several different code segments, among different programs, and across several memory devices or processing systems. In particular, some aspects of the described process (such as rewriting the code and analyzing the code) may occur in a different processing system (e.g., a computer in a data center) than the system in which the code is deployed (e.g., a computer embedded in a sensor or robot). Similarly, operational data may be identified and illustrated herein in components or modules, and may be embodied in any suitable form and organized in any suitable type of data structure. The operational data may be collected as a single data set, may be distributed in different locations, including different storage devices, or may exist, at least in part, simply as electronic signals on a system or network. The components or modules may be passive or active, including agents operable to perform a desired function.

[0216] Further examples of embodiments of the methods, systems, and devices described herein include the following non-limiting configurations, each of which may stand alone or may be combined in any permutation or combination with any one or more of the other examples provided below or throughout this disclosure.

[0217] While the present disclosure has been described in terms of particular implementations and generally associated methods, modifications and permutations of these implementations and methods will be apparent to those of ordinary skill in the art. For example, the actions described herein may be performed in an order different from that described and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. Additionally, other user interface layouts and functionality may be supported. Other variations are within the scope of the following claims.

[0218] Although the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as describing features specific to particular embodiments of a particular invention. Certain features described in the specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as functioning in a particular combination, and even initially claimed as such, in some cases one or more features of a claimed combination may be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0219] Similarly, although the figures depict operations in a particular order, this should not be understood as requiring such operations to be performed in the particular order or sequence depicted, nor as requiring all of the operations depicted, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments. It should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0220] The following examples relate to embodiments according to the present specification. Example 1 is a method comprising: accessing a synthetic three-dimensional (3D) graphical model of an object from a memory, the 3D graphical model having a photorealistic resolution; generating a plurality of different training samples from views of the 3D graphical model, the plurality of training samples being generated to simulate characteristics of real-world samples generated by a real-world sensor device by adding imperfections to the plurality of training samples; and generating a training set including the plurality of training samples, the training data being used to train an artificial neural network.

[0221] Example 2 includes the subject matter of example 1, where the plurality of training samples includes digital images and the sensor device includes a camera sensor.

[0222] Example 3 includes the subject matter of any one of examples 1 and 2, where the plurality of training samples includes a point cloud representation of an object.

[0223] Example 4 includes the subject matter of example 3, wherein the sensor device includes a LIDAR sensor.

[0224] Example 5 includes the subject matter of any one of examples 1 to 4. The subject matter further includes accessing data indicative of a parameter of the sensor device and determining, based on the parameter, defects to add to the plurality of training samples.

[0225] Example 6 includes the subject matter of example 5, where the data includes a model of the sensor device.

[0226] Example 7 includes the subject matter of any one of Examples 1 to 6. The subject matter further includes accessing data indicative of a characteristic of one or more surfaces of the object modeled by the 3D graphical model, and determining defects to add to the plurality of training samples based on the characteristic.

[0227] Example 8 includes the subject matter of example 7, where the 3D graphical model includes data.

[0228] Example 9 includes the subject matter of any one of examples 1 to 8, wherein the defects include one or more of noise or glare.

[0229] Example 10 includes the subject matter of any one of Examples 1-9, where generating the plurality of different training samples includes applying a plurality of different lighting settings to the 3D graphical model to simulate lighting in the environment, and determining defects for a subset of the plurality of training samples generated during application of a particular one of the plurality of different lighting settings, where the defects for the subset of the plurality of training samples are based on the particular lighting setting.

[0230] Example 11 includes the subject matter of any one of Examples 1-10, where generating the plurality of different training samples includes placing the 3D graphical model in a plurality of different graphical environments, the graphical environments modeling respective real-world environments, and generating a subset of the plurality of training samples while the 3D graphical model is placed in the plurality of different graphical environments.

[0231] Example 12 is a system including means for carrying out the method according to any one of examples 1 to 11.

[0232] Example 13 includes the subject matter of Example 12, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 1 to 11.

[0233] Example 14 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 1 to 11.

[0234] Example 15 is a method comprising: receiving a subject input and a reference input at a Siamese neural network, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, the weights of the first network portion being identical to the weights of the second network portion, the subject input being provided as an input to the first network portion and the reference input being provided as an input to the second network portion; and generating an output of the Siamese neural network based on the subject input and the reference input, the output of the Siamese neural network indicating a similarity between the reference input and the subject input.

[0235] Example 16 includes the subject matter of example 15, where generating the output includes determining a difference amount between the reference input and the subject input and determining whether the difference amount meets a threshold, the output identifying whether the difference amount meets the threshold.

[0236] Example 17 includes the subject matter of example 16, where determining a difference between the reference input and the subject input includes receiving a first feature vector output by the first network portion and a second feature vector output by the second network portion, and determining a difference vector based on the first feature vector and the second feature vector.

[0237] Example 18 includes the subject matter of any one of examples 15 to 17, wherein generating an output includes a one-time classification.

[0238] Example 19 includes the subject matter of any one of examples 15 to 18. The subject matter further includes training a Siamese neural network using the one or more synthetic training samples.

[0239] Example 20 includes the subject matter of example 19, where the one or more synthetic training samples are generated according to the method of any one of examples 1-11.

[0240] Example 21 includes the subject matter of any one of examples 15 to 20, wherein the reference input includes synthetically generated samples.

[0241] Example 22 includes the subject matter of Example 21, wherein the synthetically produced sample is produced according to the method of any one of Examples 1-11.

[0242] Example 23 includes the subject matter of any one of examples 15 to 22, where the subject input includes a first digital image and the reference input includes a second digital image.

[0243] Example 24 includes the subject matter of any one of examples 15 to 22, where the subject input includes a first point cloud representation and the reference input includes a second point cloud representation.

[0244] Example 25 is a system including means for performing the method according to any one of examples 15 to 24.

[0245] Example 26 includes the subject matter of Example 25, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 15 to 24.

[0246] Example 27 includes the subject matter of example 25, where the system includes one of a robot, a drone, or an autonomous vehicle.

[0247] Example 28 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 15 to 24.

[0248] Example 29 is a method comprising: providing first input data to a Siamese neural network, the first input data including a first representation of 3D space from a first pose; providing second input data to the Siamese neural network, the second input data including a second representation of 3D space from a second pose, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, weights of the first network portion being identical to weights of the second network portion, the first input data being provided as an input to the first network portion and the second input data being provided as an input to the second network portion; and generating an output of the Siamese neural network, the output including a relative pose between the first pose and the second pose.

[0249] Example 30 includes the subject matter of example 29, where the first representation of the 3D space includes a first 3D point cloud and the second representation of the 3D space includes a second 3D point cloud.

[0250] Example 31 includes the subject matter of any one of examples 29-30, where the first representation of the 3D space includes a first cloud of points and the second representation of the 3D space includes a second cloud of points.

[0251] Example 32 includes the subject matter of example 31, where the first point cloud and the second point cloud each include a respective voxelized point cloud representation.

[0252] Example 33 includes the subject matter of any one of examples 29 to 32. The subject matter further includes generating a 3D mapping of the 3D space from at least the first input data and the second input data based on the relative pose.

[0253] Example 34 includes the subject matter of any one of examples 29 to 32. The subject matter further includes determining a position of an observer in the first pose in 3D space based on the relative pose.

[0254] Example 35 includes the subject matter of example 34, wherein the observer includes an autonomous machine.

[0255] Example 36 includes the subject matter of example 35, wherein the autonomous machine includes one of a robot, a drone, or an autonomous vehicle.

[0256] Example 37 is a system including means for performing the method according to any one of examples 29 to 36.

[0257] Example 38 includes the subject matter of Example 37, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 29 to 36.

[0258] Example 39 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 29 to 36.

[0259] Example 40 is a method comprising: providing first sensor data as an input to a first portion of a machine learning model; providing second sensor data as an input to a second portion of the machine learning model, the machine learning model including a connector and a set of fully connected layers, the first sensor data being of a first type generated by the device and the second sensor data being of a different second type generated by the device, the connector receiving an output of the first portion of the machine learning model as a first input and receiving an output of the second portion of the machine learning model as a second input, the output of the connector being provided to the set of fully connected layers; and generating an output of the machine learning model from the first data and the second data, the output including a pose of the device within an environment.

[0260] Example 41 includes the subject matter of example 40, wherein the first sensor data includes image data and the second sensor data identifies a movement of the device.

[0261] Example 42 includes the subject matter of example 41, wherein the image data includes red-green-blue (RGB) data.

[0262] Example 43 includes the subject matter of example 41, wherein the image data includes 3D point cloud data.

[0263] Example 44 includes the subject matter of example 41, wherein the second sensor data includes inertial measurement unit (IMU) data.

[0264] Example 45 includes the subject matter of example 41, wherein the second sensor data includes global positioning data.

[0265] Example 46 includes the subject matter of any one of examples 40 to 45, where a first portion of the machine learning model is tuned for a first type of sensor data and a second portion of the machine learning model is tuned for a second type of sensor data.

[0266] Example 47 includes the subject matter of any one of examples 40 to 46. The subject matter further includes providing a third sensor data of a third type as an input to a third portion of the machine learning model, wherein an output is further generated based on the third data.

[0267] Example 48 includes the subject matter of any one of examples 40 to 47, wherein the pose output includes a rotational component and a translational component.

[0268] Example 49 includes the subject matter of example 48, where one of the set of fully connected layers includes a fully connected layer for determining a rotation component, and another of the set of fully connected layers includes a fully connected layer for determining a translation component.

[0269] Example 50 includes the subject matter of any one of examples 40 to 49, wherein one or both of the first and second portions of the machine learning model include respective convolutional layers.

[0270] Example 51 includes the subject matter of any one of examples 40 to 50, where one or both of the first and second portions of the machine learning model include one or more respective long short-term memory (LSTM) blocks.

[0271] Example 52 includes the subject matter of any one of examples 40 to 51, where the device includes an autonomous machine, and the autonomous machine navigates within an environment based on the pose.

[0272] Example 53 includes the subject matter of example 52, wherein the autonomous machine includes one of a robot, a drone, or an autonomous vehicle.

[0273] Example 54 is a system including means for performing the method according to any one of examples 40 to 52.

[0274] Example 55 includes the subject matter of example 54, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of examples 40 to 52.

[0275] Example 56 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 40 to 52.

[0276] Example 57 is a method comprising: requesting random generation of a set of neural networks; performing a machine learning task using each one of the set of neural networks, where the machine learning task is performed using particular processing hardware; monitoring attributes of the performance of the machine learning task for each of the set of neural networks, where the attributes include accuracy of a result of the machine learning task; and identifying a best performing neural network of the set of neural networks when used to perform the machine learning task using the particular processing hardware based on the attributes of the best performing neural network.

[0277] Example 58 includes the subject matter of example 57. The subject matter further includes providing a top performing neural network for use by the machine in executing the machine learning application.

[0278] Example 59 includes the subject matter of any one of examples 57-58. The subject matter further includes determining characteristics of the top performing neural network, requesting generation of a second set of neural networks according to the characteristics, the second set of neural networks including a plurality of different neural networks, each including one or more of the characteristics, performing a machine learning task using each one of the second set of neural networks, the machine learning task being performed using particular processing hardware, monitoring attributes of the performance of the machine learning task for each of the second set of neural networks, and identifying a top performing neural network of the second set of neural networks based on the attributes.

[0279] Example 60 includes the subject matter of any one of examples 57-59. The subject matter further includes receiving a criterion based on the parameters, wherein the best performing neural network is based on the criterion.

[0280] Example 61 includes the subject matter of any one of examples 57 to 60, wherein the attributes include attributes of specific processing hardware.

[0281] Example 62 includes the subject matter of Example 61, where the attributes of the particular processing hardware include one or more of: power consumed by the particular processing hardware during execution of the machine learning task, a temperature of the particular processing hardware during execution of the machine learning task, and a memory used to store the neural network on the particular processing hardware.

[0282] Example 63 includes the subject matter of any one of examples 57-62, where the attributes include a time to complete the machine learning task using a corresponding one of the set of neural networks.

[0283] Example 64 is a system including means for performing the method according to any one of examples 57 to 63.

[0284] Example 65 includes the subject matter of example 64, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of examples 57 to 63.

[0285] Example 66 is a computer-readable storage medium storing instructions executable by a processor for performing the method of any one of Examples 57 to 63.

[0286] Example 67 is a method comprising: identifying a neural network including a plurality of kernels, each of the kernels including a respective set of weights; reducing the plurality of kernels to a particular set of kernels by pruning a subset of the plurality of kernels according to one or more parameters; and pruning a subset of weights in the particular set of kernels to form a pruned version of the neural network, wherein pruning the subset of weights assigns one or more non-zero weights in the subset of weights to zero, and the subset of weights is selected based on original values ​​of the weights.

[0287] Example 68 includes the subject matter of example 67, where the subset of weights is pruned based on values ​​of the subset of weights that are below a threshold.

[0288] Example 69 includes the subject matter of any one of Examples 67-68. The subject matter further includes performing one or more iterations of the machine learning task using a pruned version of the neural network to regain at least a portion of the accuracy lost by pruning the kernels and weights.

[0289] Example 70 includes the subject matter of any one of examples 67 to 69. The subject matter further includes quantizing values ​​of unpruned weights in the pruned version of the neural network to generate a compact version of the neural network.

[0290] Example 71 includes the subject matter of example 70, wherein the quantization includes log-based quantization.

[0291] Example 72 includes the subject matter of example 71, where the weights are quantized from floating point values ​​to base 2 values.

[0292] Example 73 includes the subject matter of any one of examples 67 to 72. The subject matter further includes providing a pruned version of the neural network for performing the machine learning task using hardware adapted for sparse matrix operations.

[0293] Example 74 is a system including means for performing the method according to any one of examples 67 to 73.

[0294] Example 75 includes the subject matter of example 64, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of examples 67 to 73.

[0295] Example 76 is a computer readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 67 to 73.

[0296] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results.

[0297] (Item 1) accessing a synthetic three-dimensional (3D) graphical model of the object from a memory, the 3D graphical model having a photorealistic resolution; generating a plurality of different training samples from views of the 3D graphical model, the plurality of training samples being generated by adding imperfections to the plurality of training samples to simulate characteristics of real-world samples generated by a real-world sensor device; generating a training set including the plurality of training samples, the training data being used to train an artificial neural network; A method for providing the above. (Item 2) 2. The method of claim 1, wherein the plurality of training samples includes digital images and the sensor device includes a camera sensor. (Item 3) 3. The method of any one of items 1 to 2, wherein the plurality of training samples comprises a point cloud representation of the object. (Item 4) 4. The method of claim 3, wherein the sensor device includes a LIDAR sensor. (Item 5) accessing data indicative of a parameter of the sensor device; determining the defects to add to the plurality of training samples based on the parameters; 5. The method according to any one of items 1 to 4, further comprising: (Item 6) 6. The method of claim 5, wherein the data includes a model of the sensor device. (Item 7) accessing data characteristic of one or more surfaces of the object modeled by the 3D graphical model; determining the defects to add to the plurality of training samples based on the characteristics; 7. The method according to any one of items 1 to 6, further comprising: (Item 8) 8. The method of claim 7, wherein the 3D graphical model comprises the data. (Item 9) 9. The method according to any one of items 1 to 8, wherein the defects include one or more of noise or glare. (Item 10) The step of generating a plurality of different training samples comprises: applying a plurality of different lighting settings to the 3D graphical model to simulate lighting in an environment; determining the deficiencies for a subset of the plurality of training samples generated during application of a particular one of the plurality of different lighting settings, the deficiencies for the subset of the plurality of training samples being based on the particular lighting setting; 10. The method according to any one of items 1 to 9, comprising: (Item 11) The step of generating a plurality of different training samples comprises: placing said 3D graphical model in a plurality of different graphical environments, each of said graphical environments modeling a respective real-world environment; generating a subset of said plurality of training samples while said 3D graphical model is placed within a plurality of different graphical environments; 11. The method according to any one of items 1 to 10, comprising: (Item 12) 12. A system comprising means for carrying out the method according to any one of items 1 to 11. (Item 13) 13. The system of claim 12, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 1 to 11. (Item 14) 12. A computer-readable storage medium storing instructions executable by a processor for performing the method according to any one of items 1 to 11. (Item 15) receiving a subject input and a reference input at a Siamese neural network, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, weights of the first network portion being identical to weights of the second network portion, the subject input being provided as an input to the first network portion and the reference input being provided as an input to the second network portion; generating an output of the Siamese neural network based on the subject input and the reference input, the output of the Siamese neural network indicating a similarity between the reference input and the subject input; A method for providing the above. (Item 16) The step of generating the output comprises: determining a difference between the reference input and the subject input; determining whether the difference amount meets a threshold, the output identifying whether the difference amount meets the threshold; Item 16. The method according to item 15, comprising the steps of: (Item 17) The step of determining the difference between the reference input and the subject input further comprises: receiving a first feature vector output by the first network portion and a second feature vector output by the second network portion; determining a difference vector based on the first feature vector and the second feature vector; 17. The method according to item 16, comprising: (Item 18) 18. The method according to any one of items 15 to 17, wherein the step of generating an output comprises a one-time classification. (Item 19) 19. The method of any one of items 15 to 18, further comprising training the Siamese neural network using one or more synthetic training samples. (Item 20) 20. The method according to claim 19, wherein the one or more synthetic training samples are generated according to the method according to any one of claims 1 to 11. (Item 21) 21. The method of any one of items 15 to 20, wherein the reference input comprises a synthetically generated sample. (Item 22) 22. The method according to item 21, wherein the synthetically produced sample is produced according to the method according to any one of items 1 to 11. (Item 23) 23. The method of any one of items 15 to 22, wherein the subject input comprises a first digital image and the reference input comprises a second digital image. (Item 24) 23. The method of any one of items 15 to 22, wherein the subject input includes a first point cloud representation and the reference input includes a second point cloud representation. (Item 25) 25. A system comprising means for carrying out the method according to any one of items 15 to 24. (Item 26) 26. The system of claim 25, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 15 to 24. (Item 27) 26. The system of item 25, comprising one of a robot, a drone, or an autonomous vehicle. (Item 28) 25. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 15 to 24. (Item 29) providing first input data to a Siamese neural network, the first input data including a first representation of 3D space from a first pose; providing second input data to the Siamese neural network, the second input data including a second representation of 3D space from a second pose, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, weights of the first network portion being identical to weights of the second network portion, the first input data being provided as an input to the first network portion and the second input data being provided as an input to the second network portion; generating an output of the Siamese neural network, the output including a relative pose between the first pose and the second pose; A method for providing the above. (Item 30) 30. The method of claim 29, wherein the first representation of 3D space comprises a first 3D point cloud and the second representation of 3D space comprises a second 3D point cloud. (Item 31) 31. The method of any one of items 29 to 30, wherein the first representation of the 3D space comprises a first cloud of points and the second representation of the 3D space comprises a second cloud of points. (Item 32) 32. The method of claim 31, wherein the first cloud of points and the second cloud of points each comprise a respective voxelized point cloud representation. (Item 33) generating a 3D mapping of the 3D space from at least the first input data and the second input data based on the relative pose. 33. The method according to any one of items 29 to 32, further comprising: (Item 34) determining a position of an observer of said first pose in said 3D space based on said relative pose. 33. The method according to any one of items 29 to 32, further comprising: (Item 35) 35. The method of claim 34, wherein the observer comprises an autonomous machine. (Item 36) Item 36. The method of item 35, wherein the autonomous machine includes one of a robot, a drone, or an autonomous vehicle. (Item 37) A system comprising means for carrying out the method according to any one of items 29 to 36. (Item 38) 38. The system of claim 37, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 29 to 36. (Item 39) 37. A computer-readable storage medium storing instructions executable by a processor for performing the method according to any one of items 29 to 36. (Item 40) providing the first sensor data as an input to a first portion of a machine learning model; providing the second sensor data as an input to a second portion of the machine learning model, the machine learning model including a connector and a set of fully connected layers, the first sensor data being of a first type generated by a device and the second sensor data being of a different second type generated by the device, the connector receiving an output of the first portion of the machine learning model as a first input and an output of the second portion of the machine learning model as a second input, the output of the connector being provided to the set of fully connected layers; generating an output of the machine learning model from the first data and the second data, the output comprising a pose of the device within an environment; A method for providing the above. (Item 41) 41. The method of claim 40, wherein the first sensor data includes image data and the second sensor data identifies a movement of the device. (Item 42) 42. The method of claim 41, wherein the image data comprises red-green-blue (RGB) data. (Item 43) 42. The method of claim 41, wherein the image data includes 3D point cloud data. (Item 44) 42. The method of claim 41, wherein the second sensor data includes inertial measurement unit (IMU) data. (Item 45) Item 42. The method of item 41, wherein the second sensor data includes global positioning data. (Item 46) 46. ​​The method of any one of items 40 to 45, wherein the first portion of the machine learning model is tuned for the first type of sensor data and the second portion of the machine learning model is tuned for the second type of sensor data. (Item 47) 47. The method of any one of items 40 to 46, further comprising providing a third sensor data of a third type as an input to a third part of the machine learning model, the output being further generated based on the third data. (Item 48) 48. The method according to any one of items 40 to 47, wherein the pose output includes a rotational component and a translational component. (Item 49) Item 49. The method of item 48, wherein one of the set of fully connected layers includes a fully connected layer for determining the rotation component, and another of the set of fully connected layers includes a fully connected layer for determining the translation component. (Item 50) 50. The method of any one of items 40 to 49, wherein one or both of the first and second parts of the machine learning model include respective convolutional layers. (Item 51) 51. The method of any one of items 40 to 50, wherein one or both of the first and second parts of the machine learning model include one or more respective long short-term memory (LSTM) blocks. (Item 52) 52. The method of any one of items 40 to 51, wherein the device includes an autonomous machine, the autonomous machine navigating within the environment based on the pose. (Item 53) 53. The method of claim 52, wherein the autonomous machine comprises one of a robot, a drone, or an autonomous vehicle. (Item 54) 53. A system comprising means for carrying out the method according to any one of items 40 to 52. (Item 55) 55. The system of claim 54, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 40 to 52. (Item 56) 53. A computer-readable storage medium storing instructions executable by a processor for performing the method according to any one of items 40 to 52. (Item 57) requesting random generation of a set of neural networks; performing a machine learning task using each one of the set of neural networks, the machine learning task being performed using specific processing hardware; monitoring attributes of the execution of the machine learning task for each of the set of neural networks, the attributes including accuracy of a result of the machine learning task; identifying a best performing neural network of the set of neural networks when used to perform the machine learning task using the particular processing hardware based on the attributes of the best performing neural network; A method for providing the above. (Item 58) providing said best performing neural network for use by a machine in executing a machine learning application. 58. The method of claim 57, further comprising: (Item 59) determining characteristics of said best performing neural network; requesting generation of a second set of neural networks according to the characteristics, the second set of neural networks including a plurality of different neural networks, each of the neural networks including one or more of the characteristics; performing said machine learning task using each one of said second set of neural networks, said machine learning task being performed using said particular processing hardware; monitoring attributes of the performance of the machine learning task for each of the second set of neural networks; identifying a best performing neural network from said second set of neural networks based on said attributes; 59. The method according to any one of items 57 to 58, further comprising: (Item 60) 60. The method of any one of items 57 to 59, further comprising the step of receiving a criterion based on the parameters, wherein the best performing neural network is based on the criterion. (Item 61) 61. The method of any one of items 57 to 60, wherein the attributes include attributes of the particular processing hardware. (Item 62) 62. The method of claim 61, wherein the attributes of the particular processing hardware include one or more of: power consumed by the particular processing hardware during execution of the machine learning task; a temperature of the particular processing hardware during execution of the machine learning task; and memory used to store the neural network on the particular processing hardware. (Item 63) Item 63. The method of any one of items 57 to 62, wherein the attributes include a time to complete the machine learning task using the corresponding one of the set of neural networks. (Item 64) 64. A system comprising means for carrying out the method according to any one of items 57 to 63. (Item 65) 65. The system of claim 64, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 57 to 63. (Item 66) A computer-readable storage medium storing instructions executable by a processor for performing the method according to any one of items 57 to 63. (Item 67) identifying a neural network including a plurality of kernels, each of the kernels including a respective set of weights; reducing the plurality of kernels to a set of particular kernels by pruning a subset of the plurality of kernels according to one or more parameters; pruning a subset of weights in the particular set of kernels to form a pruned version of the neural network, the pruning causing one or more non-zero weights in the subset of weights to be assigned to zero, the subset of weights being selected based on original values ​​of the weights; A method for providing the above. (Item 68) Item 68. The method of item 67, wherein the subset of weights is pruned based on values ​​of the subset of weights being below a threshold. (Item 69) 69. The method of any one of items 67 to 68, further comprising performing one or more iterations of a machine learning task using the pruned version of the neural network to regain at least a portion of the accuracy lost by the pruning of the kernels and the weights. (Item 70) 70. The method of any one of items 67 to 69, further comprising the step of quantizing values ​​of unpruned weights in the pruned version of the neural network to generate a compact version of the neural network. (Item 71) Item 71. The method of item 70, wherein the quantization includes log-based quantization. (Item 72) Item 72. The method of item 71, wherein the weights are quantized from floating point values ​​to base 2 values. (Item 73) 73. The method of any one of items 67 to 72, further comprising providing the pruned version of the neural network for performing machine learning tasks using hardware adapted for sparse matrix operations. (Item 74) A system comprising means for carrying out the method according to any one of items 67 to 73. (Item 75) 65. The system of claim 64, further comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 67 to 73. (Item 76) A computer-readable storage medium storing instructions executable by a processor for performing the method according to any one of items 67 to 73.

Claims

1. generating one or more synthetic training samples, the synthetic training samples comprising a synthetic image; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set having the one or more photorealistic training samples; training a neural network using the training set, the neural network having a first network and a second network, and after the training step, the first network and the second network process the two different input images to estimate a similarity between the two different input images; A method comprising:

2. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples; 2. The method of claim 1, comprising:

3. The method of claim 1 or 2, wherein the photorealistic training samples have a lower photorealistic resolution than the resolution of the synthetic training samples from which they are generated.

4. The method of claim 1 , wherein the quality of the one or more synthetic training samples comprises brightness, noise level, or contrast.

5. The method of claim 1 , wherein the first network and the second network have the same weight.

6. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object.

2. The method of claim 1, comprising:

7. 7. The method of claim 1, wherein the first network generates a first output from one of the two different images and the second network generates a second output from the other of the two different images, and the neural network determines the similarity based on the first output and the second output.

8. To the computer processor, generating one or more synthetic training samples, the synthetic training samples comprising a synthetic image; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set having said one or more photorealistic training samples; training a neural network using the training set, the neural network having a first network and a second network, and after the training step, the first network and the second network process two different input images to estimate a similarity between the two different input images; A computer program for executing the above.

9. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples; 9. The computer program of claim 8, comprising:

10. 10. The computer program product of claim 8 or 9, wherein the photorealistic training samples have a lower photorealistic resolution than a resolution of synthetic training samples from which the photorealistic training samples are generated.

11. The computer program product of claim 8 , wherein the quality of the one or more synthetic training samples comprises brightness, noise level, or contrast.

12. 12. The computer program product of claim 8, wherein the first network and the second network have the same weights.

13. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object.

9. The computer program of claim 8, comprising:

14. 14. The computer program product of claim 8, wherein the first network generates a first output from one of the two different images and the second network generates a second output from the other of the two different images, and the neural network determines the similarity based on the first output and the second output.

15. A computer readable storage medium storing a computer program according to any one of claims 8 to 14.

16. a computer processor for executing computer program instructions; generating one or more synthetic training samples, the synthetic training samples having a synthetic image; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set having the one or more photorealistic training samples; training a neural network using the training set, the neural network having a first network and a second network, after the training, the first network and the second network process the two different input images to estimate a similarity between the two different input images; one or more non-transitory computer-readable storage media storing computer program instructions executable by said computer processor to perform operations having the steps of: An apparatus comprising:

17. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples; 17. The apparatus of claim 16, further comprising:

18. 18. The apparatus of claim 16 or 17, wherein the photorealistic training samples have a lower photorealistic resolution than a resolution of synthetic training samples from which the photorealistic training samples are generated.

19. 19. The apparatus of claim 16, wherein the first network and the second network have the same weight.

20. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object.

17. The apparatus of claim 16, further comprising:

21. 21. The apparatus of claim 16, wherein the first network generates a first output from one of the two different images and the second network generates a second output from the other of the two different images, and the neural network determines the similarity based on the first output and the second output.

Images