Deep Learning System

Abstract

To provide a machine learning system, a method, a computer program, and an apparatus for enhancing various aspects of a machine learning model.SOLUTION: A system accesses a three-dimensional (3D) graphical model 1410 having substantially photorealistic resolution of an object and generates from the 3D model a set of training images, a point group, a depth map or other training data 1425 for a 3D graphical mode. A training set generator 1420 automatically renders or generates images or other training data from the 3D model in a manner that intentionally downgrades the resolution and quality of the image obtained as a result. The training data is used by a machine learning system 1430 to train one or more machine learning models 1435.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] [Related Applications] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 675,601, filed May 23, 2018, the entirety of which is incorporated herein by reference.

[0002] The present disclosure relates generally to the field of computer systems, and more specifically to machine learning systems. [Background technology]

[0003] The worlds of computer vision and graphics are rapidly converging with the emergence of augmented reality (AR), virtual reality (VR), and mixed reality (MR) products, such as MagicLeap™, Microsoft™ HoloLens™, Oculus™ Rift™, and other VR systems such as those from Valve™ and HTC™. The current approach in such systems is to use separate graphics processing units (GPUs) and computer vision subsystems that run in parallel. These parallel systems can be assembled from pre-existing GPUs in parallel with computer vision pipelines implemented in software running on an array of processors and / or programmable hardware accelerators. [Brief explanation of the drawings]

[0004] Various objects, features, and advantages of the disclosed subject matter can be more fully understood with reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals identify like elements. The accompanying drawings are schematic and are not intended to be drawn to scale. For clarity, not every component has been labeled in every drawing. Not every component of each embodiment of the disclosed subject matter has been shown unless a drawing is necessary to enable one of ordinary skill in the art to understand the disclosed subject matter.

[0005] [Figure 1] 1 illustrates a conventional augmented or mixed reality rendering system.

[0006] [Figure 2] 1 illustrates a voxel-based augmented or mixed reality rendering system according to some embodiments.

[0007] [Figure 3] 1 illustrates the difference between dense and sparse volumetric representations according to some embodiments.

[0008] [Figure 4] 1 illustrates a composite view of a scene according to some embodiments.

[0009] [Figure 5] 1 illustrates the level of detail of an exemplary element tree structure according to some embodiments.

[0010] [Figure 6] 1 illustrates applications in which the data structure and voxel data of the present application can be utilized, according to some embodiments.

[0011] [Figure 7] 1 illustrates an exemplary network used to recognize 3D digits, according to some embodiments.

[0012] [Figure 8] 10 illustrates multiple classifications performed on the same data structure using an implicit level of detail, according to some embodiments.

[0013] [Figure 9] 1 illustrates computation elimination with a 2D convolutional neural network according to some embodiments.

[0014] [Figure 10] 10 illustrates experimental results obtained from analysis of exemplary test images, according to some embodiments.

[0015] [Figure 11] 1 illustrates hardware for culling operations according to some embodiments.

[0016] [Figure 12] 1 illustrates hardware improvements for culling operations, according to some embodiments.

[0017] [Figure 13] 1 illustrates hardware according to some embodiments.

[0018] [Figure 14] 1 illustrates an example system employing an example training set generator, according to at least some embodiments.

[0019] [Figure 15] 1 illustrates an exemplary generation of synthetic training data according to at least some embodiments.

[0020] [Figure 16] 1 illustrates an exemplary Siamese network according to at least some embodiments.

[0021] [Figure 17]1 illustrates an exemplary use of a Siamese network to perform autonomous comparisons, according to at least some embodiments.

[0022] [Figure 18] 1 illustrates an exemplary voxelization of a point cloud according to at least some embodiments.

[0023] [Figure 19] 1 is a simplified block diagram of an example machine learning model according to at least some embodiments.

[0024] [Figure 20] 1 is a simplified block diagram illustrating aspects of an exemplary training of a model, according to at least some embodiments.

[0025] [Figure 21] 1 illustrates an example robot that uses neural networks to generate a 3D map for navigation, according to at least some embodiments.

[0026] [Figure 22] 1 is a block diagram illustrating an example machine learning model for use with inertial measurement data, according to at least some embodiments.

[0027] [Figure 23] 1 is a block diagram illustrating an example machine learning model for use with image data, according to at least some embodiments.

[0028] [Figure 24] FIG. 24 is a block diagram illustrating an exemplary machine learning model that combines aspects of the models in the examples of FIGS. 22 and 23.

[0029] [Figure 25A] 25 is a graph illustrating the results of a machine learning model similar to the example machine learning model of FIG. 24. [Figure 25B] 25 is a graph illustrating the results of a machine learning model similar to the example machine learning model of FIG. 24.

[0030] [Figure 26] 1 illustrates an example system including an example neural network optimizer, according to at least some embodiments.

[0031] [Figure 27] 1 is a block diagram illustrating an example optimization of a neural network model, according to at least some embodiments.

[0032] [Figure 28] 1 is a table illustrating exemplary results generated and used during optimization of an exemplary neural network model.

[0033] [Figure 29] 10 shows graphs illustrating results generated and used during optimization of an exemplary neural network model.

[0034] [Figure 30A] 1 is a simplified block diagram illustrating an example of hybrid neural network pruning, according to at least some embodiments.

[0035] [Figure 30B] FIG. 1 is a simplified flow diagram illustrating an exemplary pruning of a neural network, according to at least some embodiments.

[0036] [Figure 31] 1 is a simplified block diagram illustrating exemplary weight quantization performed in connection with pruning a neural network, according to at least some embodiments.

[0037] [Figure 32] 1 is a table comparing the results of exemplary neural network pruning techniques.

[0038] [Figure 33A] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33B] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33C] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33D] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33E] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments. [Figure 33F] FIG. 1 is a simplified flow diagram of an exemplary computer-implemented technique associated with machine learning, according to at least some embodiments.

[0039] [Figure 34] 1 illustrates an exemplary multi-slot vector processor according to some embodiments.

[0040] [Figure 35] 1 illustrates exemplary volumetric acceleration hardware according to some embodiments.

[0041] [Figure 36] 1 illustrates an organization of voxel cubes according to some embodiments.

[0042] [Figure 37] 1 illustrates a two-level sparse voxel tree according to some embodiments.

[0043] [Figure 38] 1 illustrates a two-level sparse voxel tree according to some embodiments.

[0044] [Figure 39] 1 illustrates an exemplary voxel data storage according to some embodiments.

[0045] [Figure 40] 1 illustrates the insertion of voxels into an exemplary volumetric data structure according to some embodiments.

[0046] [Figure 41] 1 illustrates a projection of an exemplary 3D volumetric object according to some embodiments.

[0047] [Figure 42A] The positional relationship between Figures 42B and 42C is shown. [Figure 42B] 1 illustrates an exemplary operation involving an exemplary volumetric data structure. [Figure 42C] 1 illustrates an exemplary operation involving an exemplary volumetric data structure.

[0048] [Figure 43] 1 illustrates the generation of a simplified map using projection, according to some embodiments.

[0049] [Figure 44] 1 illustrates an exemplary aggregation of exemplary volumetric 3D measurements and / or simple 2D measurements from an embedded device, according to some embodiments.

[0050] [Figure 45A] The positional relationship between FIG. 45B and FIG. 45C is shown. [Figure 45B] 10 illustrates an exemplary acceleration of 2D pathfinding on a 2D 2x2 bitmap, according to some embodiments. [Figure 45C] 10 illustrates an exemplary acceleration of 2D pathfinding on a 2D 2x2 bitmap, according to some embodiments.

[0051] [Figure 46] 1 illustrates an example acceleration of collision detection using an example volumetric data structure, according to some embodiments.

[0052] [Figure 47] 1 is a simplified block diagram of an exemplary network having devices according to at least some embodiments.

[0053] [Figure 48] 1 is a simplified block diagram of an exemplary fog or cloud computing network according to at least some embodiments.

[0054] [Figure 49] 1 is a simplified block diagram of a system including an example device according to at least some embodiments.

[0055] [Figure 50] 1 is a simplified block diagram of an exemplary processing device according to at least some embodiments.

[0056] [Figure 51] 1 is a block diagram of an example processor according to at least some embodiments.

[0057] [Figure 52] 1 is a block diagram of an exemplary computing system according to at least some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0058] In the following description, numerous specific details are set forth regarding the systems and methods of the disclosed subject matter, as well as the environments in which such systems and methods may operate, to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced without such specific details, and that certain features that are well known in the art have not been described in detail to avoid complicating the disclosed subject matter. Furthermore, it will be understood that the embodiments provided below are exemplary, and that other systems and methods are contemplated that fall within the scope of the disclosed subject matter.

[0059] Various technologies based on and incorporating augmented reality, virtual reality, mixed reality, autonomous devices, and robots are emerging that may use data models representing three-dimensional spaces and geometric volumes. Descriptions of various real and virtual environments using such 3D or volumetric data traditionally involve large datasets, which some computing systems have struggled to process in a desirable manner. Furthermore, as devices such as drones, wearable devices, virtual reality systems, and the like become smaller, the memory and processing resources of such devices may also be constrained. By way of example, AR / VR / MR applications may require high frame rates for graphical representations generated using supporting hardware. However, in some applications, the GPU and computer vision subsystem of such hardware may need to process data (e.g., 3D data) at high speeds, such as up to 130 fps (7 milliseconds), to produce desirable results (e.g., to generate reliable graphical scenes at frame rates that produce reliable results and prevent users from experiencing motion sickness due to excessive latency, among other exemplary goals). Similarly, further applications may be challenged to satisfactorily process data describing large volumes while meeting the processing, memory, power, and application requirement constraints of the corresponding system, among other exemplary problems.

[0060] In some implementations, a computing system may include logic for generating and / or using sparse volumetric data defined according to a format. For example, the defined volumetric data structure may be provided for integrating computer vision and 3D rendering in various systems and applications. The volumetric representation of an object may be imaged using an optical sensor, such as a stereoscopic camera or a depth camera. The volumetric representation of the object may include multiple voxels. An improved volumetric data structure may be defined to enable recursive refinement of the corresponding volumetric representation to obtain a target resolution for the object. During refinement, empty space in the volumetric representation, which may be contained in one or more of these voxels, may be culled from the volumetric representation (and supporting operations). The empty space may be a region of the volumetric representation that does not include geometric characteristics of the object.

[0061] Thus, in an improved volumetric data structure, individual voxels in a corresponding volume may be tagged as “occupied” (due to some geometry present in the corresponding volumetric space) or “empty” (indicating that the corresponding volume is composed of free space). Such tags may be further interpreted as specifying that one or more of the corresponding subvolumes are also occupied (e.g., if a parent or higher-level voxel is tagged as occupied) or that all of the subvolumes are free space (i.e., if a parent or higher-level voxel is tagged as empty). In some implementations, tagging a voxel as empty may effectively remove the voxel and / or voxels of the corresponding subvolume from operations used to generate the corresponding volumetric representation. The volumetric data structure may follow a sparse tree structure, for example, a sparse sexaquaternary tree (SST) format. Moreover, such approaches to sparse volumetric data structures may utilize relatively less storage space than traditionally used to store volumetric representations of objects. Furthermore, compression of volumetric data may increase the feasibility of transmitting such representations and enable faster processing of such representations, among other exemplary benefits.

[0062] Volumetric data structures can be hardware accelerated to enable rapid updates to 3D renderers, eliminating delays that can occur in separate computer vision and graphics systems. Such delays can introduce latency, which, when used in AR, VR, MR, and other applications, can cause motion sickness in users, among other drawbacks. The ability to rapidly test voxels for occupancy of geometric properties within accelerated data structures enables the construction of low-latency AR, VR, MR, or other systems that can be updated in real time.

[0063] In some embodiments, the functionality of the volumetric data structure may provide in-frame warnings, such as in AR, VR, MR, and other applications when a user is likely to collide with a real or synthetic object in the imaged scene, or in computer vision applications for drones or robots when such a device is likely to collide with a real or synthetic object in the imaged scene, allowing for the processing speed provided by the volumetric data structure to warn of an impending collision.

[0064] Embodiments of the present disclosure may relate to the storage and processing of volumetric data in applications such as robotics, head-mounted displays for augmented and mixed reality headsets, and phones and tablets. In embodiments of the present disclosure, each volumetric element (e.g., voxel) within a group of voxels, and optionally physical quantities related to the voxel's geometry, are represented as a single bit. Additional parameters related to a group of 64 voxels, such as corresponding red-green-blue (RGB) or other color encoding, transparency, truncated signed distance function (TSDF) information, etc., may be associated with the voxel and stored in an associated optional 64-bit data structure (e.g., such that two or more bits are used to represent each voxel). Such a representation scheme may achieve minimal memory requirements. Furthermore, representing voxels with a single bit allows for many simplified calculations to be performed to logically or mathematically combine elements of the volumetric representation. Combining elements of a volumetric representation can include, for example, ORing planes within a volume to create a 2D projection of the 3D volumetric data, and calculating surface area by counting the number of occupied voxels in a 2.5D manifold, among other things. Comparisons can involve comparing 64-bit subvolumes (e.g., 4^3 subvolumes) using XOR logic, and volumes can be inverted. Here, objects can be merged and ORed together to create hybrid objects, among other examples.

[0065] 1 illustrates a conventional augmented or mixed reality system comprised of parallel graphics rendering and computer vision subsystems with post-rendering interconnects to account for changes due to rapid head movement and environmental changes that may produce occlusions and shadows in the rendered graphics. In one exemplary implementation, the system may include a host processor 100 supported by host memory 124 to control the execution of a graphics pipeline, a computer vision pipeline, and a post-rendering correction unit via interconnection via bus 101, an on-chip network-on-chip, or other interconnect. The interconnection allows host processor 100, running appropriate software, to control the execution of a graphics processing unit (GPU) 106, associated graphics memory 111, computer vision pipeline 116, and associated computer vision memory 124. In one example, rendering of graphics using GPU 106 via OpenGL graphics shader 107 (e.g., operating on triangle list 105) may occur at a slower rate than the computer vision pipeline. As a result, post-rendering corrections via the warp engine 108 and display / occlusion processor 109 can be performed to account for changes in head pose and occlusion scene geometry that may have occurred since the graphics were rendered by the GPU 106. The output of the GPU 106 is time-stamped so that it can be used in conjunction with correct control signals 121 from the head pose pipeline 120 and correct control signals 123 from the occlusion pipeline 122, among other examples, to generate correct graphics output to account for any changes in head pose 119 and occlusion geometry 113.

[0066] In parallel with the GPU 106, multiple sensors and cameras (e.g., including active and passive stereo cameras for depth and vision processing 117) may be connected to a computer vision pipeline 116. The computer vision pipeline 116 may include one or more of at least three stages, each of which may include multiple stages of lower-level processing. In one example, the stages of the computer vision pipeline 116 may be an image signal processing (ISP) pipeline 118, a head pose pipeline 120, and an occlusion pipeline 122. The ISP pipeline 118 may receive the outputs of the input camera sensors 117 and condition them so that they can be used for subsequent head pose and occlusion processing. The head pose pipeline 120 may receive the output of the ISP pipeline 118 and use it, along with the output 119 of an inertial measurement unit (IMU) in the headset 110, to calculate the change in head pose since the corresponding output graphics frame was rendered by the GPU 106. The output 121 of the head pose pipeline (HPP) 120, along with a user-specified mesh, can be applied to the warp engine 108 to warp the GPU output 102 to match the updated head pose position 119. The occlusion pipeline 122 receives the output of the head pose pipeline 121 and can look for new objects in the field of view, such as a hand 113 (or other exemplary object) that enters the field of view, which should generate a corresponding shadow 114 on the scene geometry. The output 123 of the occlusion pipeline 122 can be used by the display and occlusion processor 109 to accurately overlay the field of view on the output 103 of the warp engine 108.Among other example use cases and features, the display and occlusion processor 109 can use the calculated head pose 119 to generate a shadow mask for the synthetic shadow 114, and the display and occlusion processor 109 can composite the occlusion geometry of the hand 113 onto the shadow mask to generate the graphical shadow 114 on top of the output 103 of the warp engine 108 to generate the final output frame 104 for display on the augmented / mixed reality headset 110.

[0067] 2 illustrates a voxel-based augmented or mixed reality rendering system according to some embodiments of the present disclosure. The device illustrated in FIG. 2 may include a host system configured on a host CPU 200 and associated host memory 201. Such a system may communicate via a bus 204, an on-chip network, or other communication mechanism with an integrated computer vision and graphics pipeline 223 and associated integrated computer vision and graphics memory 213 containing actual and synthetic voxels to be rendered in a final scene for display on a head-mounted augmented or mixed reality display 211. The AR / MR display 211 may include multiple active and passive image sensors 214 and an inertial measurement unit (IMU) 212 used to measure changes in head pose 222 orientation.

[0068] In a combined rendering pipeline, composite geometry may be generated starting with a triangle list 204, which is processed by an OpenGL JiT (Just-in-Time) translator 205 to generate composite voxel geometry 202. The composite voxel geometry may be generated, for example, by selecting a major plane of triangles from the triangle list. 2D rasterization of each triangle within the selected plane may then be performed (e.g., in the X and Z directions). A third coordinate (e.g., Y) may be created as an attribute that is interpolated across the triangle. Each pixel of the rasterized triangle may result in the definition of a corresponding voxel. This processing may be performed by a CPU or a GPU. Once performed by the GPU, each rasterized triangle may be read back from the GPU to create a voxel where the GPU has rendered the pixel, among other example implementations. For example, composite voxels may be generated using a 2D buffer in the list, where each entry in the list stores depth information for the polygon rendered at that pixel. For example, the model may be rendered using an orthographic perspective (e.g., top-down). For example, every (x,y) provided in the exemplary buffer can represent a column of (x,y) within the corresponding voxel volume (e.g., from (x,y,0) to (x,y,4095)). Each column can then be rendered from the information as a 3D scanline using the information in each list.

[0069] Continuing with the example of FIG. 2, in some implementations, the synthetic voxel geometry 202 may be combined with measured geometry voxels 227 constructed using a simultaneous localization and mapping (SLAM) pipeline 217. The SLAM pipeline may generate output 225 using active and / or passive image sensors 214 (e.g., 214.1 and 214.2), which are first processed using an image signal processing (ISP) pipeline 215, and the output 225 may be converted to a depth image 226 by a depth pipeline 216. The active or passive image sensors 214 (214.1 and 214.2) may include active or passive stereo sensors, structured light sensors, and time-of-flight sensors, among other examples. For example, the depth pipeline 216 may process depth data from a structured light or time-of-flight sensor 214.1, or alternatively, a passive stereo sensor 214.2. In one exemplary implementation, the stereo sensor 214.2 may include a pair of passive stereo sensors, among other exemplary implementations.

[0070] The depth images generated by the depth pipeline 215 can be processed by a dense SLAM pipeline 217 using a SLAM algorithm (e.g., Kinect Fusion) to generate a voxelized model of the measured geometry voxels 227. The measured geometry voxels 227 (e.g., actual voxel geometry) can be combined with the synthetic voxel geometry 202 to provide a ray tracing accelerator 206 that can generate a 2D rendering of the scene for output to a display device (e.g., a head-mounted display 211 in a VR or AR application) via the display processor 210. In such an implementation, a complete scene model can be constructed from the measured geometry voxels 227 and the actual voxels of the synthetic geometry 202. As a result, there is no need to warp the 2D rendered geometry (e.g., as in FIG. 1 ). Such an implementation can combine a head pose tracking sensor and corresponding logic to accurately align the actual and measured geometries. For example, an exemplary head pose pipeline 221 can process head pose measurements 232 from an IMU 212 attached to a head-mounted display 212, and the output 231 of the head pose measurement pipeline can be taken into account during rendering via the display processor 210.

[0071] In some examples, a unified rendering pipeline may use measured geometry voxels 227 (e.g., actual voxel models) and synthetic geometry 202 (e.g., synthetic voxel models) to render audio reverberation models and model the physics of a real-world, virtual, or mixed reality scene. As an example, physics pipeline 218 can receive measured geometry voxels 227 and synthetic voxel geometry 202 and use ray casting accelerator 206 to calculate output audio samples for left and right earphones in head-mounted display (HMD) 211 to calculate output samples 230 using acoustic reflection coefficients embedded in the voxel data structure. Similarly, the unified voxel model composed of 202 and 227 can also be used to determine physics updates for synthetic objects in a synthetic AR / MR scene. Physics pipeline 218 receives synthetic scene geometry as input and uses ray casting accelerator 206 to calculate collisions before calculating updates 228 to synthetic geometry 202 for rendering and as a basis for future iterations of the physics model.

[0072] 2 may further comprise one or more hardware accelerators for implementing and / or utilizing a convolutional neural network (CNN) that can process either RGB video / image input from the output of the ISP pipeline 215 or volumetric scene data from the output of the SLAM pipeline 217, among other examples. A neural network classifier can be run using the hardware (HW) convolutional neural network (CNN) accelerator 207 exclusively or in a combination of a processor and the HW CNN accelerator 207 to generate the output classification 237. The availability of the HW CNN accelerator 207 to infer volumetric representations may enable labeling groups of voxels within the measured geometry voxels 227 as belonging to a particular object class, among other example uses.

[0073] Labeling the voxels (e.g., using a CNN and supporting hardware acceleration) may enable the system to recognize the objects to which they belong as corresponding to known objects. The source voxels may be removed from the measured geometry voxels 227 and replaced with a bounding box corresponding to the object and / or information about the object's origin, object pose, and object descriptors, among other example information. This may result in a much more semantically meaningful description of the scene that may be used, for example, as input by a robot, drone, or other computing system that interacts with objects in the scene, or as input by an audio system that examines the sound absorption coefficients of objects in the scene and incorporates them into an acoustic model of the scene, among other example uses.

[0074] One or more processor devices and hardware accelerators may be provided to implement the pipeline of the exemplary system shown and described in Figure 2. In some implementations, among other example implementations, all of the hardware and software elements of the combined rendering pipeline may share access to a DRAM controller 209, which in turn enables data to be stored in a shared DDR memory device 208.

[0075] FIG. 3 is provided to illustrate the difference between dense and sparse volumetric representations according to some embodiments. As shown in the example of FIG. 3, a real-world or synthetic object 300 (e.g., a rabbit statue) can be represented in terms of voxels in a dense manner, as shown at 302, or in a sparse manner, as shown at 304. The advantage of a dense representation such as 302 is uniform access speed to all voxels in the volume, but the disadvantage is the amount of storage that may be required. For example, a dense representation such as a 512^3 element volume (e.g., corresponding to 5 m at 1 cm resolution for a volume scanned using a Kinect sensor) requires 512 megabytes to store a relatively small volume using a 4-byte truncated signed distance function (TSDF) per voxel. On the other hand, an octree representation 304, embodying a sparse representation, can reduce the amount of data required to store the same volume by storing only those voxels for which actual geometry exists in the real-world scene.

[0076] Referring to FIG. 4, a composite view of an exemplary scene according to some embodiments is shown. In particular, FIG. 4 illustrates how a composite view of a scene 404 may be maintained, displayed, or subjected to further processing using parallel data structures to represent synthetic voxels 401 and measured real-world voxels 403 within equivalent bounding boxes 400 and 402 for synthetic and real-world voxel data, respectively. FIG. 5 illustrates the level of detail of a uniform 4^3 element tree structure according to some embodiments. In some implementations, using an octree representation such as that depicted in the example of FIG. 5, as little as one bit may be utilized to describe each voxel in a volume. However, a drawback of octree-based techniques may be the number of indirect memory accesses utilized to access a particular voxel within the octree. In the case of sparse voxel octrees, among other exemplary benefits, the same geometry can be advantageously implicitly represented at multiple levels of detail, enabling operations such as ray casting, game physics, CNNs, and other techniques, and culling empty parts of a scene from further computation, resulting in an overall reduction not only in terms of required storage, but also in terms of power consumption and computational load.

[0077] In one implementation, an improved voxel descriptor (also referred to herein as a “volumetric data structure”) may be provided to organize volumetric information as 4^3 (or 64-bit) unsigned integers as shown in 501 with a memory requirement of 1 bit per voxel. In this example, 1 bit per voxel is insufficient to store the truncated signed distance function value (compared to the TSDF in SLAMbench / KFusion, which utilizes 64 bits). In this example, an additional (e.g., 64-bit) field 500 may be included in the voxel descriptor. This example may be further enhanced such that the TSDF in the 64-bit field 500 is 16 bits, while implicitly providing an additional 2 bits of fractional resolution in x, y, and z in the voxel descriptor 501, making the combination of the voxel TSDF in the 64-bit field 500 and the voxel position 501 comparable to a much higher resolution TSDF such as that used in SLAMbench / KFusion or other examples. For example, the further data in the 64-bit field 500 (voxel descriptor) may be used to store subsampled RGB color information (e.g., from the scene via a passive RGB sensor), each of which is 1 byte, an 8-bit transparency value alpha, and two 1-byte reserved fields R1 and R2. These reserved fields may be application specific and may be used, for example, to store acoustic reflectivity for audio applications, stiffness for physics applications, type of object material, among other examples.

[0078] As shown in FIG. 5, voxel descriptors 501 may be logically grouped into four 2D planes. Each of these 2D planes contains 16 voxels 502. These 2D planes (or voxel planes) may describe each level of an octree-style structure based on successive decompositions in ascending powers of four, as depicted in FIG. 5. In this exemplary implementation, a 64-bit voxel descriptor is selected because it is suitable for the 64-bit bus infrastructure used in the corresponding system implementation (although other voxel descriptor sizes and formats may be provided in other system implementations and sized according to the bus or other infrastructure of the system). In some implementations, the voxel descriptor may be sized to reduce the number of memory accesses used to retrieve a voxel. For example, using a 64-bit voxel descriptor may reduce the number of memory accesses required to access a voxel at any level in the octree by a factor of two compared to a traditional octree that operates with 2^3 elements, among other exemplary considerations and implementations.

[0079] In one example, an octree may be written starting with a 4^3 root volume 503 and each non-zero entry indicating the presence of geometry in underlying layers 504, 505, and 506, in an exemplary 256^3 volume. In this particular example, four memory accesses may be used to access the lowest level in the octree. If such overhead is too high, an alternative approach may be employed to encode the top level of the octree as a larger volume, such as 64^3, as shown in 507. In this case, each non-zero entry in 507 may indicate the presence of an underlying 4^3 octree in the underlying 256^3 volume 508. As a result of this alternative organization, only two memory accesses are required to access any voxel in the 256^3 volume 508, compared to the alternative formulations shown in 503, 504, and 505. This latter approach is advantageous when the device hosting the octree structure has a larger amount of built-in memory, thereby permitting only the smaller, less frequently accessed portions of the voxel octree 508 in external memory. While this approach may be more costly from a storage perspective, for example, when a full, larger (e.g., 64^3) volume is stored in on-chip memory, the trade-off may enable faster memory access (e.g., by a factor of 2) and significantly reduced power consumption, among other exemplary benefits.

[0080] Referring to FIG. 6, a block diagram illustrating exemplary applications in which the present data structures and voxel data can be utilized, according to some embodiments, is shown. In one example, such as that shown in FIG. 5, additional information can be provided via an exemplary voxel descriptor 500. While the voxel descriptor can increase the overall memory utilized to two bits per voxel, the voxel descriptor can enable a wide range of applications in which voxel data such as that depicted in FIG. 6 can be utilized. For example, among other implementations, a shared volumetric representation 602, such as that generated using a dense SLAM system 601 (e.g., SLAMbench), can be used in rendering a scene using graphics ray casting or ray tracing 603 used in audio ray casting 604. In yet another example, the volumetric representation 602 may be used in convolutional neural network (CNN) inference 605 and may be backed by a cloud infrastructure 607. In some cases, the cloud infrastructure 607 can include detailed volumetric descriptors of objects, such as trees, furniture, or other objects (e.g., 606), that can be accessed via inference. Based on the inference or otherwise identification of the object, a corresponding detailed descriptor can be returned to the device, so that the voxels of the volumetric representation 602 can be replaced with a bounding box representation having pose information and a descriptor containing the properties of the object, among other example features.

[0081] In yet other embodiments, the voxel model described above may additionally or alternatively be utilized in some systems to construct 2D maps of the exemplary environment 608 using 3D-to-2D projections from the volumetric representation 602. These 2D maps may again be shared via cloud infrastructure and / or other network-based resources 607 and aggregated (e.g., using the same cloud infrastructure) to construct higher-quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices. In yet a further example, the 2D maps may be refined for ultra-low-bandwidth applications using projections followed by piecewise simplification 609 (e.g., assuming the width and height of the vehicle or robot are fixed). The simplified path then has only a single X,Y coordinate pair for each piecewise linear segment of the path, thereby reducing the amount of bandwidth required to communicate the path of the vehicle 609 to the cloud infrastructure 607 and aggregated to that same cloud infrastructure 607 to construct higher-quality maps using crowdsourcing techniques. These maps can be shared by the cloud infrastructure 607 to connected machines and devices.

[0082] To enable these different applications, some implementations may provide common functionality, such as through shared software libraries, and some embodiments may accelerate common functionality using hardware accelerators or processor instruction set architecture (ISA) extensions, among other examples. For example, such functionality may include inserting voxels into a descriptor, deleting voxels, or searching for voxels 610. Some implementations may support collision detection functionality 620 as well as point / voxel deletion from a volume 630, among other examples. As introduced above, the system may provide functionality for quickly generating 2D projections 640 in the X, Y, and Z directions from the corresponding volumetric representation 602 (3D volume) (e.g., which may serve as a basis for path or collision determination). In some cases, it may also be advantageous to be able to generate triangle lists from the volumetric representation 602 using histogram pyramids 650. Additionally, the system may provide functionality for fast determination of free paths 660 in the 2D and 3D representations of the volumetric space 602. Such functionality may be useful in a variety of applications: further functionality may be provided such as refining the number of voxels in a volume, counting the number of 1 bits in masked regions of the volumetric representation 602 using a population counter to determine the surface of an object, among other examples.

[0083] Referring to the simplified block diagram of FIG. 7 , an exemplary network is shown that includes a system capable of recognizing 3D digits according to at least some embodiments. For example, one of the applications shown in FIG. 6 is a volumetric CNN application 605, which is described in more detail in FIG. 7 . In FIG. 7 , the exemplary network is used to recognize 3D digits 700 generated from a dataset such as the mixed National Institute of Standards and Technology (MNIST) dataset. The digits in such a dataset can be used to train a CNN-based convolutional network classifier 710 by applying appropriate rotations and translations in X, Y, and Z to the digits before training. When used for inference on an embedded device, the trained network 710 can be used to accurately classify 3D digits in a scene 720, even when the digits are subject to rotations and translations in X, Y, and Z, among other examples. In some implementations, the operation of the CNN classifier can be accelerated by the HW CNN accelerator 207 shown in FIG. 2 . The first layer of the neural network performs multiplications using voxels of the volumetric representation 602. So multiplying by 0 is always 0, and multiplying a data value A by 1 (a voxel) is equal to A, so these arithmetic operations can be omitted.

[0084] FIG. 8 illustrates multiple classifications performed on the same data structure using implicit levels of detail. A further improvement to CNN classification using volumetric representation 602 may be that, because the octree representation implicitly includes multiple levels of detail in the octree structure, as shown in FIG. 5, multiple classifications on the same data structure may be performed using the implicit levels of detail 800, 810, and 820 in parallel, and using a single classifier 830 or multiple classifiers in parallel, as shown in FIG. 8. In conventional systems, equivalent parallel classification may be slower due to the need to resize images between classification passes. Such resizing may be omitted in implementations that apply the voxel structure described herein, because the same octree may contain the same information at multiple levels of detail. In fact, a single training dataset based on a volumetric model can cover all of these levels of detail, rather than a resized training dataset as required for conventional CNN networks.

[0085] Referring to the example of FIG. 9 , an exemplary computational elimination by a 2D CNN according to some embodiments is shown. Computational elimination can be used not only in 2D CNNs such as the one shown in FIG. 9 , but also in 3D volumetric CNNs. For example, in FIG. 9 , in the first layer, a bitmap mask 900 can be used to describe the expected “shape” of input 910 and can be applied to the incoming video stream 920. In one example, computational elimination can be used not only in 3D volumetric CNNs but also in 2D volumetric CNNs. For example, in the 2D CNN of the example of FIG. 9 , a bitmap mask 900 can be applied to the first layer of the CNN to describe the expected “shape” of input 910 and can be applied to the CNN's input data, such as the incoming video stream 820. As an example, FIG. 9 illustrates the effect of applying a bitmap mask to an image of a pedestrian for training or inference in a CNN network. Here, 901 represents the original image of the pedestrian 901, and 903 represents the corresponding version with the bitmap mask applied. Similarly, an image without pedestrians is shown at 902, and the corresponding bitmap mask version at 904. The same method can be applied to any kind of 2D or 3D object to reduce the number of operations required for CNN training or inference through knowledge of the expected 2D or 3D geometry predicted by the detector. An example of a 3D volumetric bitmap is shown at 911. The use of 2D bitmaps for inference in real scenes is shown at 920.

[0086] In the exemplary implementation of Figure 9, a conceptual bitmap is shown (at 900), and the actual bitmap is generated by averaging a series of training images for a particular class of object 910. While the example shown is two-dimensional, the proposed volumetric data format of one bit per voxel can also generate similar bitmap masks for 3D objects. Indeed, among other exemplary implementations, the method may potentially be extended to use additional bits per voxel / pixel to specify the expected color range or other characteristics of a 2D or 3D object.

[0087] FIG. 10 is a table illustrating the results of an exemplary experiment involving the analysis of 10,000 CIFAR-10 test images, according to some embodiments. In some implementations, operation elimination can be used to eliminate intermediate calculations in 1D, 2D, and 3D CNNs due to rectified linear unit (ReLU) operations, which are frequently performed in CNN networks, such as LeNet1000 shown in FIG. 10. As shown in FIG. 10, in an experiment using 10,000 CIFAR-10 test images, the percentage of data-dependent zeros generated by the ReLU unit can reach up to 85%. That is, a system can be provided that recognizes zeros and, accordingly, does not fetch corresponding data or perform corresponding multiplication operations. In this example, 85% represents the percentage of ReLU dynamic zeros generated from a modified National Institute of Standards and Technology (MNIST) test dataset. The corresponding operation elimination corresponding to these zeros can help reduce power consumption and memory bandwidth requirements, among other exemplary benefits.

[0088] Trivial operations may be culled based on a bitmap. For example, the use of such a bitmap may be in accordance with the principles and embodiments described and illustrated in U.S. Patent No. 8,713,080, entitled "Circuit for compressing data and a processor employing the same," which is incorporated herein by reference in its entirety. Some implementations may provide hardware that can use such a bitmap, such as the systems, circuits, and other implementations described and illustrated in U.S. Patent No. 9,104,633, entitled "Hardware for performing arithmetic operations," which is also incorporated herein by reference in its entirety.

[0089] FIG. 11 illustrates hardware that may be incorporated into a system to provide functionality for culling trivial operations based on a bitmap, according to some embodiments. In this example, a multi-layer neural network including repeated convolutional layers is provided. The hardware may include one or more processors, one or more microprocessors, one or more circuits, one or more computers, etc. In this particular example, the neural network includes an initial convolutional processing layer 1100, followed by a pooling process 1110, and finally, an activation function process, such as a rectified linear unit (ReLU) function 1120. The output of the ReLU unit 1120, which provides a ReLU output vector 1131, may be connected (e.g., possibly via a delay 1132) to a next convolutional processing layer 1180, which receives the ReLU output vector 1131. In one exemplary implementation, in parallel with the connection of the ReLU unit 1120 to the next convolution unit 1180, a ReLU bitmap 1130 may be generated, which represents which elements in the ReLU output vector 1131 are 0 and which elements are not 0.

[0090] In one implementation, a bitmap (e.g., 1130) may be generated or otherwise provided to inform available hardware of opportunities to eliminate operations involved in the neural network computation. For example, bits in the ReLU bitmap 1130 may be interpreted by the bitmap scheduler 1160. The bitmap scheduler 1160 instructs the multipliers in the next convolution unit 1180 to omit 0 entries in the ReLU output vector 1131 for which a corresponding binary 0 exists in the ReLU bitmap 1130, assuming that multiplication by 0 always produces 0 as an output. In parallel, memory fetches from the address generator 1140 for data / weights corresponding to 0 in the ReLU bitmap 1130 may also be omitted, since there is little value in fetching weights that will be omitted by the next convolution unit 1180. If the weights are fetched from an attached DDR DRAM storage device 1170 via the DDR controller 1150, the latency may be so long that only some on-chip bandwidth and associated power consumption may be saved. On the other hand, if the weights are fetched from on-chip RAM 1180 storage, it may be possible to avoid / omit the weight fetch operation altogether, especially if a delay corresponding to the RAM / DDR fetch delay 1132 is added at the input to the next convolution unit 1180.

[0091] 12 , a simplified block diagram is presented to illustrate an exemplary hardware improvement with circuitry and other logic for trivial operation culling (or performing operation elimination) according to some embodiments. As shown in the example of FIG. 12 , additional hardware logic may be provided to predict in advance the sign of the ReLU unit 1220 input from the immediately preceding max pooling unit 1210 or convolution unit 1200. Adding sign prediction and ReLU bitmap generation to the max pooling unit 1210 may enable the ReLU bitmap information to be predicted earlier in terms of timing, covering any delay that may occur via the address generator 1240 via the external DDR controller 1250 and DDR storage 1270 or the internal RAM storage 1271. If the delay is small enough, the ReLU bitmap can be interpreted by the address generator 1240, since it can be determined that the result of the fetch from memory will never be used, and the memory fetch associated with ReLU bitmap 0 can be omitted entirely. This modification to the scheme of FIG. 11 may provide further power savings, among other example features and functions, and may also allow for the removal of a delay stage (e.g., 1132, 1232) at the input to the next convolution unit 1280 if the delay through the DDR access path (e.g., 1240 to 1250, 1250 to 1270) or the RAM access path (e.g., 1240 to 1271) is small enough to not warrant delay stage 1232.

[0092] FIG. 13 is another simplified block diagram illustrating exemplary hardware according to some embodiments. For example, a CNN ReLU layer can generate a large number of outputs of 0 corresponding to negative inputs. In fact, negative ReLU inputs can be predictively determined by looking at the sign input to the previous layer (e.g., the pooling layer in the example of FIG. 13). Floating-point and integer operations can be explicitly signed in terms of the most significant bit (MSB). Therefore, a simple bitwise exclusive OR (XOR) operation across the vectors of inputs multiplied in a convolutional layer can predict which multiplications will produce an output of 0, as shown in FIG. 13. The resulting sign-predicted ReLU bitmap vector can be used as a basis to determine and eliminate a subset of multiplications and associated coefficient reads from memory, such as in the manner described in other examples above.

[0093] Providing for the generation of a ReLU bitmap that returns to a previous pooling or convolution stage (i.e., a stage before the corresponding ReLU stage) can result in additional power. For example, sign prediction logic can be provided to disable a multiplier when it generates a negative output that is ultimately set to 0 by the ReLU activation logic. For example, this is shown when two sign bits 1310 and 1315 of multiplier 1314 inputs 1301 and 1302 are logically combined by an XOR gate to form PreReLU bitmap bit 1303. This same signal can be used to disable the operation of multiplier 1314. Otherwise, energy would be unnecessarily consumed generating a negative output that would be set to 0 by the ReLU logic before being input for multiplication in the next convolution stage 1390, among other examples.

[0094] Note that the representations of 1300, 1301, 1302, and 1303 (notation A) show a high-level view of what is shown in the representation given in FIG. 13B. In this example, the input to block 1302 may include two floating-point operands. Input 1301 may include an explicit sign bit 1310, an exponent 1311 containing multiple bits, and a mantissa 1312, again containing multiple bits. Similarly, input 1302 may also include a sign 1315, a mantissa 1317, and an exponent 1316. In some implementations, the precision of the mantissa and exponent may vary, since the sign of result 1303 depends only on the signs of 1301 and 1302 or 1310 and 1315, respectively. In fact, neither 1301 nor 1302 need be floating-point numbers, but may be in any integer or fixed-point format, as long as they are signed numbers and the most significant bit (MSB) is effectively the sign bit, explicitly or implicitly (e.g., if the numbers are one's or two's complement).

[0095] 13, two sign inputs 1310 and 1315 may be combined using an XOR (alternatively referred to herein as ExOR or EXOR) gate to generate bitmap bit 1303, which may then be processed using hardware to identify downstream multiplications that may be omitted in the next convolution block (e.g., 1390). The same XOR output 1303 can also be used to disable multiplier 1314 if the two input numbers 1313 (e.g., corresponding to 1301) and 1318 (e.g., corresponding to 1302) have opposite signs and produce a negative output 1304 that would be set to 0 by ReLU block 1319, resulting in a 0 value in the ReLU output vector 13191 input to the next convolution stage 1390. Thus, in some implementations, PreReLU bitmap 1320 may be transmitted in parallel to bitmap scheduler 1360, which may schedule multiplications to be performed (and / or omitted) on convolution unit 1390. For example, for every 0 in bitmap 1320, the corresponding convolution operation may be omitted in convolution unit 1390. In parallel, bitmap 1320 may be consumed by an exemplary address generator 1330, which controls the fetching of weights for use in convolution unit 1390. A list of addresses corresponding to 1s in bitmap 1320 may be compiled in address generator 1330 and controlled via DDR controller 1350 to DDR storage 1370, or otherwise to on-chip RAM 1380. In either case, among other examples, weights corresponding to weights in the PreReLU bitmap 1320 may be fetched and presented to the convolution block 1390 (e.g., after some latency in terms of clock cycles to the weight input 1371), but fetching weights corresponding to 0 may be omitted.

[0096] As mentioned above, in some implementations, a delay (e.g., 1361) may be inserted between the bitmap scheduler 1360 and the convolution unit 1390 to balance the delay through the address generator 1330, the DDR controller 1350, and the DDR 1350, or the path through the address generator 1330 and the internal RAM 1380. The delay may allow the convolutions driven by the bitmap scheduler to properly align with the corresponding weights for the convolution calculation in the convolution unit 1390. Indeed, from a timing perspective, it may be possible to gain additional time by generating the ReLU bitmap earlier than the output of the ReLU block 1319. This additional time may be used to intercept reads to memory (e.g., RAM 1380 or DDR 1370) before they are generated by the address generator 1330, so that some reads (e.g., corresponding to zeros) may be saved. Since memory reads can be much more expensive than on-chip logical operations, eliminating such memory fetches can lead to very significant energy savings, among other exemplary benefits.

[0097] In some implementations, if the savings in terms of clock cycles are still insufficient to cover the DRAM access time, a block-oriented technique can be used to pre-read groups of sign bits (e.g., 1301) from the DDR. These groups of sign bits can be used with blocks of signs from the input image or intermediate convolutional layers 1302 to generate blocks of PreReLU bitmaps using a set of XOR gates 1300 (e.g., to calculate the difference between sign bits in 2D or 3D convolutions between 2D or 3D arrays / matrices, among other examples). In such implementations, an additional 1-bit storage in the DDR or on-chip RAM can be provided to store the sign of each weight, which may make it possible to cover many latency cycles to avoid reading weights from the DDR or RAM that would be multiplied by 0 from the ReLU stage. In some implementations, the additional 1-bit storage per weight in the DDR or on-chip RAM can be avoided because the signs are stored so that they can be addressed independently from the exponent and mantissa, among other example considerations and implementations.

[0098] In some implementations, it can be particularly difficult to access readily available training sets for training machine learning models, including models such as those described above. Indeed, in some cases, training sets may not exist for a particular machine learning application or may not correspond to the type of sensors generating the inputs for the model being trained, among other example problems. In some implementations, synthetic training sets can be developed and utilized to train neural networks or other deep reinforcement learning models. For example, rather than acquiring or capturing a training dataset consisting of hundreds or thousands of images of a particular person, animal, object, product, etc., a synthetic 3D representation of the subject can be generated manually (e.g., using a graphic design or 3D photo editing tool) or automatically (e.g., using a 3D scanner), and the resulting 3D model can be used as a basis for automatically generating training data related to the 3D model subject. This training data can be combined with other training data to form a training dataset composed at least in part of the synthetic training data, which can be utilized to train one or more machine learning models.

[0099] By way of example, a deep reinforcement learning model such as those introduced herein or other machine learning models may be used to enable an autonomous machine to scan shelves in a store, warehouse, or another business to assess the inventory of a particular product in the store. Thus, a machine learning model may be trained to enable an autonomous machine to detect individual products. In some cases, the machine learning model may not only identify the products on a shelf, but also identify the number of products on the shelf (e.g., using a depth model). Rather than training a machine learning model with a series of real-world images (e.g., from the same store or different stores), a synthetic 3D model of each product (or at least some of these products) may be generated (e.g., by the product provider, the machine learning model provider, or another source) for each and every product and configuration of the product that a store may carry (e.g., various displays, various poses or views (full and partial) of the product in various lighting, various orientations of the product packaging, etc.). The 3D model may be of photorealistic or near-photorealistic quality in detail and resolution. 3D models may be provided for consumption along with other 3D models to generate a variety of different views of a given object (e.g., a product), or even a collection of different objects (e.g., a collection of products on a store shelf, with various combinations of products placed next to each other in different lighting and at different orientations), among other example uses, to generate a set of synthetic training data images.

[0100] 14 , a simplified block diagram 1400 of an exemplary computing system (e.g., 1415) implementing a training set generator 1420 for generating synthetic training data used by a machine learning system 1430 to train one or more machine learning models (e.g., 1435) (such as deep reinforcement learning models, Siamese neural networks, convolutional neural networks, and other artificial neural networks) is shown. For example, a 3D scanner 1405 or other tool can be used to generate a set of 3D models 1410 (e.g., of people, interior and / or exterior architecture, landscape elements, products, furniture, transportation elements (e.g., road signs, automobiles, traffic hazards, etc.), and other examples), which can be consumed by providing them as input to the training set generator 1420. In some implementations, the training data generator 1420 can automatically render a set of training images, point cloud, depth map, or other training data 1425 from the 3D model 1410. For example, the training set generator 1420 can be programmatically configured to automatically tilt, rotate, and zoom the 3D model to fill a "complete" and diverse image collection for imaging the object by capturing several different images, and capture a collection of images of the 3D model at different resulting orientations and poses, and in different (e.g., computer-simulated) lighting, with all or part of the entire object of the 3D model captured in the images, etc.

[0101] In some implementations, synthetic training images generated from a 3D model may have photorealistic resolution comparable to the real-world objects on which they are based. In some cases, the training set generator 1420 may be configured to automatically render or generate images or other training data from a 3D model in a manner that intentionally downgrades the resolution and quality of the resulting images (compared to the high-resolution 3D model). For example, the quality of the images may be degraded by adding noise, applying filters (e.g., Gaussian filters), and adjusting one or more rendering parameters to introduce noise, reduce contrast, reduce resolution, or change brightness levels, among other adjustments, to bring the images to a quality level comparable to that which might be generated by a sensor (e.g., a 3D scanner, a camera, etc.) that is expected to provide input to the machine learning model being trained.

[0102] When constructing a dataset, particularly for training a deep neural network, several different conditions or rules may be defined and considered by the training set generator system. For example, CNNs traditionally require a large amount of data for training to produce accurate results. Synthetic data can avoid cases where the available training dataset is too small. Thus, for a particular machine learning model, a target number of training data samples may be identified, and the training set generator may be based on the amount and type of training samples to be generated to meet the desired amount of training samples. Furthermore, conditions for generating sets with sample variability exceeding a threshold may be designed and considered by the training set generator. This is to minimize overfitting of the machine learning model and provide the generalization necessary to perform well in a large number of highly diverse scenarios. Such variability may be achieved through adjustable parameters applied by the training set generator, such as the camera angle, camera height, field of view, lighting conditions, etc., used to generate individual samples from the 3D model, among other examples.

[0103] In some implementations, a sensor model (e.g., 1440) may be provided that defines aspects of a particular type or model of sensor (e.g., a particular 2D or 3D camera, LIDAR sensor, etc.). The model 1440 defines filters and other modifications to be made to raw images, point clouds, or other training data (e.g., generated from a 3D model) to simulate data generated by the modeled sensor (e.g., resolution, vulnerability to glare, sensitivity to light and dark, vulnerability to noise, etc.). In such cases, the training set generator may artificially degrade samples generated from the 3D model to mimic equivalent images or samples generated by the modeled sensor. In this way, synthetic training data samples (e.g., as generated by a real-world version of the sensor) may be generated that are comparable in quality to the data input to a trained machine learning model.

[0104] Referring to FIG. 15 , a block diagram 1500 illustrating an example generation of synthetic training data is shown. For example, a 3D model 1410 of a particular object may be generated. The 3D model may be a lifelike or photorealistic representation of the object. In the example of FIG. 15 , the model 1410 represents a cardboard package containing a set of glass bottles. A collection of images (e.g., 1505) may be generated based on the 3D model 1410, capturing various views of the 3D model 1410, including views of the 3D model in various lighting, environments, and conditions (e.g., in use / at rest, open / closed, damaged, etc.). The collection of images 1505 may be processed using a sensor filter (e.g., defined in the sensor model) of an example training data generator. Processing of the images 1505 may modify the images 1505 to degrade them, thereby generating “lifelike” images 1425 that mimic the quality and characteristics of images captured using a real sensor.

[0105] In some implementations, to aid in generating degraded versions of synthetic training data samples, a model (e.g., 1410) may include metadata to indicate the material and other characteristics of the model's object. In such implementations, the characteristics of the object defined in the model may be considered by the training data generator (e.g., in combination with the sensor model) to determine how an actual image (or point cloud) is likely to be generated by a particular sensor, taking into account lighting, the position of the sensor relative to the modeled object, and the object's characteristics (e.g., material), among other considerations. For example, model 1410 in the particular example of FIG. 15 , which models bottle packaging, may include metadata defining which portions of the 3D model (e.g., which pixels or polygons) correspond to the glass material (the bottle) and which portions correspond to the cardboard (the package). Thus, when the training data generator generates synthetic images and applies sensor filtering 1510 to the images 1505 (e.g., to model the way light reflects off various surfaces of the 3D model), the modeled sensor's response to these properties can be more realistically applied to generate reliable training data that more closely matches what would be produced by the actual sensor if used to generate the training data. For example, depending on the materials modeled in the 3D model, it may be possible to generate training data images that model the sensor's vulnerabilities, producing images with glare, noise, or other imperfections, e.g., corresponding to reflections from the glass surface of the bottle in the example of FIG. 15, but less noise or glare, corresponding to a less reflective cardboard surface. Similarly, the type, temperature, and other properties of the material modeled in the 3D model representation of the subject may affect different sensors (e.g., a camera sensor versus a LIDAR sensor) differently.Thus, an exemplary test data generation system can consider both the 3D model metadata and the particular sensor model when automatically determining filters or processing to apply to image 1505 to generate a degraded version of the image (e.g., 1425) that simulates a version of the image that would likely be produced by a real-world sensor.

[0106] Additionally, in some implementations, further post-processing of the image 1505 may include adjusting depth of field. In some 3D rendering programs, the virtual camera used in the software is perfect and can capture both near and far objects in perfect focus. However, this may not be the case for a real-world camera or sensor (and may be defined as such within the attributes of the corresponding sensor model used by the training set generator). Thus, in some implementations, a depth of field effect may be applied to the image during post-processing (e.g., using the training set generator to create imperfect but more photorealistic examples of images (e.g., 1425) by automatically identifying and selecting points where the camera should appear to be focused on the background and out of focus on the modeled subject features). Further post-processing may include adding noise on the image to simulate noisy artifacts present in photography. For example, the training set generator may include adding noise by limiting the number of light bounces that the ray tracing algorithm calculates on an object, among other example techniques. Additionally, slight pixelation may be applied on top of the rendered model in an effort to remove any overly or unrealistically smooth edges or surfaces that result from the compositing process. For example, an optical blur layer can be added to average out "blocks" of pixels, which, when combined with other post-processing operations (e.g., based on corresponding sensor models), may result in more realistic synthetic training samples.

[0107] As shown in FIG. 15 , when generating training data samples (e.g., 1425) from the example 3D model 1410, the samples (e.g., images, point clouds, etc.) may be added to or included with other real or synthetically generated training samples to build a training dataset for a deep learning model (e.g., 1435). Similarly, as shown in the example of FIG. 14 , a model trainer 1455 (e.g., of the example machine learning system 1430) can be used to train one or more machine learning models 1435. In some cases, synthetic depth images can be generated from the 3D model. The trained machine learning models 1435 can then be used by autonomous machines to perform various tasks such as object recognition, automated inventory processing, navigation, among other examples.

[0108] 14 , a computing system 1415 implementing the exemplary training set generator 1420 may include one or more data processing devices 1445, one or more computer-readable memory elements 1450, and logic implemented in hardware and / or software to implement the training set generator 1420. While the example of FIG. 14 depicts the computing system 1415 (and its components) as separate from the machine learning system 1430 used to train and / or execute the machine learning model (e.g., 1435), it should be understood that in some implementations, a single computing system may be used to implement the combined functionality of two or more of the model generator 1405, the training set generator 1420, and the machine learning system 1430, among other alternative implementations and exemplary architectures.

[0109] In some implementations, a computing system may be provided that enables one-time learning using synthetic training data. Such a system may enable object classification without the need for training on hundreds of thousands of images. One-time learning allows classification from very few training images, even a single training image in some cases. This saves time and resources in developing a training set for training a particular machine learning model. In some implementations, such as the examples shown in FIGS. 16 and 17, the machine learning model may be a neural network that learns to distinguish between two inputs rather than a model that learns to classify inputs. The output of such a machine learning model may identify the similarity between two inputs provided to the model.

[0110] In some implementations, a machine learning system may be provided that can simulate the ability to classify object categories from a small number of training examples, such as the example of FIG. 14 . Such a system may also eliminate the need to create large datasets for multiple classes to effectively train a corresponding machine learning model. Similarly, a machine learning model that does not require training for multiple classes may be selected. A machine learning model may be used to recognize an object (e.g., a product, human, animal, or other object) by providing the system with a single image of the object along with a comparison image. If the comparison image is not recognized by the system, the object is determined to be a mismatch using a machine learning model (e.g., a Siamese network).

[0111] In some implementations, a Siamese network can be used as a machine learning model trained using synthetic training data such as those introduced in the examples above. For example, FIG. 16 shows a simplified block diagram 1600 illustrating an exemplary Siamese network composed of two identical neural networks 1605a and 1605b. Each of these networks has the same weights after training. A comparison block (e.g., 1620) can be provided to evaluate the similarity of the outputs of the two identical networks and compare the determined similarity with a threshold. If the similarity is within a threshold range (e.g., below or above a given threshold), the output of the Siamese network (composed of the neural networks 1605a and 1605b and the comparison block 1620) can indicate (e.g., at 1625) whether the two inputs refer to a common subject. For example, two samples 1610 and 1615 (e.g., images, point clouds, depth images, etc.) can be provided as respective inputs to each of the two identical neural networks 1605a and 1605b. In one example, the neural networks 205a, 1605b may be implemented as ResNet-based networks (e.g., ResNet50 or another variation), and the output of each network may be a feature vector input to the comparison block 1620. In some implementations, the comparison block may generate a similarity vector from the two feature vector inputs to indicate how similar the two inputs 1610, 1615 are. In some implementations, the inputs (e.g., 1610, 1615) to such a Siamese network implementation may comprise two images representing the agent's current observations (e.g., images or depth maps generated by sensors of an autonomous machine) and a target object. A deep Siamese network is a type of two-stream neural network model for discriminative embedded learning and may enable one-time learning using synthetic training data. For example, at least one of the two inputs (e.g., 1610, 1615) may be a synthetically generated training or reference image as described above.

[0112] In some implementations, the execution of Siamese networks or other machine learning models trained using synthetic data can utilize specialized machine learning hardware, such as machine learning accelerators (e.g., Intel Movidius Neural Compute Sticks (NCS)), that can interface with a general-purpose microcomputer, among other example implementations. This system can be used in a variety of applications. For example, the network can be used in security or authentication applications, such as those in which a human, animal, or vehicle is recognized before an actuator that allows access to the human, animal, or vehicle can be triggered. As a specific example, a smart door can be equipped with an image sensor to recognize a human or animal approaching the door and can grant access (using a machine learning model) only to those that match one of a set of authorized users. Such machine learning models (e.g., trained with synthetic data) can also be used in industrial or commercial applications, such as product verification, inventory counting, and other applications that use product recognition in a store to determine whether (or how many) a product is present or whether (or how many) a product is present in a particular location (e.g., on the appropriate shelf), among other examples. For example, as shown in the example illustrated by simplified block diagram 1700 in FIG. 17 , two sample images 1705, 1710 related to consumer products may be provided as inputs to a Siamese network model having threshold determination logic 1720. The Siamese network model 1720 may determine (at 1715) whether the two sample images 1705, 1710 are likely to be images of the same product. Indeed, in some implementations, such a Siamese network model 1720 may enable product identification at various rotations and occlusions. In some cases, 3D models may be utilized to generate multiple reference images for more complex products and objects to increase the level of verification, among other example considerations and features.

[0113] In some implementations, a computing system may comprise logic and hardware adapted to perform machine learning tasks to perform point cloud registration or merging of two or more separate point clouds. Merging point clouds requires finding a transformation that aligns the contents of the point clouds. Such problems are common in applications involving autonomous machines, such as robotic perception applications and building maps of unknown environments, among other use cases.

[0114] In some implementations, using convolutional networks as a solution for finding relative poses between 2D images can provide results comparable to traditional feature-based approaches. Advances in 3D scanning technology can further create multiple datasets with 3D data useful for training neural networks. In some implementations, a machine learning model can be provided that can accept two or more different streams of inputs. Each of the two or more data inputs embodies a respective three-dimensional (3D) point cloud. The two 3D point clouds can be representations of the same physical space or object (or a virtualized version of the physical space or object) measured from two different respective poses. The machine learning model can accept these two 3D point cloud inputs and generate as output a relative or absolute pose indicator between the two 3D point cloud sources. The relative pose information can then be used to generate a global 3D point cloud representation of the environment from multiple snapshots (of the 3D point clouds) of the environment (from one or more different sensors and devices (e.g., multiple drones moving to scan the environment or the same drone)). Relative pose can also be used to determine the relative position of a particular machine within an environment by comparing 3D point cloud input measured by a particular machine with a previously generated global 3D point cloud representation of the environment, among other example uses.

[0115] In one example, a voxelized point cloud processing technique is used to create a 3D grid for sorting points. Here, convolutional layers may be applied, as shown in the example of FIG. 18. In some implementations, the 3D grid or point cloud may be embodied or represented as a voxel-based data structure as described herein. For example, in the example of FIG. 18, a voxel-based data structure (represented by 1810) may be generated from a point cloud 1805 generated from an RGB-D camera or LIDAR scan of an exemplary 3D environment. In some implementations, two point cloud inputs that may be provided to a comparative machine learning model, such as one employing a Siamese network, may be a pair of 3D voxel grids (e.g., 1810). In some cases, the two inputs may first be voxelized (converted to a voxel-based data structure (e.g., 1810)) at any one of several potential voxel resolutions, as described above.

[0116] In one example, depicted in simplified block diagram 1900 of FIG. 19 , the machine learning model may be comprised of a representation portion 1920 and a regression portion 1925. As described above, a Siamese network-based machine learning model may be configured to directly estimate relative camera poses from a pair of 3D voxel grid inputs (e.g., 1905, 1910) or other point cloud data. In some implementations, the 3D voxel grid may be advantageously used in a neural network with traditional convolutional layers, given the organized structure of the voxel grid data. The relative camera poses determined using the network may be used to merge corresponding point clouds of the voxel grid inputs.

[0117] In some implementations, the representation portion 1920 of the exemplary network may include a Siamese network with shared weights and biases. Each branch (or channel of the Siamese network) is formed by successive convolutional layers to extract feature vectors for the respective inputs 1905, 1910. Furthermore, in some implementations, a rectified linear unit (ReLU) may be provided as an activation function after each convolutional layer. In some cases, pooling layers may be omitted to ensure that spatial information in the data is preserved. The feature vectors output from the representation portion 1920 of the network may be combined and input to the regression portion 1925. The regression portion 1925 includes a pair of fully connected layers that can generate an output 1930 representing the relative pose between the two input point clouds 1905, 1910. In some implementations, the regression portion 1925 may consist of two pairs of fully connected layers, one responsible for generating rotation values ​​for the pose estimation and a second pair of layers responsible for generating translation values ​​for the pose. In some implementations, the fully connected layers of the regression portion 1925 may be followed by a ReLU activation function (except for the final layer, since the output may have negative values), among other example features and implementations.

[0118] In some implementations, such as in the example of FIG. 19 above, self-supervised learning may be performed on the machine learning model during a training phase. For example, because the goal of the network (shown in the example of FIG. 19) is to solve a regression problem, a loss function may be provided to guide the network toward that solution. A training phase may be provided to derive a loss function, such as a label-based loss function or a loss function that quantifies the alignment of two point clouds. In one example, an exemplary training phase 2020 may be implemented as shown in simplified block diagram 2000 of FIG. 20. Here, input 2005 is provided to an iterative closest point (ICP)-based method 2015 (e.g., in conjunction with a corresponding CNN 2010) to obtain a database ground truth, which is used to predict pose y, against which a loss function 2025 compares the network prediction. In such an example, the ground truth does not need to have a labeled dataset.

[0119] A trained Siamese network-based model such as that described in the examples of Figures 19 and 20 can be utilized in applications such as 3D map generation and navigation and localization. For example, as shown in the example of Figure 21, such a network can be utilized by a mobile robot (e.g., 2105) or other autonomous machine (in conjunction with machine learning hardware (e.g., NCS device 2110)) to assist in navigating within an environment. The network can also be used to generate a 3D map of the environment 2115 (e.g., which can then be used by the robot or autonomous machine), among other simultaneous localization and mapping (SLAM) applications.

[0120] In some implementations, edge-to-edge machine learning can be used to perform sensor fusion within applications. Such solutions can be applied to regress robot motion over time by fusing data from multiple different sensors. This is a well-studied problem, but current solutions drift over time or are computationally expensive. In some examples, machine learning approaches can be utilized in computer vision tasks while being less susceptible to noise in the data, lighting changes, and subject blur, among other exemplary advantages. For example, convolutional neural networks (CNNs) can be used for object recognition and optical flow calculations. System hardware running CNN-based models can employ hardware components and subsystems such as long short-term memory (LSTM) blocks to achieve additional efficiencies, such as better results for signal regression, among other examples.

[0121] In one example, a system may be provided that utilizes a machine learning model that can accept input from multiple sources of different types of data (e.g., RGB data and IMU data, etc.) to independently overcome the weaknesses of each source (e.g., monocular RGB: lack of scale, IMU: drift over time, etc.). The machine learning module may include a respective neural network (or other machine learning model) tailored to analyze each type of data source. Each type of data source may be coupled and fed into a fully connected layer stage to generate results (e.g., poses) from the multiple data streams. Such a system may be used, for example, in computing systems intended to enable autonomous navigation of machines such as robots, drones, or vehicles, among other example applications.

[0122] For example, as shown in the example of FIG. 22, IMU data may be provided as input to a network tuned for IMU data. IMU data can provide a way to track a subject's movement by measuring acceleration and orientation. However, in some cases, utilizing IMU data alone in a machine learning application can result in drift over time. In some implementations, using an LSTM to track this data relationship over time can help reduce drift. In one example, shown in simplified block diagram 2200 of FIG. 22, a subsequence of n raw accelerometer and gyroscope data elements 2205 (e.g., where each data element consists of six values ​​(three axes each from the IMU's accelerometer and gyroscope)) is used as input to an example LSTM 2210. In another example, input 2205 may be n (e.g., 10) relative poses between image frames (e.g., frame f i and f i+1 n IMU relative attitudes between

number

[0123] Referring to the example of Figure 23, a simplified block diagram 2300 of a network capable of processing a data stream of image data, such as monocular RGB data, is shown. Thus, an RGB CNN portion may be provided that is trained to compute optical flow and may feature dimensionality reduction for pose estimation. For example, several fully connected layers may be provided to reduce dimensionality, and / or the feature vectors may be reshaped as matrices, and a set of four LSTMs may be used to find correspondences between features and reduce dimensionality, among other examples.

[0124] In the example of FIG. 23 , a pre-trained optical flow CNN (e.g., 2310), such as FlowNetSimple, FlowNetCorr, Guided Optical Flow, VINet, or other optical network, can be provided to accept a pair of consecutive RGB images as input 2305. The model can be further constructed to extract a feature vector from the image pair 2305 through the optical flow CNN 2310 and then reduce that vector to obtain a pose vector corresponding to the input 2305. For example, the output of the optical network portion 2310 can be provided to a set of one or more additional convolutional layers 2315 (e.g., utilized to reduce the dimensionality from the output of the optical network portion 2310 and / or remove information used in estimating the flow vector but not required for pose estimation), and the output of these convolutional layers 2315 can be a matrix that can be flattened (at 2320) into a corresponding vector. The vector can be provided to a fully connected layer 2325 to perform a dimensionality reduction of the flattened vector (e.g., from 1536 to 512). This reduced vector may be returned to a reshape block 2330 to convert or reshape the vector into a matrix. A set of four LSTMs 2335, one for each direction of the reshaped matrix (e.g., left-to-right / top-to-bottom, right-to-left / bottom-to-top, top-to-bottom / left-to-right, and bottom-to-top / right-to-left), may then be used to track feature correspondences over time and reduce dimensionality. Among other example implementations, the output of the set of LSTMs 2335 may then be provided to a fully connected layer for rotation 2340 to generate rotation values ​​2350 based on the pair of images 2305, and to a fully connected layer for translation 2345 to generate translation values ​​2355 based on the pair of images 2305.

[0125] Referring to FIG. 24, a simplified block diagram 2400 of a sensor fusion network 2405 is shown. The sensor fusion network 2405 combines the results of an IMU neural network portion 2410 (e.g., as shown in the example of FIG. 22) and an RGB neural network portion 2415 (e.g., as shown in the example of FIG. 23). Such a machine learning model 2405 may further enable sensor fusion by selecting the best from each sensor type. For example, a machine learning model may combine a CNN and an LSTM to provide more robust results (e.g., a CNN can extract features from a pair of consecutive images, and an LSTM can obtain information about the gradual movement of the sensor). In this respect, the outputs of both the CNN and the LSTM are complementary, providing the machine with an accurate estimate of the difference between two consecutive frames (and their relative translation) and their representation in real-world units.

[0126] In the example of FIG. 24 , among other examples, the results of each sensor-specific portion (e.g., 2405, 2410) can be concatenated and provided to a fully connected layer of a combined machine learning model 2405 (or sensor fusion network) to generate a pose result that incorporates both rotational and translational pose. While IMU data and monocular RGB individually may not appear to provide enough information for a reliable solution to the regression problem, combining these data inputs as illustrated and described herein can yield more robust and reliable results (e.g., as shown in the example results shown in graphs 2500a, b of FIGS. 25A and 25B). Such a network 2405 leverages the useful information of both sensor types (e.g., RGB and IMU). For example, in this particular example, the RGB CNN portion 2415 of the network 2405 can extract information about the relative translation between successive images, while the IMU LSTM-based portion 2410 provides scale to the translation. The respective feature vectors output by each portion 2410, 2415 may be fed to a concatenation block 2420 to concatenate the vectors based on a combination of the RGB image 2305 and IMU data 2205 and feed the results to a core fully connected layer 2425, which in turn feeds the results to both a rotation fully connected layer 2430 to generate a rotation value 2440 and a translation fully connected layer 2435 to generate a translation value 2445. While the examples in FIGS. 22-24 show a fusion of RGB and IMU data, it should be understood that other data types can be substituted (e.g., replacing and supplementing IMU data with GPS data) and combined in machine learning models according to the principles described herein. Indeed, more than two data streams (and corresponding neural network portions fed to the concatenation block) may be provided in other exemplary implementations to enable a more robust solution, among other exemplary modifications and alternatives.

[0127] In some implementations, a neural network optimizer may be provided that can identify to a user or system one or more neural networks recommended for a particular application and hardware platform that uses a neural network to perform machine learning tasks. For example, as shown in FIG. 26 , a computing system 2605 may be provided that includes a microprocessor 2610 and computer memory 2615. The computing system 2605 may implement a neural network optimizer 2620. The neural network optimizer 2620 may include, among other example hardware, an execution engine that executes a set of machine learning tasks (e.g., 2635) using machine learning hardware (e.g., 2625) as described herein. Neural network optimizer 2620 may further include one or more probes (e.g., 2630) for monitoring the execution of a machine learning task as it is performed by machine learning hardware 2625 using one of a set of neural networks selected by neural network optimizer 2620 for execution on and monitored by neural network optimizer 2620. Probe 2630 may measure attributes such as the power consumed by machine learning hardware 2625 while performing the task, the temperature of the machine learning hardware while it is running, the speed at which or the time elapsed to complete a task using a particular neural network, the accuracy of the task results using a particular neural network, the amount of memory utilized (e.g., to store the neural network being used), among other example parameters.

[0128] In some implementations, the computing system 2605 can interface with a neural network generation system (e.g., 2640). In some implementations, the computing system (e.g., 2605) that evaluates neural networks and the neural network generation system 2640 can be implemented on the same computing system. The neural network generation system 2640 can enable a user to manually design neural network models (e.g., CNNs) for various tasks and solutions. In some implementations, the neural network generation system 2640 can further include a repository 2645 of previously generated neural networks. In one example, the neural network generation system 2640 (e.g., a system such as CAFFE, TensorFlow, etc.) can generate a set of neural networks 2650. This set can be randomly generated, new neural networks can be generated from scratch (e.g., based on some generalized parameters suitable for a given application or according to a common neural network type or category), and / or neural networks can be randomly selected from the repository 2645.

[0129] In some implementations, a set of neural networks 2650 may be generated by the neural network generation system 2640 and provided to the neural network optimizer 2620. The neural network optimizer 2620 may use each of the set of neural networks 2650 to cause a particular machine learning hardware (e.g., 2625) to perform a standardized set of one or more machine learning tasks. The neural network optimizer 2620 may monitor task performance in connection with the use of each of the set of neural networks 2650 by the hardware 2625. The neural network optimizer 2620 may further accept data as input to identify which parameters or characteristics, as measured by the neural network optimizer's probes (e.g., 2630), are most highly weighted or prioritized by the neural network optimizer in determining the "best" of the set of neural networks. Based on these criteria and the neural network optimizer's observations during use of each of a set of (e.g., randomly generated) neural networks, the neural network optimizer 2620 can identify and provide the best-performing neural network to a particular machine learning hardware (e.g., 2625) based on the provided criteria. In some implementations, the neural network optimizer can automatically provide this best-performing neural network to the hardware for further use, training, etc.

[0130] In some implementations, the neural network optimizer can employ evolutionary search to iteratively improve the results identified from an initial (e.g., randomly generated) set of neural networks evaluated by the neural network optimizer (e.g., 2620). For example, the neural network optimizer can identify characteristics of one or more best-performing neural networks from the initial set evaluated by the neural network optimizer. The neural network optimizer can then send a request to the neural network generator (e.g., 2640) to generate another diverse set of neural networks having characteristics similar to those identified in the best-performing neural network for the particular hardware (e.g., 2625). The neural network optimizer 2620 can then repeat its evaluation using a next set or generation of neural networks generated by the neural network generator based on the best-performing neural networks from the initial batch evaluated by the neural network optimizer. As before, neural network optimizer 2620 can identify which of this second-generation neural network performed optimally according to the provided criteria and can again determine the characteristics of the best-performing neural network in the second generation as a basis for sending a request to the neural network generator to generate a third-generation neural network for evaluation. Thus, neural network optimizer 2620 is used to iteratively evaluate neural networks as they evolve (and theoretically improve) from one generation to the next. As in the previous example, neural network optimizer 2620 can provide an indication or copy of the latest-generation best-performing neural network for use by machine learning hardware (e.g., 2625), among other example implementations.

[0131] As a specific example shown in block diagram 2700 of FIG. 27, machine learning hardware 2625, such as Movidius NCS, may be utilized with a neural network optimizer that acts as a design space exploration tool, using a neural network generator 2640 or a provider such as CAFFE to find the network with the highest accuracy, subject to hardware constraints. Such design space exploration (DSX) tools may be provided to utilize a complete API, including network graphs and bandwidth measurements. Furthermore, several extensions may be added or provided to the machine learning hardware API to derive additional parameters useful for design space exploration, such as temperature measurements, inference time measurements, among other examples.

[0132] To demonstrate the power of the DSX concept, an example is provided that explores the neural network design space for a small, always-on face detector, such as those implemented in modern mobile phones that wake up on face detection. Various neural networks may be provided to the machine learning hardware, and performance may be monitored for each use of the neural network, including the power usage of the trained network in the inference stage. The DSX tool (or neural network optimizer) can generate different neural networks for a given classification task. Data may be transferred to the hardware (e.g., in the case of the NCS, data may be transferred to and from the NCS via USB using the NCS API). With the implementation described above, the design space exploration results in finding optimal models for multiple different purposes, rather than manually editing, copying, and pasting arbitrary files. To illustrate, FIG. 28 shows a table 2800 illustrating exemplary results of the DSX tool's evaluation of multiple different randomly generated neural networks. The results include performance characteristics of the machine learning hardware (e.g., a general-purpose microprocessor connected to the NCS) during the execution of machine learning tasks using each of the neural networks (e.g., showing accuracy, execution time, temperature, size of the neural network in memory, and measured power, among other example parameters that may be measured by a DSX tool). Figure 29 shows the results of comparing validation accuracy with execution time (2900) and validation accuracy with size (2905). These relationships and ratios may be considered by the NCS, among other examples, when determining which of the evaluated neural networks is "best" for a particular machine learning platform.

[0133] Deep neural networks (DNNs) provide state-of-the-art accuracy for a variety of computer vision tasks, including image classification and object detection. However, the success of DNNs is often achieved through significant increases in computation and memory, making them challenging to deploy on resource-constrained inference edge devices. In some implementations, network compression techniques such as pruning and quantization can reduce computation and memory demands. This can also help prevent overfitting with little or no loss in accuracy, especially in the case of transfer learning on small, custom datasets.

[0134] In some implementations, a neural network optimizer (e.g., 2620) or other tool may also be provided to dynamically and automatically reduce the size of the neural network used by particular machine learning hardware. For example, the neural network optimizer may perform fine-grained pruning (e.g., pruning of connections or weights) and coarse-grained pruning (e.g., pruning of kernels, neurons, or channels) to reduce the size of the neural network stored and operated by the given machine learning hardware. In some implementations, the machine learning hardware (e.g., 2625) may include arithmetic circuitry capable of performing sparse matrix multiplication to enable the hardware to efficiently process weight-pruned neural networks.

[0135] In one implementation, the neural network optimizer 2620 or other tool can perform hybrid pruning of the neural network to prune at both the kernel and weight levels (e.g., as shown in block diagram 3000a of FIG. 30A ). For example, one or more algorithms, rules, or parameters can be considered by the neural network optimizer 2620 or other tool to automatically identify (at 3005) a set of kernels or channels that can be pruned 3010 from a given neural network. After this first channel pruning phase is completed (at 3015), weight pruning 3020 can be performed for the remaining channels 3015, as shown in the illustrative diagram of FIG. 30A . For example, rules can govern weight pruning such that a threshold is set such that weights below the threshold are pruned (at 3025) (e.g., reassigned a weight of “0”). The hybrid pruned network may then be run or iterated to recover the network's accuracy from pruning, resulting in a compact version of the network without detrimentally reducing the model's accuracy. Furthermore, in some implementations, the weights remaining after pruning may be quantized to further reduce the amount of memory required to store the pruned model's weights. For example, as shown in block diagram 3100 of FIG. 31 , a log-scale quantization 3110 may be performed such that floating-point weight values ​​(3105) are replaced with their nearest radix-2 counterparts (3115). In this way, 32-bit floating-point values ​​may be replaced with 4-bit radix-2 values, dramatically reducing the amount of memory required to store the network's weights while only minimally losing accuracy in the compact neural network (e.g., as shown in the exemplary results shown in table 3200 of FIG. 32 ), among other exemplary quantizations and features. Indeed, in the particular example of Figure 32, a study of the application of hybrid pruning is shown as applied to an exemplary neural network such as ResNet50.Furthermore, we present the application of weight quantization to a pruned, sparse, and thin ResNet50 to further reduce the model size and make it more hardware-friendly.

[0136] As shown in simplified block diagram 3000b of FIG. 30B , in one example, hybrid pruning of an example neural network may be performed by accessing an initial or baseline neural network model 3035 and (optionally) training the model with regularization (L1, L2, or L0) 3040. The importance of individual neurons (or connections) within the network may be evaluated 3045, and neurons determined to be less important may be pruned (3050) from the network. The pruned network may be fine-tuned 3055, and a final compact (or sparse) network may be generated from this pruning 3060. As shown in the example of FIG. 30B , in some cases, the network may be iteratively pruned by performing further training and pruning (e.g., 3040-3050) following fine-tuning of the pruned network 3055, among other example implementations. In some implementations, determining neuron importance 3045 may be performed using hybrid pruning techniques such as those described above. For example, fine-grained weight pruning / sparsification may be performed (e.g., by global incremental pruning using (mean+std*factor)). Coarse-grained channel pruning may be performed layer-by-layer (e.g., weight sum pruning) based on sensitivity tests and / or some target MAC. Coarse pruning may be performed before sparse pruning. For example, weight quantization may be performed to set constraints on non-zero weights to 0 or powers of 2 and / or to use 1 bit for 0 and 4 bits to represent weights. In some cases, low-precision quantization (e.g., weights and activations) may be performed, among other example techniques. Pruning techniques such as those described above may provide various example benefits. For example, compact matrices may reduce the size of stored network parameters, and decompression of weights at runtime may reduce DDR bandwidth. Accelerated computation may be provided, among other example advantages.

[0137] 33A is a simplified flow diagram 3300a of an exemplary technique for generating a training dataset including synthetic training data samples (e.g., synthetically generated images or synthetically generated point clouds). For example, a digital 3D model can be accessed 3302 from computer memory, and multiple training samples can be generated 3304 from various views of the digital 3D model. These training samples can be modified 3306 to add imperfections to simulate training samples generated by one or more real-world samples. A training dataset can be generated 3308 to include the modified synthetically generated training samples. The generated training dataset can be used to train 3310 one or more neural networks.

[0138] FIG. 33B is a simplified flow diagram 3300b of an exemplary technique for performing one-time classification using a Siamese neural network model. A subject input can be provided as an input to a first portion of the Siamese neural network model 3312, and a reference input can be provided as an input to a second portion of the Siamese neural network model 3314. The first and second portions of the model can be identical and can have identical weights. A Siamese network output can be generated 3316 from the outputs of the first and second portions based on the subject input and the reference input, such as a difference vector. For example, based on a similarity threshold for the output of the Siamese neural network model, it can be determined 3318 that the output indicates whether the subject input is appropriately similar to the reference input (e.g., to indicate that the subject of the subject input is the same as the subject of the reference input).

[0139] 33C is a simplified flow diagram 3300c of an exemplary technique for determining a relative pose using an exemplary Siamese neural network model. For example, a first input can be received 3320 as an input to a first portion of a Siamese neural network model, where the first input represents a view of 3D space (e.g., point cloud data, depth map data, etc.) from a first pose (e.g., of an autonomous machine). A second input can be received 3322 as an input to a second portion of the Siamese neural network model, where the second input represents a view of 3D space from a second pose. An output 3324 of the Siamese network can be generated based on the first and second inputs, where the output represents a relative pose between the first and second poses. Based on the determined relative pose, a machine position associated with the first and second poses and / or a 3D map in which the machine resides can be determined 3326.

[0140] FIG. 33D is a simplified flow diagram 3300d of an exemplary technique including a sensor fusion machine learning model. The sensor fusion machine learning model combines at least portions of two or more machine learning models that are tuned for use with corresponding ones of two or more different data types. First sensor data of a first type can be received as input 3330 in a first of the two or more machine learning models in the sensor fusion machine learning model. Second sensor data of a second type (e.g., generated simultaneously with the first sensor data (e.g., by a sensor on the same or a different machine)) can be received as input 3332 to a second of the two or more machine learning models. Outputs of the first and second machine learning models can be concatenated 3334, and the concatenated output can be provided 3336 to a set of fully connected layers of the sensor fusion machine learning model. The sensor fusion machine learning model can generate an output based on the first and second sensor data to define 3338 a pose of a device (e.g., a machine on which sensors generating the first and second sensor data are located).

[0141] FIG. 33E is a simplified flow diagram 3300e of an exemplary technique for generating improved or optimized neural networks tailored to specific machine learning hardware according to an evolutionary algorithm. For example, a set of neural networks can be accessed 3340, or a set of neural networks can be generated (e.g., automatically according to randomly selected attributes). A specific piece of hardware can perform machine learning tasks using the set of neural networks 3342, and performance attributes of the specific hardware for these tasks can be monitored 3344. Based on the results of this monitoring, one or more top-performing neural networks in the set can be identified 3346. Characteristics of the top-performing neural networks (for the specific hardware) can be determined 3348, and another set of neural networks including such characteristics can be generated 3350. In some cases, this new set of neural networks can be tested (e.g., via steps 3342-3348) to iteratively improve the set of neural networks being considered for use with the hardware until one or more sufficiently high-performing or optimized neural networks are identified for the specific hardware.

[0142] 33F is a simplified flow diagram 3300f of an exemplary technique for pruning a neural network. For example, a neural network can be identified 3352, and a subset of the neural network's kernels can be determined 3354 as less important or otherwise good pruning candidates. This subset of kernels can be pruned 3356 to generate a pruned version of the neural network. The remaining kernels can then be further pruned 3358 to prune a subset of weights from these remaining kernels, further pruning the neural network at both the coarse-grained and fine-grained levels.

[0143] Figure 34 is a simplified block diagram of an exemplary multi-slot vector processor (e.g., a very long instruction word (VLIW) vector processor) according to some embodiments. In this example, the vector processor may include multiple (e.g., nine) functional units (e.g., 3403-3411), which may be fed by a multi-port memory system 3400 backed by a vector register file (VRF) 3401 and a general-purpose register file (GRF) 3402. The processor includes an instruction decoder (IDEC) 3412 that decodes instructions and generates control signals to control the functional units 3403-3411. The functional units 3403-3411 are a predicated execution unit (PEU) 3403, a branch and repeat unit (BRU) 3404, load store port units (e.g., LSU0 3405 and LSU1 3406), a vector operation unit (VAU) 3407, a scalar operation unit (SAU) 3410, a compare and move unit (CMU) 3408, an integer operation unit (IAU) 3411, and a volumetric acceleration unit (VXU) 3409. In this particular implementation, the VXU 3409 can accelerate operations on volumetric data, including both arithmetic, logical, and storage / retrieval operations. While the example of FIG. 34 shows the VXU circuit 3409 as a single component, it should be understood that the functionality of the VXU (and the other functional units 3403-3411) may be distributed across multiple circuits. Furthermore, in some implementations, the functionality of VXU 3409 may be distributed within one or more of the processor's other functional units (e.g., 3403-3408, 3410, 3411), among other example implementations.

[0144] 35 is a simplified block diagram illustrating an example implementation of a VXU 3500 according to some embodiments. For example, VXU 3500 may provide at least one 64-bit input port 3501 to accept input from vector register file 3401 or general-purpose register file 3402. This input may be connected to multiple functional units, including register file 3503, address generator 3504, point addressing logic 3505, point insertion logic 3506, point deletion logic 3507, 3D to 2D projection logic in X dimension 3508, 3D to 2D projection logic in Y dimension 3509, 3D to 2D projection logic in Z dimension 3510, 2D histogram pyramid generator 3511, 3D histopyramid generator 3512, population counter 3513, 2D pathfinding logic 3514, 3D pathfinding logic 3515, and possibly further functional units for operating on 64-bit unsigned integer volumetric bitmaps. Output from block 3502 may be written back to vector register file VRF 3401 or general purpose register file GRF 3402.

[0145] Referring to the example of FIG. 36 , a representation of the organization of a 4^3 voxel cube 3600 is shown. A second voxel cube 3601 is also shown. In this example, the voxel cube can be defined in the data as a 64-bit integer 3602, where each voxel in the cube is represented by a single corresponding bit of the 64-bit integer. For example, voxel 3602 at address {x, y, z}={3, 0, 3} can be set to “1” to indicate that geometry exists at that coordinate in the volumetric space represented by voxel cube 3601. Furthermore, in this example, all other voxels (other than voxel 3602) may correspond to “empty” space and, among other examples, can be set to “0” to indicate that there is no physical geometry at those coordinates. Referring to FIG. 37 , an exemplary two-level sparse voxel tree 3700 according to some embodiments is shown. In this example, only a single “occupied” voxel is contained within the volume (e.g., at location {15, 0, 15}). The top level -0 3701 of the tree in this case contains a single voxel entry {3, 0, 3}. That voxel then points to the next level 3702 of the tree, which also contains a single voxel at element {3, 0, 3}. The entry in the data structure corresponding to level 0 of the sparse voxel tree is a 64-bit integer 3703 with one voxel set as occupied. Setting the voxel means that an array of 64-bit integers is then assigned to level 1 of the tree, corresponding to the voxel volume set in 3703. In the level 1 subarray 3704, only one of the voxels is set as occupied, and all others are set as unoccupied. Because the tree is a two-level tree in this example, level 1 represents the bottom of the tree, and as a result, the hierarchy ends here.

[0146] Figure 38 illustrates a two-level sparse voxel tree 3800 according to some embodiments, including occupied voxels at locations {15, 0, 3} and {15, 0, 15} for a particular volume. The top-level-0 of the tree 3801 in this case (which subdivides the particular volume into 64 top-level-0 voxels) includes two voxel entries: {3, 0, 0} and {3, 0, 3}, and the corresponding data 3804 indicates that the two voxels are set (or occupied). The next level of the sparse voxel tree (SVT) is provided as an array of 64-bit integers containing two subcubes 3802 and 3803, one for each voxel set at level 0. In level 1 subarray 3805, two voxels, v15 and v63, are set as occupied, and all other voxels are set as unoccupied and tree. This format is flexible. This is because the 64 entries in the next level of the tree are always allocated corresponding to each configured voxel in the upper layer of the tree. This flexibility may allow dynamically changing scene geometry to be inserted into an existing volumetric data structure in a flexible manner (i.e., not in a fixed order such as random), as long as the corresponding voxels in the upper layer are configured. Otherwise, a table of pointers would have to be maintained, resulting in high memory requirements, or the tree would have to be at least partially restructured to insert unexpected geometry.

[0147] FIG. 39 illustrates an alternative technique for storing voxels from FIG. 38 , according to some embodiments. In this example, the entire volume 3900 includes two voxels stored at global coordinates {15, 0, 3} and {15, 0, 15}, as in FIG. 23 . In this approach, rather than allocating a 64-entry array to represent all of the level-1 subcubes below level 0, only those level-1 elements that actually contain geometry (e.g., as indicated by whether the corresponding level-0 voxel is occupied) are allocated as corresponding 64-bit level-1 records. As a result, level 1 in this example has only two 64-bit entries (i.e., for each of the 64 level-1 voxels, whether occupied or empty) rather than sixty-four 64-bit entries. Thus, in this example, the first level-0 3904 is equivalent to 3804 in FIG. 38 , but the next level 3905 has 62 times smaller memory requirements than its counterpart 3805 in FIG. 38 . In some implementations, if you insert new geometry into level 0 that has no space allocated in level 1, the tree must be copied and rearranged.

[0148] In the example of Figure 39, the subvolume can be derived by counting occupied voxels in layers above the current layer. In this way, the system can determine where one higher layer ends and the next lower layer begins in the voxel data. For example, if three layer 0 voxels are occupied, the system can predict that three corresponding layer 1 entries follow in the voxel data, and that the next entry (after these three) corresponds to the first entry in layer 2, and so on. Such optimal compression can be very useful when certain parts of a scene do not change over time, or when an application requires remote transmission of volumetric data, such as from a space probe scanning the surface of Pluto, where transmitting every bit is costly and time-consuming.

[0149] FIG. 40 illustrates how voxels can be inserted into a 4^3 cube, represented as a 64-bit integer volumetric data structure entry, to reflect changes to geometry within the corresponding volume, according to some embodiments. In one example, as shown in 4000, each voxel cube can be organized as four logical 16-bit planes within a 64-bit integer. Among other example organizations, each plane corresponds to a Z value from 0 to 3; within each plane, each y value encodes four logical 4-bit displacements from 0 to 3; and finally, within each 4-bit y plane, each bit encodes four possible x values ​​from 0 to 3. Thus, in this example, to insert a voxel into the 4^3 volume, one can first shift one bit by the x value from 0 to 3, then shift that value by 0 / 4 / 8 / 12 bits to encode the y value; and finally, the z value can be represented by a shift of 0 / 16 / 32 / 48 bits, as shown in the C code expression in 4001. Finally, since each 64-bit integer can be a combination of up to 64 voxels, and each of these voxels is written separately, the new bitmap must be logically combined with the old 64-bit value read from the sparse voxel tree by ORing the old and new bitmap values, as shown at 4002.

[0150] Referring to FIG. 41 , a representation is shown to illustrate how a 3D volumetric object stored in a 64-bit integer 4100 may be projected by logically ORing in the X direction to generate a 2D pattern 4101, in the Y direction to generate a 2D output 4102, and finally in the Z direction to generate the pattern shown at 4103, according to some embodiments. FIG. 42 illustrates how bits from an input 64-bit integer may be logically ORed to generate an output projection in X, Y, and Z, according to some embodiments. In this example, table 4201 indicates, column by column, which element indexes from input vector 4200 are ORed to generate X projection output vector 4202. Table 4203 indicates, column by column, which element indexes from input vector 4200 are ORed to generate Y projection output vector 4204. Finally, 4205 indicates, column by column, which element indexes from input vector 4200 are ORed to generate Z projection output vector 4206.

[0151] The X projection logically ORs bits 0, 1, 2, and 3 from input data 4200 to generate bit 0 of X projection 4201. For example, bit 1 of 4201 may be generated by ORing bits 4, 5, 6, and 7, etc., from 4200. Similarly, bit 0 of Y projection 4204 may be generated by ORing bits 0, 4, 8, and 12 of 4200 together. Bit 1 of 4204 may be generated by ORing bits 1, 5, 9, and 13, etc., of 4200 together. Finally, bit 0 of Z projection 4206 may be generated by ORing bits 0, 16, 32, and 48 of 4200 together. Bit 1 of 4206 may be generated by ORing bits 1, 17, 33, and 49, etc., of 4200 together.

[0152] FIG. 43 shows an example of how a simplified map may be generated using projections, according to some embodiments. The goal in this scenario may be to generate a compact 2D map of a path traversed by a vehicle 4300 of height h 4310 and width w 4301 from a voxel volume 4302. Here, Y projection logic may be used to generate an initial coarse 2D map 4303 from the voxel volume 4302. In some implementations, the map may be processed to check whether a particular vehicle (e.g., a car (or autonomous vehicle), drone, etc.) of particular dimensions can pass through the path's width constraints 4301 and height constraints 4310. This may be done to ensure the path is traversable by performing a projection in Z and checking the width constraint 4301, and the projection in Y may be masked to limit the calculation to the vehicle's height 4310. With further post-processing (e.g., in software), it can be seen that for a route that is traversable and satisfies width and height constraints, only the X and Z coordinates of points A4304, B4305, C4306, D4307, E4308, and F4309 along the route can be stored or transmitted over a network to completely reconstruct a legal route that a vehicle can travel. Assuming the route can be decomposed into such piecewise segments, a route can be completely described with only one or two bytes per piecewise linear section of the route. This can be useful for fast transmission and processing of such route data (e.g., by autonomous vehicles), among other examples.

[0153] FIG. 44 illustrates how volumetric 3D measurements or simple 2D measurements from embedded devices may be aggregated by mathematical means according to some embodiments to generate a crowdsourced, high-quality map instead of taking accurate measurements using LIDAR or other expensive means. In the proposed system, multiple embedded devices 4400, 4401, etc. may be equipped with various sensors capable of taking measurements that may be transmitted to a central server 4410. Software running on the server performs an aggregation 4402 of all of the measurements and performs a numerical solution of the resulting matrix with a nonlinear solver 4403 to generate a high-precision map. This map may then be redistributed to the embedded devices. In fact, data aggregation may also include high-precision survey data from satellites 4420, airborne LIDAR surveys 4421, and terrestrial LIDAR measurements 4422 to increase the accuracy of the resulting map when these high-fidelity datasets are available. In some implementations, the maps and / or recorded measurements may be generated in, converted to, or otherwise represented using a sparse voxel data structure in a format as described herein, among other example implementations.

[0154] Figure 45 illustrates how 2D pathfinding on a 2D 2x2 bitmap may be accelerated, according to some embodiments. The principle of operation is that for connectivity to exist between points on the map in the same grid cell, the values ​​of either the x or y cell, or consecutive runs of x and y cells, must all be set to 1. Therefore, a logical AND of the bits drawn from those cells can be instantiated to test the bitmap in the grid to see if a valid path exists, and a different AND gate can be instantiated for each valid path through the NxN grid. In some cases, this approach can even achieve 2D paths on an 8x8 2D grid. 64This can introduce combinatorial complexity in that a grid may contain one or more valid paths. Therefore, in some improved implementations, the grid can be reduced to 2x2 or 4x4 tiles, allowing connectivity testing to be performed hierarchically. 2x2 bitmap 4500 contains four bits labeled b0, b1, b2, and b3. These four bits can take on values ​​0000 through 1111, with corresponding labels 4501 through 4517. Each of these bit patterns represents a different level of connectivity between the faces of the 2x2 grid, labeled 4521 through 4530. For example, if 2x2 grid 4500 contains bitmaps 1010 (7112), 1011 (7113), 1110 (7116), or 1111 (7117), then there exists bit 4521 or v0, which represents vertical connectivity between x0 and y0 of 4500. The two-input logical AND of 4500, b0 and b3, as shown in row 1 of table 4518, produces a connectivity map v0 that can be used by higher-level hardware or software to determine global connectivity through a global grid subdivided into 2x2 subgrids. If the global map contains an odd number of grid points on the x- or y-axis, the top-level grid must be padded to the next highest even grid point (e.g., resulting in the addition of an additional row of zeros to the x- and / or y-axis of the global grid). Figure 45 also shows an exemplary 7x7 grid 4550. This grid shows how to pad to 8x8 by adding an additional row 4532 and column 4534 filled with zeros. In this example, the NxN map 4550 can be progressively subsampled down to a 2x2 map to speed up pathfinding compared to other techniques (e.g., depth-first search, breadth-first search, or Dijkstra algorithm, or other graph-based approaches). For example, in this example, cell W of 4540 is populated by ORing the contents of cells A, B, C, and D, etc., of 4550. Similarly, the bits of the 2x2 cells of 4540 are ORed together to populate the cells of 4542. For pathfinding, the algorithm starts with the smallest 2x2 representation of grid 4542 and tests each of the bits.Only the portion of the 4×4 grid of 4540 (composed of four 2×2 grids) that corresponds to a 1 bit in the 2×2 grid 4542 needs to be tested for connectivity, since it can be seen that a 0 bit means that there is no corresponding 2×2 grid cell in 4540. This approach can also be used to search the 8×8 grid of 4520. For example, if cell W in 4540 contains a 0, it can be seen that there is no path in ABCD, such as 4520. This approach prunes branches from the graph search algorithm used, whether it be A*, Dijkstra, DFS, BFS, or variations thereof. Additionally, using a 2×2 organization of the hardware basic path finder 4518 can further limit the computations involved. In fact, a 4×4 basic hardware element may be constructed using five 2×2 hardware blocks in the same arrangement as 4540 and 4542, further constraining the amount of graph searching that needs to be performed. Furthermore, an 8x8 hardware based search engine can be constructed with 21 2x2 HW blocks (7118) with the same arrangement as 4542, 4540, 4500, etc., potentially in any NxN topology.

[0155] Figure 46 is a simplified block diagram illustrating how the proposed volumetric data structure may be used to accelerate collision detection, according to some embodiments. A 3D NxNxN map of geometry may be subsampled into a pyramid consisting of the lowest level of detail (LoD) 2x2x2 volume 4602, the next highest 4x4x4 volume 4601, the 8x8x8 volume 4600, and so on, up to NxNxN. If the location of a drone, vehicle, or robot 4605 in 3D space is known via a positioning means such as GPS or via relocalization from the 3D map, that location may be quickly used to test whether geometry is present in the relevant quadrant of the 2x2x2 sub-volume by appropriately scaling the drone / robot's x, y, and z positions (dividing them by the relevant number of times) and querying 4602 for the presence of geometry (e.g., checking whether the corresponding bitmap bit indicates a potential collision). If a potential collision exists (e.g. a "1" is found), further checks can be performed on volumes 4601, 4600, etc. to see if the drone / robot can move there. However, if the voxel in 4602 is empty (e.g. a "0"), the robot / drone can interpret this as free space and operate its directional controls to move freely through most of the map.

[0156] Although some of the systems and solutions described and illustrated herein are described as including or associated with multiple elements, not all elements explicitly illustrated or described may be utilized in every alternative implementation of the present disclosure. Furthermore, while one or more of the elements described herein may be located external to the system, in other examples, a particular element may be included within or as part of one or more of the other elements described and not described in the illustrated implementation. Furthermore, a particular element may be combined with other components and used for alternative or additional purposes in addition to those described herein.

[0157] Furthermore, it should be understood that the examples presented above are merely non-limiting examples provided for the purpose of illustrating certain principles and features, and do not necessarily limit or restrict potential implementations of the concepts described herein. For example, a variety of different embodiments can be realized using various combinations of the features and components described herein, including combinations realized through various implementations of the components described herein. Other implementations, features, and details should be apparent from the contents of this specification.

[0158] 47-52 are block diagrams of exemplary computer architectures that may be used in accordance with embodiments disclosed herein. Indeed, the computing devices, processors, and other logic and circuitry of the systems described herein may incorporate all or a portion of the functionality and supporting software and / or hardware circuitry for implementing such functionality. Moreover, other computer architecture designs known in the art for processors and computing systems may also be used beyond the examples shown. In general, computer architectures suitable for embodiments disclosed herein may include, but are not limited to, the configurations shown in FIGS. 47-52.

[0159] 47 illustrates an example domain topology of respective Internet of Things (IoT) networks coupled via links to respective gateways. The Internet of Things (IoT) is a concept in which a large number of computing devices are interconnected with each other and to the Internet to provide functionality and data acquisition at a very low level. Thus, as used herein, an IoT device can include, among other things, semi-autonomous devices that communicate with other IoT devices and wider networks, such as the Internet, to perform functions such as sensing or control. Such IoT devices can include logic and memory for implementing and using hash tables such as those introduced above.

[0160] IoT devices often have limited memory, size, or functionality, allowing for larger numbers to be deployed at a similar cost to fewer, larger devices. However, IoT devices may be smartphones, laptops, tablets, or PCs, or other larger devices. Additionally, IoT devices may be virtual devices, such as applications on smartphones or other computing devices. IoT devices may include IoT gateways, used for data storage, process control, and the like, to connect IoT devices to other IoT devices and cloud applications.

[0161] A network of IoT devices can include commercial and home automation devices such as water distribution systems, power distribution systems, pipeline control systems, plant control systems, light switches, thermostats, locks, cameras, alarms, and motion sensors. IoT devices can be accessed through remote computers, servers, and other systems, for example, to control the systems or access data.

[0162] The future growth of the Internet and similar networks may involve a large number of IoT devices. Therefore, in the context of the technology described herein, several innovations for such future networking address the need for all these layers to grow unimpeded and support the ability to discover and create accessible connected resources, as well as hide and compartmentalize connected resources. Any number of network protocols and communication standards can be used, with each protocol and standard designed to address a specific purpose. Furthermore, protocols are part of the mechanisms supporting human-accessible services that operate regardless of location, time, or space. Innovations include service provisioning and associated infrastructure, such as hardware and software, enhanced security, and the provision of services based on service levels and quality of service (QoS) conditions specified in service provision agreements. As will be appreciated, the use of IoT devices and networks such as those introduced in Figures 47 and 48 presents several new challenges in heterogeneous networks of connectivity, including a combination of wired and wireless technologies.

[0163] Specifically, FIG. 47 provides a simplified diagram of a domain topology that may be used for several Internet of Things (IoT) networks including IoT device 4704. IoT networks 4756, 4758, 4760, and 4762 are coupled to respective gateways 4754 via backbone links 4702. For example, several IoT devices 4704 may communicate with gateway 4754 and with each other via gateway 4754. To simplify the drawing, not all IoT devices 4704 or communication links (e.g., links 4716, 4722, 4728, or 4732) have been labeled. Backbone links 4702 may include any number of wired or wireless technologies, including optical networks, and may be part of a local area network (LAN), a wide area network (WAN), or the Internet. Furthermore, such communication links facilitate optical signal paths between both IoT devices 4704 and gateways 4754, including the use of MUX / deMUX components to facilitate interconnection of various devices.

[0164] The network topology may include any number of types of IoT networks, such as a mesh network comprising a network 4756 using Bluetooth® Low Energy (BLE) links 4722. Other types of IoT networks that may be present include a wireless local area network (WLAN) network 4758 used to communicate with IoT devices 4704 via IEEE 802.11 (Wi-Fi®) links 4728, a cellular network 4760 used to communicate with IoT devices 4704 via an LTE / LTE-A (4G) or 5G cellular network, and a low power wide-area (LPWA) network 4762, such as an LPWA network compatible with the LoRaWan specification promulgated by the LoRa Alliance, or an IPv6 over Low Power Wide-Area Network (LPWAN) network compatible with specifications promulgated by the Internet Engineering Task Force (IETF). Furthermore, each IoT network can communicate with external network providers (e.g., Tier 2 or Tier 3 providers) using any number of communication links, such as LTE cellular links, LPWA links, or links based on the IEEE 802.15.4 standard like Zigbee. Each IoT network can also operate using various network protocols and Internet application protocols, such as the Constrained Application Protocol (CoAP). Each IoT network can also be integrated with a coordinator device that provides a chain of links that form a cluster tree of linked devices and networks.

[0165] Each of these IoT networks can offer opportunities for new technological features, such as those described herein. Improved technologies and networks can enable exponential growth of devices and networks, including the use of IoT networks as fog devices or systems. As the use of such improved technologies increases, IoT networks can be developed for self-management, functional evolution, and collaboration without the need for direct human intervention. Improved technologies may even enable IoT networks to function without a centralized system. Thus, the improved technologies described herein can be used to automate and enhance network management and operational capabilities far beyond current implementations.

[0166] In one example, communications between IoT devices 4704, such as over backbone links 4702, can be secured by a distributed system for authentication, authorization, and accounting (AAA). A distributed AAA system can implement decentralized payment, credit, audit, authorization, and authentication systems across interconnected, heterogeneous network infrastructures. This can move systems and networks toward autonomous operations. In these types of autonomous operations, machines can contract human resources and even negotiate partnerships with other machine networks. This not only enables the realization of mutual objectives and balanced service delivery against outlined and planned service level agreements, but also enables solutions that provide metering, measurement, traceability, and traceability. Creating new supply chain structures and methods can enable the creation of numerous services, mining value, and disruption without any human involvement.

[0167] Such IoT networks can be further enhanced by integrating sensing technologies such as sound, light, electronic traffic, face and pattern recognition, smell, and vibration into autonomous organization among IoT devices. The integration of sensory systems can enable systematic and autonomous communication and service delivery coordination, orchestration, and Quality of Service (QoS)-based resource swarming and fusion for contracted service objectives. Some specific examples of network-based resource handling include:

[0168] For example, mesh network 4756 can be augmented with systems that perform in-line data-to-information translation. For example, a self-forming chain of processing resources, including a multi-link network, can decentralize the efficient translation of raw data into information, as well as the ability to distinguish between assets and resources and their associated management. Furthermore, appropriate components of infrastructure and resource-based trust and service indexing can be inserted to improve data integrity, quality, and reliability, and provide metrics of data reliability.

[0169] For example, a WLAN network 4758 may provide multi-standard connectivity using a system that performs standards conversion, allowing IoT devices 4704 using different protocols to communicate. Additional systems can provide seamless interconnectivity across a multi-standard infrastructure, including visible and hidden Internet resources.

[0170] For example, communications in the cellular network 4760 may be enhanced by systems that offload data, extend communications to more remote devices, or both. The LPWA network 4762 may include systems that perform non-Internet Protocol (IP)-to-IP interconnection, addressing, and routing. Additionally, each of the IoT devices 4704 may include a transceiver suitable for wide-area communications with that device. Additionally, each IoT device 4704 may include other transceivers for communications using additional protocols and frequencies. This is further described in connection with the communications environment and hardware of the IoT processing device shown in FIGS. 49 and 50.

[0171] Finally, a cluster of IoT devices may be equipped to communicate with other IoT devices as well as with a cloud network. This may enable the IoT devices to form ad-hoc networks between themselves and function as a single device, which may be referred to as a fog device. This configuration is further described in connection with Figure 48 below.

[0172] Figure 48 illustrates a cloud computing network in communication with a mesh network of IoT devices (devices 4802) operating as fog devices at the edge of the cloud computing network. The mesh network of IoT devices may be referred to as fog 4820 operating at the edge of the cloud 4800. To simplify the illustration, not all of the IoT devices 4802 are labeled.

[0173] The fog 4820 may be considered a large-scale interconnected network in which several IoT devices 4802 communicate with each other, for example, by wireless links 4822. By way of example, this interconnected network may be facilitated using interconnection specifications released by the Open Connectivity Foundation™ (OCF). These standards enable devices to discover each other and establish communications for interconnection. Other interconnection protocols may also be used, including, for example, the Optimized Link State Routing (OLSR) protocol, the Better Approach to Mobile Ad Hoc Networking (BATMAN) routing protocol, or the OMA Lightweight M2M (LWM2M) protocol, among others.

[0174] Although this example shows three types of IoT devices 4802—gateway 4804, data aggregator 4826, and sensor 4828—any combination of IoT devices 4802 and functions may be used. The gateway 4804 may be an edge device providing communication between the cloud 4800 and the fog 4820 and may also provide back-end processing functions for data obtained from the sensors 4828, such as operational data, flow data, and temperature data. The data aggregator 4826 may collect data from any number of sensors 4828 and perform back-end processing functions for analysis. Results, raw data, or both may be passed to the cloud 4800 via the gateway 4804. The sensors 4828 may be full IoT devices 4802 capable of both collecting and processing data, for example. In some cases, the sensors 4828 may have more limited functions, for example, collecting data and allowing the data aggregator 4826 or gateway 4804 to process the data.

[0175] Communications from any IoT device 4802 can be passed along a convenient path (e.g., the most convenient route) between any of the IoT devices 4802 to reach the gateway 4804. In these networks, the number of interconnections provides substantial redundancy, allowing communications to be maintained even if some IoT devices 4802 are lost. Furthermore, the use of mesh networks may enable the use of IoT devices 4802 that are very low power or located far from the infrastructure, since the range of connecting to another IoT device 4802 may be much smaller than the range of connecting to the gateway 4804.

[0176] The fog 4820 provided from these IoT devices 4802 may be presented to devices in the cloud 4800, such as server 4806, as a single device, e.g., a fog device, located at the edge of the cloud 4800. In this example, alerts coming from the fog device may be sent without being identified as coming from a particular IoT device 4802 in the fog 4820. In this way, the fog 4820 may be considered a distributed platform that provides computing and storage resources for performing processing or data-intensive tasks such as data analysis, data aggregation, and machine learning, among other things.

[0177] In some examples, IoT devices 4802 may be configured using an imperative programming style, e.g., each IoT device 4802 has specific capabilities and communication partners. However, IoT devices 4802 forming fog devices may be configured using a declarative programming style, allowing the IoT devices 4802 to reconfigure their operation and communications, e.g., determine required resources in response to conditions, queries, and device failures. As an example, a query from a user located at the server 4806 regarding the operation of a subset of equipment monitored by the IoT devices 4802 may cause the fog 4820 device to select IoT devices 4802, such as specific sensors 4828, needed to respond to the query. Data from these sensors 4828 may then be aggregated and analyzed by any combination of sensors 4828, data aggregators 4826, or gateways 4804 before being transmitted by the fog 4820 device to the server 4806 to respond to the query. In this example, IoT devices 4802 in the fog 4820 can select sensors 4828 to use based on the query, for example, adding data from a flow sensor or a temperature sensor. Additionally, if some of the IoT devices 4802 are not operational, other IoT devices 4802 in the fog 4820 device can provide similar data if available.

[0178] In other examples, the above operations and functions may be embodied by an IoT device machine in the exemplary form of an electronic processing system, in which a set or sequence of instructions may be executed to cause the electronic processing system to perform any one of the methods described herein, according to an exemplary embodiment. The machine may be an IoT device or IoT gateway, including a machine embodied in aspects of a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone or smartphone, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the machine. Furthermore, while only a single machine may be depicted and referenced in the above examples, such a machine shall also be considered to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methods described herein. Furthermore, these and similar examples of processor-based systems shall also be considered to include any set of one or more machines controlled or operated by a processor (e.g., a computer) to individually or jointly execute instructions to perform any one or more of the methods described herein. In some implementations, one or more devices may work in concert to implement functions and perform tasks described herein. In some cases, one or more host devices may supply data, provide instructions, aggregate results, or otherwise facilitate the joint operations and functions provided by multiple devices. When implemented by a single device, functions may be considered local to the device, but in implementations of multiple devices operating as a single machine, functions may be considered collectively local to the devices, and this collection of devices may provide or consume results provided by other remote machines (implemented as a single device or as a collection of devices), among other example implementations.

[0179] For example, FIG. 49 shows a diagram of a cloud computing network or cloud 4900 in communication with several Internet of Things (IoT) devices. The cloud 4900 may represent the Internet or a wide area network (WAN) or local area network (LAN), such as a company's proprietary network. The IoT devices can include any number of different types of devices grouped in various combinations. For example, a traffic control group 4906 may include IoT devices along city streets. These IoT devices may include stoplights, traffic flow monitors, cameras, weather sensors, and the like. The traffic control group 4906 or other subgroups may be in communication with the cloud 4900 via wired or wireless links 4908, such as LPWA links and optical links. Additionally, wired or wireless sub-networks 4912 may enable IoT devices to communicate with each other, for example, via local area networks and wireless local area networks. The IoT devices can communicate with remote locations, such as the cloud 4900, using another device, such as a gateway 4910 or 4928. IoT devices may also use one or more servers 4930 to facilitate communication with the cloud 4900 or gateway 4910. For example, one or more servers 4930 may operate as intermediate network nodes to support local edge cloud or fog implementations between local area networks. Additionally, the illustrated gateway 4928 may operate in a cloud-to-gateway / gateway-to-many edge device configuration, e.g., with the various IoT devices 4914, 4920, 4924 being constrained or dynamic with respect to resource allocation and usage within the cloud 4900.

[0180] Other exemplary groups of IoT devices may include, among many others, a remote weather station 4914, a local information terminal 4916, an alarm system 4918, an automated teller machine 4920, an alarm panel 4922, or a mobile vehicle such as an emergency vehicle 4924 or other vehicle 4926. Each of these IoT devices may be in communication with other IoT devices, a server 4904, another IoT fog device or system (not shown, but shown in FIG. 48), or a combination thereof. Groups of IoT devices may be deployed in a variety of residential, commercial, and industrial environments, including both private and public environments.

[0181] As can be seen in FIG. 49, numerous IoT devices may be communicating via the cloud 4900. This may enable multiple different IoT devices to autonomously request or provide information to other devices. For example, a group of IoT devices (e.g., traffic control group 4906) may request a current weather forecast from a group of remote weather stations 4914, which may provide the forecast without human intervention. Additionally, an emergency vehicle 4924 may be alerted by an automated teller machine 4920 that a robbery is in progress. As the emergency vehicle 4924 travels toward the automated teller machine 4920, it may approach the traffic control group 4906 and request clearance to the location. This clearance may be, for example, by lights turning red to block cross traffic at the intersection in enough time for the emergency vehicle 4924 to access the intersection unimpeded.

[0182] A cluster of IoT devices, such as a remote weather station 4914 or a traffic control group 4906, may be equipped to communicate with other IoT devices as well as the cloud 4900. This may enable the IoT devices to form ad-hoc networks between themselves and function as a single device that may be referred to as a fog device or system (e.g., as described above with reference to FIG. 48).

[0183] FIG. 50 is a block diagram of example components that may be present in an IoT device 5050 to implement the techniques described herein. The IoT device 5050 may include any combination of components shown in the example or mentioned in the disclosure above. The components may be implemented as an IC, portion thereof, discrete electronic devices, or other modules, logic, hardware, software, firmware, or combinations thereof that fit into the IoT device 5050 or as components that are otherwise incorporated within the chassis of a larger system. Additionally, the block diagram of FIG. 50 is intended to illustrate a high-level view of the components of the IoT device 5050. However, in other implementations, some of the shown components may be omitted, additional components may be present, and different arrangements of the shown components may be made.

[0184] The IoT device 5050 may include a processor 5052. The processor 5052 may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. The processor 5052 may be part of a system-on-chip (SoC) in which the processor 5052 and other components are formed within a single integrated circuit or within a single package, such as an Intel Edison™ or Galileo™ SoC substrate. By way of example, the processor 5052 may include an Intel® Architecture Core™-based processor, such as a Quark™, ​​Atom™, i3, i5, i7, or MCU-class processor, or another such processor available from Intel® Corporation of Santa Clara, California. However, any number of other processors may be used, such as those available from Advanced Micro Devices, Inc. (AMD) of Sunnyvale, California, MIPS-based designs from MIPS Technologies, Inc. of Sunnyvale, California, ARM-based designs licensed from ARM Holdings, Ltd. or its customers or its licensees or adopters. Processors may include units such as Apple® Inc.'s A5-A10 processors, Qualcomm® Technologies, Inc.'s Snapdragon™ processors, or Texas Instruments, Inc.'s OMAP™ processors.

[0185] The processor 5052 can communicate with the system memory 5054 via an interconnect 5056 (e.g., a bus). Any number of memory devices can be used to provide a given amount of system memory. By way of example, the memory can be random access memory (RAM) according to a Joint Electron Devices Committee (JEDEC) design, such as a DDR or mobile DDR standard (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In various implementations, the individual memory devices can be in any number of different package types, such as a single die package (SDP), a dual die package (DDP), or a quad die package (Q17P). In some examples, these devices can be soldered directly onto a motherboard to provide a low-profile solution, while in other examples, the devices are configured as one or more memory modules that similarly couple to the motherboard with a given connector. Any number of other memory implementations can be used, such as other types of memory modules, for example, different types of dual in-line memory modules (DIMMs), including, but not limited to, microDIMMs or MiniDIMMs.

[0186] Storage 5058 may also be coupled to processor 5052 via interconnect 5056 to provide persistent storage of information such as data, applications, and the operating system. In one example, storage 5058 may be implemented via a solid-state disk drive (SSDD). Other devices that may be used for storage 5058 include flash memory cards, such as SD cards, microSD cards, and xD picture cards, and USB flash drives. In low-power implementations, storage 5058 may be on-die memory or registers associated with processor 5052. However, in some examples, storage 5058 may be implemented using a micro hard disk drive (HDD). Furthermore, any number of emerging technologies may be used for storage 5058 in addition to or instead of the described technologies, such as resistive memory, phase-change memory, holographic memory, or chemical memory, among others.

[0187] The components can communicate via an interconnect 5056. The interconnect 5056 can include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies. The interconnect 5056 can be, for example, a proprietary bus used in SoC-based systems. Other bus systems can be included, such as an I2C interface, an SPI interface, a point-to-point interface, and a power bus, among others.

[0188] The interconnect 5056 can couple the processor 5052 to a mesh transceiver 5062 for communication with other mesh devices 5064. The mesh transceiver 5062 can use any number of frequencies and protocols, such as 2.4 gigahertz (GHz) transmissions based on the IEEE 802.15.4 standard, using the Bluetooth® Low Energy (BLE) standard defined by the Bluetooth® Special Interest Group, or the ZigBee® standard, among others. Any number of radios configured for a particular wireless communication protocol can be used to connect to the mesh devices 5064. For example, a WLAN unit can be used to implement Wi-Fi™ communications in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, wireless wide-area communications, for example via cellular or other wireless wide-area protocols, can occur via the WWAN unit.

[0189] The mesh transceiver 5062 can communicate using multiple standards or radios for communication at different ranges. For example, the IoT device 5050 can conserve power by using a local transceiver based on BLE or another low-power radio to communicate with nearby devices, e.g., within about 10 meters. More distant mesh devices 5064, e.g., within about 50 meters, can be reached via ZigBee or other medium-power radio. Both communication technologies may occur via a single radio at different power levels, or via separate transceivers, e.g., a local transceiver using BLE and a separate mesh transceiver using ZigBee.

[0190] A wireless network transceiver 5066 may be included to communicate with devices or services in the cloud 5000 via local area network or wide area network protocols. The wireless network transceiver 5066 may be an LPWA transceiver conforming to the IEEE 802.15.4 or IEEE 802.15.4g standards, among others. The IoT device 5050 may communicate over a wide area using LoRaWAN™ (Long Range Wide Area Network) developed by Semtech and the LoRa Alliance. The techniques described herein may be used with any number of other cloud transceivers and other technologies implementing long-range, low-bandwidth communications, such as, but not limited to, Sigfox. Additionally, other communication techniques may be used, such as time-slotted channel hopping as described in the IEEE 802.15.4e specification.

[0191] As described herein, in addition to the systems mentioned for mesh transceiver 5062 and wireless network transceiver 5066, any number of other wireless communications and protocols may be used. For example, wireless transceivers 5062 and 5066 may include LTE or other cellular transceivers that use spread spectrum (SPA / SAS) communications to implement high-speed communications. Furthermore, any number of other protocols may be used, such as providing network communications and Wi-Fi networks for medium-speed communications.

[0192] The wireless transceivers 5062 and 5066 may include radios compatible with any number of 3GPP (3rd Generation Partnership Project) specifications, particularly Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), and Long Term Evolution Advanced Pro (LTE-A Pro). Radios compatible with any number of other fixed, mobile, or satellite communication technologies and standards may also be selected. These may include, for example, any cellular wide-area wireless communication technology. Cellular wide-area wireless communication technologies may include, for example, fifth-generation (5G) communication systems, Global System for Mobile Communications (GSM) wireless communication technology, General Packet Radio Service (GPRS) wireless communication technology, Enhanced Data Rates for GSM Evolution (EDGE) wireless communication technology, and Universal Mobile Telecommunications System (UMTS) communication technology. In addition to the above standards, any number of satellite uplink technologies may be used with the wireless network transceiver 5066. Satellite uplink technologies include, for example, radios conforming to standards published by the ITU (International Telecommunications Union) or ETSI (European Telecommunications Standards Institute), among others, and the examples provided herein should therefore be understood to be applicable to a variety of other communication technologies, both existing and yet to be formulated.

[0193] A network interface controller (NIC) 5068 may be included to provide wired communications to the cloud 5000 or other devices, such as the mesh device 5064. The wired communications may provide an Ethernet connection or may be based on other types of networks, such as a controller area network (CAN), a local interconnect network (LIN), DeviceNet, ControlNet, Data Highway+, PROFIBUS, or PROFINET, among others. Additional NICs 5068 may be included to allow connection to a second network, e.g., a NIC 5068 that provides communications to the cloud via Ethernet and a second NIC 5068 that provides communications to other devices via another type of network.

[0194] The interconnect 5056 can couple the processor 5052 to an external interface 5070 used to connect external devices or subsystems. The external devices can include sensors 5072 such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, global positioning system (GPS) sensors, pressure sensors, and barometric pressure sensors. The external interface 5070 can also be used to connect the IoT device 5050 to actuators 5074 such as power switches, valve actuators, audible sound generators, and visual warning devices.

[0195] In some example options, various input / output (I / O) devices may be present within or connected to the IoT device 5050. For example, a display or other output device 5084 may be included to show information such as sensor readings or actuator positions. An input device 5086, such as a touch screen or keypad, may be included to accept input. The output device 5084 may include any number of forms of auditory or visual displays. Displays may include simple visual outputs, such as binary status indicators (e.g., LEDs) and multi-character visual outputs, or more complex outputs, such as display screens (e.g., LCD screens), where output, such as text, graphics, and multimedia objects, is generated or produced from the operation of the IoT device 5050.

[0196] The battery 5076 can provide power to the IoT device 5050, or in instances where the IoT device 5050 is mounted in a fixed location, can include a power supply that is coupled to a power grid. The battery 5076 can be a lithium-ion battery or a metal-air battery, such as a zinc-air battery, an aluminum-air battery, and a lithium-air battery.

[0197] The IoT device 5050 may include a battery monitor / charger 5078 to track the state of charge (SoCh) of the battery 5076. The battery monitor / charger 5078 may be used to monitor other parameters of the battery 5076 to provide failure predictions, such as the state of health (SoH) and state of function (SoF) of the battery 5076. The battery monitor / charger 5078 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx family of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 5078 may communicate information on the battery 5076 to the processor 5052 via the interconnect 5056. The battery monitor / charger 5078 may also include an analog-to-digital (ADC) converter that allows the processor 5052 to directly monitor the voltage of the battery 5076 or the current drawn from the battery 5076. The battery parameters may be used to determine the actions that the IoT device 5050 can perform, such as transmission frequency, mesh network operation, and sensing frequency.

[0198] To charge the battery 5076, a power generation block 5080, or other power supply device coupled to a power grid, may be coupled to a battery monitor / charger 5078. In some examples, the power generation block 5080 may be replaced with a wireless power receiver to obtain power wirelessly, for example, via a loop antenna in the IoT device 5050. The battery monitor / charger 5078 may include, among other things, a wireless battery charging circuit such as the LTC4020 chip from Linear Technologies of Milpitas, California. The particular charging circuit selected depends on the size of the battery 5076 and, therefore, the current required. Charging may be performed using, among other things, the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Alliance for Wireless Power.

[0199] Storage 5058 may include instructions 5082 in the form of software, firmware, or hardware commands for implementing the techniques described herein. While such instructions 5082 are shown as code blocks contained in memory 5054 and storage 5058, it will be understood that any of the code blocks may be replaced by hardwired circuitry incorporated, for example, in an application specific integrated circuit (ASIC).

[0200] In one example, the instructions 5082 provided via memory 5054, storage 5058, or processor 5052 may be embodied as a non-transitory machine-readable medium 5060. The non-transitory machine-readable medium 5060 includes code that instructs the processor 5052 to perform electronic operations on the IoT device 5050. The processor 5052 can access the non-transitory machine-readable medium 5060 via the interconnect 5056. For example, the non-transitory machine-readable medium 5060 may be embodied by a device described as storage 5058 in FIG. 50 or may include a specific storage unit such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 5060 may include instructions to direct the processor 5052 to perform a particular flow or sequence of actions, for example, as described in connection with the operational and functional flowcharts and block diagrams above.

[0201] FIG. 51 is an exemplary diagram of a processor according to an embodiment. Processor 5100 is an example of a type of hardware device that may be used in connection with the above implementations. Processor 5100 may be any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, multi-core processor, single-core processor, or other device for executing code. Although only one processor 5100 is shown in FIG. 51, a processing element may alternatively include two or more of the processors 5100 shown in FIG. 51. Processor 5100 may be a single-threaded core. Alternatively, in at least one embodiment, processor 5100 may be multi-threaded in that it may include two or more hardware thread contexts (or "logical processors") per core.

[0202] 51 also illustrates memory 5102 coupled to processor 5100 according to an embodiment. Memory 5102 may be any of a wide variety of memories (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. Such memory elements may include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0203] The processor 5100 may execute any type of instruction associated with an algorithm, process, or operation described herein. Generally, the processor 5100 may transform elements or items (e.g., data) from one state or condition to another state or condition.

[0204] Code 5104, which may be one or more instructions executed by processor 5100, may be stored in memory 5102, or in software, hardware, firmware, or any suitable combination thereof, or in any other internal or external component, device, element, or object as needed based on particular needs. In one example, processor 5100 may follow a program sequence of instructions indicated by code 5104. Each instruction enters front-end logic 5106 and is processed by one or more decoders 5108. The decoders may generate as their output micro-operations, such as fixed-width micro-operations of a predefined format, or other instructions, micro-instructions, or control signals that reflect the original code instruction. Front-end logic 5106 also includes register renaming logic 5110 and scheduling logic 5112, which generally allocate resources and queue operations corresponding to instructions for execution.

[0205] The processor 5100 may also include execution logic 5114 having a set of execution units 5116a, 5116b, 5116n, etc. Some embodiments may include several execution units specialized for a particular function or set of functions. Other embodiments may include only one execution unit, or one execution unit capable of performing a particular function. The execution logic 5114 performs the operations specified by the code instructions.

[0206] After completing execution of the operations specified by the code instructions, the back-end logic 5118 can retire the instructions of the code 5104. In one embodiment, the processor 5100 allows out-of-order execution but requires in-order retirement of instructions. The retirement logic 5120 can take various known forms (e.g., a reorder buffer, etc.). In this manner, the processor 5100 is transformed during execution of the code 5104 with respect to at least the outputs generated by the decoder, the hardware registers and tables utilized by the register renaming logic 5110, and any registers (not shown) modified by the execution logic 5114.

[0207] 51, a processing element may include other elements on a chip with processor 5100. For example, a processing element may include memory control logic along with processor 5100. A processing element may include I / O control logic and / or may include I / O control logic integrated with memory control logic. A processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on a chip with processor 5100.

[0208] Figure 52 illustrates a computing system 5200 arranged in a point-to-point (PtP) configuration, according to one embodiment. In particular, Figure 52 illustrates a system in which a processor, memory, and input / output devices are interconnected by several point-to-point interfaces. Generally, one or more of the computing systems described herein may be configured in the same or similar manner as computing system 5200.

[0209] Processors 5270 and 5280 may also include integrated memory controller logic (MC) 5272 and 5282, respectively, to communicate with memory elements 5232 and 5234. In alternative embodiments, memory controller logic 5272 and 5282 may be discrete logic separate from processors 5270 and 5280. Memory elements 5232 and / or 5234 may store various data used by processors 5270 and 5280 in implementing the operations and functions outlined herein.

[0210] Processors 5270 and 5280 may be any type of processor, such as those described in connection with other figures. Processors 5270 and 5280 can exchange data via point-to-point interface 5250 using point-to-point (PtP) interface circuits 5278 and 5288, respectively. Processors 5270 and 5280 can exchange data with chipset 5290 via individual point-to-point interfaces 5252 and 5254, using point-to-point interface circuits 5276, 5286, 5294, and 5298, respectively. Chipset 5290 can also exchange data with high-performance graphics circuit 5238 via high-performance graphics interface 5239 using interface circuit 5292, which may be a PtP interface circuit. In an alternative embodiment, any or all of the PtP links shown in FIG. 52 may be implemented as multi-drop buses rather than PtP links.

[0211] Chipset 5290 may be in communication with bus 5220 via interface circuit 5296. Bus 5220 may have one or more devices communicating therethrough, such as a bus bridge 5218 and I / O devices 5216. Bus bridge 5218 may be in communication with other devices via bus 5210, such as a user interface 5212 (such as a keyboard, mouse, touch screen, or other input device), a communication device 5226 (such as a modem, network interface device, or other type of communication device that may communicate over computer network 5260), an audio I / O device 5214, and / or a data storage device 5228. Data storage device 5228 may store code 5230 that may be executed by processor(s) 5270 and / or 5280. In alternative embodiments, any portion of the bus architecture may be implemented with one or more PtP links.

[0212] The computer system shown in Figure 52 is a schematic diagram of an embodiment of a computing system that may be utilized to implement various embodiments described herein. It will be appreciated that the various components of the system shown in Figure 52 may be combined in a system-on-chip (SoC) architecture or in any other suitable configuration capable of providing the functionality and features of the examples and implementations provided herein.

[0213] In a further example, a machine-readable medium can include any tangible medium that can store, encode, or carry instructions for execution by a machine and cause a machine to perform any one or more of the methods of the present disclosure, or that can store, encode, or carry data structures used in or associated with such instructions. Accordingly, a "machine-readable medium" can include, but is not limited to, solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media include, by way of example and not limitation, non-volatile memory, including semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) and flash memory devices), magnetic disks, such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. Instructions embodied by a machine-readable medium can also be transmitted or received over a communications network using a transmission medium via a network interface device utilizing any one of several transfer protocols (e.g., HTTP).

[0214] It should be understood that functional units or functions described herein may be referred to or labeled as components or modules to more specifically emphasize their implementation independence. Such components may be embodied in any number of software or hardware forms. For example, a component or module may be implemented as a hardware circuit including custom very large scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. A component or module may also be implemented in a programmable hardware device such as a field programmable gate array, programmable array logic, or programmable logic device. A component or module may also be implemented in software for execution by various types of processors. An identified component or module of executable code may include, for example, one or more physical or logical blocks of computer instructions, which may be organized as, for example, an object, procedure, or function. Nevertheless, the executable files of an identified component or module need not be physically located together but may include entirely different instructions stored in different locations that, when logically coupled together, comprise the component or module and achieve the specified purpose of the component or module.

[0215] Indeed, a component or module of executable code may be a single instruction or many instructions, and may even be distributed across several different code segments, among different programs, and across several memory devices or processing systems. In particular, some aspects of the described processes (e.g., code rewriting and code analysis) may occur on a different processing system (e.g., a computer in a data center) than the system on which the code is deployed (e.g., a computer embedded in a sensor or robot). Similarly, operational data may be identified and illustrated herein within components or modules and may be embodied in any suitable form and organized within any suitable type of data structure. Operational data may be collected as a single data set, distributed across different locations, including different storage devices, or may exist, at least in part, simply as electronic signals on a system or network. Components or modules may be passive or active, including agents operable to perform desired functions.

[0216] Further examples of embodiments of the methods, systems, and devices described herein include the following non-limiting configurations: Each of the following non-limiting examples may stand alone or may be combined in any permutation or combination with any one or more of the other examples provided below or throughout this disclosure.

[0217] While the present disclosure has been described in terms of particular implementations and generally associated methods, modifications and permutations of these implementations and methods will be apparent to those skilled in the art. For example, the actions described herein may be performed in an order different from that described and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. Furthermore, other user interface layouts and functionality may be supported. Other variations are within the scope of the following claims.

[0218] While the specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather as describing features specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as functioning in a particular combination, and even initially claimed as such, in some cases one or more features of a claimed combination may be deleted from that combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0219] Similarly, although the figures depict operations in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence depicted, or that all of the operations depicted be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments. It should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products.

[0220] The following examples relate to embodiments herein: Example 1 is a method comprising: accessing a synthetic three-dimensional (3D) graphical model of an object from a memory, the 3D graphical model having photorealistic resolution; generating a plurality of different training samples from views of the 3D graphical model, the plurality of training samples being generated to simulate characteristics of real-world samples generated by a real-world sensor device by adding imperfections to the plurality of training samples; and generating a training set including the plurality of training samples, the training data being used to train an artificial neural network.

[0221] Example 2 includes the subject matter described in Example 1, where the plurality of training samples includes digital images and the sensor device includes a camera sensor.

[0222] Example 3 includes the subject matter of any one of Examples 1 and 2, wherein the plurality of training samples includes a point cloud representation of an object.

[0223] Example 4 includes the subject matter of example 3, wherein the sensor device includes a LIDAR sensor.

[0224] Example 5 includes the subject matter of any one of Examples 1 to 4. The subject matter further includes accessing data indicative of a parameter of the sensor device and determining, based on the parameter, defects to add to the plurality of training samples.

[0225] Example 6 includes the subject matter of example 5, wherein the data includes a model of the sensor device.

[0226] Example 7 includes the subject matter of any one of Examples 1 to 6. The subject matter further includes accessing data indicative of characteristics of one or more surfaces of the object modeled by the 3D graphical model, and determining defects to add to the plurality of training samples based on the characteristics.

[0227] Example 8 includes the subject matter described in Example 7, wherein the 3D graphical model includes the data.

[0228] Example 9 includes the subject matter of any one of Examples 1 to 8, wherein the defects include one or more of noise or glare.

[0229] Example 10 includes the subject matter of any one of Examples 1 to 9, where generating the plurality of different training samples includes applying a plurality of different lighting settings to the 3D graphical model to simulate lighting in the environment, and determining defects for a subset of the plurality of training samples generated during application of a particular one of the plurality of different lighting settings, wherein the defects for the subset of the plurality of training samples are based on the particular lighting setting.

[0230] Example 11 includes the subject matter of any one of Examples 1 to 10, where generating the plurality of different training samples includes placing the 3D graphical model in a plurality of different graphical environments, the graphical environments modeling respective real-world environments, and generating a subset of the plurality of training samples while the 3D graphical model is placed in the plurality of different graphical environments.

[0231] Example 12 is a system including means for carrying out the method according to any one of Examples 1 to 11.

[0232] Example 13 includes the subject matter of Example 12, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 1 to 11.

[0233] Example 14 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 1 to 11.

[0234] Example 15 is a method comprising: receiving a subject input and a reference input at a siamese neural network, the siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, the weights of the first network portion being identical to the weights of the second network portion, the subject input being provided as an input to the first network portion and the reference input being provided as an input to the second network portion; and generating an output of the siamese neural network based on the subject input and the reference input, the output of the siamese neural network indicating a similarity between the reference input and the subject input.

[0235] Example 16 includes the subject matter of Example 15, where generating the output includes determining a difference amount between the reference input and the subject input and determining whether the difference amount meets a threshold, the output identifying whether the difference amount meets the threshold.

[0236] Example 17 includes the subject matter of Example 16, where determining a difference between the reference input and the subject input includes receiving a first feature vector output by the first network portion and a second feature vector output by the second network portion, and determining a difference vector based on the first feature vector and the second feature vector.

[0237] Example 18 includes the subject matter of any one of Examples 15 to 17, wherein generating an output includes a one-time classification.

[0238] Example 19 includes the subject matter of any one of Examples 15 to 18. The subject matter further includes training a Siamese neural network using one or more synthetic training samples.

[0239] Example 20 includes the subject matter of Example 19, wherein one or more synthetic training samples are generated according to the method of any one of Examples 1-11.

[0240] Example 21 includes the subject matter of any one of Examples 15 to 20, wherein the reference input includes synthetically generated samples.

[0241] Example 22 includes the subject matter described in Example 21, wherein the synthetically produced sample is produced according to the method described in any one of Examples 1-11.

[0242] Example 23 includes the subject matter of any one of Examples 15 to 22, wherein the subject input includes a first digital image and the reference input includes a second digital image.

[0243] Example 24 includes the subject matter of any one of examples 15 to 22, wherein the subject input includes a first point cloud representation and the reference input includes a second point cloud representation.

[0244] Example 25 is a system including means for performing the method according to any one of Examples 15 to 24.

[0245] Example 26 includes the subject matter of Example 25, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 15 to 24.

[0246] Example 27 includes the subject matter of example 25, wherein the system includes one of a robot, a drone, or an autonomous vehicle.

[0247] Example 28 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 15 to 24.

[0248] Example 29 is a method comprising: providing first input data to a Siamese neural network, the first input data including a first representation of 3D space from a first pose; providing second input data to the Siamese neural network, the second input data including a second representation of 3D space from a second pose, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, the weights of the first network portion being identical to the weights of the second network portion, the first input data being provided as input to the first network portion and the second input data being provided as input to the second network portion; and generating an output of the Siamese neural network, the output including a relative pose between the first pose and the second pose.

[0249] Example 30 includes the subject matter of example 29, wherein the first representation of the 3D space includes a first 3D point cloud and the second representation of the 3D space includes a second 3D point cloud.

[0250] Example 31 includes the subject matter of any one of Examples 29-30, where the first representation of 3D space includes a first cloud of points and the second representation of 3D space includes a second cloud of points.

[0251] Example 32 includes the subject matter of example 31, wherein the first point cloud and the second point cloud each include a respective voxelized point cloud representation.

[0252] Example 33 includes the subject matter of any one of Examples 29 to 32. The subject matter further includes generating a 3D mapping of the 3D space from at least the first input data and the second input data based on the relative pose.

[0253] Example 34 includes the subject matter of any one of examples 29 to 32. The subject matter further includes determining a position of an observer at the first pose in 3D space based on the relative pose.

[0254] Example 35 includes the subject matter of example 34, wherein the observer includes an autonomous machine.

[0255] Example 36 includes the subject matter of Example 35, wherein the autonomous machine includes one of a robot, a drone, or an autonomous vehicle.

[0256] Example 37 is a system including means for performing the method according to any one of Examples 29 to 36.

[0257] Example 38 includes the subject matter of Example 37, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 29 to 36.

[0258] Example 39 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 29 to 36.

[0259] Example 40 is a method comprising: providing first sensor data as input to a first portion of a machine learning model; providing second sensor data as input to a second portion of the machine learning model, the machine learning model including a connector and a set of fully connected layers, the first sensor data being of a first type generated by the device and the second sensor data being of a different second type generated by the device, the connector receiving an output of the first portion of the machine learning model as a first input and an output of the second portion of the machine learning model as a second input, the output of the connector being provided to the set of fully connected layers; and generating an output of the machine learning model from the first data and the second data, the output including an orientation of the device within an environment.

[0260] Example 41 includes the subject matter of example 40, wherein the first sensor data includes image data and the second sensor data identifies a movement of the device.

[0261] Example 42 includes the subject matter of example 41, wherein the image data includes red, green, and blue (RGB) data.

[0262] Example 43 includes the subject matter of example 41, wherein the image data includes 3D point cloud data.

[0263] Example 44 includes the subject matter of example 41, wherein the second sensor data includes inertial measurement unit (IMU) data.

[0264] Example 45 includes the subject matter of example 41, wherein the second sensor data includes global positioning data.

[0265] Example 46 includes the subject matter of any one of Examples 40 to 45, wherein a first portion of the machine learning model is tuned for a first type of sensor data and a second portion of the machine learning model is tuned for a second type of sensor data.

[0266] Example 47 includes the subject matter of any one of Examples 40 to 46. The subject matter further includes providing third sensor data of a third type as an input to a third portion of the machine learning model, wherein the output is further generated based on the third data.

[0267] Example 48 includes the subject matter of any one of examples 40 to 47, wherein the pose output includes a rotational component and a translational component.

[0268] Example 49 includes the subject matter of Example 48, wherein one of the set of fully connected layers includes a fully connected layer for determining a rotational component, and another of the set of fully connected layers includes a fully connected layer for determining a translational component.

[0269] Example 50 includes the subject matter of any one of Examples 40 to 49, wherein one or both of the first and second portions of the machine learning model include respective convolutional layers.

[0270] Example 51 includes the subject matter of any one of Examples 40 to 50, wherein one or both of the first and second portions of the machine learning model include one or more respective long short-term memory (LSTM) blocks.

[0271] Example 52 includes the subject matter of any one of Examples 40 to 51, wherein the device includes an autonomous machine, and the autonomous machine navigates through an environment based on the pose.

[0272] Example 53 includes the subject matter of Example 52, wherein the autonomous machine includes one of a robot, a drone, or an autonomous vehicle.

[0273] Example 54 is a system including means for performing the method according to any one of Examples 40 to 52.

[0274] Example 55 includes the subject matter of Example 54, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 40 to 52.

[0275] Example 56 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 40 to 52.

[0276] Example 57 is a method comprising: requesting random generation of a set of neural networks; performing a machine learning task using each of the set of neural networks, where the machine learning task is performed using particular processing hardware; monitoring attributes of the performance of the machine learning task for each of the set of neural networks, where the attributes include accuracy of the results of the machine learning task; and identifying a best-performing neural network of the set of neural networks when used to perform the machine learning task using the particular processing hardware based on the attributes of the best-performing neural network.

[0277] Example 58 includes the subject matter of example 57. The subject matter further includes providing a top performing neural network for use by the machine in performing machine learning applications.

[0278] Example 59 includes the subject matter of any one of Examples 57-58. The subject matter further includes determining characteristics of the top-performing neural network, requesting generation of a second set of neural networks according to the characteristics, the second set of neural networks including a plurality of different neural networks, each including one or more of the characteristics, performing a machine learning task using each one of the second set of neural networks, the machine learning task being performed using particular processing hardware, monitoring attributes of the performance of the machine learning task for each of the second set of neural networks, and identifying a top-performing neural network of the second set of neural networks based on the attributes.

[0279] Example 60 includes the subject matter of any one of Examples 57-59. The subject matter further includes receiving a criterion based on the parameters, wherein the best performing neural network is based on the criterion.

[0280] Example 61 includes the subject matter of any one of Examples 57 to 60, wherein the attributes include attributes of specific processing hardware.

[0281] Example 62 includes the subject matter of Example 61, where the attributes of the particular processing hardware include one or more of: power consumed by the particular processing hardware while performing the machine learning task; a temperature of the particular processing hardware while performing the machine learning task; and memory used to store the neural network on the particular processing hardware.

[0282] Example 63 includes the subject matter of any one of Examples 57 to 62, wherein the attributes include a time to complete the machine learning task using a corresponding one of a set of neural networks.

[0283] Example 64 is a system including means for performing the method according to any one of Examples 57 to 63.

[0284] Example 65 includes the subject matter of Example 64, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method described in any one of Examples 57 to 63.

[0285] Example 66 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 57 to 63.

[0286] Example 67 is a method comprising: identifying a neural network including a plurality of kernels, each kernel including a respective set of weights; reducing the plurality of kernels to a particular set of kernels by pruning a subset of the plurality of kernels according to one or more parameters; and pruning a subset of weights in the particular set of kernels to form a pruned version of the neural network, wherein pruning the subset of weights assigns one or more non-zero weights in the subset of weights to zero, and the subset of weights is selected based on the original values ​​of the weights.

[0287] Example 68 includes the subject matter of example 67, wherein the subset of weights is pruned based on values ​​of the subset of weights below a threshold.

[0288] Example 69 includes the subject matter of any one of Examples 67-68, further including performing one or more iterations of the machine learning task using a pruned version of the neural network to regain at least a portion of the accuracy lost due to pruning of the kernels and weights.

[0289] Example 70 includes the subject matter of any one of Examples 67 to 69. The subject matter further includes quantizing values ​​of unpruned weights in the pruned version of the neural network to generate a compact version of the neural network.

[0290] Example 71 includes the subject matter of example 70, wherein the quantization includes log-based quantization.

[0291] Example 72 includes the subject matter of example 71, where the weights are quantized from floating point values ​​to base 2 values.

[0292] Example 73 includes the subject matter of any one of Examples 67 to 72. The subject matter further includes providing a pruned version of the neural network for performing the machine learning task using hardware adapted for sparse matrix operations.

[0293] Example 74 is a system including means for performing the method according to any one of Examples 67 to 73.

[0294] Example 75 includes the subject matter of Example 64, wherein the system includes an apparatus, the apparatus including hardware circuitry for performing at least a portion of the method of any one of Examples 67 to 73.

[0295] Example 76 is a computer-readable storage medium storing instructions executable by a processor to perform the method of any one of Examples 67 to 73.

[0296] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Moreover, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results.

[0297] (Item 1) accessing a synthetic three-dimensional (3D) graphical model of the object from a memory, the 3D graphical model having photorealistic resolution; generating a plurality of different training samples from views of the 3D graphical model, the plurality of training samples being generated by adding imperfections to the plurality of training samples to simulate characteristics of real-world samples generated by a real-world sensor device; generating a training set including the plurality of training samples, the training data being used to train an artificial neural network; A method for providing (Item 2) Item 10. The method of item 1, wherein the plurality of training samples include digital images and the sensor device includes a camera sensor. (Item 3) 3. The method according to any one of items 1 to 2, wherein the plurality of training samples comprises a point cloud representation of the object. (Item 4) 4. The method of claim 3, wherein the sensor device includes a LIDAR sensor. (Item 5) accessing data indicative of a parameter of the sensor device; determining the defects to add to the plurality of training samples based on the parameters; 5. The method according to any one of items 1 to 4, further comprising: (Item 6) 6. The method of claim 5, wherein the data includes a model of the sensor device. (Item 7) accessing data characteristic of one or more surfaces of the object modeled by the 3D graphical model; determining the defects to add to the plurality of training samples based on the characteristics; 7. The method of any one of items 1 to 6, further comprising: (Item 8) 8. The method of claim 7, wherein the 3D graphical model includes the data. (Item 9) 9. The method of any one of items 1 to 8, wherein the defects include one or more of noise or glare. (Item 10) The step of generating a plurality of different training samples comprises: applying a plurality of different lighting settings to the 3D graphical model to simulate lighting in the environment; determining the defects for a subset of the plurality of training samples generated during application of a particular one of the plurality of different lighting settings, wherein the defects for the subset of the plurality of training samples are based on the particular lighting setting. 10. The method according to any one of items 1 to 9, comprising: (Item 11) The step of generating a plurality of different training samples comprises: placing said 3D graphical model in a plurality of different graphical environments, each of said graphical environments modeling a respective real-world environment; generating a subset of the plurality of training samples while the 3D graphical model is placed within a plurality of different graphical environments; 11. The method according to any one of items 1 to 10, comprising: (Item 12) 12. A system comprising means for carrying out the method according to any one of items 1 to 11. (Item 13) Item 13. The system of item 12, comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of items 1 to 11. (Item 14) 12. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 1 to 11. (Item 15) receiving a subject input and a reference input at a Siamese neural network, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, wherein weights of the first network portion are the same as weights of the second network portion, the subject input being provided as an input to the first network portion and the reference input being provided as an input to the second network portion; generating an output of the Siamese neural network based on the subject input and the reference input, the output of the Siamese neural network indicating a similarity between the reference input and the subject input; A method for providing (Item 16) The step of generating the output comprises: determining a difference between the reference input and the subject input; determining whether the difference amount meets a threshold, the output identifying whether the difference amount meets the threshold; Item 16. The method according to Item 15, comprising: (Item 17) determining the difference between the reference input and the subject input comprises: receiving a first feature vector output by the first network portion and a second feature vector output by the second network portion; determining a difference vector based on the first feature vector and the second feature vector; Item 17. The method according to item 16, comprising: (Item 18) 18. The method of any one of items 15 to 17, wherein the step of generating an output comprises a one-time classification. (Item 19) 19. The method of any one of items 15 to 18, further comprising training the Siamese neural network using one or more synthetic training samples. (Item 20) Item 19. The method according to item 18, wherein the one or more synthetic training samples are generated according to the method according to any one of items 1 to 11. (Item 21) 21. The method of any one of items 15 to 20, wherein the reference input comprises a synthetically generated sample. (Item 22) Item 22. The method according to item 21, wherein the synthetically produced sample is produced according to the method according to any one of items 1 to 11. (Item 23) 23. The method of any one of items 15 to 22, wherein the subject input comprises a first digital image and the reference input comprises a second digital image. (Item 24) 23. The method of any one of items 15 to 22, wherein the subject input comprises a first point cloud representation and the reference input comprises a second point cloud representation. (Item 25) 25. A system comprising means for carrying out the method according to any one of items 15 to 24. (Item 26) 26. The system of claim 25, wherein the system comprises an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 15 to 24. (Item 27) Item 26. The system of item 25, comprising one of a robot, a drone, or an autonomous vehicle. (Item 28) 25. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 15 to 24. (Item 29) providing first input data to a Siamese neural network, the first input data including a first representation of 3D space from a first pose; providing second input data to the Siamese neural network, the second input data including a second representation of 3D space from a second pose, the Siamese neural network including a first network portion including a first plurality of layers and a second network portion including a second plurality of layers, the weights of the first network portion being the same as the weights of the second network portion, the first input data being provided as input to the first network portion and the second input data being provided as input to the second network portion; generating an output of the Siamese neural network, the output including a relative pose between the first pose and the second pose; A method for providing (Item 30) 30. The method of claim 29, wherein the first representation of 3D space comprises a first 3D point cloud and the second representation of 3D space comprises a second 3D point cloud. (Item 31) 31. The method of any one of items 29 to 30, wherein the first representation of the 3D space comprises a first cloud of points and the second representation of the 3D space comprises a second cloud of points. (Item 32) Item 32. The method of item 31, wherein the first point cloud and the second point cloud each comprise a respective voxelized point cloud representation. (Item 33) generating a 3D mapping of the 3D space from at least the first input data and the second input data based on the relative pose; 33. The method of any one of items 29 to 32, further comprising: (Item 34) determining a position of an observer at said first pose in said 3D space based on said relative pose; 33. The method of any one of items 29 to 32, further comprising: (Item 35) Item 35. The method of item 34, wherein the observer comprises an autonomous machine. (Item 36) Item 36. The method of item 35, wherein the autonomous machine comprises one of a robot, a drone, or an autonomous vehicle. (Item 37) 37. A system comprising means for carrying out the method according to any one of items 29 to 36. (Item 38) 38. The system of claim 37, wherein the system comprises an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 29 to 36. (Item 39) 37. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 29 to 36. (Item 40) providing the first sensor data as input to a first portion of a machine learning model; providing the second sensor data as input to a second portion of the machine learning model, the machine learning model including a connector and a set of fully connected layers, the first sensor data being of a first type generated by a device and the second sensor data being of a different second type generated by the device, the connector receiving an output of the first portion of the machine learning model as a first input and an output of the second portion of the machine learning model as a second input, the output of the connector being provided to the set of fully connected layers; generating an output of the machine learning model from the first data and the second data, the output comprising a pose of the device within an environment; A method for providing (Item 41) Item 41. The method of item 40, wherein the first sensor data includes image data and the second sensor data identifies movement of the device. (Item 42) Item 42. The method of item 41, wherein the image data comprises red-green-blue (RGB) data. (Item 43) Item 42. The method according to item 41, wherein the image data includes 3D point cloud data. (Item 44) Item 42. The method of item 41, wherein the second sensor data includes inertial measurement unit (IMU) data. (Item 45) Item 42. The method of item 41, wherein the second sensor data includes global positioning data. (Item 46) 46. ​​The method of any one of items 40 to 45, wherein the first portion of the machine learning model is tuned for the first type of sensor data and the second portion of the machine learning model is tuned for the second type of sensor data. (Item 47) 47. The method of any one of items 40 to 46, further comprising providing third sensor data of a third type as input to a third part of the machine learning model, wherein the output is further generated based on the third data. (Item 48) Item 48. The method according to any one of items 40 to 47, wherein the pose output includes a rotational component and a translational component. (Item 49) Item 49. The method of item 48, wherein one of the set of fully connected layers includes a fully connected layer for determining the rotation component, and another of the set of fully connected layers includes a fully connected layer for determining the translation component. (Item 50) 50. The method of any one of items 40 to 49, wherein one or both of the first and second parts of the machine learning model include respective convolutional layers. (Item 51) 51. The method of any one of items 40 to 50, wherein one or both of the first and second parts of the machine learning model include one or more respective long short-term memory (LSTM) blocks. (Item 52) 52. The method of any one of items 40 to 51, wherein the device includes an autonomous machine, and the autonomous machine navigates within the environment based on the pose. (Item 53) Item 53. The method of item 52, wherein the autonomous machine comprises one of a robot, a drone, or an autonomous vehicle. (Item 54) 53. A system comprising means for carrying out the method according to any one of items 40 to 52. (Item 55) 55. The system of claim 54, wherein the system comprises an apparatus having hardware circuitry for performing at least a portion of the method of any one of claims 40 to 52. (Item 56) 53. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 40 to 52. (Item 57) requesting random generation of a set of neural networks; performing a machine learning task using each one of the set of neural networks, the machine learning task being performed using specific processing hardware; monitoring attributes of the performance of the machine learning task for each of the set of neural networks, the attributes including accuracy of results of the machine learning task; identifying a best-performing neural network from the set of neural networks when used to perform the machine learning task using the particular processing hardware based on the attributes of the best-performing neural network; A method for providing (Item 58) providing said best performing neural network for use by a machine in executing a machine learning application. Item 58. The method of item 57, further comprising: (Item 59) determining characteristics of said best performing neural network; requesting generation of a second set of neural networks according to the characteristics, the second set of neural networks including a plurality of different neural networks, each of the neural networks including one or more of the characteristics; performing the machine learning task using each one of the second set of neural networks, the machine learning task being performed using the particular processing hardware; monitoring attributes of the performance of the machine learning task for each of the second set of neural networks; identifying a best performing neural network from said second set of neural networks based on said attributes; 59. The method according to any one of items 57 to 58, further comprising: (Item 60) 60. The method of any one of items 57 to 59, further comprising receiving a criterion based on the parameters, wherein the best performing neural network is based on the criterion. (Item 61) 61. The method of any one of items 57 to 60, wherein the attributes include attributes of the particular processing hardware. (Item 62) 62. The method of claim 61, wherein the attributes of the particular processing hardware include one or more of: power consumed by the particular processing hardware while performing the machine learning task; a temperature of the particular processing hardware while performing the machine learning task; and memory used to store the neural network on the particular processing hardware. (Item 63) 63. The method of any one of items 57 to 62, wherein the attributes include the time to complete the machine learning task using the corresponding one of the set of neural networks. (Item 64) 64. A system comprising means for carrying out the method according to any one of items 57 to 63. (Item 65) Item 65. The system of item 64, comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of items 57 to 63. (Item 66) 64. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 57 to 63. (Item 67) identifying a neural network comprising a plurality of kernels, each of the kernels comprising a respective set of weights; reducing the plurality of kernels to a set of particular kernels by pruning a subset of the plurality of kernels according to one or more parameters; pruning a subset of weights in the particular set of kernels to form a pruned version of the neural network, wherein pruning the subset of weights assigns one or more non-zero weights in the subset of weights to zero, and the subset of weights is selected based on the original values ​​of the weights; A method for providing (Item 68) Item 68. The method of item 67, wherein the subset of weights is pruned based on values ​​of the subset of weights being below a threshold. (Item 69) 69. The method of any one of items 67-68, further comprising performing one or more iterations of a machine learning task using the pruned version of the neural network to regain at least some of the accuracy lost by pruning the kernels and the weights. (Item 70) 70. The method of any one of items 67 to 69, further comprising quantizing values ​​of unpruned weights in the pruned version of the neural network to generate a compact version of the neural network. (Item 71) Item 71. The method of item 70, wherein the quantization includes log-based quantization. (Item 72) Item 72. The method of item 71, wherein the weights are quantized from floating-point values ​​to base-2 values. (Item 73) 73. The method of any one of items 67 to 72, further comprising providing the pruned version of the neural network for performing machine learning tasks using hardware adapted for sparse matrix operations. (Item 74) A system comprising means for carrying out the method according to any one of items 67 to 73. (Item 75) Item 65. The system of item 64, comprising an apparatus having hardware circuitry for performing at least a portion of the method of any one of items 67 to 73. (Item 76) 74. A computer-readable storage medium storing instructions executable by a processor for performing the method of any one of items 67 to 73.

Claims

1. A method implemented by a computer processor, comprising: generating one or more synthetic training samples, the synthetic training samples comprising synthetic images; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set comprising the one or more photorealistic training samples; training a neural network using the training set, the neural network having a first network and a second network, and after the training step, the first network and the second network generate first and second feature vectors from the two different input images, respectively, to estimate a similarity between the two different input images; A method comprising:

2. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples to degrade the quality; 2. The method of claim 1, comprising:

3. The method of claim 1 or 2, wherein the photorealistic training samples have a lower photorealistic resolution than the resolution of the synthetic training samples from which they are generated.

4. The method of claim 1 , wherein the quality of the one or more synthetic training samples comprises brightness, noise level, or contrast.

5. The method of claim 1 , wherein the first network and the second network have the same weight.

6. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object; 2. The method of claim 1, comprising:

7. 7. The method of claim 1, wherein the first network generates the first feature vector from one of the two different images, the second network generates the second feature vector from the other of the two different images, and the neural network determines the similarity based on the first feature vector and the second feature vector.

8. To the computer processor, generating one or more synthetic training samples, the synthetic training samples comprising synthetic images; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set comprising the one or more photorealistic training samples; a step of training a neural network using the training set, the neural network having a first network and a second network, and after the training step, the first network and the second network generate a first feature vector and a second feature vector, respectively, from the two different input images to estimate a similarity between the two different input images; A computer program for executing

9. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples to degrade the quality; 9. The computer program of claim 8, comprising:

10. 10. The computer program of claim 8 or 9, wherein the photorealistic training samples have a lower photorealistic resolution than the resolution of the synthetic training samples from which they are generated.

11. The computer program product of claim 8 , wherein the quality of the one or more synthetic training samples comprises brightness, noise level, or contrast.

12. 12. The computer program of claim 8, wherein the first network and the second network have the same weight.

13. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object; 9. The computer program of claim 8, comprising:

14. 14. The computer program of claim 8, wherein the first network generates the first feature vector from one of the two different images, the second network generates the second feature vector from the other of the two different images, and the neural network determines the similarity based on the first feature vector and the second feature vector.

15. A computer readable storage medium storing a computer program according to any one of claims 8 to 14.

16. a computer processor for executing computer program instructions; generating one or more synthetic training samples, the synthetic training samples comprising a synthetic image; generating one or more photorealistic training samples by degrading the one or more synthetic training samples; forming a training set having the one or more photorealistic training samples; training a neural network using the training set, the neural network having a first network and a second network, wherein after training, the first network and the second network generate first and second feature vectors from the two different input images, respectively, to estimate a similarity between the two different input images; and one or more non-transitory computer-readable storage media storing computer program instructions executable by said computer processor to perform operations having the following: An apparatus comprising:

17. Degrading the quality of the one or more synthetic training samples comprises: applying a filter to the one or more synthetic training samples to degrade the quality; 17. The apparatus of claim 16, comprising:

18. 18. The apparatus of claim 16 or 17, wherein the photorealistic training samples have a lower photorealistic resolution than the resolution of the synthetic training samples from which they are generated.

19. 19. The apparatus of claim 16, wherein the first network and the second network have the same weight.

20. The synthetic image comprises a three-dimensional model of an object, and degrading the quality of the one or more synthetic training samples comprises: degrading the composite image based on one or more materials of the object; 17. The apparatus of claim 16, comprising:

21. 21. The apparatus of claim 16, wherein the first network generates the first feature vector from one of the two different images, the second network generates the second feature vector from the other of the two different images, and the neural network determines the similarity based on the first feature vector and the second feature vector.

Images