Computer Vision Systems

Abstract

The raycaster executes a raycasting algorithm, wherein the raycasting algorithm takes the sparse hierarchical volume data structure as an input. Executing the raycasting algorithm includes: casting a plurality of rays from a reference point into the 3D volume; and for each ray of the plurality of rays, traversing the ray to determine whether a voxel in the voxel group is intersected and occupied by the ray, wherein the ray is traversed according to an approximate traversal algorithm.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 724,446, filed on August 29, 2018, and is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to the field of computer systems, and more particularly to hash tables for use in computer vision applications. Background Art

[0004] The world of computer vision and graphics is converging rapidly with the emergence of augmented reality (AR), virtual reality (VR), and mixed reality (MR) products, such as those from Magic Leap. TM , Microsoft TM 、HoloLens TM 、Oculus TM Rift TM Those products, as well as products such as those from Valve TM and HTC TM Other VR systems such as those described in [ 1 ]. The current approach in such systems is to use separate graphics processing units (GPUs) and computer vision subsystems running in parallel. These parallel systems can be assembled from existing GPUs in parallel with a computer vision pipeline implemented in software running on an array of processors and / or programmable hardware accelerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The various objects, features and advantages of the disclosed subject matter may be more fully understood with reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, wherein like reference numerals identify like elements. The drawings are schematic and are not intended to be drawn to scale. For clarity, not every component is labeled in every figure. Nor is every component of every embodiment of the disclosed subject matter shown where a diagram is not needed to allow one of ordinary skill in the art to understand the disclosed subject matter.

[0006] FIG1 shows a conventional augmented reality or mixed reality rendering system;

[0007] Figure 2 A voxel-based augmented reality or mixed reality rendering system is shown according to some embodiments;

[0008] Figure 3 illustrates the difference between dense and sparse volume representations according to some embodiments;

[0009] Figure 4 shows a composite view of a scene according to some embodiments;

[0010] Figure 5 illustrates the level of detail in an example element tree structure according to some embodiments;

[0011] Figure 6 shows applications that can utilize the data structures and voxel data of the present application according to some embodiments;

[0012] Figure 7 shows an example network for recognizing 3D digits according to some embodiments;

[0013] Figure 8 illustrates multiple classifications performed on the same data structure using implicit levels of detail according to some embodiments;

[0014] Fig. 9 illustrates operation elimination by a 2D convolutional neural network according to some embodiments;

[0015] Fig.10 shows experimental results from analysis of example test images according to some embodiments;

[0016] Fig.11 shows hardware for culling operations according to some embodiments;

[0017] Fig.12 shows improvements to hardware for culling operations according to some embodiments;

[0018] Fig.13 shows hardware according to some embodiments;

[0019] Fig.14 An example system employing a computer vision subsystem according to some embodiments is shown;

[0020] Fig.15 shows ray casting using a hierarchical volume data structure according to some embodiments;

[0021] Figure 16A-16C shows an example of ray traversal during an example ray casting algorithm;

[0022] Figure 17A-17B shows example comparison images rendered using different ray casting techniques;

[0023] Fig.18 is a simplified block diagram illustrating the architecture of an example vision processing unit (VPU);

[0024] Fig.19 is an example diagram showing a comparison of example results of ray casting techniques;

[0025] Figure 20A-20B is an example graph illustrating a comparison of example performance characteristics of different ray casting techniques;

[0026] Figure 21A-Figure 21C is a simplified block diagram illustrating an example neural network model;

[0027] Fig. 22 is a graph illustrating example performance of a visual inertial odometry technique;

[0028] Fig.23A is a simplified block diagram of traditional relative pose estimation techniques;

[0029] Fig. 23B is a simplified block diagram of an improved relative pose estimation technique using a neural network model;

[0030] Fig.24 is a simplified block diagram illustrating an example neural network model;

[0031] Fig.25 Voxelization of an example point cloud is shown;

[0032] Figure 26A-26C is a flow chart illustrating an example technique for enhancing computer vision processing;

[0033] Fig. 27 depicts an example multi-slot vector processor according to some embodiments;

[0034] Fig.28 shows example volume acceleration hardware according to some embodiments;

[0035] Fig.29 illustrates the organization of a voxel cube according to some embodiments;

[0036] Fig.30 shows a two-level sparse voxel tree according to some embodiments;

[0037] Fig.31 shows a two-level sparse voxel tree according to some embodiments;

[0038] Fig.32 shows storage of example voxel data according to some embodiments;

[0039] Fig.33 illustrates insertion of voxels into an example volume data structure in accordance with some embodiments;

[0040] Fig.34 shows a projection of an example 3D volumetric object according to some embodiments;

[0041] Fig.35Example operations involving example volume data structures are shown;

[0042] Fig.36 shows the use of projections to generate simplified maps according to some embodiments;

[0043] Fig.37 illustrates an example aggregation of example volumetric 3D measurements and / or simple 2D measurements from an embedded device according to some embodiments;

[0044] Fig.38 shows an example speedup of 2D pathfinding on a 2D 2×2 bitmap according to some embodiments;

[0045] Fig.39 shows an example acceleration of collision detection using an example volume data structure according to some embodiments;

[0046] Fig.40 is a simplified block diagram of an exemplary network having devices according to at least some embodiments;

[0047] Fig.41 is a simplified block diagram of an exemplary fog or cloud computing network according to at least some embodiments;

[0048] Fig.42 is a simplified block diagram of a system including example devices according to at least some embodiments;

[0049] Fig.43 is a simplified block diagram of an example processing device according to at least some embodiments;

[0050] Fig.44 is a block diagram of an exemplary processor according to at least some embodiments; and

[0051] Fig.45 is a block diagram of an example computing system in accordance with at least some embodiments. DETAILED DESCRIPTION

[0052] In the following description, many specific details related to the systems and methods of the disclosed subject matter and the environments in which such systems and methods can operate, etc., are set forth in order to provide a thorough understanding of the disclosed subject matter. However, it will be apparent to those skilled in the art that the disclosed subject matter can be practiced without these specific details, and certain features that are well known in the art are not described in detail in order to avoid complicating the disclosed subject matter. In addition, it should be understood that the embodiments provided below are exemplary, and it is contemplated that there are other systems and methods within the scope of the disclosed subject matter.

[0053] Various technologies are emerging and including augmented reality, virtual reality, mixed reality, autonomous devices and robots, which can utilize data models representing volumes of three-dimensional space and geometric shapes. The description of various real and virtual environments using such 3D or volumetric data traditionally has large data sets, and some computing systems have always been difficult to process the large data sets in a desired manner. In addition, when devices such as drones, wearable devices, virtual reality systems, etc. become smaller, the memory and processing resources of such devices may also be limited. As an example, AR / VR / MR applications may require high frame rates for graphical representations generated using supporting hardware. However, in some applications, the GPU and computer vision subsystems of such hardware may need to process data (e.g., 3D data) at a high rate (such as up to 130fps (7 milliseconds)) to produce desired results (e.g., generate a credible graphical scene with a frame rate that produces credible results, prevent motion sickness of users caused by excessive waiting time, and other example goals). Additional applications can be similarly challenged to satisfactorily process data describing large volumes while meeting the limitations in the processing, memory, power, application requirements of the corresponding system, and other example problems.

[0054] In some implementations, logic may be provided for a computing system to generate and / or use sparse volume data defined according to a format. For example, a defined volume data structure may be provided to unify computer vision and 3D rendering in various systems and applications. A volume representation of an object may be captured using an optical sensor, such as, for example, a stereo camera or a depth camera. The volume representation of an object may include multiple voxels. An improved volume data structure may be defined that enables a corresponding volume representation to be recursively subdivided to obtain a target resolution for the object. During subdivision, blank space in the volume representation may be removed from the volume representation (and supporting operations), which blank space may be included in one or more voxels in the voxel. Blank space may be an area of ​​the volume representation that does not include geometric attributes of the object.

[0055] Thus, in an improved volume data structure, individual voxels within a corresponding volume may be marked as "occupied" (by some geometry present within the corresponding volume space) or "empty" (indicating that the corresponding volume consists of empty space). Such labels may additionally be interpreted as specifying that one or more of its corresponding subvolumes are also occupied (e.g., if a parent voxel or a higher-level voxel is marked as occupied), or that all of its subvolumes are empty space (i.e., in the case where a parent voxel or a higher-level voxel is marked as empty). In some implementations, marking a voxel as empty may allow the voxel and / or its corresponding subvolume voxel to be effectively removed from operations used to generate a corresponding volume representation. The volume data structure may be in a sparse tree structure, such as in a sparse sexaquaternary tree (SST) format. Further, such a sparse volume data structure approach may utilize relatively less storage space than conventional methods for storing volume representations of objects. Additionally, compression of volume data may increase the transmission viability of such representations and enable faster processing of such representations, among other example benefits.

[0056] The volumetric data structure can be hardware accelerated to quickly allow updates to the 3D renderer, eliminating delays that may occur in separate computer vision and graphics systems. Such delays can result in latency that can cause user motion sickness and other additional disadvantages when used in AR, VR, MR, and other applications. The ability to quickly test the occupancy of voxels in the accelerated data structure allows the construction of low-latency AR, VR, MR, or other systems that can be updated in real time.

[0057] In some embodiments, the capabilities of the volumetric data structure may also provide intra-frame warnings. For example, in AR, VR, MR, and other applications, when a user may collide with a real or synthetic object in the imaged scene, or in computer vision applications for drones or robots, when such devices may collide with a real or synthetic object in the imaged scene, the processing speed provided by the volumetric data structure allows warning of an impending collision.

[0058] Embodiments of the present disclosure may be associated with the storage and processing of volume data in applications such as robots, head-mounted displays for augmented reality and mixed reality head-mounted devices, and phones and tablets. Embodiments of the present disclosure represent each volume element (e.g., voxel) within a set of voxels and, optionally, a physical quantity associated with the geometry of the voxel as a single bit. Additional parameters associated with a set of 64 voxels may be associated with the voxels, such as corresponding red, green, blue (RGB) or other color encoding, transparency, truncated signed distance function (TSDF) information, etc., and stored in an associated and optional 64-bit data structure (e.g., so that two or more bits are used to represent each voxel). Such a representation scheme may achieve minimal memory requirements. In addition, representing voxels by a single bit allows the execution of many simplified calculations to logically or mathematically combine elements from the volume representation. Combining elements from the volume representation may include, for example, ORing a plane in the volume to create a 2D projection of 3D volume data, and calculating the surface area by counting the number of voxels occupied in the 2.5D manifold. For comparison, exclusive-OR (XOR) logic may be used to compare 64-bit subvolumes (eg, 4^3 subvolumes), and volumes may be inverted, where objects may be merged by ORing them together to create a hybrid object, among other examples.

[0059] FIG1 shows a conventional augmented reality or mixed reality system consisting of parallel graphics rendering and computer vision subsystems with post-rendering connections to account for changes due to rapid head motion and changes in the environment that may produce occlusions and shadows in the rendered graphics. In one example implementation, the system may include a host processor 100 supported by a host memory 124 to control the execution of a graphics pipeline, a computer vision pipeline, and a post-rendering correction device through an interconnect via a bus 101, an on-chip network on-chip, or other interconnect. The interconnect allows the host processor 100 to run appropriate software to control the execution of a graphics processing unit (GPU) 106, associated graphics memory 111, a computer vision pipeline 116, and associated computer vision memory 124. In one example, the rendering of graphics using the GPU 106 via an OpenGL graphics shader 107 (e.g., operating on a triangle list 105) may occur at a slower rate than the computer vision pipeline. As a result, post-rendering corrections via the warp engine 108 and display / occlusion processor 109 may be performed to account for changes that may have occurred in the head pose and occluded scene geometry since the graphics were rendered by the GPU 106. The output of the GPU 106 is time-stamped so that it can be combined with correction control signals 121 from the head pose pipeline 120 and correction control signals 123 from the occlusion pipeline 123, respectively, to produce correct graphics output to account for any changes in head pose 119 and occlusion geometry 113, as well as any changes in other examples.

[0060] Multiple sensors and cameras (e.g., including active stereo cameras and passive stereo cameras for depth and vision processing 117) in parallel with the GPU 106 can be connected to the computer vision pipeline 116. The computer vision pipeline 116 can include one or more of at least three stages, each of which can contain multiple stages of lower-level processing. In one example, the stages in the computer vision pipeline 116 can be an image signal processing (ISP) pipeline 118, a head pose pipeline 120, and an occlusion pipeline 122. The ISP pipeline 118 can take the output of the input camera sensor 117 and condition them so that they can be used for subsequent head pose and occlusion processing. The head pose pipeline 120 can take the output of the ISP pipeline 118 and use it with the output 119 of the inertial measurement unit (IMU) in the head-mounted device 110 to calculate the change in head pose since the corresponding output graphics frame rendered by the free GPU 106. The output 121 of the head pose pipeline (HPP) 120 may be applied to the warp engine 108 along with a user specified mesh to warp the GPU output 102 so that it matches the updated head pose position 119. The occlusion pipeline 122 may take the output of the head pose pipeline 121 and look for new objects in the field of view, such as a hand 113 (or other example object) entering the field of view that should produce corresponding shadows 114 on the scene geometry. The output 123 of the occlusion pipeline 122 may be used by the display and occlusion processor 109 to properly overlay the field of view on top of the output 103 of the warp engine 108. The display and occlusion processor 109 uses the calculated head pose 119 to generate a shadow mask for synthesizing shadows 114, and the display and occlusion processor 109 can synthesize the occlusion geometry of the hand 113 on top of the shadow mask to produce a graphic shadow 114 on top of the output 103 of the warp engine 108, and produce (multiple) final output frames 104 for display on the augmented / mixed reality headset 110, as well as other example use cases and features.

[0061] Figure 2 A voxel-based augmented reality or mixed reality rendering system according to some embodiments of the present disclosure is shown. Figure 2The apparatus depicted in may include a host system comprised of a host CPU 200 and associated host memory 201. Such a system may communicate with a unified computer vision and graphics pipeline 223 and associated unified computer vision and graphics memory 213 via a bus 204, an on-chip network, or other communication mechanism, and the associated unified computer vision and graphics memory 213 contains real voxels and synthetic voxels to be rendered in a final scene for display on a head-mounted augmented reality or mixed reality display 211. The AR / MR display 211 may also include a plurality of active and passive image sensors 214, and an inertial measurement unit (IMU) 212 for measuring changes in the orientation of the head pose 222.

[0062] In the combined rendering pipeline, synthetic geometry can be generated starting from a triangle list 204, which is processed by an OpenGL JiT (Just-in-Time) translator 205 to produce a synthetic voxel geometry 202. For example, a synthetic voxel geometry can be generated by selecting the main plane of a triangle from a triangle list. 2D rasterization (e.g., in the X and Z directions) can then be performed on each triangle in the selected plane. A third coordinate (e.g., Y) can be created as an attribute to be interpolated across triangles. Each pixel of the rasterized triangle can result in the definition of a corresponding voxel. The process can be performed by a CPU or a GPU. When performed by a GPU, each rasterized triangle can be read back from the GPU to create a voxel where the GPU draws the pixel, as well as other example implementations. For example, a 2D buffer of a list can be used to generate synthetic voxels, wherein each entry in the list stores the depth information of the polygon rendered at the pixel. For example, an orthogonal viewpoint (e.g., top-down) can be used to render a model. For example, each (x, y) provided in the example buffer may represent a column at (x, y) in the corresponding voxel volume (e.g., from (x, y, 0) to (x, y, 4095). The information in each list may then be used to render each column from the information into a 3D scan line.

[0063] continue Figure 2In some implementations, the synthesized voxel geometry 202 can be combined with the measured geometry voxels 227 constructed using a simultaneous localization and mapping (SLAM) pipeline 217. The SLAM pipeline can use active sensors and / or passive image sensors 214 (e.g., 214.1 and 214.2), which are first processed using an image signal processing (ISP) pipeline 215 to produce an output 225, which can be converted into a depth image 226 by a depth pipeline 216. The active or passive image sensors 214 (214.1 and 214.2) can include active or passive stereo sensors, structured light sensors, time of flight sensors, and other examples. For example, the depth pipeline 216 can process depth data from a structured light or time of flight sensor 214.1 or alternatively from a passive stereo sensor 214.2. In one example implementation, the stereo sensor 214.2 can include a pair of passive stereo sensors, as well as other example implementations.

[0064] The depth image generated by the depth pipeline 215 can be processed by the dense SLAM pipeline 217 using a SLAM algorithm (e.g., Kinect Fusion) to produce a voxelized model of the measured geometry voxels 227. A ray tracing accelerator 206 can be provided, which can combine the measured geometry voxels 227 (e.g., real voxel geometry) with the synthetic voxel geometry 202 to produce a 2D rendering of the scene for output to a display device (e.g., a head mounted display 211 in a VR or AR application) via a display processor 210. In such an implementation, a complete scene model can be constructed from the real voxels of the measured geometry voxels 227 and the synthetic geometry 202. Therefore, there is no requirement to bend the 2D rendered geometry (e.g., as in FIG. 1). Such an implementation can be combined with a head pose tracking sensor and corresponding logic to correctly align the real geometry and the measured geometry. For example, the example head pose pipeline 221 may process head pose measurements 232 from an IMU 212 mounted in a head mounted display 212 , and the output 231 of the head pose measurement pipeline may be considered during rendering via the display processor 210 .

[0065] In some examples, the unified rendering pipeline can also use the measured geometry voxels 227 (e.g., real voxel models) and the synthetic geometry 202 (e.g., synthetic voxel models) to render audio reverberation models and model the physics of real-world scenes, virtual reality scenes, or mixed reality scenes. As an example, the physics pipeline 218 can take the measured geometry voxels 227 and the synthetic geometry 202 voxel geometry and use the ray casting accelerator 206 to calculate the output audio samples for the left and right headphones in the head-mounted display (HMD) 211 to calculate the output samples 230 using the acoustic reflection coefficients built into the voxel data structure. Similarly, the unified voxel model composed of 202 and 227 can also be used to determine physical updates for synthetic objects in the composite AR / MR scene. The physics pipeline 218 takes the composite scene geometry as input and uses the ray casting accelerator 206 to calculate collisions before calculating updates 228 to the synthetic geometry 202 for rendering and as a basis for future iterations of the physics model.

[0066] In some implementations, a system (such as Figure 2 The system shown in ) can additionally be provided with one or more hardware accelerators to implement and / or utilize a convolutional neural network (CNN) that can process RGB video / image input from the output of the ISP pipeline 215, volumetric scene data from the output of the SLAM pipeline 217, and other examples. The neural network classifier can be run exclusively using the hardware (HW) convolutional neural network (CNN) accelerator 207 or in a combination of the processor and the HW CNN accelerator 207 to produce an output classification 237. The availability of the HW CNN accelerator 207 to perform reasoning on the volumetric representation can allow groups of voxels in the measured geometry voxels 227 to be labeled as belonging to a specific object category, among other example uses.

[0067] Labeling voxels (e.g., using a CNN and supporting hardware acceleration) can allow those objects to which those voxels belong to be recognized by the system as corresponding to known objects, and the source voxels can be removed from the measured geometry voxels 227 and replaced by a bounding box corresponding to the object and / or replaced by information related to the object's origin, the object's pose, object descriptors, and other example information. This can result in a semantically far more meaningful description of the scene, which can, for example, be used as input by a robot, drone, or other computing system to interact with objects in the scene, or by an audio system to look up the sound absorption coefficients of objects in the scene and reflect them in an acoustic model of the scene, among other example uses.

[0068] One or more processor devices and hardware accelerators may be provided to implement Figure 2 In some implementations, all of the hardware and software elements of the combined rendering pipeline may share access to a DRAM controller 209, which in turn allows data to be stored in a shared DDR memory device 208, as well as other example implementations.

[0069] Presentation Figure 3 To illustrate the difference between dense volume representation and sparse volume representation according to some embodiments. Figure 3 As shown in the example of , a real-world object or synthetic object 300 (e.g., a figurine of a rabbit) can be described in the form of voxels in a dense manner as shown in 302 or in a sparse manner as shown in 304. The advantage of a dense representation such as 302 is the uniform speed of access to all voxels in the volume, but the disadvantage is the amount of storage that may be required. For example, for a dense representation such as a 512^3 element volume (e.g., corresponding to 5m in a volume scanned at 1cm resolution using a Kinect sensor), 512 megabytes are required to store a relatively small volume using a 4-byte truncated signed distance function (TSDF) for each voxel. The octree representation 304 embodies a sparse representation, in that on the other hand, the octree representation 304 can store only those voxels that have actual geometry in the real-world scene, thereby reducing the amount of data required to store the same volume.

[0070] Go to Figure 4 , showing a composite view of an example scene according to some embodiments. Specifically, Figure 4 A composite view of a scene 404 is shown how it may be maintained, displayed, or subjected to further processing using parallel data structures to represent synthetic voxels 401 within an equivalent bounding box 400 for synthetic voxel data and real-world measured voxels 403 within an equivalent bounding box 402 for real-world voxel data, respectively. Figure 5 4^3 element tree structure according to some embodiments. In some implementations, as little as 1 bit can be used to describe each voxel in a volume represented using an octree, such as in Figure 5 However, a disadvantage of octree-based techniques can be the number of indirect memory accesses utilized to access a particular voxel in the octree. In the case of a sparse voxel octree, the same geometry can be implicitly represented at multiple levels of detail, advantageously allowing operations such as ray casting, game physics, CNNs, and other techniques to allow blank portions of the scene to be culled from further computation, resulting in an overall reduction in not only required storage, but also in power consumption and computational load, among other example advantages.

[0071] In one implementation, an improved voxel descriptor (also referred to herein as a "volume data structure") may be provided for organizing volume information into 4^3 (or 64-bit) unsigned integers, such as shown in 501 with a memory requirement of 1 bit per voxel. In this example, 1 bit per voxel is insufficient to store the truncated signed distance function value (compared to the TSDF in SLAMbench / KFusion using 64 bits). In this example, an additional (e.g., 64-bit) field 500 may be included in the voxel descriptor. This example may be further enhanced so that when the TSDF in the 64-bit field 500 is 16 bits, an additional 2 bits of fractional resolution in x, y, and z may be implicitly provided in the voxel descriptor 501 so that the combination of the voxel TSDF and the voxel position 501 in the 64-bit field 500 is equivalent to a TSDF with much higher resolution, such as used in SLAMbench / KFusion or other examples. For example, additional data in the 64-bit field 500 (voxel descriptor) can be used to store 1-byte each of subsampled RGB color information (e.g., from a scene via a passive RGB sensor) and an 8-bit transparency value alpha, as well as two 1-byte reserved fields R1 and R2 that can be application specific and can be used to store, for example, acoustic reflectivity for audio applications, stiffness for physics applications, object material type, and other examples.

[0072] like Figure 5 As shown, the voxel descriptor 501 can be logically grouped into four 2D planes, each of which includes 16 voxels 502. Figure 5 As represented, these 2D planes (or voxel planes) can describe each level of an octree type structure based on a continuous decomposition of ascending powers of 4. In this example implementation, a 64-bit voxel descriptor is chosen because it is a good match to the 64-bit bus infrastructure used in the corresponding system implementation (although other voxel descriptor sizes and formats can be provided in other system implementations and the sizes can be designed based on the bus or other infrastructure of the system). In some implementations, the size of the voxel descriptor can be designed to reduce the number of memory accesses used to obtain voxels. For example, compared to a traditional octree operating on 2^3 elements, a 64-bit voxel descriptor can be used to reduce the number of memory accesses required to access a voxel at any level in the octree by 1 / 2, as well as other example considerations and implementations.

[0073] In one example, an octree may be described starting with a 4^3 root volume 503, and each non-zero entry therein for encoding the geometry presented in the layers 504, 505, and 506 below is depicted in an example 256^3 volume. In this particular example, four memory accesses may be used to access the lowest level in the octree. In cases where such overhead is too high, an alternative approach may be taken to encode the highest level of the octree as a larger volume, such as 64^3, as shown in 507. In this case, each non-zero entry in 507 may indicate the presence of the underlying 4^3 octree in the underlying 256^3 volume 508. The result of this alternative organization is that only two memory accesses are required to access any voxel in the 256^3 volume 508, compared to the alternative equations shown in 503, 504, and 505. The latter approach is advantageous in cases where the device hosting the octree has a large amount of embedded memory, thereby allowing only the lower and less frequently accessed portions of the voxel octree 508 to be in external memory. This approach is more expensive in terms of storage, for example, where a full, larger (e.g., 64^3) volume is stored in on-chip memory, but the tradeoff can allow faster memory access (e.g., 2x speed) and much lower power consumption, among other example advantages.

[0074] Go to Figure 6 , shows a block diagram of an example application that can utilize data structures and voxel data of the present application according to some embodiments. In one example, such as Figure 5 As shown in , additional information can be provided by the example voxel descriptor 500. While the voxel descriptor can increase the overall memory utilization to 2 bits per voxel, the voxel descriptor can enable a wide range of applications that can utilize voxel data, such as Figure 6 . For example, a shared volume representation 602 (such as generated using a dense SLAM system 601 (e.g., SLAMbench)) can be used to render a scene using graphics ray casting or ray tracing 603, can be used for audio ray casting 604, and other implementations. In yet other examples, the volume representation 602 can also be used in convolutional neural network (CNN) reasoning 605 and can be backed up by a cloud infrastructure 607. In some instances, the cloud infrastructure 607 can contain detailed volume descriptors of objects (such as a tree, a piece of furniture, or other objects (e.g., 606)) that can be accessed via reasoning. Based on the identification of the object in an inference or other manner, the corresponding detailed descriptor can be returned to the device, allowing the voxels of the volume representation 602 to be replaced by a bounding box representation with pose information and a descriptor containing object properties and other example features.

[0075] In yet other embodiments, in some systems, the voxel model discussed above may be utilized additionally or alternatively, using 3D to 2D projections to construct a 2D map 608 of the example environment from the volume representation 602. These 2D maps may be shared again and aggregated (e.g., using the same cloud infrastructure) via a communication machine via a cloud infrastructure and / or other network-based resources 607 to build higher quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices. In yet further examples, projections may be used and then optimized for ultra-low bandwidth applications by simplification 609 piece by piece (e.g., assuming that the width and height of the vehicle or robot are fixed). The simplified path may then have only a single X, Y coordinate pair at each piece-by-piece linear segment of the path, thereby reducing the amount of bandwidth required to transmit the path of the vehicle 609 to the cloud infrastructure 607 and aggregate in the same cloud infrastructure 607 to build higher quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices.

[0076] To enable these different applications, in some implementations, common functionality may be provided, such as through a shared software library, which may be accelerated in some embodiments using a hardware accelerator or processor instruction set architecture (ISA) extension, as well as other examples. For example, such functionality may include inserting voxels into a descriptor, deleting voxels, or finding voxels 610. In some implementations, collision detection functionality 620 may also be supported, as well as deleting points / voxels from a volume 630, as well as other examples. As described above, the system may be provided with functionality for quickly generating 2D projections 640 in the X, Y, and Z directions from a corresponding volume representation 602 (3D volume), for example, which may be used as a basis for path or collision determination. In some cases, it may also be advantageous to be able to generate a triangle list from a volume representation 602 using a histogram pyramid 650. Further, the system may be provided with functionality for quickly determining free paths 660 in 2D and 3D representations of a volume space 602. Such functionality may be useful in a range of applications. Further functionality may be provided, such as accounting for the number of voxels in the volume, using a population counter to count the number of bits with a value of 1 in a mask region of the volume representation 602 to determine the surface of the object, and other examples.

[0077] Go to Figure 7 A simplified block diagram of an example network including a system equipped with the functionality to recognize 3D numbers according to at least some embodiments. For example, Figure 6 One of the applications shown in FIG. 6 is a volumetric CNN application 605, which is Figure 7, where an example network is used to recognize 3D digits generated from a dataset such as the mixed National Institute of Standards and Technology (MNIST) dataset 700. The digits in such a dataset can be used for such training by applying appropriate rotations and translations in X, Y, and Z to the digits before training a CNN-based convolutional network classifier 710. When used for inference in an embedded device, the trained network 710 can be used to classify 3D digits in a scene with high accuracy 720, even when the digits are subjected to rotations and translations in X, Y, and Z, among other examples. In some implementations, the digits can be trained by Figure 2 The HW CNN accelerator 207 shown accelerates the operation of the CNN classifier. As the first layer of the neural network performs multiplication operations using voxels in the volume representation 602, these arithmetic operations can be skipped since multiplication by zero always equals zero and a data value A times 1 (voxel) equals A.

[0078] Figure 8 602. Multiple classifications are shown performed on the same data structure using implicit level of detail. A further improvement to CNN classification using volume representation 602 may be that the octree representation is Figure 5 The octree structure shown in Figure 1 implicitly contains multiple levels of detail, so a structure such as Figure 8 As shown in , a single classifier 830 or multiple classifiers in parallel can be used to perform multiple classifications on the same data structure using implicit detail levels 800, 810, and 820 in parallel. In conventional systems, comparable parallel classifications can be slow due to the image scaling required between classification passes. Because the same octree can contain the same information at multiple levels of detail, such scaling can be abandoned in the implementation of the voxel structure discussed in this article. In fact, a single training data set based on a volume model can cover all levels of detail, rather than a scaled training data set such as required in a conventional CNN network.

[0079] Go to Fig. 9 In some embodiments, an example operation elimination is shown by a 2D CNN. Fig. 9 As shown in , operation elimination can be used for 3D volumetric CNNs as well as 2D CNNs. For example, in Fig. 9 In the first layer, a bitmap mask 900 can be used to describe the expected "shape" of the input 910 and can be applied to the incoming video stream 920. In one example, operation elimination can be used not only for 3D volume CNN, but also for 2D volume CNN. For example, in Fig. 9In the example 2D CNN of , a bitmap mask 900 can be applied to the first layer of the CNN to describe the expected "shape" of the input 910, and can be applied to the input data of the CNN, such as an incoming video stream 820. As an example, Fig. 9 The effect of applying a bitmap mask to an image of a pedestrian for training or reasoning in a CNN network is shown in , where 901 represents the original image of a pedestrian 901 and 903 represents the corresponding version with the bitmap mask applied. Similarly, an image without a pedestrian is shown in 902 and the corresponding bitmap mask version is shown in 904. The same method can be applied to any type of 2D or 3D object to reduce the number of operations required for CNN training or reasoning through knowledge of the expected 2D or 3D geometry expected by the detector. An example of a 3D volume bitmap is shown in 911. The use of a 2D bitmap for reasoning in a real scene is shown in 920.

[0080] exist Fig. 9 In an example implementation of , a conceptual bitmap is shown (at 900), while a real bitmap is generated by averaging a series of training images for a particular class of objects 910. The example shown is two-dimensional, however, similar bitmap masks can also be generated for 3D objects in the proposed volume data format with 1 bit per voxel. In fact, the method can also potentially be extended to use additional bits per voxel / pixel to specify the expected color range or other characteristics of 2D or 3D objects, as well as other example implementations.

[0081] Fig.10 is a table showing the results of an example experiment involving analysis of 10,000 CIFAR-10 test images according to some embodiments. In some implementations, operation elimination can be used to eliminate the Fig.10 The intermediate computations in 1D, 2D, and 3D CNNs are caused by the frequent rectified linear unit (ReLU) operations in LeNet 1000 shown in . Fig.10 As shown in , in experiments using 10,000 CIFAR-10 test images, the percentage of data-dependent zero values ​​generated by the ReLU unit can be as high as 85%, which means that in the case of zero values, a system can be provided that recognizes the zero value and, in response, does not obtain the corresponding data and perform the corresponding multiplication operation. In this example, the 85% represents the percentage of ReLU dynamic zero values ​​generated from the modified National Institute of Standards and Technology database (MNIST) test data set. The elimination of corresponding operations corresponding to these zero values ​​can be used to reduce power consumption and memory bandwidth requirements, among other example benefits.

[0082] Unimportant operations may be eliminated based on the bitmap. For example, such bitmaps may be used in accordance with the principles and embodiments discussed and illustrated in U.S. Pat. No. 8,713,080, entitled “Circuit for compressing data and a processor employing the same,” the contents of which are incorporated herein by reference in their entirety. Some implementations may provide hardware capable of using such bitmaps, such as the systems, circuit systems, and other implementations discussed and illustrated in U.S. Pat. No. 9,104,633, entitled “Hardware for performing arithmetic operations,” the contents of which are also incorporated herein by reference in their entirety.

[0083] Fig.11 Hardware that can be incorporated into a system to provide functionality for bitmap-based culling of insignificant operations is shown in accordance with some embodiments. In this example, a multi-layer neural network is provided that includes repeated convolutional layers. The hardware may include one or more processors, one or more microprocessors, one or more circuits, one or more computers, etc. In this particular example, the neural network includes an initial convolution processing layer 1100, followed by a pooling process 1110, and finally an activation function process (such as a rectified linear unit (ReLU) function) 1120. The output of the ReLU unit 1120, which provides a ReLU output vector 1131, can be connected to a subsequent convolution processing layer 1180 (e.g., possibly via a delay 1132), which receives the ReLU output vector 1131. In an example implementation, a ReLU bitmap 1130 can also be generated in parallel with the connection of the ReLU unit 1120 to the subsequent convolution unit 1180, which indicates which elements in the ReLU output vector 1131 are zero values ​​and which elements are non-zero.

[0084] In one implementation, a bitmap (e.g., 1130) may be generated or otherwise provided to inform enabled hardware of opportunities to eliminate operations involving computations of a neural network. For example, bits in the ReLU bitmap 1130 may be interpreted by a bitmap scheduler 1160 that instructs the multipliers in the subsequent convolutional unit 1180 to skip zero-valued entries in the ReLU output vector 1131 given that multiplication by zero will always produce zero as output, given the presence of a corresponding binary zero in the ReLU bitmap 1130. In parallel, memory fetches from the address generator 1140 for data / weights corresponding to zero values ​​in the ReLU bitmap 1130 may also be skipped, since there are few values ​​in the fetched weights that will be skipped by the subsequent convolutional unit 1180. If the weights are fetched from an attached DDR DRAM memory device 1170 via the DDR controller 1150, the latency may be so high that only some on-chip bandwidth and associated power consumption may be saved. On the other hand, if the weights are fetched from on-chip RAM 1180 storage, it may be possible to bypass / skip the entire weight fetch operation, especially if a delay corresponding to the RAM / DDR fetch delay 1132 is added at the input to the subsequent convolution unit 1180.

[0085] Go to Fig.12 , presents a simplified block diagram according to some embodiments to illustrate improvements to example hardware equipped with circuitry and other logic for eliminating unimportant operations (or performing operation elimination). Fig.12 As shown in the example of , additional hardware logic may be provided to predict the sign of the ReLU unit 1220 input in advance from the leading Max-Pooling unit 1210 or the Convolution unit 1200. Adding sign prediction and ReLU bitmap generation to the Max-Pooling unit 1210 may allow the ReLU bitmap information to be predicted earlier from a timing perspective to cover delays that may occur through the address generator 1240, through the external DDR controller 1250 and DDR storage 1270, or the internal RAM storage 1271. If the latency is low enough, the ReLU bitmap may be interpreted in the address generator 1240 and the memory fetch associated with the ReLU bitmap zero value may be skipped entirely, since it may be determined that the result fetched from memory will never be used. Fig.11 This modification of the scheme can save additional power and may also allow removal of delay stages (e.g., 1132, 1232) at the input to a subsequent convolution unit 1280 if the latency through the DDR access path (e.g., 1240 to 1250 to 1270) or the RAM access path (e.g., 1240 to 1271) is low enough so as not to warrant delay stage 1232, as well as other example features and functionality.

[0086] Fig.13 is another simplified block diagram illustrating example hardware according to some embodiments. For example, a CNN ReLU layer may produce a high number of output zero values ​​corresponding to negative inputs. In practice, negative ReLU inputs can be identified by looking at the previous layers (e.g., Fig.13 Floating point and integer arithmetic can be explicitly signed in a most significant bit (MSB) manner so that a simple bitwise exclusive OR (XOR) operation across the input vectors to be multiplied in the convolutional layer can predict which multiplications will produce an output value of zero, such as Fig.13 The resulting ReLU bitmap vector of predicted symbols may be used as a basis for determining a subset of multiplications to eliminate and the associated coefficient readings from memory, such as in the manner described in other examples above.

[0087] Providing for returning the generation of the ReLU bitmap to a previous pooling or convolution stage (i.e., a stage before the corresponding ReLU stage) can result in additional power. For example, when the multiplier will produce a negative output that will ultimately be set to a zero value by the ReLU activation logic, sign prediction logic can be provided for disabling the multiplier. For example, this is shown where the two sign bits 1310 and 1315 of the multiplier 1314 inputs 1301 and 1302 are logically combined through an XOR gate to form a pre-ReLU (PreReLU) bitmap bit 1303. This same signal can be used to disable the operation of the multiplier 1314, which would otherwise unnecessarily consume energy generating a negative output that would be set to a zero value by the ReLU logic before being input for multiplication in the next convolution stage 1390, as well as other examples.

[0088] Note that the representations of 1300, 1301, 1302 and 1303 (symbol A) show that Fig.13 1302. In this example, the input to block 1302 may include two floating point operands. Input 1301 may include an explicit sign bit 1310, a mantissa 1311 including a plurality of bits, and an exponent 1312 also including a plurality of bits. Similarly, input 1302 may similarly include a sign 1315, a mantissa 1317, and an exponent 1316. In some implementations, the mantissa and exponent may have different precisions because the sign of the result 1303 depends only on the signs of 1301 and 1302, or 1310 and 1315, respectively. In fact, neither 1301 nor 1302 need to be floating point numbers, but may be any integer or fixed point format as long as they are signed numbers and the most significant bit (MSB) is effectively a sign bit, either explicitly or implicitly (e.g., if the number is in 1's complement or 2's complement, etc.).

[0089] continue Fig.13 In the example of, two sign inputs 1310 and 1315 may be combined using an exclusive-OR (XOR, sometimes alternatively denoted herein as ExOR or EXOR) gate to generate a bitmap bit 1303, which may then be processed using hardware to identify downstream multiplications that may be omitted in the next convolution block (e.g., 1390). The same exclusive-OR output 1303 may also be used to disable the multiplier 1314 in the case where the two input numbers 1313 (e.g., corresponding to 1301) and 1318 (e.g., corresponding to 1302) have opposite signs and would produce a negative output 1304 (which would be set to a zero value by the ReLU block 1319), resulting in a zero value in the RELU output vector 13191 to be input to the subsequent convolution stage 1390. Thus, in some implementations, the PreReLU bitmap 1320 may be transmitted in parallel to a bitmap scheduler 1360, which may schedule multiplications to be run (and / or omitted) on the convolution unit 1390. For example, for each zero value in the bitmap 1320, the corresponding convolution operation may be skipped in the convolution unit 1390. In parallel, the bitmap 1320 may be used by an example address generator 1330, which controls the acquisition of weights for use in the convolution unit 1390. A list of addresses corresponding to the 1 values ​​in the bitmap 1320 may be compiled in the address generator 1330 and control the path to the DDR storage 1370 via the DDR controller 1350, or control the path to the on-chip RAM 1380. In either case, weights corresponding to 1 values ​​in PreReLU bitmap 1320 may be retrieved and presented to convolution block 1390 (e.g., after some latency in clock cycles to weight input 1371), while retrieval of weights corresponding to zero values ​​may be omitted, among other examples.

[0090] As described above, in some implementations, a delay (e.g., 1361) may be interposed between the bitmap scheduler 1360 and the convolution unit 1390 to balance the delay through the address generator 1330, the DDR controller 1350 and the DDR 1350, or the delay of the path through the address generator 1330 and the internal RAM 1380. The delay may enable the convolution driven by the bitmap scheduler to be properly aligned in time with the corresponding weights for the convolution calculation in the convolution unit 1390. In fact, from a timing perspective, generating the ReLU bitmap earlier than the output of the ReLU block 1319 may allow additional time to be gained, which may be used to intercept reads to the memory (e.g., RAM 1380 or DDR 1370) before the address generator 1330 generates them, so that some reads (e.g., corresponding to zero values) may be discarded. Because memory reads may be much higher than logic operations on the chip, eliminating such memory fetches may result in very significant energy savings, among other example advantages.

[0091] In some implementations, if the savings in clock cycles is still not enough to cover the DRAM access time, a block-oriented technique can be used to read groups of sign bits (e.g., 1301) from the DDR in advance. These groups of sign bits can be used together with the sign blocks from the input image or intermediate convolution layer 1302 to generate PreReLU bitmap blocks using a set (multiple) of XOR gates 1300 (e.g., to calculate the difference between sign bits in a 2D or 3D convolution between 2D or 3D arrays / matrices, among other examples). In such implementations, an additional 1-bit of storage can be provided in the DDR or on-chip RAM for storing the sign of each weight, but this can allow many cycles of latency to be covered in a manner that avoids reading weights from the DDR or RAM that will be multiplied by the zero value from the ReLU stage. In some implementations, the additional 1-bit storage per weight in the DDR or on-chip RAM can be avoided because the signs are stored in a manner that they can be addressed independently of the exponent and mantissa, among other example considerations and implementations.

[0092] In one example, the system can be further enhanced to utilize DDR access, which can have natural burst access for maximum data transfer rate. It may not be feasible to save energy by skipping individual DDR weight accesses in this content because they may be shorter than the burst. Therefore, in some instances, the burst can be skipped when all bitmap bits corresponding to a particular burst transaction are 0. However, this may not happen frequently and therefore, the resulting power and bandwidth savings may be limited. In yet other implementations, a register programmable threshold can be set for the number of bitmap bits in the burst so that if more than N bits in the bitmap burst are zero values, the burst will be skipped completely. This may have the effect of slightly degrading the overall CNN classification accuracy, but may be acceptable for energy saving considerations.

[0093] Steering Fig.14 , a simplified block diagram 1400 illustrating an example environment of a machine 1405 that performs analysis of a 3D space is shown. In some implementations, the machine can be equipped with machine executable logic implemented in hardware and / or circuitry to utilize volumetric data to describe a 3D space in one or various applications or actions (such as, as discussed herein, SLAM programs, ray casting, odometry, collision detection, 2D or 3D path planning, and other examples). As discussed herein, the volumetric data can be embodied as a sparse tree representation. In some instances, hash tables can also or alternatively be used to perform faster processing of volumetric data by trading off increased memory usage (e.g., when hash tables use less memory than dense arrays).

[0094] The machine 1405 may include hardware and logic as discussed herein configured to process and handle volume data represented in a sparse tree volume data structure. Fig.14 In the particular example shown in , machine 1405 may be implemented as an autonomous or semi-autonomous machine that is capable of processing volumetric data describing a 3D scene and utilizing that information to autonomously move within the scene (e.g., change its position within the scene and / or change the orientation (e.g., aiming) of one or more machine elements (e.g., sensors, cameras, pointers, actuators, tools, etc.) based on the terrain present in the scene. By doing so, the machine may detect objects (e.g., 1410a-c) and autonomously navigate within or interact with the scene based on the detected objects. In some implementations, machine 1305 may be embodied as an autonomous vehicle (for carrying passengers or cargo), an aerial drone, a land-based drone, or a water-based drone, a robot, and other examples.

[0095] In one example implementation, machine 1405 may include a computing system 1406 implemented using a data processor 1415 (such as one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units or other matrix arithmetic processors, hardware accelerators (e.g., volumetric processing accelerators, machine learning accelerators), and other example general or special processing hardware), and further implemented using one or more memory elements (e.g., 1420). Additional logic blocks may be provided, which are implemented in hardware circuitry, firmware, or software, such as volumetric processing logic 1425, computer vision engine 1430, and actuators 1435. In some implementations, machine 1405 may additionally include one or more sensors (e.g., 1440) to measure 3D space (e.g., lidar, time of flight sensor, realsense sensor, etc.). Such sensors 1440 may be used to generate volumetric data describing a 3D environment to develop a map of the volume and to compare locally observed geometry detected using the sensors 1440 to reference data describing occupancy geometry of an expected or previously observed version of the volume. The sensor data may be processed by the computer vision engine 1430 in conjunction with ray casting, odometry, pose estimation, or other functionality (e.g., which may be implemented as a submodule of the computer vision engine 1430). In some instances, the volume processing logic 1425 may have logic for performing one or more different volume processing operations or tasks, such as tasks related to the functionality of the computer vision engine 1430. In one example, at least a portion of the volume processing logic 1425 and / or the computer vision engine 1430 may be implemented using a volume acceleration unit, such as the volume acceleration unit (VXU) described herein. In some implementations, all or part of the logic of the volume processing logic 1425 and the computer vision engine 1430 may be combined. The volume processing logic may take as input the volume data implemented in the VOLA data 1450 and parse the data 1450 to determine opportunities to forgo analysis of certain portions of the volume of interest that are identified in the data 1450 as not occupied by structures or geometric shapes. In some instances, the results generated by the volume processing logic 1425 and / or the computer vision engine may cause one or more actuators of the machine 1405 to be triggered such that one or more motors, engines, or other drives and / or one or more steering mechanisms are activated and the machine itself or a particular tool of the machine moves within the volume in accordance with its design. For example, the volume processing logic may provide input to one or more actuators to enable a drone or autonomous vehicle to autonomously navigate within a volume as understood by the machine through processing of the volume data.

[0096] Among its applications, for example, in the field of robotics, synchronized localization and mapping (SLAM) applications can be used to facilitate autonomous movement of robots and drones in environments, as well as other examples. In some cases, SLAM can utilize maps generated offline. However, autonomous robots usually need to build and interact with maps in real time, and use limited available resources to do this. Most methods of computationally intensive SLAM are designed for high-performance desktop systems, and the required FLOPs, memory, and power far exceed the low-cost embedded devices that are typically recommended or may be integrated in modern robots and drones, especially when considering the computationally intensive ray casting steps in traditional SLAM pipelines. To address this problem, an optimized low-power, low-memory implementation of the ray casting algorithm can be implemented in hardware and / or software to be integrated in the SLAM pipeline. For example, the volume acceleration data structure (VOLA) introduced in this article can be used for 3D volume representation, and additional techniques and features can utilize the structure of VOLA and the architecture for implementing improved SLAM applications to achieve specific performance / memory efficiency.

[0097] Although the worlds of computer vision and graphics are separate, they merge in the field of robotics. For both fields, it is a major challenge to represent volumetric data in 3D space in a way that saves the limited resources available in embedded systems. For example, input systems for computer vision (such as light detection and ranging (LiDAR)) can generate a point cloud depth map of the environment, which can quickly become too large for the embedded system memory to handle. A common application that poses a problem is SLAM applications, whose purpose is to create a map of the robot's surroundings and simultaneously locate the robot therein. In autonomous applications, building highly detailed maps will be useless if the autonomous agent cannot effectively interact with the map in real time. One of the most important interactions may be collision detection, which is particularly important in drone applications (for example, for obvious safety reasons). A common method for ray / geometry intersection is ray casting, which typically requires a large number of operations per second. Ray casting represents one of the most expensive components in a typical SLAM SW pipeline, and therefore despite the limitations of embedded platforms, its performance and efficiency need to be improved to make real-time embedded SLAM possible.

[0098] In some implementations, an improved, lightweight implementation of a ray casting algorithm for embedded platforms based on the use of a VOLA data structure, which is specifically designed to reduce the memory requirements of 3D models, can be utilized. For example, an embedded implementation of a ray casting algorithm can be provided, particularly for SLAM applications. Implementing such an algorithm can improve the utilization of computing resources such as caches and SIMD instructions to exploit the consistency of image and object space, among other example advantages. Dense or sparse hierarchical grid data structures (e.g., VOLA-based structures) can be utilized to produce volumetric compression of 3D scenes. Such solutions can also improve power efficiency while maintaining sufficient quality and accuracy.

[0099] As described above, a VOLA type data structure can be implemented as a compact data structure organized as a hierarchical bit array that allows efficient compression of volumetric data for use in embedded devices, where the use of full point clouds in such devices is unrealistic or even impossible due to memory limitations. The spatial partitioning VOLA structure can combine the hierarchical structure of the octree with the volumetric approach using a modular algorithm and a bit count applied to the bit array. In contrast to octrees, kd trees, and other 3D representations, VOLA may not use a pointer-based tree structure, but a hierarchical grid structure. Since compression is a primary consideration, VOLA encodes only occupancy data, achieving a maximum compression of 1 bit per voxel, making it extremely compact. Therefore, VOLA may not be well suited for accurate volume rendering by ray casting, but is very effective in applications such as SLAM, where a simple map of the surrounding must be constructed and ray cast, and where realism is not a must or priority. In some implementations, each element in VOLA is a 64-bit integer that divides the represented volume into 4x4x4 sub-cells, and uses a modular algorithm to map each element from a one-dimensional bit array to three dimensions. If some points are occupied within the cell, the corresponding sub-cell bit is set to 1, otherwise it is set to zero. In the next level, only occupied sub-cells are represented so that only occupied space is encoded. In this way, passing from a higher level to a lower level in the VOLA structure means a four-fold increase in resolution, as well as other examples.

[0100] The use of the VOLA format has been demonstrated in several applications, with particular efficiency when encoding aerial scans of major cities, for example, for LiDAR datasets, the maximum size can be reduced by a factor of 70. Other common applications that can benefit from the VOLA format are CNN inference, path planning, and obstacle avoidance. Finally, the memory footprint can be further reduced by using hash tables, among other example applications.

[0101] Example ray casting algorithms can be implemented as non-recursive ray tracing algorithms that cast only primary rays, and they are particularly popular as direct volume rendering techniques. Some of the main advantages of volume rendering techniques are their excellent image quality and the possibility to render images without explicitly defining surface geometry. On the contrary, the main disadvantage is their high computational cost. There are several approaches regarding data structures for representing 3D volumes and algorithms for voxel traversal. In addition, over the years, many additional techniques have been developed to handle specific sub-problems and improve the performance of the algorithms (such as space jumps, coherent group traversal, frustum culling, etc.).

[0102] The type of optimization used in ray casting is usually related to the volume representation chosen. The most basic representations of volumes are uniform grids and hierarchical grids. Hierarchical grids can be organized as loosely nested grids, recursive or multi-resolution grids, and macrocells or multi-grids. As another example, binary space partitioning (BSP) can be used to recursively subdivide space using arbitrary partitioning planes. In BSP, kd trees are a special case that only use axis-aligned splitting planes. Kd trees allow adaptive partitioning of volumes, but are more difficult to update than octrees, especially for large volumes. Although the performance of grouped kd tree traversals outperforms grids for static scenes, the cost of rebuilding the tree makes them less suitable for truly dynamic scenes. In contrast, grids can be created and modified at a higher rate, making them attractive for animated scenes even though they have a higher traversal cost. Another effective representation for rendering voxelized geometry is sparse voxel octrees, bounding volume hierarchies (BVHs), and other examples.

[0103] In some implementations, the VOLA structure can be utilized to efficiently skip empty cells using spatial jumps. The strategy for skipping empty spaces may be related to the volume representation below. In one example, ray acceleration by distance coding (RACD) can be used for spatial jumps to store the jump distance to the next possible opaque voxel for each voxel, which can be performed as a preprocessing step. In some implementations, a hardware architecture for spatial jumps can be implemented by calculating an occupancy map of the rendered volume only for the first level of a multi-level grid structure similar to VOLA. A method for skipping empty nodes based on a GPU octree can also be implemented. In some implementations, a proximity cloud can be used to skip empty spaces in a conventional 3D grid, where empty voxels are filled with scene-related information indicating proximity to surrounding objects. For example, an algorithm based on a proximity cloud can be implemented in a system that is particularly dedicated to GPU ray casting for large volume rendering, where the best image order and object order empty space skipping strategies are combined.

[0104] Volume traversal implementations may depend at least in part on the volume representation chosen. Most approaches have evolved from planar grid-based algorithms that determine the closest neighbor voxel to visit in each iteration based on the distance to the next axis-aligned plane. Other implementations may traverse an octree that employs only integer operations throughout the traversal. In another example, an efficient octree representation of the data may be utilized in conjunction with an efficient sparse voxel octree, as well as other examples.

[0105] In a traditional raycaster, color may be accumulated during the traversal of occupied voxels until an opacity threshold is reached (e.g., opacity accumulation), allowing early termination of the ray traversal. In this way, transparent materials can be taken into account to render realistic volumetric data. In a front-back implementation using VOLA, these types of features can be discarded because the required information is about occupancy in 3D space, and because VOLA may naturally omit the encoding of color information (although VOLA can be extended with an additional bit array of voxel color, at the expense of increased memory usage). In such an implementation, the algorithm flow can stop immediately when a voxel is encountered, without detailing any color information, opacity, texture, etc.

[0106] In one implementation, the ray traversal algorithm used during ray casting can be implemented to specifically exploit the structure of the VOLA-based representation. Fig.15 , two different 3D volume representations can be used and compared for a specific example raycaster: a dense layered grid (1505) and the VOLA format (1510), or a sparse layered grid. Both data structures in this example have 4 3 The shape of the trees differs in that they are organized as multi-level bit arrays. Data is encoded in one bit per voxel format and is stored according to the geographical location in the grid. Since empty and occupied space are represented, the size of a dense grid is predictable given the resolution of the last level. However, since empty sub-tiles are not stored, the actual occupancy in the VOLA format may depend on the geometry. This results in a very small memory footprint, but at the same time makes the positioning of individual voxels more complex and computationally more expensive. For example, to locate a sub-tile bit at a certain level, it may be necessary to know the number of bits that precede the corresponding bit that was set at a higher level. Such operations significantly slow down the calculation and must be handled carefully during the ray casting algorithm to avoid incurring unacceptable overhead. As Fig.15 As shown in , as represented at 1505 and 1510, in theory, it is simpler for a dense layered grid (1505) to position voxels depending on the level than for a sparse layered grid (1510).

[0107] In one example, a computing system may implement a ray traversal algorithm based on a hierarchical sparse grid structure (e.g., VOLA) in conjunction with the performance of performing a ray casting operation (e.g., in a SLAM application). In some implementations, a more efficient approximate hierarchical ray traversal algorithm may be implemented using a VOLA structure to represent a 3D space in a ray casting application. In some implementations, a ray casting traversal algorithm used in a ray casting application may implement a hierarchical version of a traditional traversal algorithm (such as John Amanatides and Andrew Wo's fast voxel traversal algorithm for ray tracing). For example, in an example implementation of a traversal algorithm, a ray intersection test operation (e.g., an axis-aligned bounding box (AABB) test) is performed on the boundary volume when the boundary volume is aligned with the axis of the reference system. If some externally positioned portions of the volume are empty, the L0 VOLA element may be checked to easily trim the volume to be ray casted, thereby reducing it to a sub-AABB to be intersected (e.g., as shown in 1515). Such applications may include aerial scanning, where the volume in the z dimension is almost blank. Scan a virtual image using perspective projection. Since each ray is independent, multiple threads may be used to process different rays simultaneously. In some implementations, a thread pool is initialized and then scheduled based on available runtime resources. In fact, in some implementations, it may be more convenient to assign rectangular sub-slices of the virtual image to threads rather than individual pixels or entire rows, due to cache data locality and context switching overhead.

[0108] In one example, an algorithm may be implemented using a computing system (and in some cases entirely in hardware), where p is the ray-AABB intersection point (or alternatively, a point within the volume), and d is the ray direction; the ray equation is expressed as r(t) = p + td. Solving the equation to find t for the axis-aligned plane orthogonal to the x component yields t x =(1 / d x )x+(-p x / d x ), where d x is the distance from the point to the plane. Now (t DeltaX , t DeltaY , t DeltaZ ) is defined as the t value at which the ray is moved for a voxel width in a particular direction, and uses the t value closest to the intersection of the axis-aligned plane x , t y , t z Value initialization (t maxX , t MaxY , t MaxZ ) variables, a simple algorithm can be defined to handle the traversal of a ray through a uniform grid. Pseudocode for this algorithm is shown below and is Fig.16AAn example visual traversal is depicted in:

[0109]

[0110]

[0111] As shown in FIG. 16 , a conventional traversal algorithm is shown whereby a ray 1605 (e.g., one of many rays to be cast through a volume (e.g., 1610)) is cast through a volume 1610. Fig.16A In FIG. 1 , black squares (e.g., 1615, 1620) represent occupied voxels in volume 1610 that are not hit by ray 1605, while gray squares (e.g., 1625) represent the first occupied voxel hit by ray 1605. In a conventional traversal, as Fig.16A As shown in , each voxel traversed by ray 1605 is checked by system logic (e.g., each check is represented by a corresponding point (e.g., 1630)) to determine whether the voxel is occupied. The traversal can be extended to a hierarchical structure (such as VOLA), which can be effectively used to jump empty space. In one example, the hierarchical dense grid and VOLA formats share the same data organization, so the algorithm can be applied to both indiscriminately. As for the basic algorithm, 3D coordinates can be used to move through space while changing the resolution level depending on the voxels encountered: the main difference between the two is how the contents of the voxel are tested. For example, going to Fig. 16B , when an occupied voxel at a certain level (e.g., 1635) is encountered, the next level in the hierarchy is accessed (e.g., for t Max and t Delta Variables perform new initialization operations), and subvoxels within an occupied higher level voxel traversed by ray 1605 can be individually checked (e.g., represented by points (e.g., 1640)). Boundaries can be determined using a modular algorithm based on regular volume subdivision. When a higher level voxel (e.g., 1645) is determined to be empty, checking of the traversed constituent subvoxels can be skipped. For example, when passing from one level to another, spatial jumps are given by the form r(t)=p+t min The 3D linear step size of d is determined by minis the t value from a higher level. Because of the traversal approach, tracking positions at different levels can be achieved more efficiently than with traditional traversal algorithms. For example, at each hierarchical level, traversal is performed the same as traditional traversal, but when passing from one level to the next, the first position is found in linear steps using the current t parameter, moving from one occupied voxel to the next (skipping any intermediate voxels in the ray path that are not occupied (e.g., 1645)). In either case, the traversal algorithm continues until the system first encounters an occupied voxel at the highest resolution level (e.g., 1625).

[0112] In one example, the hierarchical implementation of the traversal algorithm can be modified to speed up execution in situations where accuracy is not a primary concern and accuracy can be sacrificed in favor of speed. Fig. 16C As shown in , in some implementations, the ray traversal algorithm first determines which coarser-grained voxel the ray occupies and traverses first. The ray traversal algorithm can then focus only on evaluating the high-resolution (lower-level) voxel representations in the occupied lower-resolution (higher-level) voxels. However, as opposed to checking each sub-voxel in an occupied voxel, an approximate approach can be adopted in which the sub-voxels are checked at intervals (e.g., x) that cover more than one voxel (e.g., checking whether every second or third voxel traversed by the ray occupies the volume, rather than checking every traversed voxel). For example, a fixed-point implementation can be used for the data. After identifying the principal component of the ray direction, a ray of a certain level is advanced so that it covers a distance equal to the sub-voxel size in the principal component of the determined direction (e.g., the x, y, or z component). In doing so, the occupied / unoccupied decision for the new voxel to be traversed is fast. However, the trade-off is that the approximate layered traversal algorithm does not strictly visit all voxels traversed by a ray, which may result in hit misses for occupied voxels, which may produce artifacts under certain conditions (e.g., when the density of voxels is low or when rendering empty objects with thin surfaces), as well as other side effects. Fig. 16C As shown in , the interval 1655 for scanning voxels within occupied higher-level voxels can be defined as voxels in the x-direction. Therefore, when a raycast is performed on an occupied higher-level voxel (e.g., 1660, 1665, 1670), only a portion of the traversed voxels (e.g., 1675, 1680, 1685) are checked. The example pseudo code further illustrates such an embodiment:

[0113]

[0114]

[0115] Level_Increase_Value[Maximum Load+1]←DistanceInitialization(r d, maximum load);

[0116]

[0117]

[0118] and Fig. 16C The example of shows an approximate hierarchical traversal algorithm that checks voxels along defined subvoxel intervals, other intervals can be selected and utilized in the algorithm based on the application and the resolution sensitivity of the application (e.g., rendering or SLAM application). Turning to FIG. 17, for example, a standard hierarchical traversal algorithm (e.g., as combined with Fig. 16B , where each subvoxel in an occupied voxel traversed by a ray is checked) is compared to an example result of an approximate hierarchical traversal algorithm (e.g., as combined with Fig. 16C 1705, 1710, where only a portion of the traversed sub-voxels in the occupied higher level voxels are examined, shows a comparative rendering of an example Asian dragon model. In this example, images 1705, 1710 are simply rendered as depth maps from the original ray casting data of the 3D Asian dragon model. Image 1705 represents the result of the standard hierarchical traversal algorithm, while image 1710 represents the result of the approximate hierarchical traversal algorithm. Although image 1710 generated from the approximate algorithm omits some voxels on the surface (manifested as graphical artifacts), the similarity between the two images 1705, 1710 indicates that the computational efficiency achieved by the approximate hierarchical traversal algorithm may be a very worthwhile trade-off in some applications.

[0119] The illustration of FIG. 17 is just one example, but the results achieved by approximate hierarchical traversal can be compared with traditional algorithms that place a premium on accuracy in rendering images more generally. For such comparisons, for each camera pose, two hierarchical traversal algorithms can be used to produce grayscale images. The intensity of each pixel in the image is an integer, which is related to the distance between the virtual camera and the hit voxel. Therefore, the two images can be compared based on the difference between the intensity values, while considering the use of an accurate, unified algorithm as the basis truth. Since the background (light leakage) does not provide valuable information, only the rays that hit the voxel in the accurate ray caster can be considered for comparison (which can only lead to missed detections, not false positives). The percentage of different results can be calculated by averaging hundreds of camera poses in space and frames of different scenes. In this example, VOLA models can be used for testing, such as models generated from aerial scans of the River Liffey (located in Dublin, Ireland) and three models from the Stanford 3D Scan Repository: Asian Dragon, Lucy Angel, and Thai Statue. Since the use of floating point arithmetic and conversions may result in very small pixel intensity differences (as small as one unit), thus impractically increasing the difference count, a threshold can be implemented where only cases where the pixel intensity difference is greater than 1 are considered. Example results are shown in Table 1. Note that the percentage of different pixels shown in the table is expressed as the fraction of pixels that were actually hit by the exact raycaster only. For those pixels that differ in intensity between the two images, the high value of the mean confirms that the considered case is a correct miss. The percentage difference depends on the density of voxels in the used model: in general, the higher the pixel density, the lower the approximation.

[0120] Model Different pixels (%) Average strength difference Asian Dragon 5.51 77.62 Lucy Angel 3.61 81.08 Thai statuette 3.97 63.82 Dublin Lifestyle 5.84 32.24

[0121] Table 1: The percentage of pixels that differ between the exact rendered image and the approximately rendered image, as the fraction of hit pixels in the exact image. The mean intensity difference for misses is also reported.

[0122] In some implementations, the raycaster can be implemented in hardware such as a vision processing unit (VPU). Such hardware devices can be specifically implemented to provide high-performance machine vision and artificial intelligence (AI) computing in a severely power-constrained environment. Fig.18It is a simplified block diagram of a multi-core always-on system-on-chip (SoC) 1805 that can be implemented as a multi-core always-on system-on-chip (SoC) that provides highly sustainable performance efficiency for a range of machine learning and vision applications with low latency and low power requirements. The architecture of the example processing device 1805 can use a combination of a low-power very long instruction word (VLWI) processor (e.g., 1810) supporting vector and SIMD operations with hardware acceleration for image processing and computer vision cores (e.g., provided by one or more accelerator devices (e.g., 1815)), supported by a very high bandwidth memory subsystem (e.g., 1820) and one or more RISC processors 1825, 1830 (e.g., Leon 32-bit RISC processors according to the SPARC architecture). The system is divided into three main architectural units: a media subsystem (MSS), a CPU subsystem (CSS), and a microprocessor array (UPA). In this example, the UPA contains a VLWI processor (e.g., twelve custom vector processors (e.g., 1840a-l)), which is implemented as a SHAVE (stream hybrid architecture vector engine) processor in some cases.

[0123] To easily access the SoC from a PC, a machine learning accelerator (e.g., Movidius TM Neural Compute Stick TM (NCS)) to program the vector processor over a USB3 connection. A software application (e.g., running on a general-purpose CPU or dedicated hardware or firmware of a computing system) manages the creation of a hierarchical grid structure starting from a set of .vol (VOLA format) files and then sends the contents to the vector processor unit over USB (or other interface (e.g., 1845)). In addition, it handles the visualization of rendered frames from the accelerator (e.g., using the SDL2 (Simple Direct Media Layer) library). In some implementations, the real ray casting algorithm can be performed by a set of dedicated vector processors (e.g., 1840a-l), each of which processes a small rectangular patch area of ​​the virtual image. In order to fairly utilize the computational load, the vector processors (e.g., 1840a-l) can be scheduled asynchronously and dynamically to reduce the total idle time. Data about the virtual camera position and orientation is passed to the threads via a shared structure that is dynamically updated by the system, as well as other example system implementations and programs.

[0124] In one example test, exact (standard) and approximate versions of the unified mesh and hierarchical mesh algorithms were tested and compared in terms of performance. To take advantage of vector operations available in specific architectures used to execute ray casting applications (e.g., architectures using the SHAVE instruction set architecture (ISA)), the hierarchical implementations were refactored using built-in functions (intrinsics) available in the system compiler and tested separately. In one example, an aerial scan of the O'Connell Bridge Building (OCBH) in Dublin and two models from the Stanford 3D Scan Repository were used. The results obtained from such example tests are given in Fig.19 It is shown in Fig.19 The average, minimum, and maximum frame rate measurements are shown in . The measurements shown were taken under the same conditions for all tests, performing circular motion of the virtual camera while looking at a fixed point in the center of the volume. Fig.19 As shown in the graph 1900 of , the approximate implementation consistently outperforms the exact implementation, with frame rates being improved by 30% to 75%. By using intrinsic functions, the maximum speedup factors for the exact and approximate SIMD raycaster versions reach 12.78 and 17.74, respectively.

[0125] Using sparse hierarchical grid structures (e.g., VOLA) in ray casting applications can achieve additional benefits, for example, in terms of performance and memory footprint of the data structures. For example, from a topological point of view, dense and sparse hierarchical grids can effectively share the same organization in space, but in the case of sparse grids, only occupied data is actually stored. This results in a very small memory footprint (table), but at the same time increases the time required to locate a specific voxel in the hierarchical grid, because all previous levels must be checked in order to identify the set bits in the sparse grid table and count them. For example, Table 2 reports a comparison between the memory footprint of different formats of the same model. The dense grid and VOLA formats shown in the table are obtained directly from a sample polygon file format (PLY) object. As can be seen, the VOLA structure ensures a substantial size reduction, with the maximum size here being reduced by about 1 / 11140 relative to the PLY format. The dense grid occupancy can be considered to be completely deterministic in the sense that it depends only on the depth of the hierarchical grid; in contrast, the VOLA structure occupancy depends on the voxels that are actually occupied in space. As Fig. 20AThe graph shown in 2000a shows the average frame rate for three different VOLA models and the same algorithm and optimizations for dense grids. It may be noticed that there is no longer any real gain from using the approximate implementation; also, the benefit of introducing the optimization of the SHAVE intrinsics is very low. This can be explained by the fact that the adopted techniques are aimed at optimizing the mesh traversal performance of the algorithm, but the bottleneck now lies mainly in the high time required to test the occupancy of voxels in the VOLA structure. As a result, the proportion of time spent in the actual traversal is much lower here than for dense grids. The performance of the VOLA implementation is about 1 / 11 to 1 / 30 lower than that for dense grids. However, correlating the obtained performance with the memory occupation (average - FPS / MB), it can be concluded that the VOLA implementation is much more efficient and it can therefore be shown (for example, in Fig. 20B Example Table 2000b) constitutes a valid choice in highly memory-constrained environments.

[0126] Model PLY Dense Grid VOLA Asian Dragon 133MB 17.04MB 46.6KB Lucy Angel 508MB 17.04MB 45.6KB Thai statuette 220MB 17.04MB 58.4KB

[0127] Table 2: Comparison of memory usage of different data formats for the same 3D model.

[0128] In some implementations, a sparse hierarchical data structure for implementing ray casting can be utilized to implement an embedded implementation of a ray casting algorithm. In addition, an approximate ray traversal algorithm can be employed to achieve processing performance benefits. In addition, performance and memory usage can be traded between a dense (VOLA) approach and a sparse (VOLA) approach, showing consistent advantages of the VOLA structure in FPS / MB efficiency at the expense of reduced performance. Therefore, systems that utilize sparse hierarchical grid structures to represent 3D volumes in ray casting applications may be considered preferred, particularly in environments where memory is highly constrained. In fact, the improved system implementations discussed herein exhibit very high power efficiency, up to two orders of magnitude higher than traditional ray casting implementations, as well as other example advantages.

[0129] Computer vision systems can utilize ray casting and rendering results in conjunction with autonomously determining the position and orientation of an autonomous device (such as a robot, autonomous drone, or vehicle), among other examples. In fact, accurate position and orientation estimates are critical to the navigation of autonomous robots. Although this is a well-studied problem, existing solutions rely on statistical filters that typically require good parameter initialization or calibration and are computationally expensive. As discussed herein, improved systems can implement improved end-to-end machine learning methods, including incorporating multiple data sources (e.g., monocular RGB images and inertial data) to independently overcome the weaknesses of each source. In some implementations, improved systems utilizing such functionality can achieve odometry results that are computationally cheaper and have an accuracy that is at or only slightly below the state-of-the-art, allowing such systems to implement valuable potential solutions for resource-constrained devices, as well as other example uses and advantages.

[0130] Motion estimation is one of the main pillars of mobile robotics. It provides the robot with the ability to know its position and orientation in an unknown environment, and can be combined with mapping methods to develop simultaneous localization and mapping (SLAM). In fact, within such implementations, the SLAM pipeline utilized can adopt the improved ray casting technique discussed in this article. The robot can use different data sources to perform such motion estimation depending on the type of sensor: in proprioception, it provides robot internal information such as inertial measurement unit (IMU); or in external perception, it provides information about the robot's surrounding environment (such as cameras or LiDAR sensors). Due to the autonomous nature of the robot, it should be able to perform such motion estimation on-board in real time, which is a particular challenge when the device resources are limited. Therefore, it is desirable to find a solution that can run in embedded devices under such constraints.

[0131] Deep learning can be used to solve the pose estimation problem in odometry applications. For example, a camera captures the surroundings of a robot or other autonomous device and can be used to track the robot's motion through visual odometry (VO). Classical VO methods estimate motion based on geometric constraints and can be divided into two groups: sparse feature-based methods and direct methods. On the one hand, sparse feature-based methods extract and match feature points to estimate motion between frames. In addition, some VO methods add and maintain feature maps in order to correct drift due to the presence of outliers and noisy images. On the other hand, direct and semi-direct methods use all image pixels to estimate pose by minimizing the photometric error between consecutive images.

[0132] However, classical VO methods typically require external information (such as camera height or templates) to sense scale and recover distances in real units. In addition, traditional VO systems have been shown to be unreliable in the presence of rapid motion or when illumination changes suddenly. To address this lack of reliability, camera information can be combined with inertial sensors, which can provide acceleration and angular rate information. These sensors typically provide data at a much higher frequency than the camera (e.g., about 10 times faster). Therefore, inertial information can be used to overcome the weaknesses of VO systems in the case of rapid camera motion, as well as other example improvements and issues.

[0133] Visual-inertial odometry (VIO) systems utilize visual and inertial information to provide position and orientation estimates. In some implementations, visual-inertial data fusion is accomplished using a probabilistic filter approach such as an extended Kalman filter (EKF) or an unscented Kalman filter (UKF). Visual-inertial data fusion is performed by an EKF-based system, which they use to compare different fusion models that use only gyroscope data or that use gyroscope and accelerometer data. Other variants of the EKF can be implemented, such as a multi-state constrained Kalman filter (MSCKF), such as a system in which several past camera poses are used to detect static features and add constraints to the state vector, and other examples.

[0134] In some implementations, systems equipped with deep learning capabilities can be used to overcome weaknesses of classic VO methods, such as lack of robustness to blurry or noisy images, or when changes in lighting or occlusion occur. For example, convolutional neural networks (CNNs) can perform well even for blurry and noisy images, providing a robust method for extracting image features. CNNs can also be used to compute the optical flow between two consecutive images. Optical flow represents the change in the position of an object on the camera view, so it is related to the motion that the camera experiences between two consecutive frames. In one example, the image features extracted by the optical flow network can be used in a deep learning system together with two long short-term memory (LSTM) layers to implement a monocular VO system in an end-to-end deep learning manner, which can outperform classic monocular VO methods (e.g., LIBVISO2-based VO), as well as other examples.

[0135] VIO methods based on probabilistic filters for sensor fusion may require a painstaking and complex calibration process to bring the camera and inertial measurement unit (IMU) measurements into the same reference coordinate system. For example, the calibration process can be performed in real time while the tracking system is running, which increases the complexity of the filtering process. In addition, some parameters of the IMU are difficult to model, such as noise scaling within the measurements found in most commercial IMUs. Deep learning techniques can be further used to solve problems in the sensor fusion process. For example, LSTM can be used to track past IMU raw measurements (accelerometers and gyroscopes) to estimate the robot's posture, which can then be fused with the VO system. LSTM can be used in VINet to extract encoded features from the raw measurements of the IMU. These encoded features can be combined with features extracted from CNN in a feature vector that is tracked over time by a second LSTM that can provide a posture estimate of the robot. In some implementations, the VINet approach may be superior to traditional methods, such as traditional optimization-based sensor fusion methods, as well as other example benefits.

[0136] In one example, the improved system can implement an end-to-end trainable neural network architecture that combines vision and inertial deep learning sub-models. For example, in a vision-based CNN sub-model, an RGB image can be provided as input. In a second inertial-based deep learning sub-model, IMU raw measurements are provided as input. In one example, an end-to-end neural network architecture can utilize a combination of neural network sub-models, and the combined network (and its sub-components) can be trained in an end-to-end manner, thereby eliminating any need for calibration or preprocessing.

[0137] In some implementations, a neural network of an end-to-end machine learning odometry system can be trained to produce pose estimates at each frame relative to the previous frame. Each pose estimate represents a transformation, typically expressed as an element of a special Euclidean set of transformations SE(3). All transformations represented in SE(3) (Equation 1) can be composed of a rotation matrix and a translation vector, which is the rotation matrix part of the special orthogonal set SO(3).

[0138] Equation 1:

[0139]

[0140] Finding the transformation in SE(3) is not straightforward for the network because R must be orthogonal to the constraints. Therefore, to simplify the learning process, the estimated transformation is expressed in the Lie algebra of SE(3) se(3) (Equation 2).

[0141] Equation 2:

[0142]

[0143] The pose estimate in se(3) can be a 6-dimensional vector and may not be orthogonal. Once estimated, the pose in se(3) can be converted to the transformation of SE(3) by performing an exponential mapping se(3)->SE(3) (Equation 10):

[0144] Equation 3:

[0145]

[0146] Equation 4:

[0147]

[0148] Equation 5:

[0149]

[0150] Equation 6:

[0151]

[0152] Equation 7:

[0153]

[0154] Equation 8:

[0155]

[0156] Equation 9:

[0157]

[0158] Equation 10:

[0159] se(3)→SE(3): exp(ω|t)=(R|Vt)

[0160] where the matrices R and V can be calculated using equations 8 and 9, respectively. A, B, C, and θ can be derived using equations 4, 5, 6, and 3. The ωx matrix consists of the ω values ​​(Equation 7).

[0161] In one example, a vision-based neural network model 2105 may be utilized, such as in Fig.21AIt takes as input 2110 two consecutive RGB images stacked together to form an input tensor of size 512x384 with 6 channels. This image size can contain enough features while obtaining a shallow CNN. In one example, a layer 2115 of a computer vision neural network (such as FlowNetS) can be used to extract features of the image, which can be advantageously used for motion estimation, among other examples. Such a network (e.g., 2115) employed as part of a neural network model 2105 can be trained on a synthetic dataset to learn how to estimate the optical flow between frames, which represents the movement of the robot over time, among other examples.

[0162] exist Fig.21A In the example of , FlowNetS layer 2115 can occupy the ninth convolutional layer of the network 2105, followed by an additional convolutional layer 2120 to reduce the output size of the CNN to a 2x3x256 tensor. After the CNN layer 2125, the extracted features can be combined using a series of fully connected layers 2130 to generate an output 6-dimensional vector pose 2135, which represents the transformation of the current frame (t) relative to the previous frame (t-1), expressed in the Lie algebra of SE(3).

[0163] exist Fig. 21B In another example network 2140 shown in , only inertial data is used as input 2142 to the network 2140. In one example, the input 2145 can be IMU data, embodied as a subsequence of ten 6-dimensional vectors with xyz raw data components from the accelerometer and gyroscope components of the device. In this example, the (ten) measured input subsequences can be sorted in time order (the last one is the most recent), thereby encoding the motion experienced by the sensor over time.

[0164] continue Fig. 21BIn an example, a long short-term memory (LSTM) artificial recurrent neural network portion 2150 can be used as a regression layer in the model 2140 to track measurements on a subsequence and extract motion information because it is able to store short-term and long-term dependencies resulting from past inputs in its hidden states. Each input is combined with the hidden state as it passes through the LSTM 2150 to find the temporal correspondence between the current measurement and the past measurements. In one example, the LSTM 2150 can have 1 layer, 1024 units, and can be followed by four fully connected layers 2155 that output a 6-dimensional vector representing the transformation experienced (e.g., by a robot) from the last element to the first element of the subsequence. These architectures and parameters can be selected due to their performance in Monte Carlo analysis, which can be used to explore and optimize the selection of different combinations of layers, as well as other example implementations.

[0165] As described above, the improved machine learning system can utilize the neural network model 2160 to perform odometry calculations, and the improved neural network model combines Figure 21A-21B The example network portions 2105, 2140 shown in FIG. 2160 are used to utilize both visual sensors and inertial sensors that may be present on a robot or other autonomous device. Thus, the input to the model 2160 may include visual data and inertial data, such as a pair of consecutive RGB images and a subsequence of ten inertial measurements. In this example, the virtual neural network portion 2105 is coupled to the Fig.21A The virtual neural network portion 2160 is identical to the one presented in the example of , up to its third fully connected layer. Similarly, the inertial odometry (IO) network portion 2140 is used up to its second fully connected layer. In this way, the neural network model 2160 maintains both the VO and IO networks until the last layer that provides useful features. The visual feature vector and the inertial feature vector are then merged into a 128-dimensional vector and passed through three fully connected layers 2165 to output pose estimates 2170. As before, each pose estimate represents the transformation that the robot has undergone at the current frame relative to the previous frame.

[0166] In one example implementation, the data used to train the example neural network model 2160 can be part of a compiled dataset (e.g., the raw data portion of the KITTI Vision Benchmark Suite). In one example, the odometry dataset consists of 22 sequences, of which the first 11 sequences have ground truth transformations. In this example, sequences 11-22 are intended for use as evaluation and therefore do not provide ground truth. Sequences 00, 02, 08, and 09, which contain the most frames, are used for training, while sequences 05, 07, and 10 are used for evaluation. In some implementations, the training data can be enhanced by randomly applying Gaussian noise, Gaussian blur, and intensity variations to the images, such as by applying Gaussian noise (e.g., mean = 0, standard deviation = [0, 32]) and pixel intensity variations (e.g., [-25%, 25%]) to 2 / 3 of the data, and applying Gaussian blur and kernels 3, 5, and 7 to the remaining one-third of the data. After augmenting the data, the training dataset (e.g., 22,912 image frames in total) as well as the ground truth (e.g., at 10 Hz) can be sampled. The IMU data may arrive at a higher frequency (e.g., 100 Hz), meaning that there may be multiple IMU measurements per image frame. However, there may also be frames where some IMU data is missing. In this case, the first IMU measurement of the frame is used to fill in the missing measurements to fill the subsequence.

[0167] In one example, the loss function used (Equation 11) represents the Euclidean distance between each estimated relative pose and its respective ground truth, denoted by se(3):

[0168] Equation 11

[0169]

[0170] ω、 t and denote the estimated and ground truth rotation and translation in se(3), respectively. The parameter β helps balance the different magnitudes between ω and t and can be fixed to 0.1 in all training. Nesterov accelerated gradient (NAG) can be used as an optimizer (Equations 12 and 13) to speed up convergence relative to standard gradient descent, thereby measuring the gradient m of the loss function not in the local position but slightly forward in the direction of momentum.

[0171] Equation 12

[0172]

[0173] Equation 13

[0174] w se(3) =w se(3)-m

[0175] Where β is used as a friction factor to prevent momentum from becoming too large, and λ is the learning rate. The weights w are then updated according to m se(3) For training, a friction factor β = 0.9 was used. In some cases, implementing an exponential schedule (Eq. 14) leads to faster convergence and can be easier to implement compared to other methods such as performance scheduling.

[0176] Equation 14

[0177] In(t) = In 0 2 -t / r

[0178] In one example, the initial learning rate (λ 0 ) is 10-5, and the step size (r) is 50. Using these parameters, the learning rate is divided by 2 every 50 iterations. In an example comparison, all networks have been implemented on TensorFlow and trained using NVIDIA GeForce GTX Titan X GPUs. To reduce training time, the FlowNetS weights can be frozen during training.

[0179] The networks discussed in this paper can be evaluated and their performance compared. In this example, VO and VIO are compared with existing methods using the same type of data. For example, an improved VO network (e.g., as described in this paper combined with Fig.21A ) and compare it with traditional solutions. In one example, the comparison can be performed using the metric proposed in the KITTI odometry development kit and the network executed on sequences 05, 07, and 10, obtaining the absolute pose of each frame relative to the first frame. The root mean square error (RMSE) can then be calculated for different trajectory lengths (100m, 200m, 300m, ..., 800m) over the sequence. These results are shown in Table 3, along with traditional VISO2 M and DeepVO networks. As shown in the example of Table 3, an improved VO network such as discussed in this paper can outperform traditional networks (e.g., VISO2M) in terms of translation and rotation errors for sequences 05 and 10, and the comparison is performed in the translation of sequence 07.

[0180]

[0181] Table 3. All errors represent the average RMSE for all possible sequence lengths. rel is a translation error and r rel It's a rotation error.

[0182] Steering Fig. 22 , showing that in this article (for example, in combination with Fig. 21C ) and compare them with the results of the conventional solution. For example, Fig. 22 The estimated trajectory achieved using the improved VIO network for the ground truth is shown (the corresponding example endpoint translation and rotation errors are shown in Table 4).

[0183] Improved VO technology Conventional VO technique Translation (m) / (%) 37.20 / 6.68 6.44 / 1.15 Rotation (degrees) 15.64 1.05

[0184] Table 4. Endpoint position and orientation errors of the improved VIO network model compared to the traditional solution. Translation errors are expressed as the absolute error of the end point and the percentage of this error relative to the total distance covered in frames 0-800.

[0185] Although a pure inertial odometry (IO) network implementation may show large drift errors over time, when it is combined with a visual odometry network (to form a visual inertial odometry network, such as described in the above example), the drift may be greatly reduced. In addition, the visual inertial odometry (VIO) network may show better performance when the robot is turning, outperforming the visual odometry network. This illustrates how the IMU can compensate for large displacements of objects in the camera. Improved visual inertial odometry networks may outperform traditional solutions. Such networks can be used in a variety of applications from robots to autonomous drones, among other examples.

[0186] In addition to the above example features, the improved computer vision system can additionally use a sparse hierarchical voxel grid representation (e.g., VOLA) to estimate the relative pose between a pair of point clouds using a convolutional neural network (CNN). As discussed in this article, the voxel grid is a more efficient memory solution compared to point clouds, so it can be used on embedded systems (e.g., autonomous devices) that may have limited memory and computing resources. A new loss function using transformation error can be used with 2D convolutions on 3D voxel grids. In fact, such solutions can provide such efficiency while maintaining accuracy comparable to traditional, state-of-the-art implementations of iterative closest points (ICP). In fact, the improved, voxel grid-based solution can achieve improvements over ICP because the use of voxel grids limits the impact of high-density regions on alignment errors, as well as other example uses and advantages.

[0187] In recent years, 3D scanning technology has become more accessible to everyone due to the availability of a wide variety of sensors capable of scanning 3D data in higher quantities and quality. These 3D sensors produce a point cloud, which is a set of unorganized three-dimensional points in a unified coordinate system that describes spatial information used to represent 3D data. However, the scanning range of available 3D sensors such as LiDAR, sonar, and RGB-D is limited. To overcome this limited range, multiple scans need to be acquired and merged to create a 3D map. The fusion of multiple scans requires that each scan is represented in the same coordinate system, so their information is displayed in an organized manner. Therefore, in order to represent all scans in the same coordinate system, it is crucial to know the location where the scan was performed. This position can be expressed as a displacement between a reference frame and the current scan. The displacement, known as the camera pose, can be expressed as a transformation ΔT formed by a translation t and a rotation R, so ΔT = (R|t). The displacement of the sensor can be extracted by using the 3D information available in the scan or with the support of other types of sensors such as GPS, IMU, or wheel odometry on a ground vehicle. In some cases, odometry solutions such as those discussed in this article can be applied to help determine such placement. Once the position of the camera is known, the scans can be represented on the same coordinate system, creating a 3D map that contains information from all scans. The problem of finding the camera position is called relative pose estimation, which has historically been difficult to implement on resource-constrained devices.

[0188] In one example, a computing system may be configured to solve the relative pose estimation problem by implementing and using a deep neural network (DNN). Due to noise and sensor limitations, the size of the point cloud may be different between two scans. However, from the design stage, the voxel grid representation of the point cloud has been fixed. Since the computational complexity of 3D convolution is large, an improved solution may alternatively use an alternative 2D representation of a 3D voxel grid, thereby allowing the use of a computationally efficient 2D convolutional neural network (CNN). In some implementations, a 2D convolutional neural network may be used to find the transformation between voxel grid pairs. The loss function may be determined based on the error obtained by transforming the point cloud using the ground truth and the network prediction. The performance of such an implementation may be to compare its results with the iterative closest point (ICP) method of the latest technology using an RGB-D SLAM dataset. In addition, the relative pose estimation between the scans of the sensor that produces the two point clouds may be used to find the transformation that produces the best alignment between the generated point clouds. Therefore, such relative pose estimation solutions discussed herein may be alternatively referred to as "point cloud alignment" solutions.

[0189] Steering Figure 23A-23BCurrently, the most frequent 3D point cloud alignment methods are based on variants of the Iterative Closest Point (ICP) 2315, which iteratively estimates the transformation that minimizes the distance between corresponding points in two different point clouds 2305, 2310, e.g. Fig.23A As shown in . This iterative process can be computationally expensive and, in addition, it is sensitive to the initial alignment and overlap between the point clouds 2305, 2310 and their sizes. Depending on the method of calculating and minimizing the alignment difference between the two point clouds, a variety of ICP-based methods can be found, where this distance can be done between planes or curves instead of between points. In order to make 3D alignment more efficient, the ICP solution can be extended to extract key points from the descriptor of the 3D point cloud using features. Factors such as rotation and translation invariance or noise robustness are important for these key points, and works using histogram-based descriptors, 3DSIFT, SHOT and NARF are examples.

[0190] Where RGB-D images are available, the RGB images can be used to improve the estimation of camera displacement. This can be used to implement methods for monocular or binocular or multi-camera pose estimation. In some implementations of the improved system, the RGB images are used as input to a convolutional neural network (CNN) that extracts complex features from the image data using a set of convolutional layers and then regresses the pose through a set of fully connected layers. In addition, in some implementations, a long short-term memory (LSTM) layer can be added to this topology to form a recurrent neural network (RNN) to handle temporal dependencies between images taken in long trajectories.

[0191] Other implementations can utilize different types of inputs as well as RGB images. For example, optical flow between pairs of images can be used to estimate relative poses. In addition, considering this situation from a 2.5D perspective, information in the depth dimension can also be used in the case where the neural network uses RGB-D images as input, as well as other examples. Some implementations can utilize CNNs on 3D data to perform deep learning on this 3D data. For example, the point cloud alignment problem can be evaluated by first extracting feature points from the point cloud and then feeding the obtained features to a deep learning method. For example, a "superpoint" descriptor can be used as a basic unit for matching point clouds, and the descriptor can be used as input to a deep neural network autoencoder that retains geometric shape information for finding transformations between point clouds. In other implementations, interest points can be used, which represent clusters of point clouds where there are a large number of points. In addition, different problems such as classification or segmentation using this strategy can be used, where the nearest neighbor map and KD tree are used as descriptors of the point cloud, respectively.

[0192] In some implementations, the point cloud can be represented as a subset of points represented as a list of (x; y; z) coordinates, which are then fed into fully connected layers that look for interactions between points in order to classify three-dimensional objects. Fig. 23B As represented by the simplified block diagram in , the point clouds 2305, 2310 can be converted into corresponding voxel grids (VG) 2320, 2325 by a voxelization process 2330, which converts each point cloud 2305, 2310 on a 3D grid with a fixed size for each cell. This fixes the amount of memory required and generates neighbor relationships between voxels. The voxel grid-based inputs 2320, 2325 can then be used to provide input to the convolutional neural network 2350 to generate relative pose results 2355. Strategies for using this type of data with deep learning can be applied to object classification and orientation problems, as well as other applications. Traditional solutions and systems do not use voxel grids (VG) and deep learning to solve the relative pose estimation problem. The VG representation using 3D space has the properties of having a fixed size and a small memory footprint at the design stage. In addition, by using low-power hardware accelerators for CNN execution and other machine learning operations (e.g., Movidius TM Neural Compute Stick TM ), this approach can be applied to resource-constrained devices, among other example benefits.

[0193] Go to Fig.24 The simplified block diagram shown in , represents an example neural network topology for determining relative pose estimates based on a voxel grid representation (e.g., 2320, 2325) of the 3D space being analyzed. In one example implementation, a 2D convolutional neural network can be implemented to take as input a pair of voxel grid structures 2320, 2325 to find the relative pose between scans by training a CNN 2350 to align their corresponding point clouds (the point clouds are generated from the scans and the voxel grid structures are based on them). In practice, each voxel grid structure 2320, 2325 can be generated or extracted from a corresponding one of the point clouds by a voxelization process performed by the system. A 3D to 2D projection 2405 can then be performed by the system to adapt the input for use in a 2D convolutional layer within the network 2350. Once a pair of 2D projections are introduced in the network, a relative pose ΔT 2355 between them is predicted.

[0194] As in Fig.24As shown in the example of , where the voxel grid inputs 2320, 2325 are sliced ​​in 2D projection and converted to 2D projection inputs (2410, 2415, respectively), the 2D projection inputs 2410, 2415 can be provided as inputs to the representation portion 2420 of the CNN 2350. The representation portion 2420 of the network 2350 can be implemented as a Siamese network having multiple groups (e.g., pairs) of corresponding convolutional layers (e.g., 2425a, b). Fig.24 In the notation used in , for each layer in the representation portion 2420, Conv(i; j; k) represents a convolutional layer with i outputs, a kernel size of j×j with a stride of k, and “same” zero padding. The outputs from the representation portion 2420 can be concatenated to form a feature vector 2430, which is introduced as an input to the regression portion 2435 of the network 2350. For example, the regression portion 2435 can be implemented to include a set of fully connected layers 2440, where the final fully connected layer generates each of the rotation (ΔR) component and the translation (Δt) component used to generate the relative pose result 2355, respectively. In Fig.24 In the notation used in the representation of , FC(i) represents a fully connected layer with i neurons.

[0195] As mentioned above, a point cloud is a type of data formed by a set of unorganized points, so it is first represented as a 3D grid in a process called voxelization. Voxelization converts a point cloud into a voxel grid, where the data is structured in voxels, as 2D images are structured in pixels accordingly. Essentially, it is a cubic 3D grid with a shape of [height, width, depth], where each cube called a voxel contains a small region of information from the point cloud. If the point falls in that region, the voxel occupancy is set to 1, otherwise it is 0. This means that the VG contains only 1 bit per voxel, i.e. the occupancy of the VG. Fig.25 An example of a point cloud representation 2505 of a 3D space and an example of a corresponding voxel grid representation 2510 generated by voxelization from the point cloud representation 2505 are shown in FIG. The VOLA structure can be viewed as a special form of a voxel grid data structure in which only data representing those voxels of the 3D space occupied by certain geometric shapes are stored. Thus, the VOLA voxel grid structure, such as described in the examples herein, reduces the amount of stored data useful for data transfer and memory-constrained environments.

[0196] As described above, in some implementations, before introducing a voxel grid-based input (e.g., 2320, 2325), a 3D to 2D projection step may be performed to make the 3D representation suitable for use with a convolutional layer with a 2D kernel. This reduces the computational cost of the network (e.g., 2350) and makes it simpler, allowing it to be easily deployed to embedded devices with memory constraints, as well as other example applications. 2D projections can be used on 3D data, such as in multi-view CNNs, where 2D image projections are extracted from different angles of a 3D model to be used with a well-established 2D CNN. However, this approach does not typically preserve the spatial structure of the 3D volume because it focuses on classification problems that may not require this information. Other implementations may use an anisotropic detection kernel to project a 3D object into 2D, similar to the "X-ray scanning" process of convolving a 3D VG into a 2D plane. This projection may help capture the 3D global structure of the 3D volume for the pose estimation problem we are working on. Since the voxel grid is represented as a (h) representing the (X; Y; Z) dimensions in the real world, the projection may be used to capture the 3D global structure of the 3D volume. VG ;w VG ;d VG ), so in an example solution, the voxel grid structure can be converted into a 2D projection by slicing the voxel grid along the Y dimension, generating a voxel grid of size (h VG , d VG ) VG The resulting 2D images are then stacked together to generate a VG The size of the channel is (h VG , d VG ) image. In this example, w VG is used as the slicing direction, since in related applications it may be desirable that the vertical direction of the object aid in the alignment of the point cloud. Other implementations based on considerations of the intended application may alternatively perform similar operations along other dimensions of the 3D voxel grid, among other example alternatives.

[0197] As above Fig.24As described in the example of , the computing system can implement a convolutional neural network (CNN) in hardware and / or software and use the CNN to estimate the relative pose between a pair of voxel grids. CNN 2350 takes the 2D projections of the corresponding voxel grids as input and outputs vectors representing the rotation (ΔR) and translation (Δt) of the relative pose. The representation portion 2420 of CNN 2350 can extract a feature vector 2430 from each pair of inputs. The feature vector 2430 can contain practical information that is later used to predict the relative pose. The feature vector 2430 is formed by a Siamese network with two identical convolutional layer branches, where weights are shared between each other. In some implementations, after each convolutional layer in the representation portion 2430, there is a rectified linear unit (ReLU) as an activation function. In some instances, a pooling layer is not used to ensure that the spatial information of the data is retained. For the regression part 2435 of the CNN 2350, the outputs of the two branches of the Siamese network are flattened and concatenated to form a vector that is introduced into the regression part, which is the part responsible for estimating the network predictions. The regression part is formed by a set of fully connected layers 2440, each followed by a ReLU activation function. The output of the network has two components (the rotation part and the translation part of the prediction), so in the last segment 2450 of the network 2350, there is a separation on two fully connected layers, one for each prediction part.

[0198] Continuing with the above example, the output 2355 of the CNN 2350 may consist of a vector representing the relative pose between pairs of voxel grids. This vector is typically defined as a transformation matrix ΔT of a special Euclidean group of transformations SE(3): a set of elements consisting of rotation elements and translation vectors of a special orthogonal group SO(3). When making predictions in the SE(3) group, there may be some problems since the SO(3) rotation components must be orthogonal matrices. In some implementations, the Lie algebra se(3) of the SE(3) group is used as a solution to avoid the orthogonality requirement since it represents instantaneous transformations without orthogonality constraints. To convert between se(3) and SE(3), an exponential mapping may be used:

[0199] Equation 15:

[0200] se(3)→SE(3):exp(ω│u)=(R│t)=ΔT

[0201] Because of the Lie array, the network's pose prediction (w|u) is represented by two vectors: w = (w1; w2; w3) for rotation and u = (u1; u2; u3) for translation, which are converted to ΔT. The computational steps to perform this conversion utilize Equations 3-9 above.

[0202] In some implementations, the network 2350 can be trained using a learning phase controlled by a loss function that guides the network to the correct solution. The loss function can measure the quality of the network's predictions, penalizing or rewarding backpropagation of learning throws. In one example, the proposed loss function applies two different transformations T gt and The difference in the position of the points in the point cloud is then quantified. The process involves taking a generic point cloud, which has points evenly distributed in 3D space, then applying two different transformations and comparing the difference in the point positions. These two transformations are compared with the network predictions. and the basic facts T gt As a result, the closer the distance between these point clouds, the more similar their predictions of the ground truth are. And the transformation T gt The corresponding point p i The position of the point corresponding to the transformation T The Euclidean distance between the positions of is used as the error metric. Once this distance is determined for each point, the average can be calculated to represent the loss. The loss function can be expressed as:

[0203] Equation 16:

[0204]

[0205] in L represents point i 2 norm, and n is the number of points in the point cloud.

[0206] There are many datasets that focus on the relative pose estimation problem and can be used to test the effectiveness of CNNs used in pose estimation solutions. Such datasets can be used to represent ground truth that provides information about the localization of the data, which is necessary to understand the relative pose. In one example, point clouds are needed to evaluate the network. This type of data can be found in two ways: from 3D datasets that directly provide point clouds produced by 3D laser sensors (e.g., LiDAR; or RGB-D datasets), where point clouds can be extracted from depth maps by a straightforward process, converting each pixel with depth information into a point.

[0207] For implementations that utilize deep learning, such as the one discussed in this article, a large amount of data may be required to train the network during the learning phase. To meet this requirement, different datasets can be found and differentiated according to their applications. For example, the KITTI dataset, the Oxford Robot Car dataset, the Malaga Urban dataset, and others may focus on autonomous driving applications where cars are used to move cameras / sensors. To give another example: the 7-scene dataset, the ICL-NUIM dataset, and others may focus on handheld devices in indoor environments where they hold and move the camera with their hands. In another example, the TUM dataset can be utilized, which, in addition to the handheld sequences, can also contain data obtained from a wheeled robot moving on the ground, as well as other example datasets that can be used during the training of the network.

[0208] As an example, when applying deep learning to indoor localization of ground robots, the TUM dataset can be exploited because it provides a sufficient amount of such data to train a deep neural network (DNN). The dataset can include multiple sequences organized in categories based on their application. In one example, only the "Robotic SLAM" sequence is used. To obtain a point cloud from a depth map, the tools provided in the dataset are used to evaluate the performance of our approach.

[0209] As described in the paper, the point cloud from the TUM dataset can be voxelized (e.g., only points within [4, 4, 8] meters from the camera in the x, y, and z directions are put into a VG of size [64, 64, 128]). To train a network from scratch, in one example, the first three sequences of the "Robotic SLAM" category can be used (e.g., fr2 / pioneer360, fr2 / pioneerslam, and fr2 / pioneerslam2), leaving the fourth group (e.g., fr2 / pioneerslam3) untrained in order to evaluate the network with unknown data.

[0210] Furthermore, in order to increase the amount of data used for training and to make the network robust to large displacements of the sensor, training pairs can be created by using different step sizes in the data sequence, in the form of (F i , F i+s ), where F is the frame with index i, i∈0,1,...,n, where n is the length of the sequence, and s is the step size, s∈3,5,7,10. These values ​​can be chosen to increase the separation between the used frames.

[0211] In one example, the network model can be implemented with the TensorFlow framework and trained with a graphics processing unit (GPU), vector processing unit, or other computer processor. In one example, the performance of the trained network is analyzed by using tools provided in the TUM dataset, where two different types of errors are provided: (i) relative pose error (RPE), which measures the local accuracy of the trajectory within a fixed interval Δ, which is the error corresponding to trajectory drift; and (ii) absolute trajectory error (ATE), which measures the global consistency of the estimated trajectory by comparing the absolute distance between the estimated trajectory and the ground truth trajectory.

[0212] To compare the performance of the network with existing methods, a traditional implementation of iterative closest point (ICP) can be used. The errors determined in the example evaluation of the CNN-based relative pose estimation methods discussed in this paper are listed in Table 5, and the results of the traditional ICP implementation and the combined CNN-ICP method (combining the improved CNN-based solution with the ICP part) are also used for comparison. It can be seen that in terms of relative pose estimation, the rotation error of CNN is higher than that of the method using ICP, but the translation error of CNN is the lowest. This means that although the rotation error is large, the impact on the pose is small, making the translation error the largest. In some instances, the translation relative pose estimation error can be considered sufficient for comparison because the rotation relative pose estimation error appears as a translation error when the camera moves. Finally, although the ATE error of CNN is larger, the trajectory obtained by the network is smooth and follows the same "shape" as the ground truth, while the ICP trajectory is generally more unstable. The higher ATE of CNN can be regarded as the result of training the CNN on a small set of data where there is no evaluation rotation scale.

[0213]

[0214] Table 5: Relative pose error (RPE) and absolute trajectory error (ATE) for each relative pose estimation technique (expressed as root mean square error (RMSE), mean and standard deviation (STD)). In this example, RPE is calculated at a fixed delta interval of 5 frames.

[0215] As described above, the improved computing system can implement an end-to-end method (in hardware and / or software) to find the relative pose between voxel grids using a loss function based on the transformation error between the point clouds. The trajectory estimated by the CNN of such a system may be smoother than that of ICP (or even a combination of CNN+ICP). Although such a system can utilize a relatively large amount of data to train the corresponding deep learning model, in other implementations, the size of the training data set can be increased. It should be understood that the method can be used in other applications, including handheld cameras, drones, and cars. In addition, the implementation can utilize point clouds obtained by LiDAR. In some implementations, the size of the parameters of the network can also be reduced or otherwise simplified, as well as other example modifications.

[0216] Figure 26A-26C 2600a-c are flow charts illustrating example techniques for enhancing computer vision processing by the system. Fig.26A In the example of , an improved technique for performing ray casting is shown. For example, a sparse hierarchical volumetric (grid-based) data structure (e.g., a VOLA-based structure) can be accessed 2602, and a ray casting algorithm can be performed 2604 using the volumetric data structure. Multiple rays can be cast 2606 into the volume represented by the volumetric data structure to perform the ray casting algorithm. The ray casting may include a ray traversal step - in which case, based on the interval that caused the ray to be approximately traversed, fewer than all voxels intersected by the ray are checked to see if they contain geometry (or are "occupied"). Based on the traversal, it is approximately determined whether the ray intersects with an occupied voxel, and if so, which occupied voxel the ray intersects first. For example, as part of a SLAM pipeline, rendering application, or other example use, the ray casting process is repeated for each ray.

[0217] Go to Fig.26B , an improved technique for determining the posture of a device (e.g., one or more sensors of the device) is shown. The device can be an autonomous device capable of autonomously navigating and moving in an environment. In one example, visual data (e.g., visual data generated by a camera of the device) can be input 2620 to a first portion of a neural network model, which is implemented as a visual odometry neural network. In addition, inertial data (e.g., inertial data generated by an IMU of the device) can also be input 2622 to the model at a second portion of the neural network model, which is implemented as an inertial odometry neural network portion. The respective outputs of the visual odometry network portion and the inertial odometry network portion can be cascaded and provided as input to a first fully connected layer in a set of fully connected layers within the neural network model. An output representing the posture of the device can be generated 2626 by the neural network model based on the combined input of the visual data and the inertial data.

[0218] Go to Fig.26C , showing an improved technique for determining relative pose estimates. For example, point cloud data captured to describe a view of an environment (and to describe two volumes (e.g., sub-volumes of the entire environment)) may be converted (at 2630, 2632) into respective voxelized volumetric data structures (e.g., VOLA-based structures), each volumetric data structure representing a corresponding volume. These volumetric data structures may then be sliced ​​or otherwise converted (at 2634, 2636) into 2D data structures (e.g., individual sets of 2D slices), and the 2D slices may be provided (at 2638) as input to train a convolutional neural network model to determine relative pose estimates. In practice, the convolutional neural network model may generate 2640 a result representing a relative pose estimate based on the volumes described in the two voxelized volumetric data structures.

[0219] Fig. 27 2701 is a simplified block diagram of an example multi-slot vector processor (e.g., a very long instruction word (VLIW) vector processor) according to some embodiments. In this example, the vector processor may include multiple (e.g., 9) functional units (e.g., 2703-2711), which may be fed by a multi-port memory system 2700, backed up by a vector register file (VRF) 2701 and a general register file (GRF) 2702. The processor includes an instruction decoder (IDEC) 2712, which decodes instructions and generates control signals for controlling the functional units 2703-2711. Functional units 2703-2711 are predicated execution unit (PEU) 2703, branch and repeat unit (BRU) 2704, load store port units (e.g., LSU0 2705 and LSU1 2706), vector arithmetic unit (VAU) 2707, scalar arithmetic unit (SAU) 2710, compare and move unit (CMU) 2708, integer arithmetic unit (IAU) 2711, and volume acceleration unit (VXU) 2800. In this particular implementation, VXU 2800 can accelerate operations on volume data, including both store / retrieve operations, logical operations, and arithmetic operations. Although in Fig. 27 In the example of FIG. 27 , the VXU circuit system 2800 is shown as a single component, but it should be understood that the functionality of the VXU (and any of the other functional units 2703-2711) can be distributed among multiple circuit systems. Further, in some implementations, the functionality of the VXU 2800 can be distributed within one or more of the other functional units of the processor (e.g., 2703-2708, 2710, 2711), as well as other example implementations.

[0220] Fig.282800 , a 3D histogram pyramid generator 2811 , a 3D histogram pyramid generator 2812 , a population counter 2813 , a 2D path finding logic 2814 , a 3D path finding logic 2815 , and possible additional functional units to operate on a 64-bit unsigned integer volume bitmap. The output from block 2802 may be written back to the vector register file VRF 2801 or general register file GRF 2802 register file.

[0221] Go to Fig.29 An example is presented in which a representation of the organization of a 4^3 voxel cube 2900 is presented. A second voxel cube 2901 is also presented. In this example, the voxel cube may be defined in data such as a 64-bit integer 2902, where each individual voxel within the cube is represented by a single corresponding bit in the 64-bit integer. For example, voxel 2912 at address {x,y,z}={3,0,3} may be set to "1" to indicate the presence of a geometric shape at that coordinate within the volumetric space represented by voxel cube 2901. Further, in this example, all other voxels (except voxel 2902) may correspond to "blank" space and may be set to "0" to indicate that no physical geometry exists at those coordinates, as well as other examples. Go to Fig.30 , shows an example two-level sparse voxel tree 3000 according to some embodiments. In this example, only a single "occupied" voxel is included in the volume (e.g., at position {15,0,15}). In this case, the upper level-0 of the tree 3001 contains a single voxel entry {3,0,3}. This voxel in turn points to the next level of the tree 3002, which contains a single voxel in the element {3,0,3}. The entry in the data structure corresponding to level 0 of the sparse voxel tree is a 64-bit integer 3003 with one voxel set to occupied. This set voxel means that an array of 64-bit integers is subsequently allocated to level 1 of the tree corresponding to the voxel volume set in 3003. In the level 1 subarray 3004, only one of the voxels is set to occupied, while all other voxels are set to unoccupied. In this example, since the tree is a two-level tree, level 1 represents the bottom of the tree, so that the hierarchical structure ends here.

[0222] Fig.31 A two-level sparse voxel tree 3100 is shown containing occupied voxels at positions {15,0,3} and {15,0,15} of a particular volume in accordance with some embodiments. In this case (subdividing the particular volume into 64 upper level 0 voxels), the upper level 0 of the tree 3101 contains two voxel entries {3,0,0} and {3,0,3} with corresponding data 3104, which show that two voxels are set (i.e., occupied). The next level of the sparse voxel tree (SVT) is provided as an array of 64-bit integers, which contains two sub-cubes 3102 and 3103, one for each voxel set in level 0. In the level 1 sub-array 3105, two voxels are set to occupied: v15 and v63, and all other voxels are set to unoccupied and the tree. This format is flexible because the 64 entries in the next level of the tree are always allocated corresponding to each set voxel in the upper level of the tree. This flexibility can allow dynamically changing scene geometry to be inserted into an existing volume data structure in a flexible manner (i.e., such as randomly, rather than in a fixed order), as long as the corresponding voxels in the upper layer have been set. If the corresponding voxels in the upper layer are not set, either a table of pointers will be maintained, resulting in higher memory requirements, or the tree will need to be at least partially rebuilt in order to insert the unforeseen geometry.

[0223] Fig.32 A method for storing data from a Fig.31 In this example, the overall volume 3200 contains Fig.31 , stored at global coordinates {15,0,3} and {15,0,15}. In this approach, rather than allocating a 64-entry array to represent all subcubes in level 1 below level 0, only those elements that actually contain geometry in level 1 (e.g., as indicated by whether the corresponding level 0 voxel is occupied or unoccupied) are allocated as corresponding 64-bit level 1 records, in this example, so that level 1 has only two 64-bit entries instead of 64 entries (i.e., for each of the 64 level 1 voxels, whether occupied or unoccupied). Thus, in this example, the first level 0 3204 is equivalent to Fig.31 23, while the next level 3205 is 1 / 62 of the corresponding 3105 in Figure 23 in terms of memory requirements. In some implementations, if there is no space allocated in level 1 for the new geometry to be inserted into level 0, the tree must be copied and rearranged.

[0224] exist Fig.32In the example of , subvolumes can be derived by counting occupied voxels in layers above the current layer. In this way, the system can determine where in the voxel data one higher level ends and where the next lower level begins. For example, if three level 0 voxels are occupied, the system can expect that three corresponding level 1 entries will follow in the voxel data, and the next entry (after these three) corresponds to the first entry in level 2, and so on. Such optimal compaction can be very useful in cases where some parts of the scene do not change over time, or in applications where volumetric data needs to be transmitted remotely (such as from a space probe scanning the surface of Pluto, where the transmission of each bit is expensive and time consuming).

[0225] Fig.33 The manner in which voxels may be inserted into a 4^3 cube represented as a 64-bit integer volume data structure entry is shown according to some embodiments to reflect changes in geometry within the corresponding volume. In one example, as shown in 3300, each voxel cube may be organized as four logical 16-bit planes within a 64-bit integer. Each plane corresponds to a Z value of 0 to 3, and each y value within each plane codes for 4 logical 4-bit shifts 0 to 3, and finally each bit within each 4-bit y plane codes for 4 possible values ​​of x (0 to 3), as well as other example organizations. Thus, in this example, to insert a voxel into a 4^3 volume, the x value 0 to 3 may first be offset by one bit, then the value may be offset by 0 / 4 / 8 / 12 bits to encode the y value, and finally the z value may be represented by offsetting by 0 / 16 / 32 / 48 bits, as shown in the C code expression in 3301. Finally, since each 64-bit integer can be a combination of up to 64 voxels, each of which is written separately, the new bitmap must be logically combined with the old 64-bit value read from the sparse voxel tree by ORing the old bitmap value with the new bitmap value as shown in 3302.

[0226] Go to Fig.34 , shows a representation according to some embodiments to illustrate how a 3D volume object stored in a 64-bit integer 3400 can be projected by a logical OR operation in the X direction to produce a 2D pattern 3401, projected by a logical OR operation in the Y direction to produce a 2D output 3402, and finally projected by a logical OR operation in the Z direction to produce the pattern shown in 3403. Fig.353500 to produce an output projection in the X, Y, and Z directions. In this example, table 3501 shows column by column which element indices from input vector 3500 are ORed to produce an x-projection output vector 3502. Table 3503 shows column by column which element indices from input vector 3500 are ORed to produce a y-projection output vector 3504. Finally, table 3505 shows column by column which element indices from input vector 3500 are ORed to produce a z-projection output vector 3506.

[0227] The X-projection logically ORs bits 0, 1, 2, 3 from input data 3500 to produce bit 0 of the X-projection 3501. For example, bit 1 in 3501 can be produced by XORing bits 4, 5, 6, and 7 from 3500, and so on. Similarly, bit 0 in the Y-projection 3504 can be produced by ORing bits 0, 4, 8, and 12 together from 3500. And bit 1 of 3504 can be produced by ORing bits 1, 5, 9, and 13 together from 3500, and so on. Finally, bit 0 in the Z-projection 3506 can be produced by ORing bits 0, 16, 32, and 48 together from 3500. And bit 1 of 3506 can be produced by ORing bits 1, 17, 33, and 49 together from 3500, and so on.

[0228] Fig.36An example of how projection can be used to generate a simplified map is shown in accordance with some embodiments. In this scenario, the goal can be to generate a compact 2D map of a path from a voxel volume 3600 along which a vehicle 3602 of height h 3610 and width w 3601 travels. Here, Y projection logic can be used to generate an initial rough 2D map 3603 from the voxel volume 3602. In some implementations, the map can be processed to check whether a particular vehicle (e.g., a car (or autonomous car), a drone, etc.) of a particular dimension can traverse the width constraints 3601 and height constraints 3610 of the path. This can be performed by performing a projection in the Z direction to check the width constraint 3601 and the projection in the Y direction can be masked to limit the calculation of the vehicle height 3610 to ensure that the path is traversable. It can be seen through additional post-processing (e.g., in software) that for paths that are traversable and only X and Z satisfy width and height constraints, the coordinates of points A 3604, B 3605, C 3606, D 3607, E 3608, and F 3609 along the path can be stored or transmitted over the network to fully reconstruct a legal path along which the vehicle can travel. Assuming that the path can be parsed into such segmented fragments, it is possible to fully describe the path with only one or two bytes per segmented linear region of the path. This can facilitate rapid transmission and processing of such path data (e.g., by autonomous vehicles), as well as other examples.

[0229] Fig.37It is shown how volumetric 3D measurements or simple 2D measurements from embedded devices can be mathematically aggregated in order to generate high-quality crowdsourced maps as an alternative to precision measurements using LIDAR or other expensive devices, according to some embodiments. In the proposed system, multiple embedded devices 3700, 3701, etc. can be equipped with various sensors capable of taking measurements, which can be transmitted to a central server 3710. Software running on the server performs aggregation 3702 of all measurements and numerically solves the result matrix by a nonlinear solver 3703 to produce a highly accurate map, which can then be redistributed back to the embedded devices. In fact, data aggregation can also include high-precision survey data from satellites 3720, aerial LIDAR surveys 3721, and ground LIDAR measurements 3722 to increase the accuracy of the resulting map when these high-fidelity data sets are available. In some implementations, maps and / or recorded measurements may be generated in a sparse voxel data structure having a format such as that described herein, converted to a sparse voxel data structure having a format such as that described herein, or otherwise expressed using a sparse voxel data structure having a format such as that described herein, as well as other example implementations.

[0230] Fig.38 is a diagram showing how 2D pathfinding on a 2D 2×2 bitmap can be accelerated according to some embodiments. The principle of operation is that in order for connectivity to exist between points on the map having the same grid cells, the values ​​of cells that are consecutively arranged in the x or y or x and y directions must all be set to a value of 1. Therefore, a logical AND of the bits drawn from those cells can be instantiated to test whether there is a valid path in the bitmap in the grid, and a different AND gate can be instantiated for each valid path through the N×N grid. In some instances, this approach can introduce combinatorial complexity because even an 8×8 2D grid can contain 34 4-1 valid path. Therefore, in some improved implementations, the grid can be reduced to 2×2 or 4×4 patches, which can be tested for connectivity hierarchically. The 2×2 bitmap 3800 contains 4 bits labeled b0, b1, b2, and b3. The 4 bits can take values ​​from 0000 to 1111, with corresponding labels 3801 to 3817. Each of these bit patterns represents a different level of connectivity between the faces of the 2×2 grid labeled 3821 to 3830. For example, when the 2×2 grid 3800 contains bitmaps 1010 (3812), 1011 (3813), 1110 (3816), or 1111 (3817), there is 3821 or v0 representing the vertical connectivity between x0 and y0 in 3800. The 2-input logical AND or b0 and b3 in 3800 as shown in row 1 of table 3818 generates v0 in a connectivity map that can be used in higher level hardware or software to determine global connectivity through a global grid that has been subdivided into 2×2 subgrids. If the global map contains an odd number of grid points on the x-axis or y-axis, the top-level grid will need to be padded to the next highest even number of grid points (e.g., such that an additional 1 row of zero values ​​will be needed to be added to the x-axis and / or y-axis on the global grid). Fig.38An exemplary 7×7 grid 3850 is further shown, which shows how to fill to 8×8 by adding additional rows 3832 and columns 3834 filled with zero values. In order to speed up path finding compared to other techniques (e.g., depth-first search, breadth-first search, or Dijkstra's algorithm, or other graph-based methods), this example can gradually subsample the N×N map 3850 to a 2×2 map. For example, in this example, cell W in 3840 is filled by ORing the contents of cells A, B, C, and D in 3850, and so on. In turn, the bits in the 2×2 cells in 3840 are ORed to fill the cells in 3842. For path finding, the algorithm starts with the minimum 2×2 representation of the grid 3842 and tests each of the bits. Because we know that a zero-valued bit means that the corresponding 2×2 grid cell does not exist in 3840, only the portion of the 4×4 grid (composed of four 2×2 grids) in 3840 that corresponds to a bit in the 2×2 grid 3842 needs to be tested for connectivity. This method can also be used to search the 8×8 grid in 3820, for example, if cell W in 3840 contains a zero value, then we know that there is no path in ABCD in 3820, etc. This method prunes branches from the graph search algorithm used, whether the graph search algorithm used is A*, Dijkstra, DFS, BFS, or a variant thereof. In addition to this, the use of a hardware-based pathfinder 3818 with a 2×2 organization can further limit the associated computations. In fact, the 4×4-based hardware element can be composed using 5 2×2 hardware blocks with the same arrangement as 3840 and 3842 to further limit the amount of graph search that needs to be performed. Furthermore, an 8x8 hardware based search engine may be constructed using 21 2x2 hardware blocks (3818) having the same arrangement as 3842, 3840, 3800, etc. for potentially any NxN topology.

[0231] Fig.39is a simplified block diagram according to some embodiments, which shows how the proposed volumetric data structure can be used to accelerate collision detection. A 3D N×N×N map of geometry can be sub-sampled into a pyramid consisting of the lowest level of detail (LoD) 2×2×2 volume 3902, the next highest 4×4×4 volume 3901, the 8×8×8 volume 3900, and so on up to the N×N×N volume. If the position of the drone, vehicle or robot 3905 is known in 3D space via a positioning method such as GPS or via repositioning from a 3D map, then the position can then be quickly used to test the presence or absence of the geometry in the quadrant of the relevant 2×2×2 sub-volume by scaling the x, y and z positions of the drone / robot appropriately (dividing them by 2 relevant times) and querying 3902 for the presence of the geometry (e.g., checking whether the corresponding bitmap bit is 1 indicating a possible collision). If there is a possible collision (e.g., a "1" was found), then further checks can be performed in volumes 3901, 3900, etc. to confirm whether the drone / robot can move. However, if the voxel in 3902 is free (e.g., a "0"), then the robot / drone can interpret the same voxel as free space and manipulate the directional control to move freely through most of the map.

[0232] Although some of the systems and solutions described and shown herein have been described as comprising or being associated with multiple elements, not all elements clearly shown or described are used in each alternative implementation of the present disclosure. In addition, one or more of the elements may be positioned outside the system, and in other instances, some elements may be included in one or more of the other elements described in other elements and other elements not described in the implementation shown, or as a part of one or more of the other elements described in other elements and other elements not described in the implementation shown. Further, some elements may be combined with other components and components for alternative purposes or additional purposes other than those described herein.

[0233] Further, it should be understood that the examples presented above are non-limiting examples provided only for the purpose of illustrating certain principles and features, and do not necessarily limit or constrain the possible embodiments of the concepts described herein. For example, various combinations of the features and components described herein (including combinations implemented by various implementations of the components described herein) may be utilized to implement various different embodiments. Other implementations, features, and details should be understood from the content of this specification.

[0234] Figure 40-Figure 45is a block diagram of an exemplary computer architecture that may be used in accordance with the embodiments disclosed herein. In practice, the computing devices, processors, and other logic and circuitry of the systems described herein may include all or a portion of the functionality and supporting software and / or hardware circuitry to implement such functionality. Further, other computer architecture designs for processors and computing systems known in the art may also be used beyond the examples shown here. In general, computer architectures suitable for the embodiments disclosed herein may include, but are not limited to: Figure 40-Figure 45 The configuration shown in .

[0235] Fig.40 An example domain topology of a corresponding Internet of Things (IoT) network coupled to a corresponding gateway via a link is shown. The Internet of Things (IoT) is a concept in which a large number of computing devices are interconnected with each other and connected to the Internet to provide very low-level functions and data collection. Therefore, as used herein, an IoT device may include a semi-autonomous device that performs functions (such as sensing or control, etc.) to communicate with other IoT devices and a wider network (such as the Internet). Such IoT devices may be equipped with logic and memory to implement and use hash tables (such as those introduced above).

[0236] Typically, IoT devices are limited in memory, size, or functionality, allowing a larger number of IoT devices to be deployed at a similar cost to a smaller number of larger devices. However, IoT devices can be smartphones, laptops, tablets or PCs, or other larger devices. Further, IoT devices can be virtual devices, such as applications on smartphones or other computing devices. IoT devices can include IoT gateways for coupling IoT devices to other IoT devices and coupling IoT devices to cloud applications for data storage, process control, etc.

[0237] A network of IoT devices may include commercial and home automation devices such as water systems, power distribution systems, assembly line control systems, factory control systems, light switches, thermostats, locks, cameras, alarms, motion sensors, etc. IoT devices may be accessed by a remote computer, server, or other system, for example, to control the system or access data.

[0238] The future growth of the Internet and similar networks is likely to involve a very large number of IoT devices. Therefore, in the context of the technology discussed herein, many innovations for this future networking will address the need for unimpeded growth of all these layers, discovering connected resources and making them accessible, and supporting the ability to hide and partition connected resources. Any number of network protocols and communication standards can be used, each of which is designed to address specific goals. In addition, protocols are part of the structure that supports human-accessible services that operate regardless of location, time, or space. The innovations include service delivery and associated infrastructure, such as hardware and software, security enhancements, and the provision of services based on quality of service (QoS) specified in service levels and service delivery agreements. As will be understood, IoT devices and networks (such as Fig.40 and Fig.41 The use of ) presents many new challenges in heterogeneous connectivity networks that include a combination of wired and wireless technologies.

[0239] Fig.40 Specifically, a simplified diagram of a domain topology is provided, which can be used for multiple Internet of Things (IoT) networks including IoT devices 4004, wherein IoT networks 4056, IoT networks 4058, IoT networks 4060, and IoT networks 4062 are coupled to corresponding gateways 4054 via backbone links 4002. For example, multiple IoT devices 4004 can communicate with gateway 4054 and communicate with each other through gateway 4054. In order to simplify the drawings, not every IoT device 4004 or communication link (e.g., link 4016, 4022, 4028, or 4032) is labeled. Backbone link 4002 can include any number of wired or wireless technologies (including optical networks) and can be part of a local area network (LAN), a wide area network (WAN), or the Internet. In addition, such communication links facilitate optical signal paths between IoT devices 4004 and gateway 4054, including the use of multiplexing / demultiplexing (MUXing / deMUXing) components that facilitate the interconnection of various devices.

[0240] The network topology may include any number of types of IoT networks, such as a mesh network provided using a Bluetooth Low Energy (BLE) link 4022 in conjunction with a network 4056. Other types of IoT networks that may be present include a wireless local area network (WLAN) network 4058 for communicating over IEEE 802.11 Link 4028 communicates with IoT device 4004; cellular network 4060 for communicating with IoT device 4004 via LTE / LTE-A (4G) or 5G cellular network; and low power wide area (LPWA) network 4062, for example, an LPWA network compatible with the LoRaWan specification promulgated by the LoRa Alliance, or IPv6 on a low power wide area network (LPWAN) network compatible with the specification promulgated by the Internet Engineering Task Force (IETF). In addition, the corresponding IoT network can communicate with an external network provider (e.g., a layer 2 or layer 3 provider) using any number of communication links, such as an LTE cellular link, an LPWA link, or a network based on the IEEE 802.15.4 standard (such as ). The corresponding IoT network may also operate using various network and Internet application protocols, such as Constrained Application Protocol (CoAP). The corresponding IoT network may also be integrated with a coordinator device that provides a chain of links that forms a cluster tree of linked devices and networks.

[0241] Each of these IoT networks can provide opportunities for new technical features (such as those described herein). Improved technology and networks can enable exponential growth of devices and networks, including using IoT networks as fog devices or systems. With the growth of the use of such improved technology, IoT networks can be developed for self-management, functional evolution, and collaboration without direct human intervention. Improved technology can even enable IoT networks to work without a central control system. Therefore, the improved technology described herein can be used for automation and enhanced network management and operation functions far beyond the current implementation.

[0242] In an example, communications between IoT devices 4004, such as communications over backbone link 4002, can be protected by a decentralized system for authentication, authorization, and accounting (AAA). In a decentralized AAA system, distributed payment, loan, audit, authorization, and authentication systems can be implemented across interconnected heterogeneous network infrastructures. This allows systems and networks to move toward autonomous operations. In these types of autonomous operations, machines can even contract human resources and negotiate partnerships with other machine networks. This can allow for the achievement of common goals and balanced service delivery relative to outlined, planned service level agreements, as well as solutions that provide metering, measurement, traceability, and traceability. Creating new supply chain structures and approaches can create a large number of services without human involvement, tap the value of a large number of services, and collapse a large number of services.

[0243] Such IoT networks can be further enhanced by integrating sensing technologies such as sound, light, electronic traffic, facial and pattern recognition, smell, vibration, etc. into autonomous organizations between IoT devices. The integration of sensing systems can allow for systematic and autonomous communication and coordination of service delivery relative to contractual service goals, resource aggregation and convergence based on service orchestration and quality of service (QoS). Some of the various examples of network-based resource processing include the following.

[0244] For example, the mesh network 4056 can be enhanced by a system that performs inline data to information transformation. For example, a self-forming chain of processing resources including a multi-link network can distribute the transformation of raw data to information in an efficient manner, as well as having the ability to distinguish between assets and resources and the associated management of each asset and resource. In addition, appropriate components of the infrastructure and resource-based trust and service indexes can be inserted to improve data integrity, quality, assurance and provide data confidence metrics.

[0245] For example, WLAN network 4058 can provide multi-standard connectivity using a system that performs standard conversion, thereby enabling IoT devices 4004 to communicate using different protocols. Other systems can provide seamless interconnection across a multi-standard infrastructure that includes visible Internet resources and hidden Internet resources.

[0246] For example, communications in the cellular network 4060 may be enhanced by systems that migrate data, extend communications to more remote devices, or migrate data and extend communications to more remote devices. The LPWA network 4062 may include systems that perform non-Internet Protocol (IP) to IP interconnection, addressing, and routing. In addition, each IoT device 4004 may include a suitable transceiver for wide area communications with the device. Further, each IoT device 4004 may include other transceivers for communicating using additional protocols and frequencies. Fig.42 and Fig.43 This is further discussed in the communication environment and hardware of IoT processing devices depicted in .

[0247] Finally, clusters of IoT devices can be equipped to communicate with other IoT devices as well as cloud networks. This can allow IoT devices to form ad-hoc networks between devices, allowing them to function as a single device, which can be referred to as a fog device. Fig.41 This configuration is discussed further.

[0248] Fig.41A cloud computing network is shown communicating with a mesh network of IoT devices (devices 4102) operating as fog devices at the edge of the cloud computing network. The mesh network of IoT devices may be referred to as fog 4120, operating at the edge of cloud 4100. To simplify the diagram, not every IoT device 4102 is labeled.

[0249] The fog 4120 can be considered as a massively interconnected network in which several IoT devices 4102 communicate with each other, for example, via radio links 4122. As an example, the Open Connectivity Foundation (OCF) TM The interconnection network is facilitated by the interconnection specification published by the Open Compute Foundation (OCF). The standard allows devices to discover each other and establish communications for interconnection. Other interconnection protocols may also be used, including, for example, the Optimized Link State Routing (OLSR) protocol, the Best Solution for Mobile Ad Hoc Networks (BATMAN) routing protocol, or the OMA Lightweight M2M (LWM2M) protocol, etc.

[0250] Although three types of IoT devices 4102 are shown in this example: gateway 4104, data aggregator 4126, and sensor 4128, any combination of IoT devices 4102 and functions can be used. Gateway 4104 can be an edge device that provides communication between cloud 4100 and fog 4120, and can also provide back-end processing functions for data obtained from sensor 4128 (such as motion data, traffic data, temperature data, etc.). Data aggregator 4126 can collect data from any number of sensors 4128 and perform back-end processing functions for analysis. Result data, raw data, or both can be passed to cloud 4100 through gateway 4104. Sensor 4128 can be a full IoT device 4102, for example, an IoT device that can both collect data and process data. In some cases, sensor 4128 can be more limited in function, for example, collecting data and allowing data aggregator 4126 or gateway 4104 to process data.

[0251] Communications from any IoT device 4102 may be passed along a convenient path (e.g., the most convenient path) between any IoT device 4102 to reach the gateway 4104. In these networks, the number of interconnections provides a large amount of redundancy, which allows communications to be maintained even with the loss of several IoT devices 4102. Further, the use of a mesh network may allow the use of very low power or IoT devices 4102 located far from the infrastructure, as the distance to connect to another IoT device 4102 may be much less than the distance to connect to the gateway 4104.

[0252] The fog 4120 provided from these IoT devices 4102 may be presented to devices in the cloud 4100 (such as servers 4106) as a single device, e.g., a fog device, located at the edge of the cloud 4100. In this example, alerts from the fog device may be sent without being identified as coming from a specific IoT device 4120 within the fog 4102. In this manner, the fog 4120 may be considered a distributed platform that provides computing and storage resources to perform processing or data-intensive tasks, such as data analysis, data aggregation, and machine learning.

[0253] In some examples, the IoT devices 4102 may be configured using an imperative programming style, for example, each IoT device 4102 having specific functionality and communication partners. However, the IoT devices 4102 forming the fog devices may be configured in a declarative programming style, which allows the IoT devices 4102 to reconfigure their operations and communications, such as determining required resources in response to conditions, queries, and device failures. As an example, a query from a user located at a server 4106 related to the operation of a subset of devices monitored by the IoT device 4102 may cause the fog 4120 device to select the IoT device 4102 (such as a specific sensor 4128) required to answer the query. The data from these sensors 4128 may then be aggregated and analyzed by any combination of the sensor 4128, the data aggregator 4126, or the gateway 4104 before being sent by the fog 4120 device to the server 4106 to answer the query. In this example, the IoT devices 4102 in the fog 4120 may select the sensors 4128 used based on a query (such as adding data from a flow sensor or a temperature sensor). Furthermore, if some of the IoT devices 4102 are inoperable, other IoT devices 4102 in the fog 4120 devices may provide similar data if available.

[0254] In other examples, the operations and functions described above may be embodied by an IoT device machine in the form of an electronic processing system, in which an instruction set or sequence of instructions may be executed to cause the electronic processing system to perform any of the methods discussed herein, according to an example embodiment. The machine may be an IoT device or an IoT gateway, including a machine embodied by aspects of a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile phone or a smart phone, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by the machine. In addition, although only a single machine is described and referenced in the above examples, such machines should also be considered to include any set of machines that execute a set (or multiple sets) of instructions individually or jointly to perform any one or more of the methods discussed herein. In addition, these examples and examples similar to processor-based systems should be considered to include any set of one or more machines controlled or operated by a processor (e.g., a computer) to execute instructions individually or jointly to perform any one or more of the methodologies discussed herein. In some implementations, one or more devices may be operated collaboratively to implement functions and perform tasks described herein. In some cases, one or more host devices may supply data, provide instructions, aggregate results, or otherwise facilitate joint operations and functionality provided by multiple devices. When a function is implemented by a single device, it can be considered a function local to the device; in an implementation where multiple devices operate as a single machine, the function can be considered to be local to the device collectively, and the collection of devices can provide or consume results provided by other remote machines (implemented as a single device or a collection of devices), as well as other example implementations.

[0255] For example, Fig.42A diagram of a cloud computing network or cloud 4200 in communication with multiple Internet of Things (IoT) devices is shown. Cloud 4200 may represent the Internet, or may be a local area network (LAN), or a wide area network (WAN), such as a company's proprietary network. IoT devices may include any number of different types of devices, grouped in various combinations. For example, a traffic control group 4206 may include IoT devices along streets in a city. These IoT devices may include traffic lights, traffic flow monitors, cameras, weather sensors, etc. Traffic control group 4206 or other subgroups may communicate with cloud 4200 via a wired or wireless link 4208 (such as an LPWA link, an optical link, etc.). Further, a wired or wireless subnetwork 4212 may allow IoT devices to communicate with each other, such as via a local area network, a wireless local area network, etc. IoT devices may use another device (such as gateway 4210 or gateway 4228) to communicate with a remote location (such as cloud 4200); IoT devices may also use one or more servers 4230 to facilitate communication with cloud 4200 or with gateway 4210. For example, one or more servers 4230 can operate as intermediate network nodes to support local edge cloud or fog implementations between local area networks. In addition, the depicted gateway 4228 can operate in a cloud-to-gateway-to-many edge devices configuration (such as various IoT devices 4214, 4220, 4224 are limited or dynamic in the allocation and use of resources in the cloud 4200).

[0256] Other example groups of IoT devices may include a remote weather station 4214, a local information terminal 4216, an alarm system 4218, an ATM 4220, an alarm panel 4222, or a mobile vehicle (such as an emergency vehicle 4224 or other vehicle 4226), etc. Each of these IoT devices can be connected to other IoT devices, to the server 4204, to another IoT fog device or system (not shown, but Fig.41 ), or a combination thereof. The group of IoT devices can be deployed in various residential settings, commercial settings, and industrial settings (including both in private environments or public environments).

[0257] As from Fig.42As can be seen in , a large number of IoT devices can communicate through the cloud 4200. This can allow different IoT devices to autonomously request or provide information to other devices. For example, a group of IoT devices (e.g., traffic control group 4206) can request the current weather forecast from a group of remote weather stations 4214, which can provide the forecast without human intervention. In addition, the emergency vehicle 4224 can be warned by the ATM 4220 that the theft is in progress. When the emergency vehicle 4224 moves toward the ATM 4220, it can access the traffic control group 4206 to request permission to pass to the location, for example, by turning the signal light to red to block crossing traffic at the intersection for a sufficient time to allow the emergency vehicle 4224 to enter the intersection unimpeded.

[0258] A cluster of IoT devices (such as a remote weather station 4214 or a traffic control group 4206) can be equipped to communicate with other IoT devices as well as the cloud 4200. This can allow the IoT devices to form an ad hoc network between the devices, allowing them to function as a single device, which can be referred to as a fog device or system (e.g., as described above with reference to Fig.41 described above).

[0259] Fig.43 4350 for implementing the techniques described herein. The IoT device 4350 may include any combination of the components shown in the examples or referenced in the above disclosure. These components may be implemented as ICs, portions of ICs, discrete electronic devices or other modules, logic, hardware, software, firmware, or a combination thereof suitable for use in the IoT device 4350, or as components that are otherwise incorporated into a chassis of a larger system. Additionally, Fig.43 The block diagram is intended to depict a high-level view of the components of the IoT device 4350. However, some of the components shown may be omitted, additional components may be present, and different arrangements of the components shown may occur in other implementations.

[0260] IoT device 4350 may include processor 4352, which may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. Processor 4352 may be part of a system on a chip (SoC), where processor 4352 and other components are formed as a single integrated circuit, or a single package (such as Edison from Intel). TM ) or Galileo TM )SoC board). As an example, the processor 4352 may include a Core Architecture TM (CoreTM ) processors (such as Quark TM 、Atom TM , i3, i5, i7 or MCU class processors), or available from However, any number of other processors may be used, such as processors available from Advanced Micro Devices, Inc. (AMD) of Sunnyvale, California, MIPS-based designs from MIPS Technologies, Inc. of Sunnyvale, California, ARM-based designs licensed from ARM Holdings, Inc., or processors obtained from customers, licensees, or adopters of the aforementioned companies. The processor may include units such as: The company's A5-A10 processors, from Qualcomm Snapdragon TM (Snapdragon TM ) processor or OMAP from Texas Instruments TM processor.

[0261] Processor 4352 can communicate with system memory 4354 via interconnect 4356 (e.g., a bus). Any number of memory devices can be used to provide a quantitative system memory. As an example, the memory can be a random access memory (RAM) designed according to the Joint Electronic Device Engineering Council (JEDEC), such as DDR or mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). In various implementations, the individual memory devices can be any number of different package types, such as a single die package (SDP), a dual die package (DDP), or a quad die package (Q17P). In some examples, these devices can be directly soldered to the motherboard to provide a thinner solution, while in other examples, the devices are configured as one or more memory modules, which are then coupled to the motherboard via a given connector. Any number of other memory implementations can be used, such as other types of memory modules, for example, different types of dual in-line memory modules (DIMMs), including but not limited to microDIMMs or MiniDIMMs.

[0262] In order to provide persistent storage of information (such as data, applications, operating systems, etc.), storage 4358 may also be coupled to processor 4352 via interconnect 4356. In an example, storage 4358 may be implemented via a solid state disk drive (SSDD). Other devices that may be used for storage 4358 include flash memory cards (such as SD cards, microSD cards, xD picture cards, etc.) and USB flash drives. In a low-power implementation, storage 4358 may be an on-die memory or register associated with processor 4352. However, in some examples, storage 4358 may be implemented using a micro hard disk drive (HDD). In addition, in addition to or in lieu of the described technology, any number of new technologies may be used for storage 4358, such as resistive memory, phase change memory, holographic memory, or chemical memory, etc.

[0263] Components may communicate via interconnect 4356. Interconnect 4356 may include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies. Interconnect 4356 may be a proprietary bus used, for example, in SoC-based systems. Other bus systems may be included, such as an I2C interface, an SPI interface, a point-to-point interface, a power bus, and the like.

[0264] The interconnect 4356 may couple the processor 4352 to a mesh transceiver 4362 for communication with other mesh devices 4364, for example. The mesh transceiver 4362 may use any number of frequencies and protocols, such as 2.4 gigahertz (GHz) transmissions under the IEEE 802.15.4 standard, using a wireless communication protocol such as that provided by Bluetooth. Bluetooth SIG Low Power (BLE) standard, or standards, etc. Any number of radios configured for a particular wireless communication protocol may be used for connection to mesh device 4364. For example, a WLAN unit may be used to implement Wi-FiTm communications in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. Additionally, wireless wide area communications, such as in accordance with cellular or other wireless wide area protocols, may occur via a WWAN unit.

[0265] Mesh transceiver 4362 can communicate using multiple standards or radios for different ranges of communication. For example, IoT device 4350 can communicate with close (e.g., within about 10 meters) devices using a local transceiver based on BLE or another low-power radio to save power. Mesh devices 4364 farther away (e.g., within about 50 meters) can be reached via ZigBee or other intermediate power radios. These two communication technologies can occur through a single radio at different power levels, or can occur through separate transceivers, such as a local transceiver using BLE and a separate mesh transceiver using ZigBee.

[0266] A wireless network transceiver 4366 may be included to communicate with devices or services in the cloud 4300 via a local area network protocol or a wide area network protocol. The wireless network transceiver 4366 may be an LPWA transceiver that complies with IEEE802.15.4 or IEEE 802.15.4g standards, etc. The IoT device 4350 can communicate over a wide area using LoRaWANTm (Long Range Wide Area Network) developed by Semtech and the LoRa Alliance. The techniques described herein are not limited to these techniques, but may be used with any number of other cloud transceivers that implement long-distance, low-bandwidth communications (such as Sigfox and other technologies). Further, other communication technologies may be used, such as time-division channel hopping described in the IEEE 802.15.4e specification.

[0267] In addition to the systems mentioned for mesh transceiver 4362 and wireless network transceiver 4366 as described herein, any number of other radio communications and protocols may be used. For example, radio transceivers 4362 and 4366 may include LTE or other cellular transceivers that use spread spectrum (SPA / SAS) communications to achieve high-speed communications. Further, any number of other protocols may be used, such as LTE for medium-speed communications and supply network communications. network.

[0268] Radio transceivers 4362 and 4366 may include radios compatible with any number of 3GPP (3rd Generation Partnership Project) specifications, particularly Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), and Long Term Evolution-Advanced Plus (LTE-A Pro). It may be noted that radios compatible with any number of other fixed, mobile, or satellite communication technologies and standards may be selected. These may include, for example, any cellular wide area wireless communication technology, which may include, for example, a 5th generation (5G) communication system, a global mobile communication (GSM) radio communication system, a general packet radio service (GPRS) radio communication technology, or an enhanced data rate for GSM evolution (EDGE) radio communication technology, a UMTS (Universal Mobile Telecommunications System) communication technology, and in addition to the standards listed above, any number of satellite uplink technologies may be used for the wireless network transceiver 4366, including, for example, radios that comply with standards issued by ITU (International Telecommunication Union) or ETSI (European Telecommunications Standards Institute), etc. The examples provided herein may therefore be understood to be applicable to various existing and yet to be developed various other communication technologies.

[0269] A network interface controller (NIC) 4368 may be included to provide wired communications to the cloud 4300 or to other devices such as mesh device 4364. Wired communications may provide Ethernet connectivity, or may be based on other types of networks such as controller area network (CAN), local interconnect network (LIN), device network (DeviceNet), control network (ControlNet), data highway +, field bus (PROFIBUS) or industrial Ethernet (PROFINET), etc. Additional NICs 4368 may be included to allow connection to a second network, for example, NIC 4368 provides communications to the cloud via Ethernet and a second NIC 4368 provides communications to other devices via another type of network.

[0270] Interconnect 4356 may couple processor 4352 to external interface 4370, which is used to connect external devices or subsystems. External devices may include sensors 4372, such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, global positioning system (GPS) sensors, pressure sensors, barometric pressure sensors, etc. External interface 4370 may further be used to connect IoT device 4350 to actuator 4374 (such as a power switch, a valve actuator, an audible sound generator, a visual warning device, etc.).

[0271] In some optional examples, various input / output (I / O) devices may be present within the IoT device 4350, or may be connected to the IoT device 4350. For example, a display or other output device 4384 may be included to display information, such as sensor readings or actuator positions. Input devices 4386 (such as a touch screen or keypad) may be included to accept input. Output devices 4384 may include any number of audio or visual display forms, including: simple visual outputs, such as binary status indicators (e.g., LEDs); multi-character visual outputs; or more complex outputs, such as display screens (e.g., LCD screens) with outputs of characters, graphics, multimedia objects, etc. generated or produced from the operation of the IoT device 4350.

[0272] The battery 4376 can power the IoT device 4350, but in examples where the IoT device 4350 is installed in a fixed location, the IoT device 4350 can have a power source coupled to the power grid. The battery 4376 can be a lithium-ion battery, a metal-air battery (such as a zinc-air battery, an aluminum-air battery, a lithium-air battery), or the like.

[0273] A battery monitor / charger 4378 may be included in the IoT device 4350 to track the state of charge (SoCh) of the battery 4376. The battery monitor / charger 4378 may be used to monitor other parameters of the battery 4376 to provide failure prediction, such as the state of health (SoH) and state of function (SoF) of the battery 4376. The battery monitor / charger 4378 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx family of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 4378 may pass information about the battery 4376 to the processor 4352 via the interconnect 4356. The battery monitor / charger 4378 may also include an analog-to-digital (ADC) converter that allows the processor 4352 to directly monitor the voltage of the battery 4376 or the current from the battery 4376. Battery parameters may be used to determine actions that the IoT device 4350 may perform, such as transmission frequency, mesh network operation, sensing frequency, and the like.

[0274] A power block 4380 or other power source coupled to the grid may be coupled to the battery monitor / charger 4378 to charge the battery 4376. In some examples, the power block 4380 may be replaced with a wireless power receiver to obtain power wirelessly, for example, through a loop antenna in the IoT device 4350. A wireless battery charging circuit (such as the LTC4020 chip from Linear Technology, Inc. of Milpitas, California, etc.) may be included in the battery monitor / charger 4378. The specific charging circuit selected depends on the size of the battery 4376 and therefore on the current required. Charging may be performed using the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Alliance for Wireless Power, etc.

[0275] Storage 4358 may include instructions 4382 in the form of software, firmware, or hardware commands for implementing the techniques disclosed herein. Although such instructions 4382 are illustrated as code blocks included in memory 4354 and storage 4358, it is understood that any of the code blocks may be replaced with hardwired circuitry, such as built into an application specific integrated circuit (ASIC).

[0276] In an example, the instructions 4382 provided via the memory 4354, the storage 4354, or the processor 4352 may be embodied as a non-transitory machine-readable medium 4360, which includes code for instructing the processor 4352 to perform electronic operations in the IoT device 4350. The processor 4352 may access the non-transitory machine-readable medium 4360 via the interconnect 4356. For example, the non-transitory machine-readable medium 4360 may be provided by a processor 4352 for performing an electronic operation in the IoT device 4350. Fig.43 The storage 4358 may be embodied in the device described, or may include a specific storage unit, such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 4360 may include instructions for instructing the processor 4352 to perform, for example, a specific sequence of actions or flow of actions as described with reference to the (multiple) flow charts and (multiple) block diagrams of the operations and functions depicted above.

[0277] Fig.44 4400 is an example of a processor according to an embodiment. Processor 4400 is an example of a type of hardware device that can be used in conjunction with the above implementation. Processor 4400 can be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or other device for executing code. Although in Fig.44 Only one processor 4400 is shown in the figure, but the processing element may alternatively include more than one Fig.44 4400. The processor 4400 may be a single-threaded core, or for at least one embodiment, the processor 4400 may be multi-threaded in that each of its cores may include more than one hardware thread context (or "logical processor").

[0278] Fig.44 Also shown is a memory 4402 coupled to the processor 4400 according to an embodiment. The memory 4402 can be any of a wide variety of memories (including various layers of a memory hierarchy) known or otherwise available to those skilled in the art. Such memory elements can include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0279] The processor 4400 may execute any type of instructions associated with the algorithms, processes, or operations described in detail herein. In general, the processor 4400 may transform an element or article (eg, data) from one state or thing to another state or thing.

[0280] Code 4404 (which may be one or more instructions for execution by processor 4400) may be stored in memory 4402, or may be stored in software, hardware, firmware, or any suitable combination thereof, or in any other internal or external component, device, element, or object, as appropriate and based on specific needs. In one example, processor 4400 may follow a program sequence of instructions indicated by code 4404. Each instruction enters front-end logic 4406 and is processed by one or more decoders 4408. The decoder may generate micro-operations such as fixed-width micro-operations in a predefined format as its output, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. Front-end logic 4406 also includes register renaming logic 4410 and scheduling logic 4412, which generally allocates resources and queues operations corresponding to instructions for execution.

[0281] The processor 4400 may also include an execution logic 4414 having a set of execution units 4416a, 4416b, 4416n, etc. Some embodiments may include a large number of execution units dedicated to a specified function or set of functions. Other embodiments may include only one execution unit or one execution unit that can perform a specific function. The execution logic 4414 performs the operations specified by the code instructions.

[0282] After execution of the operations specified by the code instructions is complete, the back-end logic 4418 may retire the instructions of the code 4404. In one embodiment, the processor 4400 allows out-of-order execution but requires in-order retirement of instructions. The retirement logic 4420 may take various known forms (e.g., a reorder buffer, etc.). In this way, the processor 4400 is transformed during execution of the code 4404, at least for the outputs generated by the decoders, the hardware registers and tables utilized by the register renaming logic 4410, and any registers (not shown) modified by the execution logic 4414.

[0283] Although not in Fig.44 4400, but the processing element may include other elements that are on the chip with the processor 4400. For example, the processing element may include memory control logic with the processor 4400. The processing element may include I / O control logic and / or may include I / O control logic that is integrated with the memory control logic. The processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on the chip with the processor 4400.

[0284] Fig.45 The figure shows a computing system 4500 arranged in a point-to-point (PtP) configuration according to an embodiment. Specifically, Fig.45 A system is shown in which processors, memory, and input / output devices are interconnected by a number of point-to-point interfaces. In general, one or more of the computing systems described herein can be configured in the same or similar manner as computing system 4500.

[0285] The processors 4570 and 4580 may also each include integrated memory controller logic (MC) 4572 and 4582 to communicate with the memory elements 4532 and 4534. In alternative embodiments, the memory controller logic 4572 and 4582 may be discrete logic separate from the processors 4570 and 4580. The memory elements 4532 and / or 4534 may store various data to be used by the processors 4570 and 4580 to obtain the operations and functions outlined herein.

[0286] Processors 4570 and 4580 may be any type of processor, such as those discussed in connection with other figures. Processors 4570 and 4580 may exchange data via a point-to-point interface 4550 using point-to-point (PtP) interface circuits 4578 and 4588, respectively. Processors 4570 and 4580 may each exchange data with a chipset 4590 via respective point-to-point interfaces 4552 and 4554 using point-to-point interface circuits 4576, 4586, 4594, and 4598. Chipset 4590 may also exchange data with a high-performance graphics circuit 4538 via a high-performance graphics interface 4539 using interface circuit 4592 (which may be a PtP interface circuit). In an alternative embodiment, Fig.45 Any or all of the PtP links illustrated in FIG. 5 may be implemented as a multi-drop bus instead of a PtP link.

[0287] Chipset 4590 may communicate with bus 4520 via interface circuit 4596. Bus 4520 may have one or more devices communicating over the bus, such as bus bridge 4518 and I / O devices 4516. Bus bridge 4518 may communicate with other devices via bus 4510, such as user interface 4512 (such as a keyboard, mouse, touch screen, or other input device), communication device 4526 (such as a modem, network interface device, or other type of communication device that can communicate over computer network 4560), audio I / O device 4514, and / or data storage device 4528. Data storage device 4528 may store code 4530 that may be executed by processor 4570 and / or 4580. In alternative embodiments, any portion of the bus architecture may be implemented using one or more PtP links.

[0288] Fig.45 The computer system depicted in is a schematic illustration of an embodiment of a computing system that can be used to implement the various embodiments discussed herein. It will be appreciated that Fig.45 The components of the systems depicted in the can be combined in a system-on-a-chip (SoC) architecture or in any other suitable configuration capable of achieving the functions and features of the examples and implementations provided herein.

[0289] In a further example, a machine-readable medium also includes any tangible medium that can store, encode or carry instructions for execution by a machine and cause the machine to perform any one or more of the methods of the present disclosure, or that can store, encode or carry data structures utilized by or associated with such instructions. "Machine-readable media" may therefore include, but are not limited to, solid-state memory, optical media, and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, by way of example, but not limited to: semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The instructions embodied by the machine-readable medium may be transmitted or received further over a communication network using a transmission medium, via a network interface device, using any of a variety of transmission protocols (e.g., HTTP).

[0290] It should be understood that the functional units or capabilities described in this specification may be referred to or marked as components or modules, thereby emphasizing the independence of their implementation. Such components may be embodied in any number of software or hardware forms. For example, a component or module may be implemented as a hardware circuit, which includes a customized very large scale integration (VLSI) circuit or gate array, a readily available semiconductor such as a logic chip, a transistor, or other discrete components. A component or module may also be implemented in a programmable hardware device, such as a field programmable gate array, a programmable array logic, a programmable logic device, etc. A component or module may also be implemented in software executed by various types of processors. The identified components or modules of the executable code may, for example, include one or more physical or logical boxes of computer instructions, which may be organized into, for example, objects, processes, or functions. However, the executable files of the identified components or modules do not have to be physically together, but may include different instructions stored in different locations, which, when logically combined together, include components or modules, and achieve the claimed purpose for the components or modules.

[0291] In fact, the components or modules of the executable code can be a single instruction or many instructions, and can even be distributed over several different code segments, distributed between different programs and distributed across several memory devices or processing systems. Specifically, some aspects of the described process (such as code rewriting and code analysis) may occur on a processing system (e.g., a computer in a data center) different from the processing system in which the code is deployed (e.g., a computer embedded in a sensor or robot). Similarly, operational data can be identified and illustrated herein within a component or module, and can be embodied in any suitable form and organized in any suitable type of data structure. The operational data can be collected as a single data set, or can be distributed over different locations (including over different storage devices), and can exist on a system or network at least in part only as electronic signals. Components or modules can be passive or active, including agents that can be operated to perform desired functions.

[0292] Additional examples of the presently described method, system, and device embodiments include the following non-limiting configurations. Each of the following non-limiting examples may exist independently or may be combined with one or more of the other examples provided below or throughout the present disclosure in any arrangement or combination.

[0293] Although the present disclosure has been described in terms of certain implementations and generally associated methods, changes and permutations of these implementations and methods will be apparent to those skilled in the art. For example, the actions described herein may be performed in an order different from the order described and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown, or sequentially to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous. In addition, other user interface layouts and functions may also be supported. Other variations fall within the scope of the appended claims.

[0294] Although this specification contains many specific implementation details, these specific implementation details should not be interpreted as limitations on the scope of any invention or claimable rights, but rather as descriptions of features for specific embodiments of specific inventions. Certain features described in the context of separate embodiments in this specification may also be implemented in a single embodiment in combination. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although multiple features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be deleted from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0295] Similarly, although multiple operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or sequentially, or requiring that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged in multiple software products.

[0296] The following examples are related to embodiments according to the present specification. Example 1 is a device comprising: a data processing device; a memory for storing a volume data structure representing a three-dimensional (3D) volume, wherein the volume data structure comprises a sparse hierarchical volume data structure, and the volume data structure comprises a set of higher-level voxels, wherein each higher-level voxel represents a respective sub-volume in the 3D volume at a first resolution, each higher-level voxel comprises a set of lower-level voxels, and the set of lower-level voxels represents a sub-volume of each higher-level voxel at a second, higher resolution; and a ray caster executable by the data processing device to perform a ray casting algorithm, wherein the ray casting algorithm takes the volume data structure as input, and performing the ray casting algorithm comprises: casting a plurality of rays from a reference point into the 3D volume; for each ray of the plurality of rays, traversing the ray to determine whether a voxel in the voxel set is intersected and occupied by the ray, wherein the ray is traversed according to an approximate traversal.

[0297] Example 2 includes the subject matter of Example 1, wherein traversing the ray comprises determining from the volume data structure that only a subset of the set of higher-level voxels contains respective lower-level voxels occupied by the geometric shape.

[0298] Example 3 includes the subject matter of Example 2, wherein traversing the ray further comprises: determining a subset of lower level voxels intersected by the ray, wherein fewer than all lower level voxels intersected by the ray are in the subset based on the approximation traversal.

[0299] Example 4 includes the subject matter of any of Examples 2-3, wherein traversing the rays comprises skipping analysis of lower-level voxels of each higher-level voxel that are determined based on the volumetric data structure to not contain occupied lower-level voxels.

[0300] Example 5 includes the subject matter of Example 4, wherein the analysis comprises checking whether lower level voxels within a subset of the higher level voxel subsets and intersecting the ray are occupied at multiple points on the ray corresponding to the higher level voxel subset, wherein the subset of lower level voxels within the higher level voxel subset and intersecting the ray is not checked based on an approximate traversal.

[0301] Example 6 includes the subject matter of Example 5, wherein the plurality of points correspond to intervals defined based on an approximate traversal.

[0302] Example 7 includes the subject matter of Example 6, wherein the defined intervals include distances associated with lengths of lower-level voxels in a particular dimension.

[0303] Example 8 includes the subject matter of Example 7, wherein the ray caster further determines a specific direction, wherein the specific dimension comprises a primary directional component of the ray.

[0304] Example 9 includes the subject matter of any of Examples 4-8, wherein analysis of the lower level voxels in each subset of higher level voxels includes accessing a memory to retrieve a corresponding portion of a volume data structure, wherein portions of the volume data structure corresponding to higher level voxels determined not to contain occupied lower level voxels are not retrieved from the memory.

[0305] Example 10 includes the subject matter of any of Examples 1-9, wherein the ray casting algorithm is part of a SLAM pipeline implemented using the circuit system.

[0306] Example 11 includes the subject matter of any of Examples 1-9, wherein the ray casting algorithm is performed in conjunction with a graphics rendering application.

[0307] Example 12 includes the subject matter of any of Examples 1-10, wherein the volumetric data structure includes a first entry for representing the presence of a geometric shape in each higher-level voxel, and further includes a second group of entries, wherein each entry in the second group of entries represents a respective lower-level voxel in one of a subset of the higher-level voxels.

[0308] Example 13 includes the subject matter of Example 12, wherein the first entry and the second entry include a binary value for each voxel described in the respective entry, and the respective binary values ​​indicate whether the corresponding voxel is occupied.

[0309] Example 14 includes the subject matter of any of Examples 1-13, wherein the data processing device comprises a vision processing unit (VPU).

[0310] Example 15 includes the subject matter of any of Examples 1-14, wherein the device comprises one of a robot or a drone.

[0311] Example 16 is a method comprising: accessing a volume data structure from a computer memory, wherein the volume data structure represents a three-dimensional (3D) volume, the volume data structure comprising a sparse hierarchical volume data structure, and the volume data structure comprising a set of higher-level voxels, wherein each higher-level voxel represents a respective sub-volume in the 3D volume at a first resolution, and each higher-level voxel comprises a set of lower-level voxels representing a sub-volume of each higher-level voxel at a second, higher resolution; and executing a ray casting algorithm, wherein the ray casting algorithm takes the volume data structure as input, and executing the ray casting algorithm comprises: casting a plurality of rays from a reference point into the 3D volume; and for each of the plurality of rays, traversing the ray to determine whether a voxel in the voxel set is intersected and occupied by the ray, wherein the ray is traversed according to an approximate traversal.

[0312] Example 17 includes the subject matter of Example 16, wherein traversing the ray comprises determining from the volume data structure that only a subset of the set of higher-level voxels contains respective lower-level voxels occupied by the geometric shape.

[0313] Example 18 includes the subject matter of Example 17, wherein traversing the ray further comprises: determining a subset of lower level voxels intersected by the ray, wherein based on the approximate traversal, less than all lower level voxels intersected by the ray are in the subset.

[0314] Example 19 includes the subject matter of any of Examples 17-18, wherein traversing the ray comprises skipping analysis of lower-level voxels in each higher-level voxel that are determined based on the volumetric data structure to not contain occupied lower-level voxels.

[0315] Example 20 includes the subject matter of Example 19, wherein the analysis comprises checking whether lower level voxels within a subset of the higher level voxel subsets and intersecting the ray are occupied at multiple points of the ray corresponding to the higher level voxel subset, wherein the subset of lower level voxels within the higher level voxel subset and intersecting the ray is not checked based on an approximate traversal.

[0316] Example 21 includes the subject matter of Example 20, wherein the plurality of points correspond to intervals defined based on an approximate traversal.

[0317] Example 22 includes the subject matter of Example 21, wherein the defined intervals comprise distances associated with lengths of lower-level voxels in a particular dimension.

[0318] Example 23 includes the subject matter of Example 22, wherein the ray caster is further for determining a specific direction, wherein the specific dimension comprises a primary directional component of the ray.

[0319] Example 24 includes the subject matter of any of Examples 19-23, wherein analysis of the lower level voxels in each subset of higher level voxels includes accessing a memory to retrieve a corresponding portion of a volume data structure, wherein portions of the volume data structure corresponding to higher level voxels determined not to contain occupied lower level voxels are not retrieved from the memory.

[0320] Example 25 includes the subject matter of any of Examples 16-24, wherein the ray casting algorithm is part of a SLAM pipeline implemented using the circuit system.

[0321] Example 26 includes the subject matter of any of Examples 16-24, wherein the ray casting algorithm is performed in conjunction with a graphics rendering application.

[0322] Example 27 includes the subject matter of any of Examples 16-26, wherein the volumetric data structure includes a first entry for representing the presence of a geometric shape in each higher-level voxel, and further includes a second group of entries, wherein each entry in the second group of entries represents a respective lower-level voxel in one of a subset of the higher-level voxels.

[0323] Example 28 includes the subject matter of Example 27, wherein the first entry and the second entry include a binary value for each voxel described in the respective entry, and the respective binary values ​​indicate whether the corresponding voxel is occupied.

[0324] Example 29 is a system comprising means for performing the method of any one of claims 16-28.

[0325] Example 30 includes the subject matter of Example 29, wherein the apparatus comprises a machine-readable storage medium having instructions stored thereon, wherein the instructions are executable by the machine to perform at least a portion of the method of any of Examples 16-28.

[0326] Example 32 is a device comprising: a data processing device; a memory; an odometry engine executable by the data processing device to: input visual data to a first network portion of a neural network model, wherein the first network portion generates a first value based on the visual data; input inertial data to a second portion of the neural network model, wherein the second network portion generates a second value based on the inertial data; provide the first value and the second value as input to a fully connected layer group of the neural network model; and generate an output of the neural network model from the fully connected layer, wherein the output includes a posture value based on the visual data and the inertial data.

[0327] Example 32 includes the subject matter of Example 31, wherein the first network portion includes a visual odometry neural network portion and the second network portion includes an inertial odometry neural network portion.

[0328] Example 33 includes the subject matter of any of Examples 31-32, wherein the first network portion comprises a plurality of convolutional neural network layers and a first fully connected layer group.

[0329] Example 34 includes the subject matter of Example 33, wherein the first network portion comprises a layer of a convolutional neural network (CNN) based on FlowNetS.

[0330] Example 35 includes the subject matter of any of Examples 31-34, wherein the second network portion comprises a long short-term memory (LSTM) based neural network and a second fully connected layer group.

[0331] Example 36 includes the subject matter of any of Examples 31-35, wherein the visual data comprises pairs of consecutive RGB images.

[0332] Example 37 includes the subject matter of any of Examples 31-36, wherein the inertial data comprises a subsequence of inertial data generated by an inertial measurement unit (IMU) device.

[0333] Example 38 includes the subject matter of any of Examples 31-37, wherein the range measurement engine further concatenates the first and second values ​​to generate an input for the fully connected layer group.

[0334] Example 39 includes the subject matter of any of Examples 31-38, further comprising a controller to determine a direction of movement of the autonomous device based on the posture value.

[0335] Example 40 includes the subject matter of Example 39, wherein the autonomous device comprises one of a robot or a drone.

[0336] Example 41 includes the subject matter of any of Examples 39-40, further comprising an autonomous device.

[0337] Example 42 is a method comprising: receiving visual data describing an environment; receiving inertial data describing motion of a device; providing the visual data as input to a first network portion of a neural network model, wherein the first network portion generates a first value based on the visual data; providing the inertial data as input to a second portion of the neural network model, wherein the second network portion generates a second value based on the inertial data; providing the first value and the second value as input to a fully connected layer group of the neural network model; and generating an output of the neural network model from the fully connected layer, wherein the output includes a posture value based on the visual data and the inertial data.

[0338] Example 43 includes the subject matter of Example 42, wherein the first network portion includes a visual odometry neural network portion and the second network portion includes an inertial odometry neural network portion.

[0339] Example 44 includes the subject matter of any of Examples 42-43, wherein the first network portion comprises a plurality of convolutional neural network layers and a first fully connected layer group.

[0340] Example 45 includes the subject matter of Example 44, wherein the first network portion comprises a layer of a convolutional neural network (CNN) based on FlowNetS.

[0341] Example 46 includes the subject matter of any of Examples 42-45, wherein the second network portion comprises a long short-term memory (LSTM) based neural network and a second fully connected layer group.

[0342] Example 47 includes the subject matter of any of Examples 42-46, wherein the visual data comprises pairs of consecutive RGB images.

[0343] Example 48 includes the subject matter of any of Examples 42-47, wherein the inertial data comprises a subsequence of inertial data generated by an inertial measurement unit (IMU) device.

[0344] Example 49 includes the subject matter of any of Examples 42-48, further comprising concatenating the first and second values ​​to generate an input for the fully connected layer group.

[0345] Example 50 includes the subject matter of any of Examples 42-49, further comprising determining a direction of motion of the device based on the gesture value.

[0346] Example 51 includes the subject matter of Example 50, further comprising actuating movement of the device in the direction.

[0347] Example 52 is a system comprising means for performing the method of any one of claims 42-51.

[0348] Example 53 includes the subject matter of Example 22, wherein the apparatus comprises a machine-readable storage medium having instructions stored thereon, wherein the instructions are executable by the machine to perform at least a portion of the method of any of Examples 42-51.

[0349] Example 54 is a system comprising: an autonomous device comprising: a processor; an inertial measurement unit (IMU) device for generating inertial data; a camera sensor for generating visual data; an odometry engine for: providing the visual data as input to a first network portion of a neural network model, wherein the first network portion generates a first value based on the visual data; providing the inertial data as input to a second portion of the neural network model, wherein the second network portion generates a second value based on the inertial data; providing the first value and the second value as input to a fully connected layer group of the neural network model; and generating an output of the neural network model from the fully connected layer, wherein the output includes a posture value based on the visual data and the inertial data.

[0350] Example 55 includes the subject matter of Example 54, further comprising a controller for determining a direction of movement of the autonomous device based on the posture value.

[0351] Example 56 includes the subject matter of Example 55, further comprising an actuator for causing the autonomous device to move autonomously in a direction based at least in part on the posture value.

[0352] Example 57 includes the subject matter of any of Examples 54-55, wherein the first network portion includes a visual odometry neural network portion and the second network portion includes an inertial odometry neural network portion.

[0353] Example 58 includes the subject matter of any of Examples 54-57, wherein the first network portion comprises a plurality of convolutional neural network layers and a first fully connected layer group.

[0354] Example 59 includes the subject matter of Example 58, wherein the first network portion comprises a layer of a convolutional neural network (CNN) based on FlowNetS.

[0355] Example 60 includes the subject matter of any of Examples 54-59, wherein the second network portion comprises a long short-term memory (LSTM) based neural network and a second fully connected layer group.

[0356] Example 61 includes the subject matter of any of Examples 54-60, wherein the visual data comprises pairs of consecutive RGB images.

[0357] Example 62 includes the subject matter of any of Examples 54-61, wherein the inertial data comprises a subsequence of inertial data generated by an inertial measurement unit (IMU) device.

[0358] Example 63 includes the subject matter of any of Examples 54-62, wherein the range measurement engine is further for concatenating the first value and the second value to generate an input for the fully connected layer group.

[0359] Example 64 includes the subject matter of any of Examples 54-63, further comprising a controller for determining a direction of movement of the autonomous device based on the posture value.

[0360] Example 65 includes the subject matter of Example 64, wherein the autonomous device comprises one of a robot or a drone.

[0361] Example 66 is a device comprising: a data processing device; a memory for storing a first volume data structure representing a first 3D volume and a second volume data structure representing a second 3D volume; and a relative pose estimation tool executable by the data processing device, for: converting the first volume data structure into a first set of 2D grids; converting the second volume data structure into a second set of 2D grids; providing a voxel grid pair as input to a convolutional neural network (CNN), wherein the voxel grid pair includes one of the first set of 2D grids and one of the second set of 2D grids; and generating a relative pose value for the voxel grid pair using the CNN.

[0362] Example 67 includes the subject matter of Example 66, wherein the relative pose estimation tool is further used to determine a loss function for the CNN, wherein the loss function is applied in generating the relative pose value.

[0363] Example 68 includes the subject matter of any of Examples 66-67, wherein relative pose values ​​are determined for a set of 2D mesh pairs from the first set of 2D meshes and the second set of 2D meshes to determine a relative pose estimate for the first 3D volume and the second 3D volume.

[0364] Example 69 includes the subject matter of any of Examples 66-68, wherein the first volume data structure represents a point cloud of a first 3D volume and the second volume data structure represents a point cloud of a second 3D volume.

[0365] Example 70 includes the subject matter of any of Examples 66-69, wherein the CNN comprises a 2D CNN.

[0366] Example 71 includes the subject matter of any of Examples 66-70, wherein the CNN comprises a representation part and a regression part, the voxel grid pair is provided as input to the representation part, the representation part generates a feature vector based on the voxel grid pair, the feature vector is provided as input to the regression part, and the relative posture value comprises an output of the regression part.

[0367] Example 72 includes the subject matter of Example 71, wherein the representation portion comprises a Siamese network and the regression portion comprises a fully connected layer.

[0368] Example 73 includes the subject matter of any of Examples 66-72, further comprising an autonomous device.

[0369] Example 74 includes the subject matter of Example 73, wherein the autonomous device comprises one of a robot or a drone.

[0370] Example 75 is a method comprising: accessing a first volume data structure for representing a first 3D volume and a second volume data structure for representing a second 3D volume from a computer memory; converting the first volume data structure into a first set of 2D meshes; converting the second volume data structure into a second set of 2D meshes; providing the first set of 2D meshes and the second set of 2D meshes as inputs to a convolutional neural network (CNN); and generating relative pose values ​​based on the first 3D volume and the second 3D volume using the CNN.

[0371] Example 76 includes the subject matter of Example 75, further comprising: determining a loss function for the CNN; and applying the loss function to the generation of relative pose values.

[0372] Example 77 includes the subject matter of any of Examples 75-76, wherein the CNN includes a 2D convolutional layer.

[0373] Example 78 includes the subject matter of any of Examples 75-77, wherein the CNN comprises a representation part and a regression part, the first set of 2D grids and the second set of 2D grids are provided as input to the representation part, the representation part generates a feature vector as output, the feature vector is provided as input to the regression part, and the relative pose value comprises the output of the regression part.

[0374] Example 79 includes the subject matter of Example 78, wherein the representation portion comprises a Siamese network and the regression portion comprises a fully connected layer.

[0375] Example 80 is a system comprising an apparatus for performing the method as described in any of claims 75-79.

[0376] Example 81 is a non-transitory machine-readable storage medium having instructions stored thereon, which, when executed by a machine, causes the machine to: access a first volume data structure for representing a first 3D volume and a second volume data structure for representing a second 3D volume from a computer memory; convert the first volume data structure into a first set of 2D meshes; convert the second volume data structure into a second set of 2D meshes; provide the first set of 2D meshes and the second set of 2D meshes as inputs to a convolutional neural network (CNN); and use the CNN to generate relative pose values ​​based on the first 3D volume and the second 3D volume.

[0377] Example 82 includes the subject matter of Example 81, further comprising: determining a loss function for the CNN; and applying the loss function to the generation of relative pose values.

[0378] Example 83 includes the subject matter of any of Examples 81-82, wherein the CNN includes a 2D convolutional layer.

[0379] Example 84 includes the subject matter of any of Examples 81-83, wherein the CNN comprises a representation part and a regression part, the first set of 2D grids and the second set of 2D grids are provided as input to the representation part, the representation part generates a feature vector as output, the feature vector is provided as input to the regression part, and the relative pose value comprises the output of the regression part.

[0380] Example 85 includes the subject matter of Example 84, wherein the representation portion comprises a Siamese network and the regression portion comprises a fully connected layer.

[0381] Example 86 includes the subject matter of any of Examples 81-85, wherein one or both of the first volumetric data structure and the second volumetric data structure are generated based on voxelization of corresponding point cloud data.

[0382] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results.

Claims

1. A method for autonomously determining the location of a device, wherein: The method comprises: receiving visual data describing the environment; Receive inertial data describing the device's motion; providing the visual data as input to a first network portion of a neural network model, wherein the first network portion generates a first value based on the visual data; providing the inertial data as input to a second network portion of the neural network model, wherein the second network portion generates a second value based on the inertial data; providing the first value and the second value as input to a set of fully connected layers of the neural network model; and An output of the neural network model is generated from the fully connected layer, wherein the output includes a pose value based on the visual data and the inertial data.

2. The method of claim 1, wherein: The first network portion includes a visual odometry neural network portion, and the second network portion includes an inertial odometry neural network portion.

3. The method according to any one of claims 1 to 2, wherein: The first network portion includes a plurality of convolutional neural network layers and a first fully connected layer group.

4. The method as claimed in claim 3, wherein: The first network part includes layers of a convolutional neural network (CNN) based on FlowNetS.

5. The method according to any one of claims 1 to 4, wherein: The second network part includes a long short-term memory (LSTM) based neural network and a second fully connected layer group.

6. The method according to any one of claims 1 to 5, wherein: The visual data comprises pairs of consecutive RGB images.

7. The method according to any one of claims 1 to 6, wherein: The inertial data includes a subsequence of inertial data generated by an inertial measurement unit (IMU) device.

8. The method of any one of claims 1-7, further comprising concatenating the first value and the second value to generate an input for the group of fully connected layers.

9. The device of any one of claims 1-8, further comprising determining a direction of movement of the device based on the posture value.

10. The method of claim 9, further comprising actuating movement of the device in the direction.

11. A system comprising means for performing the method according to any one of claims 1 to 10.

12. The system of claim 11, wherein the apparatus comprises a machine-readable storage medium having instructions stored thereon, and the instructions are executable by a machine to perform at least a portion of the method of any one of claims 1-10.

13. A system comprising: Autonomous equipment, including: processor; Inertial measurement unit (IMU) devices for generating inertial data; Camera sensors for generating visual data; Ranging engine for: providing the visual data as input to a first network portion of a neural network model, wherein the first network portion generates a first value based on the visual data; providing the inertial data as input to a second network portion of the neural network model, wherein the second network portion generates a second value based on the inertial data; providing the first value and the second value as input to a set of fully connected layers of the neural network model; and An output of the neural network model is generated from the fully connected layer, wherein the output includes a pose value based on the visual data and the inertial data.

14. The system of claim 13, further comprising a controller for determining a direction of movement of the autonomous device based on the posture value.

15. The system of claim 14, further comprising an actuator for causing the autonomous device to move autonomously in the direction based at least in part on the posture value.

16. The system of any one of claims 13 to 15, wherein: The first network portion includes a visual odometry neural network portion, and the second network portion includes an inertial odometry neural network portion.

17. The system of any one of claims 13 to 16, wherein: The first network portion includes a plurality of convolutional neural network layers and a first fully connected layer group.

18. The system as claimed in claim 17, wherein: The first network part includes layers of a convolutional neural network (CNN) based on FlowNetS.

19. The system of any one of claims 13 to 18, wherein: The second network part includes a long short-term memory (LSTM) based neural network and a second fully connected layer group.

20. The system of any one of claims 13-19, wherein: The visual data comprises pairs of consecutive RGB images.

21. The system of any one of claims 13-20, wherein: The inertial data includes a subsequence of inertial data generated by an inertial measurement unit (IMU) device.

22. The system of any one of claims 13 to 21, wherein: The odometry engine is further configured to concatenate the first value and the second value to generate an input for the group of fully connected layers.

23. The system of any one of claims 13-22, further comprising a controller for determining a direction of movement of the autonomous device based on the posture value.

24. The system of claim 23, wherein: The autonomous device includes one of a robot or a drone.

Images