Dynamic culling of matrix operations

Abstract

An output of a first layer of the plurality of layers within the neural network is identified. A bitmap is determined from the output, the bitmap comprising a binary matrix. A particular subset of operations to skip for a second layer of the plurality of layers is determined based on the bitmap. Operations other than the particular subset of operations are performed for the second layer while skipping the particular subset of operations.

Description

This application is a divisional application of the application with PCT international application number PCT / US2017 / 047695, international application date August 19, 2017, Chinese national application number 201780064277.2, and titled “Dynamic Culling of Matrix Operations”. Related Applications

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 377,471, filed on August 19, 2016, and the U.S. Provisional Patent Application is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates generally to the field of computer systems, and more particularly, to computer vision and augmented or mixed reality systems. Background Art

[0003] With the emergence of technologies such as Magic Leap TM , Microsoft TM HoloLens TM 、0culus TM Rift TM Augmented reality (AR), virtual reality (VR) and mixed reality (MR) products, such as those from Valve TM and HTC TM With the advent of VR systems such as the VR headset, the worlds of computer vision and graphics are rapidly converging. Existing approaches in such systems use separate graphics processing units (GPUs) and computer vision subsystems running in parallel. These parallel systems can be assembled from pre-existing GPUs in parallel with computer vision pipelines implemented in software running on arrays of processors and / or arrays of programmable hardware accelerators. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The various objects, features, and advantages of the disclosed subject matter may be more fully understood with reference to the following detailed description of the disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals represent like elements. The drawings are schematic and are not intended to be drawn to scale. For clarity, not every component is labeled in every drawing. Not every component of every embodiment of the disclosed subject matter is shown, and illustration is not necessary to allow one of ordinary skill in the art to understand the disclosed subject matter.

[0005] Figure 1 shows a conventional augmented or mixed reality rendering system;

[0006] Figure 2 A voxel-based augmented or mixed reality rendering system according to some embodiments is presented;

[0007] Figure 3 Demonstrating the difference between dense volume representation and sparse volume representation according to some embodiments;

[0008] Figure 4 A composite view showing a scene according to some embodiments;

[0009] Figure 5 illustrates the level of detail in an example element tree structure according to some embodiments;

[0010] Figure 6 Demonstrating applications that can utilize the data structures and voxel data of the present application according to some embodiments;

[0011] Figure 7 An example network for recognizing 3D digits according to some embodiments is shown;

[0012] Figure 8 shows multiple classifications performed on the same data structure using an implicit level of detail according to some embodiments;

[0013] Fig. 9 Demonstrating operation elimination for a 2D convolutional neural network according to some embodiments;

[0014] Fig.10 presents experimental results from analysis of example test images according to some embodiments;

[0015] Fig.11 Hardware for culling operations according to some embodiments is shown;

[0016] Fig.12 Demonstrates refinements to hardware for culling operations according to some embodiments;

[0017] Fig.13 Hardware according to some embodiments is shown;

[0018] Fig.14 An example circuit system for generating a bitmap according to some embodiments is shown;

[0019] Fig.15 shows an example use of a bitmap associated with an example matrix multiplication operation according to some embodiments;

[0020] Fig.16 shows the use of example three-dimensional bitmaps according to some embodiments;

[0021] Fig.17 An example volume data structure is shown for use as a bitmap in accordance with some embodiments;

[0022] Fig.18shows a flowchart illustrating the generation and use of an example volume data structure according to some embodiments;

[0023] Fig.19 depicts an example multi-slot vector processor according to some embodiments;

[0024] Fig. 20 shows example volume acceleration hardware according to some embodiments;

[0025] Fig.21 illustrates the organization of a voxel cube according to some embodiments;

[0026] Fig. 22 A two-level sparse voxel tree according to some embodiments is shown;

[0027] Fig.23 A two-level sparse voxel tree according to some embodiments is shown;

[0028] Fig.24 shows storage of example voxel data according to some embodiments;

[0029] Figure 25A-25B are simplified flowcharts illustrating example techniques involving selective elimination of operations from a set of example operations;

[0030] Fig.26 is a simplified block diagram of an exemplary network having devices according to at least some embodiments;

[0031] Fig. 27 is a simplified block diagram of an exemplary fog or cloud computing network according to at least some embodiments;

[0032] Fig.28 is a simplified block diagram of a system including example devices according to at least some embodiments;

[0033] Fig.29 is a simplified block diagram of an example processing device according to at least some embodiments;

[0034] Fig.30 is a block diagram of an exemplary processor according to at least some embodiments; and

[0035] Fig.31 is a block diagram of an example computing system in accordance with at least some embodiments. DETAILED DESCRIPTION

[0036] In the following description, many specific details are set forth regarding the systems and methods of the disclosed subject matter and the environments in which these systems and methods can operate, etc., in order to provide a thorough understanding of the disclosed subject matter. However, it will be clear to those skilled in the art that the disclosed subject matter can be practiced without these specific details, and certain features well known in the art are not described in detail to avoid complicating the disclosed subject matter. In addition, it should be understood that the embodiments provided below are exemplary, and it is contemplated that other systems and methods exist within the scope of the disclosed subject matter.

[0037] Various technologies based on and in combination with augmented reality, virtual reality, mixed reality, autonomous devices and robots are emerging, which can utilize data models representing volumes of three-dimensional space and geometric structures. The description of various real and virtual environments using such 3D or volumetric data traditionally involves large data sets, and some computing systems have been struggling to process the large data sets in a desired manner. Further, as devices such as drones, wearable devices, virtual reality systems, etc. become smaller and smaller, the memory and processing resources of such devices may also be constrained. As an example, AR / VR / MR applications may require high frame rates for rendering graphics generated using supporting hardware. However, in some applications, the GPU and computer vision subsystems of such hardware may need to process data (e.g., 3D data) at high rates such as up to 130fps (7 milliseconds) in order to produce the desired results (e.g., to generate a credible graphics scene with a frame rate that produces credible results, prevents users from motion sickness caused by excessive waiting times, and other example goals). Additional applications can be similarly challenged to satisfactorily process data describing large volumes while meeting constraints in the processing, memory, power, application requirements of the corresponding system and other example problems.

[0038] In some embodiments, a computing system may be provided with logic for generating and / or using sparse volume data defined according to a format. For example, a defined volume data structure may be provided to unify computer vision and 3D rendering in various systems and applications. For example, an optical sensor such as a stereo camera or a depth camera may be used to capture a volume representation of an object. The volume representation of an object may include multiple voxels. An improved volume data structure may be defined that enables recursive subdivision of the corresponding volume representation to obtain a target resolution for the object. During subdivision, blank space in the volume representation that may be included in one or more voxels may be removed from the volume representation (and supporting operations). The blank space may be an area in the volume representation that does not include geometric properties of the object.

[0039] Thus, in the improved volume data structure, individual voxels within the corresponding volume may be marked as "occupied" (due to the presence of some geometric structure within the corresponding volume space) or as "blank" (indicating that the corresponding volume consists of blank space). Such marking may additionally be interpreted as indicating that one or more of its corresponding subvolumes are also occupied (e.g., if the parent or higher level voxel is marked as occupied) or that all of its subvolumes are blank space (i.e., in the case where the parent or higher level voxel is marked as blank). In some embodiments, marking a voxel as blank may allow the voxel and / or its corresponding subvolume voxel to be effectively removed from the operation used to generate the corresponding volume representation. The volume data structure may be based on a sparse tree structure, such as, for example, based on a sparse six-quad tree (SST) format. Further, this method for a sparse volume data structure may utilize relatively less storage space than conventionally used methods to store a volume representation of an object. Additionally, compressing the volume data may increase the feasibility of transmitting such a representation and enable faster processing of such a representation, among other example benefits.

[0040] The volume data structure can be hardware accelerated to quickly allow updates to the 3D renderer, eliminating delays that may occur in separate computer vision and graphics systems. Such delays can cause latency that can cause motion sickness to users when used in AR, VR, MR, and other applications, as well as other additional disadvantages. The ability to quickly test the geometric property occupancy of voxels in the accelerated data structure allows the construction of low-latency AR, VR, MR, or other systems that can be updated in real time.

[0041] In some embodiments, the capabilities of the volumetric data structure may also provide intra-frame warnings. For example, in AR, VR, MR, and other applications, when a user may collide with a real or synthetic object in an imaged scene, or in computer vision applications for drones or robots, when such devices may collide with a real or synthetic object in an imaged scene, the processing speed provided by the volumetric data structure allows for warnings of impending collisions.

[0042] Embodiments of the present disclosure may be related to storing and processing volume data in applications such as robots, head-mounted displays for augmented and mixed reality headsets, and phones and tablet computers. Embodiments of the present disclosure represent each voxel in a set of volume elements (e.g., voxels) and optionally physical quantities related to the geometry of the voxels as a single bit. Additional parameters related to a set of 64 voxels may be associated with voxels such as corresponding red, green, blue (RGB) or other coloring encodings, transparency, truncation signed distance functions (TSDFs), and are stored in associated and optional 64-bit data structures (e.g., so that each voxel is represented using two or more bits). This representation scheme can achieve minimal memory requirements. In addition, representing voxels by a single bit allows many simplified calculations to be performed to logically or mathematically combine elements from volume representation. Combining elements from volume representation may include, for example: performing or operations (OR-ing) on ​​planes in the volume to create 2D projections of 3D volume data, and calculating surface area by counting the number of occupied voxels in a 2.5D manifold, etc. For comparison, 64-bit subvolumes (eg, 4^3 subvolumes) may be compared using XOR logic, and volumes may be inverted, wherein objects may be merged by ORing them together to create a hybrid object, among other examples.

[0043] FIG. 1 illustrates a conventional augmented or mixed reality system consisting of a parallel graphics rendering subsystem and a computer vision subsystem with a post-rendering connection to account for changes due to rapid head movement and changes in the environment that may produce occlusions and shadows in the rendered graphics. In an example embodiment, the system may include a host processor 100 supported by a host memory 124 to control the execution of a graphics pipeline, a computer vision pipeline, and a post-rendering correction device via an interconnect via a bus 101, an on-chip network-on-chip, or other interconnect. The interconnect allows the host processor 100 to run appropriate software to control the execution of a graphics processing unit (GPU) 106, associated graphics memory 111, a computer vision pipeline 116, and associated computer vision memory 124. In one example, rendering graphics using the GPU 106 via an OpenGL graphics shader 107 (e.g., operating on a triangle list 105) can be performed at a slower rate than the computer vision pipeline. Thus, post-rendering corrections may be performed via the warp engine 108 and display / occlusion processor 109 to account for changes in head pose and occlusion scene geometry that may have occurred since the graphics were rendered by the GPU 106. Among other examples, the output of the GPU 106 is time stamped so that it can be used in conjunction with the correct control signals 121 and 123 from the head pose pipeline 120 and the occlusion pipeline 123, respectively, to produce the correct graphics output, thereby accounting for any changes in head pose 119 and occlusion geometry 113.

[0044] In parallel with the GPU 106, multiple sensors and cameras (e.g., including active stereo cameras and passive stereo cameras 117 for depth and vision processing) can be connected to the computer vision pipeline 116. The computer vision pipeline 116 can include one or more of at least three stages, each of which can contain multiple lower-level processing stages. In one example, the stages in the computer vision pipeline 116 can be an image signal processing (ISP) pipeline 118, a head pose pipeline 120, and an occlusion pipeline 122. The ISP pipeline 118 can obtain the output of the input camera sensor 117 and adjust it so that the output can be used for subsequent head pose and occlusion processing. Since the corresponding output graphics frame is rendered by the GPU 106, the head pose pipeline 120 can obtain the output of the ISP pipeline 118 and use it with the output pose. The output 121 of the head pose pipeline (HPP) 120 can be applied to the warp engine 108 together with the user-specified mesh to distort the GPU output 102 so that it matches the updated head pose position 119. The occlusion pipeline 122 can take the output of the head pose pipeline 121 and look for new objects in the field of view, such as a hand 113 (or other example object) entering the field of view, which should produce corresponding shadows 114 on the scene geometry. The output 123 of the occlusion pipeline 122 can be used by the display and occlusion processor 109 to properly overlay the field of view on the output 103 of the warp engine 108. Among other example use cases and features, the display and occlusion processor 109 uses the calculated head pose 119 to produce a shadow mask for synthesizing shadows 114, and the display and occlusion processor 109 can synthesize the occlusion geometry of the hand 113 on top of the shadow mask to produce graphical shadows 114 on top of the output 103 of the warp engine 108, and produce the final (multiple) output frames 104 for display on the augmented / mixed reality headset 110.

[0045] Figure 2 A voxel-based augmented or mixed reality rendering system according to some embodiments of the present disclosure is presented. Figure 2 The apparatus depicted in may include a host system comprised of a host CPU 200 and associated host memory 201. Such a system may communicate via a bus 204, an on-chip network, or other communication mechanism with a unified computer vision and graphics pipeline 223 and associated unified computer vision and graphics memory 213, which contains real and synthetic voxels to be rendered in a final scene for display on a head-mounted augmented or mixed reality display 211. The AR / MR display 211 may also include a plurality of active and passive image sensors 214 and an inertial measurement unit (IMU) 212 for measuring orientation changes of a head pose 222.

[0046] In a combined rendering pipeline, a composite geometry may be generated starting from a triangle list 204, which is processed by an OpenGL JiT (Just-in-Time) translator 205 to produce a composite voxel geometry 202. The composite voxel geometry may be generated, for example, by selecting a primary plane of a triangle from a triangle list. 2D rasterization may then be performed for each triangle in the selected plane (e.g., in the X and Z directions). A third coordinate (e.g., Y) may be created as an attribute to be interpolated across triangles. Each pixel of a rasterized triangle may result in the definition of a corresponding voxel. This processing may be performed by a CPU or a GPU. In addition to other example embodiments, when performed by a GPU, each rasterized triangle may be read back from the GPU to create a voxel in which the GPU draws a pixel. For example, a 2D buffer of a list may be used to generate composite voxels, wherein each entry of the list stores depth information for a polygon rendered at this pixel. For example, a model may be rendered using a positive projection viewpoint (e.g., top-down). For example, each (x, y) provided in the example buffer may represent a column at (x, y) in the corresponding voxel volume (e.g., from (x, y, 0) to (x, y, 4095)). Each column may then be rendered as a 3D scan line using the information in each list according to the information.

[0047] continue Figure 2 In some embodiments, the synthetic voxel geometry 202 can be combined with the measured geometry voxels 227 constructed using a simultaneous localization and mapping (SLAM) pipeline 217. The SLAM pipeline can use active sensors and / or passive image sensors 214 (e.g., 214.1 and 214.2), which are first processed using an image signal processing (ISP) pipeline 215 to produce an output 225, which can be converted into a depth image 226 by a depth pipeline 216. Among other examples, the active or passive image sensors 214 (214.1 and 214.2) can include active or passive stereo sensors, structured light sensors, time-of-flight sensors. For example, the depth pipeline 216 can process depth data from a structured light or time-of-flight sensor 214.1, or alternatively a passive stereo sensor 214.2. In one example embodiment, among other example embodiments, the stereo sensor 214.2 can include a passive stereo sensor pair.

[0048] The depth image generated by the depth pipeline 215 can be processed by the dense SLAM pipeline 217 using a SLAM algorithm (e.g., Kinect fusion) to produce a voxelized model of the measured geometry voxels 227. A ray tracing accelerator 206 can be provided, which can combine the measured geometry voxels 227 (e.g., real voxel geometry) with the synthetic voxel geometry 202 to produce a 2D rendering of the scene for output to a display device (e.g., a head mounted display 211 in a VR or AR application) via a display processor 210. In this embodiment, a complete scene model can be constructed from the real voxels of the measured geometry voxels 227 and the synthetic geometry 202. Therefore, there is no need to distort the 2D rendered geometry (e.g., as shown in FIG. 1). This embodiment can be combined with a head pose tracking sensor and corresponding logic to properly align the real geometry with the measured geometry. For example, the example head pose pipeline 221 may process head pose measurements 232 from an IMU 212 mounted in a head mounted display 212, and the output 231 of the head pose measurement pipeline may be taken into account via the display processor 210 during rendering.

[0049] In some examples, the unified rendering pipeline can also use the measured geometry voxels 227 (e.g., real voxel models) and the synthesized geometry 202 (e.g., synthesized voxel models) to render audio reverberation models and model the physics of real-world scenes, virtual reality scenes, or mixed reality scenes. As an example, the physics pipeline 218 can use the measured geometry voxels 227 and the synthesized geometry 202 voxel geometry and use the ray casting accelerator 206 to calculate the output audio samples of the left and right headphones in the head-mounted display (HMD) 211 to calculate the output samples 230 using the acoustic reflection coefficients built into the voxel data structure. Similarly, the unified voxel model composed of 202 and 227 can also be used to determine physical updates for synthetic objects in the composite AR / MR scene. The physics pipeline 218 takes the composite scene geometry as input and uses the ray casting accelerator 206 to calculate collisions, and then calculates updates 228 to the synthesized geometry 202 for rendering and as a basis for future iterations of the physics model.

[0050] In some embodiments, such as, among other examples, Figure 2Systems such as the illustrated system may additionally be provided with one or more hardware accelerators to implement and / or utilize a convolutional neural network (CNN) that may process RGB video / image inputs from the output of the ISP pipeline 215, volumetric scene data from the output of the SLAM pipeline 217. A neural network classifier may be run exclusively using a hardware (HW) convolutional neural network (CNN) accelerator 207 or in a combination of a processor and a HW CNN accelerator 207 to produce an output classification 237. Among other example uses, the availability of the HW CNN accelerator 207 for inference of a volumetric representation may allow groups of voxels in the measured geometry voxels 227 to be labeled as belonging to a particular object class.

[0051] Labeling voxels (e.g., using a CNN and supporting hardware acceleration) can allow those objects to which those voxels belong to be recognized by the system as corresponding to known objects, and the source voxels can be removed from the measured geometry voxels 227 and replaced by a bounding box corresponding to the object and / or information about the object's origin, the object's pose, object descriptors, and other example information. This can result in a more semantically meaningful description of the scene, which can be used, for example, as input to a robot, drone, or other computing system to interact with objects in the scene, or as input to an audio system to find the sound absorption coefficients of objects in the scene and reflect them in an acoustic model of the scene, among other example uses.

[0052] One or more processor devices and hardware accelerators may be provided to implement Figure 2 In some embodiments, among other example embodiments, all hardware and software elements of the combined rendering pipeline may share access to a DRAM controller 209 , which in turn allows data to be stored in a shared DDR memory device 208 .

[0053] Figure 3 is presented to illustrate the difference between dense volume representation and sparse volume representation according to some embodiments. Figure 3As shown in the example of , a real-world or synthetic object 300 (e.g., a statue of a rabbit) can be described in terms of voxels in a dense manner as shown in 302 or in a sparse manner as shown in 304. A dense representation such as 302 has the advantage of uniform access speed to all voxels in the volume, but has the disadvantage of the amount of storage that may be required. For example, for a dense representation such as a 512^3 element volume (e.g., for a volume scanned using a Kinect sensor, 1 cm resolution corresponds to 5m), 512M bytes are used to store a relatively small volume, with a 4-byte truncated signed distance function (TSDF) per voxel. On the other hand, an octree representation 304 embodying a sparse representation can store only those voxels for which actual geometry exists in the real-world scene, thereby reducing the amount of data required to store the same volume.

[0054] Go to Figure 4 , showing a composite view of an example scene according to some embodiments. Specifically, Figure 4 It shows how a composite view of a scene 404 may be maintained, displayed, or subjected to further processing using parallel data structures to represent synthetic voxels 401 and real-world measured voxels 403 within equivalent bounding boxes 400 and 402 of synthetic voxel data and real-world voxel data, respectively. Figure 5 shows the level of detail in a uniform 4^3 element tree structure according to some embodiments. In some implementations, as little as 1 bit can be used to describe each voxel in a volume represented using an octree, such as in Figure 5 However, a disadvantage of octree-based techniques can be the number of indirect memory accesses used to access a specific voxel in the octree. In the case of sparse voxel octrees, the same geometry can be implicitly represented at multiple levels of detail that advantageously allow operations such as ray casting, game physics, CNNs and other techniques to allow empty parts of the scene to be culled from further calculations, resulting in not only a reduction in required storage, but also an overall reduction in power consumption and computational load, among other example advantages.

[0055] In one embodiment, an improved voxel descriptor (also referred to herein as a "volume data structure") may be provided to organize volume information as 4^3 (or 64-bit) unsigned integers, such as shown in 501, where the memory requirement per voxel is 1 bit. In this example, 1 bit per voxel is insufficient to store truncated signed distance function values ​​(compared to the TSDF in the SLAM platform / K fusion that utilizes 64 bits). In this example, an additional (e.g., 64-bit) field 500 may be included in the voxel descriptor. This example may be further enhanced so that although the TSDF in the 64-bit field 500 is 16 bits, an additional 2 bits of fractional resolution in x, y, and z may be implicitly provided in the voxel descriptor 501 so that the combination of the voxel TSDF in the 64-bit field 500 and the voxel position 501 is equivalent to a higher resolution TSDF such as used in the SLAM platform / K fusion or other examples. For example, among other examples, additional data in the 64-bit field 500 (voxel descriptor) can be used to store subsampled RGB color information (e.g., from a scene via a passive RGB sensor), one byte per piece of information, and having an 8-bit transparency value α and two 1-byte reserved fields R1 and R2, which can be application-specific and can be used to store, for example, acoustic reflectivity for audio applications, rigidity for physical applications, object material type, etc.

[0056] like Figure 5 As shown, the voxel descriptors 501 can be logically grouped into four 2D planes, each containing 16 voxels 502. These 2D planes (or voxel planes) can describe each level of the octree-style structure based on a continuous decomposition of ascending powers of 4, such as Figure 5 . In this example implementation, 64-bit voxel descriptors are selected because they match well with the 64-bit bus infrastructure used in the corresponding system implementation (although other voxel descriptor sizes and formats may be provided in other system implementations and sized according to the bus or other infrastructure of the system). In some implementations, the voxel descriptors may be sized to reduce the number of memory accesses used to obtain voxels. For example, among other example considerations and implementations, 64-bit voxel descriptors may be used to reduce the number of memory accesses required to access a voxel at any level in an octree by a factor of 2 compared to a conventional octree operating on 2^3 elements.

[0057] In one example, an octree may be described starting with a 4^3 root volume 503, and each non-zero entry in which a code for the presence of geometry in bottom layers 504, 505, and 506 is depicted in the example 256^3 volume. In this particular example, four memory accesses may be used to access the lowest level in the octree. In cases where this overhead is too high, an alternative approach may be taken to encode the highest level of the octree as a larger volume such as 64^3, as shown in 507. In this case, each non-zero entry in 507 may indicate the presence of a bottom layer 4^3 octree in the bottom layer 256^3 volume 508. As a result of this alternative organization, only two memory accesses are required to access any voxel in the 256^3 volume 508, compared to the alternative formulations shown in 503, 504, and 505. The latter approach is advantageous in cases where the device hosting the octree structure has a large amount of embedded memory, thereby allowing only the lower and less frequently accessed portions of the voxel octree 508 to be in external memory. This approach may be more expensive in terms of storage (e.g., where the entire, larger (e.g., 64^3) volume is stored in on-chip memory), but the tradeoff may allow faster memory access (e.g., 2x) and lower power consumption, among other example advantages.

[0058] Go to Figure 6 , shows a block diagram illustrating an example application that can utilize the data structure and voxel data of the present application according to some embodiments. In one example, such as Figure 5 In the example shown, additional information may be provided by the example voxel descriptor 500. Although the voxel descriptor may increase the total memory utilized to 2 bits per voxel, the voxel descriptor may enable a wide range of applications that may utilize voxel data, such as Figure 6 . For example, a shared volume representation 602, such as generated using a dense SLAM system 601 (e.g., a SLAM platform), can be used to render a scene using graphics raycasting or ray tracing 603, for audio raycasting 604, and other embodiments. In still other examples, the volume representation 602 can also be used for convolutional neural network (CNN) inference 605 and can be backed up by a cloud infrastructure 607. In some instances, the cloud infrastructure 607 can include detailed volume descriptors of objects such as trees, furniture, or other objects (e.g., 606) that can be accessed via inference. Based on inference or otherwise identifying an object, a corresponding detailed descriptor can be returned to the device, allowing voxels of the volume representation 602 to be replaced by a bounding box representation with pose information and a descriptor containing object attributes and other example features.

[0059] In still other embodiments, the voxel model discussed above may be additionally or alternatively utilized in some systems to construct a 2D map of an example environment 608 using a 3D to 2D projection from a volume representation 602. These 2D maps may again be shared and aggregated (e.g., using the same cloud infrastructure) via a communication machine, via a cloud infrastructure, and / or other network-based resources 607 to build higher quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices. In still further examples, for ultra-low bandwidth applications, a 2D map may be refined using projection and then piecewise simplification 609 (e.g., assuming a fixed width and height of a vehicle or robot). The simplified path may then have only a single X, Y coordinate pair for each piecewise linear segment of the path, thereby reducing the amount of bandwidth required to communicate the path of the vehicle 609 to the cloud infrastructure 607, and the simplified path may be aggregated in this same cloud infrastructure 607 to build higher quality maps using crowdsourcing techniques. These maps may be shared by the cloud infrastructure 607 to connected machines and devices.

[0060] To enable these different applications, in some embodiments, common functionality may be provided, such as through a shared software library, which may be accelerated using hardware accelerators or processor instruction set architecture (ISA) extensions and other examples in some embodiments. For example, such functionality may include inserting voxels into a descriptor, deleting voxels, or finding voxels 610. In some embodiments, collision detection functionality 620, and deleting points / voxels from a volume 630, and other examples may also be supported. As described above, the system may be provided with functionality for quickly generating 2D projections 640 in the X, Y, and Z directions based on a corresponding volume representation 602 (3D volume), which may be used as a basis for path or collision determination. In some cases, it may also be advantageous to be able to generate a triangle list from a volume representation 602 using a histogram pyramid 650. Further, the system may be provided with functionality for quickly determining a free path 660 in a 2D and 3D representation of a volume space 602. This functionality may be useful in various applications. Further functionality may be provided, such as detailing the number of voxels in the volume, counting the number of 1 bits in a mask region of the volume representation 602 using a population counter to determine the surface of the object, and other examples.

[0061] Go to Figure 7 , showing an example network including a system equipped with 3D digital recognition capabilities according to at least some embodiments. For example, Figure 6 One of the applications shown is a volumetric CNN application 605, which Figure 7, where an example network is used to recognize 3D digits generated from a dataset such as the Mixed National Institute of Standards and Technology (MNIST) dataset 700. By applying appropriate rotations and translations in X, Y, and Z to the digits in such a dataset prior to training, the digits can be used to train a CNN-based convolutional network classifier 710. When used for inference in an embedded device, the trained network 710 can be used to classify 3D digits in a scene with high accuracy, even when the digits are subjected to rotations and translations in X, Y, and Z, among other examples 720. In some embodiments, the operation of the CNN classifier can be performed by Figure 2 6. When the first layer of the neural network performs multiplications using voxels in the volume representation 602, these arithmetic operations can be skipped because multiplication by zero is always zero and the multiplication of a data value A by one (voxel) is equal to A.

[0062] Figure 8 Multiple classifications are shown for the same data structure using implicit levels of detail. Further refinement of the CNN classification using volume representation 602 can be: Figure 5 The octree representation in the octree structure shown implicitly contains multiple levels of detail, so a single classifier 830 or multiple parallel classifiers can be used to perform multiple classifications on the same data structure in parallel using the implicit levels of detail 800, 810 and 820, such as Figure 8 As shown. In conventional systems, fairly parallel classification can be slow because the image needs to be resized between classification channels. This resizing can be abandoned in embodiments that apply the voxel structure discussed herein, because the same octree can contain the same information at multiple levels of detail. In fact, a single training dataset based on the volumetric model can cover all levels of detail instead of resizing the training dataset as would be required in a conventional CNN network.

[0063] Go to Fig. 9 An example of operation elimination is shown using a 2D CNN according to some embodiments. Operation elimination can be used on a 3D volume CNN as well as on a 2D CNN, such as Fig. 9 For example, in Fig. 9 In the first layer, a bitmap mask 900 can be used to describe the expected "shape" of the input 910 and can be applied to the incoming video stream 920. In one example, the operation elimination can be used not only on the 3D volume CNN but also on the 2D volume CNN. Fig. 9In the example 2D CNN of , a bitmap mask 900 can be applied to the first layer of the CNN to describe the expected "shape" of the input 910, and can be applied to the input data of the CNN (such as, the incoming video stream 820). As an example, Fig. 9 The effect of applying a bitmap mask to an image of a pedestrian for training or inference in a CNN network is shown in 901, where 901 represents the original image of pedestrian 901 and 903 represents the corresponding version with the bitmap mask applied. Similarly, an image without a pedestrian is shown in 902, and the corresponding bitmap mask version is shown in 904. The same method can be applied to any type of 2D or 3D object to reduce the number of operations required for CNN training or inference by understanding the expected 2D or 3D geometry expected by the detector. An example of a 3D volume bitmap is shown in 911. The use of a 2D bitmap for inference in a real scene is shown in 920.

[0064] exist Fig. 9 In an example embodiment of the invention, a conceptual bitmap is shown (at 900), while a real bitmap is generated by averaging a series of training images of a particular classification of objects 910. The example shown is two-dimensional, but similar bitmap masks can also be generated for 3D objects using the proposed volume data format with one bit per voxel. In fact, the method can potentially be extended to use additional bits per voxel / pixel to specify the expected color range or other characteristics of 2D or 3D objects, among other example embodiments.

[0065] Fig.10 is a table showing the results of an example experiment involving analysis of 10,000 CIFAR-10 test images according to some embodiments. In some embodiments, due to the frequent use of rectified linear unit (ReLU) operations in CNN networks such as LeNet 1000, operation elimination can be used to eliminate intermediate computations in 1D, 2D, and 3D CNNs, such as Fig.10 As shown. Fig.10 As shown, in an experiment using 10,000 CIFAR-10 test images, the percentage of data-related zeros generated by the ReLU unit can reach 85%, which means that in the case of zeros, a system that recognizes zeros and does not fetch corresponding data and perform corresponding multiplication operations in response can be provided. In this example, 85% represents the percentage of ReLU dynamic zeros generated according to the modified National Institute of Standards and Technology database (MNIST) test data set. The elimination of corresponding operations corresponding to these zeros can be used to reduce power consumption and memory bandwidth requirements, as well as other example benefits.

[0066] Trivial operations may be eliminated based on a bitmap. For example, the use of such a bitmap may be based on the principles and embodiments discussed and described in U.S. Pat. No. 8,713,080, entitled "Circuit for compressing data and a processors using the same," which is incorporated herein by reference in its entirety. Some embodiments may provide hardware capable of using such a bitmap, such as systems, circuit systems, and other embodiments discussed and described in U.S. Pat. No. 9,104,633, entitled "Hardware for performing arithmetic operations," which is incorporated herein by reference in its entirety.

[0067] In one example implementation, a computing system may be equipped with logic and / or executable firmware and / or software logic implemented in hardware circuitry for obtaining a bitmap corresponding to data that will be the subject of one or more operations. The operation may include an arithmetic operation (e.g., multiplication, addition, etc.) to be performed on the data, wherein the data serves as an operand (or multiple operands). Alternatively (or additionally), the operation may include fetching the data. For example, the data may embody a three-dimensional (3D) matrix composed of multiple matrix elements. Matrix arithmetic may be performed on the 3D matrix. Further, the individual elements of the matrix may be fetched from memory (e.g., to perform the corresponding arithmetic operation). In some implementations, the bitmap may be based on the trivial property of zero multiplication (i.e., multiplication by operand "0" always produces a product "0"). The bitmap may be identified or correspond to a 3D matrix element with a value "0". Processing the bitmap (e.g., by enabled hardware) may allow actual multiplication operations involving zero operands (and even the fetching of operands to be multiplied) to be skipped. Specifically, 3D matrix multiplication may involve element-by-element multiplication of elements in a 3D matrix-non-zero elements may be multiplied as usual, while multiplication steps involving zero elements are skipped. In some cases, the expected result of the multiplication step (i.e., in the case where one of the operands is "0", the expected result is also "0") may be simply provided by the logic of the system (e.g., using a multiplexer configured to return zero when such an operation is identified from a corresponding bitmap), rather than deriving the result by the multiplication operation performed. In some cases, a bitmap may be pre-derived and provided to a system controller so as to utilize the bitmap to eliminate, skip or eliminate some operations to be performed. In other instances, a bitmap may be dynamically generated during the operation of the system and based on the results of a prior operation. For example, a bitmap may be applied in a convolutional neural network (CNN), where the result of one CNN network serves as the basis for a bitmap adopted in the operation of a subsequent CNN layer, as well as other examples.

[0068] Fig.11 Hardware that can be incorporated into a system to provide functionality for eliminating trivial operations based on a dynamically generated bitmap according to some embodiments is shown. In this example, a multi-layer neural network including repeated convolutional layers is provided. The hardware may include one or more processors, one or more microprocessors, one or more circuits, one or more computers, etc. In this particular example, the neural network includes an initial convolution processing layer 1100, followed by a pooling process 1110, and finally an activation function process (such as a rectified linear unit (ReLU) function 1120). The output of the ReLU unit 1120, which provides a ReLU output vector 1131, can be connected to a subsequent convolution processing layer 1180 (e.g., possibly via a delay 1132), which receives the ReLU output vector 1131. In an example implementation, a ReLU bitmap 1130 can also be generated in parallel with the connection of the ReLU unit 1120 to the subsequent convolution unit 1180, and the ReLU bitmap 1130 indicates which elements in the ReLU output vector 1131 are zero and which are non-zero.

[0069] In one embodiment, a bitmap (e.g., 1130) may be generated or otherwise provided to inform enabled hardware of opportunities to eliminate operations involved in the computation of a neural network. For example, the bits in the ReLU bitmap 1130 may be interpreted by the bitmap scheduler 1160, which instructs the multipliers in the subsequent convolutional unit 1180 to skip zero entries in the ReLU output vector 1131 for which there are corresponding binary zeros in the ReLU bitmap 1130, given that multiplication by zero will always produce zero as output. In parallel, memory fetches of data / weights corresponding to zeros in the ReLU bitmap 1130 may also be skipped from the address generator 1140, since there is little value in fetching weights that will be skipped by the subsequent convolutional unit 1180. If the weights were to be fetched from the attached DDR DRAM storage device 1170 via the DDR controller 1150, the latency may be so high that only some on-chip bandwidth and associated power consumption may be saved. On the other hand, if the weights are fetched from the on-chip RAM 1180 storage device, the entire weight fetch operation can be bypassed / skipped, especially if a delay corresponding to the RAM / DDR fetch delay 1132 is added to the subsequent convolution unit 1180 at the input.

[0070] Go to Fig.12 , a simplified block diagram is provided to illustrate a refinement of example hardware equipped with circuitry and other logic for eliminating trivial operations (or performing operation elimination) according to some embodiments. Fig.12As shown in the example of , additional hardware logic may be provided to predict the sign of the ReLU unit 1220 input in advance from a preceding max pooling unit 1210 (corresponding to a max pooling layer in a CNN) or a convolution unit 1200 (corresponding to a convolution layer in a CNN). Adding sign prediction and ReLU bitmap generation to the max pooling unit 1210 may allow the ReLU bitmap information to be predicted earlier from a timing perspective to cover delays that may occur through the address generator 1240, through an external DDR controller 1250 and DDR memory 1270, or internal RAM memory 1271. If the latency is low enough, the ReLU bitmap may be interpreted in the address generator 1240, and the memory fetch associated with the ReLU bitmap zero may be skipped entirely, since it may be determined that the result of the fetch from the memory will never be used. In addition to other example features and functionality, Fig.11 This modification of the scheme may save additional power and may also allow the removal of delay stages (e.g., 1132, 1232) at the input to a subsequent convolution unit 1280 if the latency through the DDR access path (e.g., 1240 to 1250 to 1270) or the RAM access path (e.g., 1240 to 1271) is low enough that delay stage 1232 is not warranted.

[0071] Fig.13 is another simplified block diagram illustrating example hardware according to some embodiments. For example, a CNN ReLU layer may produce a large number of output zeros corresponding to negative inputs. In fact, this can be seen by looking at the previous layers (e.g., Fig.13 Floating point and integer operations can be explicitly signed based on the most significant bit (MSB), so a simple bitwise exclusive OR (XOR) operation across the input vectors to be multiplied in the convolutional layer can predict which multiplications will produce output zero, such as in Fig.13 The resulting sign-predicted ReLU bitmap vector can be used as a basis for determining a subset of multiplications and associated coefficients to read from memory for elimination such as described in other examples above.

[0072] Providing the generation of the ReLU bitmap back to a previous pooling or convolution stage (i.e., a stage prior to the corresponding ReLU stage) may yield additional power and time savings. For example, sign prediction logic may be provided to disable the multiplier when it would produce a negative output, which would ultimately be set to zero by the ReLU activation logic. For example, this shows the case where the two sign bits 1310 and 1315 of the inputs 1301 and 1302 of the multiplier 1314 are logically combined by an XOR gate to form the PreReLU bitmap bit 1303. Among other examples, this same signal may be used to disable the operation of the multiplier 1314, which would otherwise unnecessarily consume energy generating a negative output, which would be set to zero by the ReLU logic before being input for multiplication in the next convolution stage 1390.

[0073] It should be noted that the representations 1300, 1301, 1302 and 1303 (labeled A) show Fig.13 A higher level view shown in the representation marked B in FIG. 1302. In this example, the input to block 1302 may include two floating point operands. Input 1301 may include: an explicit sign bit 1310, a mantissa 1311 including a plurality of bits, and an exponent again including a plurality of bits 1312. Similarly, input 1302 may also include a sign 1315, a mantissa 1317, and an exponent 1316. In some embodiments, the mantissa and the exponent may have different precisions, because the sign of the result 1303 depends only on the signs of 1301 and 1302, or 1310 and 1315, respectively. In fact, neither 1301 nor 1302 need to be floating point numbers, but may be any integer or fixed point format as long as they are signed numbers and the most significant bit (MSB) is actually an explicit or implicit sign bit (e.g., if the number is one's or two's complement, etc.).

[0074] continue Fig.13, the two sign inputs 1310 and 1315 may be combined using an exclusive-OR (XOR, sometimes alternatively denoted herein as ExOR or EXOR) gate to generate a bitmap bit 1303, which may then be processed using hardware to identify downstream multiplications that may be omitted in the next convolution block (e.g., 1390). In the case where the two input numbers 1313 (e.g., corresponding to 1301) and 1318 (e.g., corresponding to 1302) have opposite signs and would produce a negative output 1304, the same exclusive-OR output 1303 may also be used to disable the multiplier 1314, which will be set to zero by the ReLU block 1319, resulting in a zero value in the ReLU output vector 13191, which will be input to the subsequent convolution stage 1390. Thus, in some embodiments, the PreReLU bitmap 1320 may be transmitted in parallel to a bitmap scheduler 1360, which may schedule multiplications to be run (and / or omitted) on the convolution unit 1390. For example, for each zero in the bitmap 1320, the corresponding convolution operation can be skipped in the convolution unit 1390. In parallel, the bitmap 1320 can be consumed by the example address generator 1330, which controls the fetching of weights used in the convolution unit 1390. A list of addresses corresponding to ones in the bitmap 1320 can be compiled in the address generator 1330 and control the path to the DDR memory device 1370 via the DDR controller 1350, or to the on-chip RAM 1380. In either case, the weight corresponding to one in the PreReLU bitmap 1320 can be fetched and presented (e.g., after a certain wait time with respect to the clock cycle of the weight input 1371) to the convolution block 1390, while the fetching of weights corresponding to zero can be omitted, among other examples.

[0075] As described above, in some embodiments, a delay (e.g., 1361) may be inserted between the bitmap scheduler 1360 and the convolution unit 1390 to balance the delays through the address generator 1330, the DDR controller 1350, and the DDR 1350, or to balance the path through the address generator 1330 and the internal RAM 1380. The delay may enable the convolution driven by the bitmap scheduler to be properly aligned in time with the corresponding weights of the convolution calculation in the convolution unit 1390. In fact, from a timing perspective, generating the ReLU bitmap earlier than at the output of the ReLU block 1319 may allow for additional time that may be used to intercept the read to the memory (e.g., RAM 1380 or DDR 1370) before it is generated by the address generator 1330, so that some reads (e.g., corresponding to zeros) may be discarded. Among other example advantages, since memory reads may be much higher than on-chip logic operations, eliminating such memory fetches may result in very significant energy savings.

[0076] In some embodiments, if there is still not enough savings in clock cycles to cover the DRAM access time, a block-oriented technique can be used to read groups of sign bits (e.g., 1301) from the DDR in advance. These groups of sign bits can be used together with the signs from the input image or the intermediate convolutional layer 1302 to generate multiple parts of the PreReLU bitmap using blocks 1300 of XOR gates. In such an embodiment, an additional 1-bit storage in the DDR or on-chip RAM can be provided to store the sign of each weight, but this can allow many latency cycles to be covered in a way that reading weights from the DDR or RAM that will multiply the ReLU stage by zero is never avoided. In some embodiments, among other example considerations and embodiments, the additional 1-bit storage per weight in the DDR or on-chip RAM can be avoided because the signs are stored in such a way that they are independently addressable from the exponent and mantissa.

[0077] In one example, the system can be further enhanced to take advantage of DDR accesses, which can perform natural burst accesses to achieve maximum data transfer rates. Saving energy by skipping individual DDR weight accesses may not be feasible in this context because the weight accesses may be shorter than the burst. Therefore, in some instances, the burst may be skipped when all bitmap bits corresponding to a particular burst transaction are zero. However, this may not occur frequently, and the resulting power savings and bandwidth savings may be limited. In still other embodiments, a register programmable threshold may be set for the number of bitmap bits in a burst, so that if more than N bits in the bitmap burst are zero, the burst will be skipped entirely. This may have the effect of slightly reducing the overall CNN classification accuracy, but may be acceptable for energy conservation.

[0078] Go to Fig.14 , a simplified block diagram 1400 is provided to illustrate an example implementation of logic for generating a neural network based on an example neural network (such as, Figure 11-13 1400 ). In this representation 1400 , operand A 1405 is provided as input to the ReLU layer. Comparator 1410 may identify when the sign of A is equal to or less than 0. In a ReLU operation, if A is less than or equal to 0, the output of the ReLU operation will be 0. On the other hand, if A is greater than 0, the value of A is returned as a result (e.g., using multiplexer 1415 ). In one example, additional circuitry may be provided to tap the output of comparator 1410 (which is used to generate a true binary “1” or a false binary “0” based on whether A is greater than “0”) so that (e.g., for each element A of a 2D or 3D matrix) the output of comparator 1410 provides a corresponding binary value 1420 for incorporation into the bitmap for use in excluding operations involving “0” elements in the resulting ReLU output matrix 1425.

[0079] Additionally or alternatively (in addition to or alternatively to the circuitry for providing the bitmap value from comparator 1410), some embodiments may support pre-ReLU bitmap generation (such as described above in Fig.13 1450). For example, as listed in table 1430, the sign of A may be predicted based on the sign of the operands (e.g., in a preceding multiplication operation) to generate A. Accordingly, an XOR gate 1435 may be provided that may receive as input at least a portion of the inputs X 1440 and Y 1445 to the multiplication operation 1450. For example, a sign bit (e.g., MSB) of each of the inputs 1440, 1445 may be provided to the XOR gate 1435 to generate a "0" when there is a sign mismatch (meaning that the resulting product A will be negative) and to generate a "1" when the sign matches (and a positive value of A will result from the multiplication operation 1450). The output 1455 of the XOR gate 1435 may thus be provided to generate a bitmap bit that is identical to what will be generated from the output 1420, as well as other example embodiments.

[0080] Fig.151500 is a simplified block diagram showing the principle of use of an example bitmap according to at least some of the embodiments described herein. For example, a matrix multiplication (e.g., between matrix A (1505) and B (1510)) may involve element-by-element multiplication of the elements of the two matrices 1505, 1510 to generate the resulting matrix C (1515). In this example, a bitmap matrix 1520 may be provided, which corresponds to matrix A and indicates whether the element is "0" for each element in matrix A (e.g., A-1). The bitmap 1520 may be processed by a scheduler or other controller logic to determine that only the multiplication operations of non-zero matrix A elements (e.g., elements A, F, G, and H) will be performed, thereby allowing the multiplication operations involving the remaining "0" elements (e.g., B, C, D, E, I) to be skipped. In addition to skipping the multiplication operation itself, it is also possible to skip taking out zero elements from memory (because there is actually no reason to take out the element in view of the skipping of the corresponding multiplication operation (multiple) of the element). The fetching of only non-zero elements may result in a compressed register (e.g., 1525) from which the multiplication circuitry will pull elements for the remaining (non-skipped) multiplication operations, as in Fig.15 Indicated in .

[0081] Go to Fig.16 , a simplified block diagram 1600 is shown, which illustrates an example implementation of a bitmap 1605 for a 3D matrix 1610 (or other matrix having a dimension greater than 2). For example, in this example, the 3D matrix 1610 may correspond to the red layer, green layer, and blue layer of a 2D image. In order to implement a corresponding bitmap such as discussed above, a 3D bitmap 1605 may be generated to correspond to the 3D matrix 1610 and identify the elements within the 3D matrix 1610 that are "0". For example, a "1" value in the 3D bitmap 1605 may indicate a non-zero value in the corresponding element of the 3D matrix 1610, while a "0" in the bitmap 1605 may correspond to a 0 element in the matrix 1610 (although in other implementations, the opposite convention may apply). For other bitmap implementations, a mapping may be defined to indicate which bit of the bitmap 1605 corresponds to which element in the corresponding matrix 1610. In this example, based on the mapping, only the non-zero elements of matrix 1610 can be fetched from memory. In one example, each plane (e.g., 1635, 1640, 1645) within bitmap matrix 1605 can be mapped to a corresponding plane (e.g., 1620, 1625, 1630) in a corresponding 3D matrix.

[0082] A compressed register 1615 may be constructed using a 3D bitmap in conjunction with scheduling of multiplication operations involving non-zero elements of the 3D matrix 1610. In one example, non-zero elements of each plane (e.g., 1620, 1625, 1630) may be added to the register 1615 in succession. The number of "1s" in each plane (e.g., 1635, 1640, 1645) may be counted to identify the number of entries to be provided to the compressed register, and an offset address (e.g., 1650) may be determined to identify the start of the bitmap for a particular one of the planes of the 3D matrix. For example, the offset address 1650 may be determined based on the base address 1655 and the size (non-zero value) in the corresponding bitmap plane. For example, offset_base_address(1650)=base_addr(1655)+sizeof(BmA)+sizeof(BmB)+sizeof(BmC). In this way, a sequence of "1"s from each bitmap plane (e.g., 1635, 1640, 1645) can be used to identify offsets for accessing corresponding matrix elements to be provided to a multiplication circuit system using principles such as those discussed in the following patents: U.S. Patent No. 8,713,080, entitled "Circuit for compressioning data and a processors using the same"; and / or U.S. Patent No. 9,104,633, entitled "Hardware for performing arithmetic operations".

[0083] Go to Fig.17, a simplified block diagram 1700 is shown, which illustrates the use of a volume data structure as a bitmap for eliminating insignificant operations within a set of operations involving the volume data structure. For example, a volume data structure may represent a 3D space, and thus a multidimensional matrix may be embodied. The volume data structure may be considered to be a dynamically generated bitmap, because logic may be provided in a device (e.g., a controller, an autonomous vehicle, a robot, etc.) to generate and update a volume data structure to reflect image data captured by one or more sensors or scanners (e.g., using LIDAR, a stereo camera, a radar, ultrasound, etc.). The image data may be provided to the SLAM pipeline logic. As new images are provided, the output of the SLAM logic may be similarly continuously updated, and new geometries may be identified, and as new geometry results are acquired, new geometries may be dynamically added or inserted into (new or existing) volume data structures. In fact, a depth map may be dynamically generated in response to the acquired imaging data, and represented in a volume data structure format based on SST. The binary nature of the bits included in the integer value of a volume data structure entry can be used as a bitmap (e.g., to identify which downstream weights are to be fetched from DDR memory and which convolutions or other arithmetic operations are to be performed in response to multiple "1" bits in the volume data structure bitmap, while operations corresponding to "0" in the bitmap result are eliminated).

[0084] For example, in Fig.17 , a volume data structure may represent a volume that is subdivided into four parts in X-coordinates, Y-coordinates, and Z-coordinates. In other embodiments, the volume data structure may represent the volume using a Cartesian coordinate system. In this example, the volume data structure may correspond to the division of a particular volume into 64 cubic sub-volumes or voxels. Binary values ​​can be used to identify whether a corresponding one of the sub-volumes is occupied by a geometric structure (e.g., "1") or unoccupied (e.g., "0"), resulting in (at least) 64 bits of 1s and 0s to represent the 3D coordinates within the particular volume. In this example, the 4x4x4 division of the volume represented by the volume data structure actually produces a four-dimensional bitmap (or "VOLA") corresponding to the 64 binary values ​​of the volume data structure 1705.

[0085] The volume data structure 1705 may be operated on in conjunction with various applications. For example, matrix arithmetic operations may be performed on a volume data structure (e.g., comprising 64-bit integer values ​​corresponding to the bits of 64 binary values ​​for representing the presence of geometric structure or empty space within each voxel). The binary nature of these values ​​may allow the same volume data structure 1705 to be used as a 4D bitmap that may be processed by the scheduler circuitry to identify operations that may be skipped (e.g., multiplication operations involving "0" elements within the volume data structure 1705). As an example, in the case where the volume data structure 1705 is to be multiplied by another matrix (e.g., the weight matrix 1710, which may be used in a convolution operation of a CNN), the VOLA-based bitmap may also be used to identify which elements from the other matrix are to be fetched from memory, thereby omitting the fetching of elements to be multiplied by the "0" elements of the volume data structure 1705.

[0086] Briefly go to Fig.18 1800, in some embodiments, a VOLA-based bitmap may be dynamically generated as a volume data structure 1705 is generated from "raw" data captured from a scan or resolution of a particular volume. As an illustrative example, an optical sensor 1715 such as a stereo camera or a depth camera may generate data that may be provided to a SLAM pipeline 1720 to generate a volume data structure 1705. In this example, the volume data structure 1705 may be provided as both an input to a CNN 1725 and an input to a memory controller 1730 (e.g., a memory controller for a double data rate (DDR) memory 1735). Matrix multiplication operations may be arranged to be performed on the volume data structure 1705 in conjunction with the layers of the CNN 1725. To enhance the performance of these operations, the volume data structure 1705 may also be used by the memory controller 1730 as a multi-dimensional (i.e., greater than 2D) bitmap to select and retrieve only those elements of the corresponding weight (or other) matrix (e.g., 1710) to be multiplied by the non-zero elements of the volume data structure.

[0087] Furthermore, if Fig.17As shown in the example of , in some embodiments, a bitmap may be provided or generated for each of two (or more) multidimensional matrices to be multiplied together in an application. For example, in the case where a first input matrix (e.g., volume data structure 1705) is multiplied with a weight matrix (e.g., 1710), the two matrices may have corresponding bitmaps for being provided to a scheduler, memory controller, or other control logic to selectively omit or skip some of the operations that would otherwise be performed to complete the multiplication of the two or more matrices. In this example, since the volume data structure 1705 can act as its own bitmap, an additional multidimensional (e.g., 3D) bitmap matrix 1750 may be provided to indicate "0" values ​​in the weight matrix 1710. Both the VOLA-based bitmap and the bitmap 1750 may be provided to the scheduler to select only those multiplication operations where both operands are non-zero, and to escape all multiplication operations involving one or more zero elements (and corresponding memory fetches from either of the matrices 1705, 1710). For example, the packed registers 1755 may be populated with weight values ​​(from the matrix 1710) for scheduling multiplication operations with the volume data structure 1705, among other example implementations.

[0088] Fig.19 1 is a simplified block diagram showing an example multi-slot vector processor (e.g., a very long instruction word (VLIW) vector processor) according to some embodiments. In this example, the vector processor may include multiple (e.g., 9) functional units (e.g., 1903 to 1911), which may be fed by a multi-port memory system 1900, supported by a vector register file (VRF) 1901 and a general register file (GRF) 1902. The processor includes an instruction decoder (IDEC) 1912, which decodes instructions and generates control signals for controlling the functional units 1903 to 1911. Functional units 1903 to 1911 are a predicate execution unit (PEU) 1903, a branch repeat unit (BRU) 1904, a load store port unit (e.g., LSU0 1905 and LSU1 1906), a vector operation unit (VAU) 1907, a scalar operation unit (SAU) 1910, a compare move unit (CMU) 1908, an integer operation unit (IAU) 1911, and a volume acceleration unit (VXU) 1909. In this particular embodiment, the VXU 1909 can accelerate operations on volume data, including storage / retrieval operations, logic operations, and operation operations. Although in Fig.19In the example of FIG. 1 , the VXU circuit system 1909 is shown as a single component, but it should be understood that the functionality of the VXU (and any other functional units 1903 to 1911) can be distributed among multiple circuit systems. Further, in some embodiments, among other example embodiments, the functionality of the VXU 1909 can be distributed among one or more of the other functional units (e.g., 1903 to 1908, 1910, 1911) of the processor in some embodiments.

[0089] Fig. 20 2 is a simplified block diagram illustrating an example implementation of a VXU 2000 according to some embodiments. For example, the VXU 2000 may provide at least one 64-bit input port 2001 to accept input from a vector register file 1901 or a general register file 1902. This input may be connected to a plurality of functional units including: a register file 2003, an address generator 2004, a point addressing logic 2005, a point insertion logic 2006, a point deletion logic 2007, a 3D to 2D projection logic in the X dimension 2008, a 3D to 2D projection logic in the Y dimension 2009, a 3D to 2D projection logic in the X dimension 2010, a 2D histogram pyramid generator 2011, a 3D histogram pyramid generator 2012, a population counter 2013, a 2D path finding logic 2014, a 3D path finding logic 2015, and possible additional functional units for operating on a 64-bit unsigned integer volume bitmap. The output from block 2002 may be written back to the vector register file VRF 1901 or general register file GRF 1902 register file.

[0090] Go to Fig.21 , showing an organized representation of a 4^3 voxel cube 2100. A second voxel cube 2101 is also shown. In this example, the voxel cube may be defined in the data as a 64-bit integer 2102, where each individual voxel within the cube is represented by a single corresponding bit in the 64-bit integer. For example, voxel 2012 at address {x, y, z} = {3, 0, 3} may be set to "1" to indicate that there is geometry at this coordinate within the volumetric space represented by voxel cube 2101. Further, in this example, among other examples, all other voxels (next to voxel 2102) may correspond to "blank" space and may be set to "0" to indicate that there is no physical geometry at those coordinates. Go to Fig. 22, shows an example two-level sparse voxel tree 2200 according to some embodiments. In this example, only a single "occupied" voxel is included within the volume (e.g., in position {15, 0, 15}). In this case, the upper level 0 2201 of the tree contains a single voxel entry {3, 0, 3}. This voxel in turn points to the next level 2202 of the tree, which contains a single voxel in the element {3, 0, 3}. The entry in the data structure corresponding to level 0 of the sparse voxel tree is a 64-bit integer 2203 in which one of the voxels is set to occupied. The set voxel means that an array of 64-bit integers is then allocated in the tree level 1 corresponding to the voxel volume set in 2203. In the level 1 subarray 2204, only one of these voxels is set to occupied, while all other voxels are set to unoccupied. In this example, since the tree is a two-level tree, level 1 represents the bottom of the tree, so that the hierarchy ends here.

[0091] Fig.23 A two-level sparse voxel tree 2300 is shown containing occupied voxels in positions {15, 0, 3} and {15, 0, 15} of a particular volume in accordance with some embodiments. In this case (which subdivides the particular volume into 64 upper level 0 voxels), the upper level 0 2301 of the tree contains two voxel entries {3, 0, 0} and {3, 0, 3}, where the corresponding data 2304 shows that the two voxels are set (or occupied). The next level of the sparse voxel tree (SVT) is provided as an array of 64-bit integers, the array containing two sub-cubes 2302 and 2303, one for each voxel set in level 0. In the level 1 sub-array 2305, two voxels are set to be occupied (v15 and v63), and all other voxels are set to be unoccupied and the tree. This format is flexible because the 64 entries in the next level of the tree are always allocated corresponding to each set voxel in the upper level of the tree. This flexibility can allow dynamically changing scene geometry to be inserted in potentially any manner (e.g., randomly rather than according to a fixed or defined order) as long as the corresponding voxels in the upper layer have been set. If not, a pointer table will be maintained, resulting in higher memory requirements, or the tree will need to be at least partially rebuilt in order to insert the unforeseen geometry.

[0092] Fig.24 A method for storing data from a Fig.23 In this example, the total volume 2400 contains the voxels stored in Fig.23In this approach, rather than allocating a 64-entry array to represent all of the subcubes in level 1 that are below level 0, only those elements of level 1 that actually contain geometry (e.g., as indicated by whether the corresponding level 0 voxel is occupied) are allocated as corresponding 64-bit level 1 records, so that in this example, level 1 has only two 64-bit entries instead of sixty-four (i.e., one for each of the 64 level 1 voxels, whether occupied or empty). Thus, in this example, the first level 02404 is equivalent to Fig.23 2304 in the 2nd level, and the next level 2405 is more memory-intensive than Fig.23 The corresponding 2305 in is 62 times smaller. In some embodiments, if a new geometry is to be inserted into level 0 and space has not yet been allocated in level 1, the tree must be copied and rearranged.

[0093] exist Fig.24 In the example of , subvolumes can be derived by counting occupied voxels in the layer above the current layer. In this way, the system can determine where in the voxel data one higher layer ends and the next lower layer begins. For example, if three layer 0 voxels are occupied, the system can expect that three corresponding layer 1 entries in the voxel data will follow, and the next entry (after these three) corresponds to the first entry in layer 2, and so on. This optimal compression can be very useful in cases where some parts of the scene do not change over time or in applications where volumetric data needs to be transmitted remotely, such as from a space probe scanning the surface of Pluto, where the transmission of each bit is expensive and time-consuming.

[0094] Figure 25A-25B 2500a-b illustrate example techniques involving selective elimination of operations from a set of example operations. Fig.25AIn an example of , a volume data structure (or VOLA) may be generated 2505 and volume data elements may be stored in a memory for use in a variety of different applications. The volume data structure may model a geometry within a particular volume, wherein the volume is subdivided into a plurality of voxels. The volume data structure may represent a particular volume and the geometry within the particular volume at a plurality of levels of detail. For example, each voxel at the lowest level of detail may be subdivided into voxels at a second level of detail that is immediately above the lower level of detail. The subvolume represented by the voxels at the second level of detail may be similarly subdivided into voxels at the next level of detail, and so on, until a desired number of levels of detail are provided and modeled within the volume data structure. As discussed herein, entries representing each level of detail in the modeled levels of detail may be provided within the volume data structure. A first entry may represent a first, lowest level of detail, wherein each voxel at the first level of detail is represented by a corresponding binary value in the first entry indicating whether a geometric volume exists in the corresponding voxel. The volume data structure may omit entries corresponding to higher levels of detail for voxels that are found not to be processed for geometry at the lowest level of detail, so as to sparsely represent multiple levels of detail for a particular volume within the volume data structure.

[0095] In some embodiments, given the binary nature of the values ​​of the volume data structure, the volume data structure can handle a large number of zero values ​​and can be treated as a bitmap (e.g., a 3D bitmap), which can be used to reduce the number of operations to be performed in conjunction with an application that uses the volume data structure as an input. For example, the volume data structure can be retrieved 2510 from a memory and processed 2515 to identify a subset of operations that can be skipped (e.g., element-by-element multiplication operations, or fetching multiplicand elements to be multiplied with matrix elements of the volume data structure) based on the zero values ​​present within the volume data structure. For example, a scheduler, controller, or other logic and circuit system can cause 2520 a set of operations to be performed while skipping those operations in the subset. This can result in efficiency gains in power, time, and computing resource utilization (e.g., processor and memory utilization), among other example advantages.

[0096] Go to Fig.25B, the output of a first layer of a neural network (e.g., a CNN) may be identified 2525. It should be understood that the "first" layer need not necessarily be the first layer in sequence within the neural network, but rather that the "first" is used to label multiple layers for naming in the current discussion. A bitmap may be determined 2520 from the output values ​​of the first layer, and a subset of operations in a different second layer within the neural network may be identified 2535 from the bitmap, and the subset of operations may be skipped based on information within the bitmap (e.g., based on corresponding zero values ​​in either or both operands to be multiplied in the subset of operations). In some cases, the output of the first layer may be predictably identified from inputs provided to another layer (e.g., a third layer), wherein a zero value in the output of the first layer may be predicted based on inputs to or outputs from a preceding layer, as well as other example embodiments. As in Fig.25A In the example of, after identifying 2535 a subset of operations that can be skipped based on the generated bitmap, operations of the second layer can be performed, wherein the subset of operations are skipped according to information of the bitmap, as well as other example implementations.

[0097] Although some systems and solutions described and shown herein have been described as comprising or being associated with multiple elements, not all elements clearly shown or described can be used in each alternative embodiment of the present disclosure. In addition, one or more elements described herein can be located outside the system, and in other examples, some elements can be included in or as part of one or more of other elements described in other descriptions and other elements not described in the shown embodiments. Further, some elements can be combined with other components and used for alternative or additional purposes other than those described herein.

[0098] Further, it should be understood that the examples presented above are non-limiting examples provided only for the purpose of illustrating certain principles and features, and do not necessarily limit or constrain potential embodiments of the concepts described herein. For example, various combinations of the features and components described herein, including combinations implemented by various implementations of the components described herein, may be utilized to implement various different embodiments. Other implementations, features, and details should be understood based on the content of this specification.

[0099] Figure 26 to Figure 31is a block diagram of an exemplary computer architecture that may be used in accordance with the embodiments disclosed herein. In practice, the computing devices, processors, and other logic and circuitry of the systems described herein may incorporate all or a portion of the functionality and supporting software and / or hardware circuitry for implementing such functionality. Further, in addition to the examples shown here, other computer architecture designs for processors and computing systems known in the art may also be used. In general, suitable computer architectures for the embodiments disclosed herein may include, but are not limited to Figure 26 to Figure 31 The configuration shown in .

[0100] Fig.26 An example domain topology of a corresponding Internet of Things (IoT) network coupled via a link to a corresponding gateway is shown. The Internet of Things (IoT) is a concept in which a large number of computing devices are interconnected to each other and to the Internet to provide functionality and data acquisition at a very low level. Therefore, as used herein, an IoT device may include a semi-autonomous device that performs functions such as sensing or control, communicates with other IoT devices and a wider network (such as the Internet).

[0101] Typically, IoT devices are limited in memory, size, or functionality, allowing a larger number to be deployed for a similar cost to a smaller number of larger devices. However, IoT devices can be smartphones, laptops, tablets, PCs, or other larger devices. Further, IoT devices can be virtual devices, such as applications on smartphones or other computing devices. IoT devices can include IoT gateways that couple IoT devices to other IoT devices and cloud applications for data storage, process control, etc.

[0102] A network of IoT devices may include commercial and home automation devices such as water distribution systems, power distribution systems, assembly line control systems, factory control systems, light switches, thermostats, locks, cameras, alarms, motion sensors, etc. IoT devices may be accessed by remote computers, servers, and other systems, for example, to control the systems or access data.

[0103] The future development of the Internet and similar networks is likely to involve a very large number of IoT devices. Therefore, in the context of the technologies discussed in this article, a large number of innovations for such future networking will address the need for: unimpeded growth of all these layers; discovery and manufacturing of accessible connected resources; and the ability to support hiding and partitioning connected resources. Any number of network protocols and communication standards may be used, each of which is designed to address specific goals. Further, the protocols are part of the structure that supports human-accessible services that operate regardless of location, time, or space. The innovations include service delivery and associated infrastructure, such as hardware and software; security enhancements; and the provision of services based on quality of service (QoS) terms specified in service levels and service delivery agreements. As will be appreciated, the implementation of IoT devices and networks (such as in Fig.26 and Fig. 27 The use of wireless networks (such as those introduced in ) poses a number of new challenges in heterogeneous connected networks that include a combination of wired and wireless technologies.

[0104] Fig.26 Specifically provided is a simplified diagram of a domain topology that can be used for multiple Internet of Things (IoT) networks including IoT devices 2604, wherein IoT networks 2656, 2658, 2660, 2662 are coupled to corresponding gateways 2654 via backbone links 2602. For example, multiple IoT devices 2604 can communicate with gateway 2654 and can communicate with each other through gateway 2654. To simplify the drawings, not every IoT device 2604 or communication link (e.g., link 2616, 2622, 2628, or 2632) is labeled. Backbone link 2602 can include any number of wired or wireless technologies (including optical networks) and can be part of a local area network (LAN), a wide area network (WAN), or the Internet. In addition, such communication links facilitate optical signal paths between IoT devices 2604 and gateway 2654, including the use of multiplexing / demultiplexing components that facilitate interconnection of the devices.

[0105] The network topology may include any number of types of IoT networks, such as a mesh network provided using Bluetooth Low Energy (BLE) link 2622 in conjunction with network 2656. Other types of IoT networks that may exist include: Link 2628 is a wireless local area network (WLAN) network 2658 for communicating with IoT device 2604, a cellular network 2660 for communicating with IoT device 2604 via an LTE / LTE-A (4G) or 5G cellular network, and a low power wide area (LPWA) network 2662 (e.g., an LPWA network compatible with the LoRaWan specification promulgated by the LoRa Alliance, or an IPv6-based low power wide area network (LPWAN) network compatible with the specification promulgated by the Internet Engineering Task Force (IETF)). Further, the corresponding IoT network can communicate with an external network provider (e.g., a Tier 2 or Tier 3 provider) using any number of communication links, such as an LTE cellular link, an LPWA link, or a cellular network based on the IEEE 802.15.4 standard (e.g., ). The corresponding IoT network may also operate using various network and Internet application protocols, such as Constrained Application Protocol (CoAP). The corresponding IoT network may also be integrated with a coordinator device that provides a chain of links that forms a cluster tree linking devices and networks.

[0106] Each of these IoT networks can provide opportunities for new technical features, such as those described in this article. Improved technology and networks can enable exponential growth of devices and networks, including the use of IoT networks as fog devices or systems. As the use of such improved technology grows, IoT networks can be developed for self-management, functional evolution, and collaboration without direct human intervention. Improved technology can even enable IoT networks to operate without a centralized control system. Therefore, the improved technology described in this article can be used to automate and enhance network management and operation functions far beyond current implementations.

[0107] In an example, communications between IoT devices 2604, such as through backbone link 2602, can be protected by a decentralized system for authentication, authorization, and billing (AAA). In a decentralized AAA system, distributed payment, credit, audit, authorization, and authentication systems can be implemented across interconnected heterogeneous network infrastructures. This allows systems and networks to move toward autonomous operation. In these types of autonomous operations, machines can even sign human resource contracts and negotiate partnerships with other machine networks. This can allow for the achievement of common goals and service delivery that is balanced with the outlined planned service level agreements, and implement solutions that provide metering, measurement, traceability, and traceability. Creating new supply chain structures and methods can enable the creation of a large number of services, mining their value, and collapsing without any human involvement.

[0108] Such IoT networks can be further enhanced by integrating sensing technologies such as sound, light, electronic traffic, facial and pattern recognition, smell, vibration, etc. into autonomous organizations in IoT devices. The integration of sensing systems can allow for systematic and autonomous communication and coordination of service delivery rather than a collection and convergence of resources based on contractual service goals, orchestration, and quality of service (QoS). Some of the individual examples of network-based resource processing include the following.

[0109] For example, mesh network 2656 can be enhanced by a system that performs inline data to information transformation. For example, a self-forming chain of processing resources including a multi-link network can distribute the transformation of raw data to information in an efficient manner, as well as the ability to distinguish assets and resources and the associated management of each. In addition, appropriate components of infrastructure and resource-based trust and service indexing can be inserted to improve data integrity, quality, assurance and deliver data confidence metrics.

[0110] For example, the WLAN network 2658 can provide multi-standard connectivity using a system that performs standard conversion, thereby enabling IoT devices 2604 to communicate using different protocols. Further systems can provide seamless interconnection between multi-standard infrastructures including visible Internet resources and hidden Internet resources.

[0111] For example, communications in the cellular network 2660 may be enhanced by systems that offload data, extend communications to more remote devices, or both. The LPWA network 2662 may include systems that perform interconnection non-Internet Protocol (IP), addressing, and routing to IP. Further, each of the IoT devices 2604 may include an appropriate transceiver for wide area communications with the device. Further, each IoT device 2604 may include other transceivers for communicating using additional protocols and frequencies. Fig.28 and Fig.29 This point is discussed further in the communication environment and hardware of IoT processing devices depicted in Figure 2.

[0112] Finally, clusters of IoT devices can be configured to communicate with other IoT devices as well as cloud networks. This can allow IoT devices to form ad-hoc networks between devices, allowing them to function as a single device, which can be referred to as a fog device. Fig. 27 Let's discuss this configuration further.

[0113] Fig. 27A cloud computing network is shown communicating with a mesh networked IoT device (device 2702) operating as a fog device at the edge of the cloud computing network. The mesh networked IoT device may be referred to as fog 2720, which operates at the edge of cloud 2700. To simplify the drawing, not every IoT device 2702 is labeled.

[0114] Fog 2720 can be considered as a large-scale interconnected network in which multiple IoT devices 2702 communicate with each other, for example, via radio links 2722. As an example, the Open Connectivity Foundation TM The interconnection specification published by the Open Communications Foundation (OCF) facilitates such interconnected networks. This standard allows devices to discover each other and establish interconnected communications. Other interconnection protocols may also be used, including, for example, the Optimized Link State Routing (OLSR) protocol, the Best Method for Mobile Ad Hoc Networking (BATMAN) routing protocol, or the OMA Lightweight M2M (LWM2M) protocol.

[0115] Three types of IoT devices 2702 are shown in this example, namely gateways 2704, data aggregators 2726, and sensors 2728, but any combination of IoT devices 2702 and functions can be used. Gateway 2704 can be an edge device that provides communication between cloud 2700 and fog 2720, and can also provide back-end processing functions for data obtained from sensors 2728 (such as motion data, flow data, temperature data, etc.). Data aggregators 2726 can collect data from any number of sensors 2728 and perform back-end processing functions for analysis. Results, raw data, or both can be passed along cloud 2700 through gateway 2704. Sensor 2728 can be, for example, a full IoT device 2702 that can both collect data and process data. In some cases, sensor 2728 may be more limited in functions such as collecting data and allowing data aggregators 2726 or gateways 2704 to process the data.

[0116] Communications from any IoT device 2702 can be passed along a convenient path (e.g., the most convenient path) between any IoT device 2702 to reach the gateway 2704. In these networks, the many interconnections provide a large amount of redundancy, allowing communications to be maintained even if many IoT devices 2702 are lost. Further, the use of mesh networks can allow the use of IoT devices 2702 that are very low power or located at a distance from the infrastructure, because the range of connection to another IoT device 2702 may be much less than the range of connection to the gateway 2704.

[0117] The fog 2720 provided from these IoT devices 2702 may be presented to devices in the cloud 2700, such as servers 2706, as a single device (e.g., fog device) located at the edge of the cloud 2700. In this example, alerts from the fog device may be sent without being identified as coming from a specific IoT device 2702 within the fog 2720. In this manner, the fog 2720 may be considered a distributed platform that provides computing and storage resources to perform processing or data-intensive tasks, such as data analysis, data aggregation, and machine learning.

[0118] In some examples, the IoT devices 2702 may be configured using an imperative programming style, for example, where each IoT device 2702 has specific functionality and communication partners. However, the IoT devices 2702 that form the fog devices may be configured in a declarative programming style, allowing the IoT devices 2702 to reconfigure their operations and communications, such as determining required resources in response to conditions, queries, and device failures. As an example, a query from a user at a server 2706 about the operation of a subset of devices monitored by the IoT devices 2702 may cause the fog 2720 device to select the IoT devices 2702 required to answer the query, such as specific sensors 2728. The data from these sensors 2728 may then be aggregated and analyzed by any combination of sensors 2728, data aggregators 2726, or gateways 2704 before being sent by the fog 2720 device to the server 2706 to answer the query. In this example, the IoT devices 2702 in the fog 2720 may select the sensors 2728 used based on the query, such as adding data from a flow sensor or a temperature sensor. Further, if some of the IoT devices 2702 are inoperable, other IoT devices 2702 in the fog 2720 devices may provide similar data if available.

[0119] In other examples, according to example embodiments, the above operations and functions may be implemented by an IoT device machine in the example form of an electronic processing system, where a set or series of instructions may be executed to cause the electronic processing system to perform any of the methods discussed herein. The machine may be an IoT device or IoT gateway, including machines implemented by various aspects of a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile phone, or a smart phone, or any machine capable of executing instructions (sequential or otherwise) that specify that the machine take an action. Further, although only a single machine may be depicted and referenced in the above examples, such a machine should also be considered to include any collection of machines that independently or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein. Further, these and similar examples to processor-based systems should be considered to include any collection of one or more machines that are controlled or operated by a processor (e.g., a computer) to independently or jointly execute instructions to perform any one or more of the methods discussed herein. In some embodiments, one or more multiple devices may operate in coordination to implement functions and perform tasks described herein. In some cases, one or more host devices may supply data, provide instructions, aggregate results, or otherwise facilitate joint operations and functionality provided by multiple devices. Although functions may be considered local to the device when implemented by a single device, functions may be collectively considered local to the device in embodiments where multiple devices operate as a single machine, and this collection of devices may provide or consume results provided by other remote machines (implemented as a single device or collection of devices), as well as other example embodiments.

[0120] For example, Fig.28A diagram of a cloud computing network or cloud 2800 in communication with multiple Internet of Things (IoT) devices is shown. Cloud 2800 may represent the Internet, or may be a local area network (LAN), or a wide area network (WAN), such as a company's proprietary network. IoT devices may include any number of different types of devices grouped in various combinations. For example, a traffic control group 2806 may include IoT devices along a city street. These IoT devices may include traffic lights, traffic flow monitors, cameras, weather sensors, etc. Traffic control group 2806 or other subgroups may communicate with cloud 2800 via a wired or wireless link 2808 (such as an LPWA link, an optical link, etc.). Further, a wired or wireless subnetwork 2812 may allow IoT devices to communicate with each other, such as via a local area network, a wireless local area network, etc. IoT devices may use another device such as a gateway 2810 or 2828 to communicate with a remote location such as cloud 2800; IoT devices may also use one or more servers 2830 to facilitate communication with cloud 2800 or gateway 2810. For example, the one or more servers 2830 can operate as intermediate network nodes to support local edge cloud or fog implementations in a local area network. Further, the depicted gateway 2828 can operate in a cloud-to-gateway-to-multi-edge device configuration, such as various IoT devices 2814, 2820, 2824 being constrained or dynamically allocated and using resources in the cloud 2800.

[0121] Other example groupings of IoT devices may include a remote weather station 2814, a local information terminal 2816, an alarm system 2818, an ATM 2820, an alarm panel 2822, or a mobile vehicle, such as an emergency vehicle 2824 or other vehicle 2826, etc. Each of these IoT devices may be connected to other IoT devices, to the server 2804, to another IoT fog device or system (not shown, but Fig. 27 IoT devices can be deployed in a variety of residential, commercial, and industrial settings (including both private and public environments).

[0122] As from Fig.28As can be seen in FIG, a large number of IoT devices can communicate through the cloud 2800. This can allow different IoT devices to autonomously request or provide information to other devices. For example, a group of IoT devices (e.g., traffic control group 2806) can request the current weather forecast from a remote weather station group 2814, which can provide forecasts without human intervention. Further, an emergency vehicle 2824 can be warned by an ATM 2820 that a theft is occurring. When the emergency vehicle 2824 moves toward the ATM 2820, it can access the traffic control group 2806 to request permission to reach the location, for example, by turning the lights red to block cross traffic at the intersection for a sufficient time to allow the emergency vehicle 2824 to enter the intersection unimpeded.

[0123] A cluster of IoT devices, such as a remote weather station 2814 or a traffic control group 2806, can be equipped to communicate with other IoT devices and with the cloud 2800. This can allow the IoT devices to form an ad hoc network between the devices, allowing them to function as a single device, which can be referred to as a fog device or system (e.g., as described above with respect to Fig. 27 described).

[0124] Fig.29 2950 for implementing the techniques described herein. The IoT device 2950 may include any combination of components shown in the examples or referenced in the above disclosure. These components may be implemented as ICs, portions thereof, discrete electronic devices, or other modules, logic, hardware, software, firmware, or combinations thereof adapted in the IoT device 2950, ​​or as components otherwise incorporated into a chassis of a larger system. Additionally, Fig.29 The block diagram is intended to depict a high-level view of the components of the IoT device 2950. However, some of the components shown may be omitted, additional components may be present, and different arrangements of the components shown may occur in other implementations.

[0125] IoT device 2950 may include processor 2952, which may be a microprocessor, a multi-core processor, a multi-threaded processor, an ultra-low voltage processor, an embedded processor, or other known processing element. Processor 2952 may be part of a system on a chip (SoC), in which processor 2952 and other components are formed as a single integrated circuit or a single package, such as Edison from Intel. TM or Galileo TM SoC board. As an example, the processor 2952 may include a Architecture Core TM Processors such as Quark TM、Atom TM , i3, i5, i7, or MCU-class processors, or available from However, any number of other processors may be used, such as those available from Advanced Micro Devices, Inc. of Sunnyvale, California, a MIPS-based design from MIPS Technologies, Inc. of Sunnyvale, California, an ARM-based design licensed by ARM Holdings Ltd. or its customers or their licensees or adopters. The processor may include, for example, a processor from The company's A5-A10 processors, from Snapdragon for Tech Companies TM processors, or OMAP from Texas Instruments TM A unit such as a processor.

[0126] The processor 2952 can communicate with the system memory 2954 via an interconnect 2956 (e.g., a bus). Any number of memory devices can be used to provide a quantitative system memory. As an example, the memory can be a random access memory (RAM) that complies with the Joint Electron Device Engineering Council (JEDEC) design (such as DDR or mobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4)). In various embodiments, each memory device can be any number of different package types, such as a single die package (SDP), a dual die package (DDP), or a quad die package (Q17P). In some examples, these devices can be directly soldered to the motherboard to provide a lower profile solution, while in other examples, these devices are configured as one or more memory modules, which are in turn coupled to the motherboard via a given connector. Any number of other memory implementations can be used, such as other types of memory modules, for example, different types of dual in-line memory modules (DIMMs), including but not limited to microDIMMs or MiniDIMMs.

[0127] In order to provide persistent storage of information such as data, applications, operating systems, etc., storage device 2958 may also be coupled to processor 2952 via interconnect 2956. In an example, storage device 2958 may be implemented via a solid state disk drive (SSDD). Other devices that may be used for storage device 2958 include flash memory cards (such as SD cards, micro SD cards, xD picture cards, etc.), and USB flash drives. In low-power implementations, storage device 2958 may be on-die memory or registers associated with processor 2952. However, in some examples, storage device 2958 may be implemented using a micro hard disk drive (HDD). Further, in addition to or in place of the described technology, any number of new technologies may be used for storage device 2958, such as resistive memory, phase change memory, holographic memory, or chemical memory, etc.

[0128] These components can communicate via interconnect 2956. Interconnect 2956 can include any number of technologies, including Industry Standard Architecture (ISA), Extended ISA (EISA), Peripheral Component Interconnect (PCI), Peripheral Component Interconnect Extended (PCIx), PCI Express (PCIe), or any number of other technologies. Interconnect 2956 can be, for example, a proprietary bus used in a SoC-based system. Other bus systems can be included, such as an I2C interface, an SPI interface, a point-to-point interface, and a power bus, etc.

[0129] The interconnect 2956 may couple the processor 2952 to a mesh transceiver 2962 for communication with other mesh devices 2964. The mesh transceiver 2962 may use any number of frequencies and protocols, such as 2.4 gigahertz (GHz) transmissions in accordance with the IEEE 802.15.4 standard, using a protocol such as that provided by Special Interest Groups Defined Low Energy (BLE) standard or Standards, etc. Any number of radios configured for a particular wireless communication protocol may be used for connection with the mesh device 2964. For example, a WLAN unit may be used to implement Wi-Fi according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard. TM Additionally, wireless wide area communications (eg, in accordance with a cellular or other wireless wide area protocol) may occur via the WWAN unit.

[0130] The mesh transceiver 2962 can communicate using multiple standards or radios for different ranges of communication. For example, the IoT device 2950 can communicate with close devices (e.g., within about 10 meters) using a local transceiver based on BLE or another low-power radio to save power. Mesh devices 2964 that are farther away can be reached via ZigBee or other intermediate power radios, for example, within about 50 meters. The two communication technologies can occur on a single radio at different power levels, or can occur on separate transceivers (e.g., a local transceiver using BLE and a separate mesh transceiver using ZigBee).

[0131] A wireless network transceiver 2966 may be included to communicate with devices or services in the cloud 2900 via a local area network protocol or a wide area network protocol. The wireless network transceiver 2966 may be a LPWA transceiver that complies with the IEEE 802.15.4 or IEEE 802.15.4g standard, etc. The IoT device 2950 may use the LoRaWAN developed by Semtech and the LoRa Alliance. TM (Long Range Wide Area Network) to communicate over a wide area. The techniques described herein are not limited to these techniques, but may be used with any number of other cloud transceivers implementing long range low bandwidth communications, such as Sigfox and other technologies. Further, other communication techniques described in the IEEE 802.15.4e specification, such as time slot channel hopping, may be used.

[0132] In addition to the systems mentioned for mesh transceiver 2962 and wireless network transceiver 2966, any number of other radio communications and protocols may be used, as described herein. For example, radio transceivers 2962 and 2966 may include LTE or other cellular transceivers that use spread spectrum (SPA / SAS) communications to implement high-speed communications. Further, any number of other protocols may be used, such as LTE for medium-speed communications and providing network communications. network.

[0133] Radio transceivers 2962 and 2966 may include radios compatible with any number of 3GPP (3rd Generation Partnership Project) specifications, particularly long term evolution (LTE), long term evolution-advanced (LTE-A) and long term evolution-advanced professional edition (LTE-A Pro). It may be noted that radios compatible with any number of other fixed, mobile or satellite communication technologies and standards may be selected. These may include, for example, any cellular wide area radio communication technology, which may include, for example, the 5th generation (5G) communication system, global system for mobile communications (GSM) radio communication technology, general packet radio service (GPRS) radio communication technology, or GSM evolution enhanced data rate (EDGE) radio communication technology, UMTS (Universal Mobile Telecommunications System) communication technology, in addition to the standards listed above, wireless network transceiver 2966 may use any number of satellite uplink technologies, including, for example, radios that conform to the standards issued by ITU (International Telecommunication Union) or ETSI (European Telecommunications Standards Institute), etc. Therefore, the examples provided herein are understood to be applicable to various other existing and unpublished communication technologies.

[0134] A network interface controller (NIC) 2968 may be included to provide wired communications to the cloud 2900 or other devices such as mesh devices 2964. The wired communications may provide an Ethernet connection, or may be based on other types of networks such as a controller area network (CAN), a local interconnect network (LIN), a device network (DeviceNet), a control network (ControlNet), a data highway +, a process field bus (PROFIBUS) or a process field network (PROFINET), etc. Additional NICs 2968 may be included to allow connection to a second network, for example, a NIC 2968 that provides communications to the cloud via Ethernet, and a second NIC 2968 that provides communications to other devices via another type of network.

[0135] The interconnect 2956 can couple the processor 2952 to an external interface 2970, which is used to connect external devices or subsystems. The external devices can include sensors 2972, such as accelerometers, level sensors, flow sensors, optical light sensors, camera sensors, temperature sensors, global positioning system (GPS) sensors, pressure sensors, barometric pressure sensors, etc. The external interface 2970 can be further used to connect the IoT device 2950 to an actuator 2974, such as a power switch, a valve actuator, an audible sound generator, a visual warning device, etc.

[0136] In some optional examples, various input / output (I / O) devices may be present within or connected to the IoT device 2950. For example, a display or other output device 2984 may be included to display information such as sensor readings or actuator positions. An input device 2986 such as a touch screen or keypad may be included to accept input. The output device 2984 may include any number of forms of audio or visual displays, including simple visual outputs such as binary status indicators (e.g., LEDs) and multi-character visual outputs, or more complex outputs such as display screens (e.g., LCD screens), where outputs of characters, graphics, multimedia objects, etc. are generated or produced based on the operation of the IoT device 2950.

[0137] The battery 2976 can power the IoT device 2950, ​​but in the example where the IoT device 2950 is installed in a fixed location, the battery can have a power source coupled to the power grid. The battery 2976 can be a lithium-ion battery or a metal-air battery, such as a zinc-air battery, an aluminum-air battery, a lithium-air battery, etc.

[0138] A battery monitor / charger 2978 may be included in the IoT device 2950 to track the state of charge (SoCh) of the battery 2976. The battery monitor / charger 2978 may be used to monitor other parameters of the battery 2976 to provide fault prediction, such as the state of health (SoH) and state of function (SoF) of the battery 2976. The battery monitor / charger 2978 may include a battery monitoring integrated circuit, such as the LTC4020 or LTC2990 from Linear Technologies, the ADT7488A from ON Semiconductor of Phoenix, Arizona, or the UCD90xxx series of ICs from Texas Instruments of Dallas, Texas. The battery monitor / charger 2978 may communicate information about the battery 2976 to the processor 2952 via the interconnect 2956. The battery monitor / charger 2978 may also include an analog-to-digital (ADC) converter that allows the processor 2952 to directly monitor the voltage of the battery 2976 or the current from the battery 2976. The battery parameters may be used to determine actions that the IoT device 2950 may perform, such as transmission frequency, mesh network operation, sensing frequency, and the like.

[0139] A power block 2980 or other power source coupled to the grid can be coupled to the battery monitor / charger 2978 to charge the battery 2976. In some examples, the power block 2980 can be replaced with a wireless power receiver to obtain power wirelessly, for example, through a loop antenna in the IoT device 2950. A wireless battery charging circuit (such as the LTC4020 chip from Linear Technology Corporation of Milpitas, California, etc.) can be included in the battery monitor / charger 2978. The specific charging circuit selected depends on the size of the battery 2976 and therefore the current required. Charging can be performed using the Airfuel standard promulgated by the Airfuel Alliance, the Qi wireless charging standard promulgated by the Wireless Power Consortium, or the Rezence charging standard promulgated by the Wireless Power Consortium.

[0140] The storage device 2958 may include instructions 2982 in the form of software, firmware, or hardware commands to implement the techniques described herein. Although such instructions 2982 are illustrated as code blocks included in the memory 2954 and the storage device 2958, it is understood that any of the code blocks may be replaced with hard-wired circuits, such as built into an application specific integrated circuit (ASIC).

[0141] In an example, the instructions 2982 provided via the memory 2954, the storage device 2958, or the processor 2952 may be implemented as a non-transitory machine-readable medium 2960, including code for directing the processor 2952 to perform electronic operations in the IoT device 2950. The processor 2952 may access the non-transitory machine-readable medium 2960 via the interconnect 2956. For example, the non-transitory machine-readable medium 2960 may be provided by a processor 2952 for performing an electronic operation in the IoT device 2950. Fig.29 The storage device 2958 may be embodied in the device described, or may include a specific storage unit, such as an optical disk, a flash drive, or any number of other hardware devices. The non-transitory machine-readable medium 2960 may include instructions for directing the processor 2952 to perform a specific sequence of actions or flow of actions, for example, as described in the (multiple) flow charts and (multiple) block diagrams regarding the above-mentioned operations and functions.

[0142] Fig.30 3000 is an example of a processor according to an embodiment. Processor 3000 is an example of a type of hardware device that can be used in conjunction with the above-described embodiments. Processor 3000 can be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a network processor, a multi-core processor, a single-core processor, or other device that executes code. Although in Fig.30 Only one processor 3000 is shown, but the processing elements may alternatively include Fig.30More than one processor 3000 is shown in FIG. Processor 3000 may be a single-threaded core, or for at least one embodiment, processor 3000 may be multi-threaded in that it may include more than one hardware thread context (or "logical processor") per core.

[0143] Fig.30 Also shown is a memory 3002 coupled to the processor 3000 according to an embodiment. The memory 3002 can be any of a wide range of memories (including various layers in a memory hierarchy) known or otherwise available to those skilled in the art. Such memory elements can include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0144] The processor 3000 may execute any type of instructions associated with the algorithms, processes or operations described in detail herein.In general, the processor 300 may transform an element or object (eg, data) from one state or thing to another state or thing.

[0145] The code 3004 (which may be one or more instructions to be executed by the processor 3000) may be stored in the memory 3002, or may be stored in software, hardware, firmware, or any suitable combination thereof, or (where appropriate and based on specific needs) in any other internal or external component, device, element, or object. In one example, the processor 3000 may follow the sequence of program instructions indicated by the code 3004. Each instruction enters the front-end logic 3006 and is processed by one or more decoders 3008. The decoder may generate micro-operations as its output, such as fixed-width micro-outputs in a predetermined format, or may generate other instructions, micro-instructions, or control signals reflecting the original code instructions. The front-end logic 3006 also includes register renaming logic 3010 and scheduling logic 3012, which generally allocates operations corresponding to instructions for execution to resources and queues.

[0146] The processor 3000 may also include an execution logic 3014 having a set of execution units 3016a, 3016b, 3016n, etc. Some embodiments may include a large number of execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit, or one execution unit that can perform a specific function. The execution logic 3014 performs the operations specified by the code instructions.

[0147] After completing execution of the operations specified by the code instructions, back-end logic 3018 may retire the instructions of code 3004. In one embodiment, processor 3000 allows out-of-order execution but requires in-order retirement of instructions. Retirement logic 3020 may take various known forms (e.g., a reorder buffer, etc.). In this manner, processor 3000 is transformed during execution of code 3004, based at least on the output generated by the decoders, the hardware registers and tables utilized by register renaming logic 3010, and any registers modified by execution logic 3014 (not shown).

[0148] Although not in Fig.30 3000, but the processing element may include other on-chip elements with the processor 3000. For example, the processing element may include memory control logic as well as the processor 3000. The processing element may include I / O control logic and / or may include I / O control logic integrated with the memory control logic. The processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on the chip with the processor 3000.

[0149] Fig.31 A computing system 3100 arranged in a point-to-point (PtP) configuration according to an embodiment is shown. Specifically, Fig.31 A system is shown in which processors, memory, and input / output devices are interconnected by a number of point-to-point interfaces. Generally, one or more of the computing systems described herein may be configured in the same or similar manner as computing system 3100.

[0150] The processors 3170 and 3180 may each include integrated memory controller logic (MC) 3172 and 3182 for communicating with the memory elements 3132 and 3134. In alternative embodiments, the memory controller logic 3172 and 3182 may be discrete logic separate from the processors 3170 and 3180. The memory elements 3132 and / or 3134 may store various data to be used by the processors 3170 and 3180 to implement the operations and functions outlined herein.

[0151] Processors 3170 and 3180 may be any type of processor, such as those discussed in connection with other figures. Processors 3170 and 3180 may exchange data via point-to-point interface 3150 using point-to-point (PtP) interface circuits 3178 and 3188, respectively. Each of processors 3170 and 3180 may exchange data with chipset 3190 via separate point-to-point interfaces 3152 and 3154 using point-to-point interface circuits 3176, 3186, 3194, and 3198. Chipset 3190 may also exchange data with high-performance graphics circuit 3138 via high-performance graphics interface 311139 using interface circuit 3192 (which may be a PtP interface circuit). In an alternative embodiment, Fig.31 Any or all of the PtP links shown may be implemented as a multi-drop bus rather than as PtP links.

[0152] Chipset 3190 may communicate with bus 3120 via interface circuit 3196. Bus 3120 may have one or more devices that communicate over the bus, such as bus bridge 3118 and I / O devices 3116. Via bus 3110, bus bridge 3118 may communicate with other devices, such as user interface 3112 (such as a keyboard, mouse, touch screen, or other input device), communication device 3126 (such as a modem, network interface device, or other type of communication device that can communicate over computer network 3160), audio I / O device 3114, and / or data storage device 3128. Data storage device 3128 may store code 3130, which may be executed by processor 3170 and / or 3180. In alternative embodiments, any portion of the bus architecture is implemented with one or more PtP links.

[0153] Fig.31 The computer system depicted in is a schematic diagram of an embodiment of a computing system that can be used to implement the various embodiments discussed herein. It will be understood that Fig.31 The various components of the system described in the above may be combined in a system on a chip (S o C) architecture or in any other suitable configuration that can implement the functions and features of the examples and embodiments provided herein.

[0154] In a further example, a machine-readable medium also includes any tangible medium capable of storing, encoding or carrying instructions for execution by a machine and causing the machine to perform any one or more of the methods of the present disclosure, or capable of storing, encoding or carrying data structures utilized by or associated with such instructions. "Machine-readable media" may therefore include, but are not limited to, solid-state memory, and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, which includes, by way of example, but is not limited to, semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as built-in hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Instructions embodied by machine-readable media may be further transmitted or received over a communication network, via a network interface device, using a transmission medium, using any of a plurality of transmission protocols (e.g., HTTP).

[0155] It should be understood that the functional unit or capability described in this specification may be referred to as or marked as a component or module to more particularly emphasize its implementation independence. These components may be implemented in the form of any number of software or hardware. For example, a component or module may be implemented as a hardware circuit, and the hardware circuit includes a custom very large scale integration (VLSI) circuit or gate array, a semiconductor available from stock, such as a logic chip, a transistor, or other discrete components. Components or modules may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, etc. Components or modules may also be implemented in software executed by various types of processors. The identified components or modules of executable code may include one or more physical blocks or logic blocks of, for example, computer instructions, which may be organized as, for example, objects, programs, or functions. However, the executable portion of the identified components or modules does not need to be physically located together, but may include different instructions stored in different locations, which instructions include the components or modules when logically connected together and realize the purpose of the components or modules described.

[0156] In fact, the components or modules of executable code can be a single instruction, or many instructions, and can even be distributed in several different code segments, between different programs, and across several memory devices or processing systems. Specifically, some aspects of the described process (such as code rewriting and code analysis) can occur on different processing systems (for example, in a computer in a data center), rather than on a processing system where the code is deployed (for example, in a computer embedded in a sensor or robot). Similarly, operational data can be identified and displayed in a component or module herein, and can be embodied in any suitable form and organized in a data structure of any suitable type. Operational data can be collected as a single data set, or can be distributed in different locations, including distribution on different storage devices, and can exist only as electronic signals in a system or network at least in part. The components or modules can be passive or active, including agents that can be operated to perform desired functions.

[0157] Additional examples of the presently described method, system, and device embodiments include the following non-limiting configurations. Each of the following non-limiting examples may stand alone or may be combined in any enumeration or combination with any one or more of the other examples provided below or throughout this disclosure.

[0158] Although the present disclosure has been described in terms of certain embodiments and generally associated methods, alternatives and permutations of these embodiments and methods will be apparent to those skilled in the art. For example, the actions described herein may be performed in a different order than described, and still achieve the desired result. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing may be advantageous. In addition, other user interface layouts and functions may be supported. Other variations are within the scope of the following claims.

[0159] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or content that may be claimed, but should be interpreted as descriptions of features that are peculiar to specific embodiments of specific inventions. Certain features described in this specification in the context of a single embodiment may also be implemented in a single embodiment in combination. On the contrary, the individual features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Moreover, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be separated from the combination in some cases, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0160] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the specific order shown or in an ordered order, or all the operations shown can be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Moreover, the separation of the different system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can usually be integrated together in a single software product or encapsulated in multiple software products.

[0161] The following examples relate to embodiments according to the present specification. Example 1 is a machine-accessible storage medium having instructions stored thereon, which, when executed on a machine, cause the machine to: retrieve a volume data structure from a memory, wherein the volume data structure represents a geometric structure within a specific volume in multiple levels of detail, the specific volume including a plurality of voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure for representing the geometric structure in each voxel in the plurality of voxels at the first level of detail, the first entry including a binary value for each voxel in the plurality of voxels at the first level of detail, and the binary value indicating whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel; process the volume data structure to identify a first portion of an operation set to be skipped based at least in part on the binary value of the first entry, wherein the operation set includes operations associated with the volume data structure; and cause a second portion of the operation set other than the first portion to be executed, wherein the operations in the first portion are to be skipped.

[0162] Example 2 may include at least a portion of the subject matter of Example 1, wherein the set of operations includes a three-dimensional (3D) matrix multiplication operation, and the volume data structure includes a 3D matrix.

[0163] Example 3 may include at least a portion of the subject matter of Example 2, wherein the 3D matrix multiplication operation is performed in the convolution operation using the volume data structure.

[0164] Example 4 may include at least part of the subject matter of any of Examples 1-3, wherein the set of operations includes a memory fetch.

[0165] Example 5 may include at least a portion of the subject matter of Example 4, wherein the memory fetch comprises fetching weight values ​​in a 3D weight matrix to be multiplied with the volume data structure.

[0166] Example 6 may include at least a portion of the subject matter of any of Examples 1-5, wherein the volume data structure further includes a set of second entries, each of the second entries representing a voxel at a second level of detail in a plurality of levels of detail, each of the second entries corresponding to a corresponding one of the voxels at the first level of detail identified by a corresponding one of the binary values ​​as containing a geometric structure, and each of the second entries including a binary value for each of the plurality of voxels at the second level of detail indicating whether a geometric structure exists within the corresponding voxel at the second level of detail.

[0167] Example 7 may include at least a portion of the subject matter of Example 6, wherein the binary value in the first entry indicates that a first number of voxels at a first level of detail are occupied by geometry and a second number of voxels at the first level of detail are blank, and the set of second entries consists of the first number of second entries, and in the set of second entries, entries are omitted for voxels at the second level of detail that represent sub-volumes of voxels at the first level of detail that are indicated as blank.

[0168] Example 8 may include at least a portion of the subject matter of any of Examples 6-7, wherein the voxels at the first level of detail include 64 voxels, the binary value in the first entry includes 64 binary values, and the voxels at the second level of detail represent 64 subvolumes of each of the 64 voxels at the first level of detail.

[0169] Example 9 may include at least a portion of the subject matter of any of Examples 1-8, wherein skipping the first portion of the operation includes disabling at least a portion of circuitry for performing the first portion of the operation.

[0170] Example 10 may include at least a portion of the subject matter of any of Examples 1-9, wherein each voxel in the volume data structure is represented by at least two bits, and one of the at least two bits includes a binary value indicating whether the corresponding voxel includes a geometric structure.

[0171] Example 11 is a machine-accessible storage medium having instructions stored thereon that, when executed on a machine, cause the machine to: identify an output of a first layer of a plurality of layers within a neural network; determine a bitmap comprising a binary matrix from the output; determine a particular subset of operations to be skipped for a second layer of the plurality of layers based on the bitmap; and cause operations other than the particular subset of operations to be performed for the second layer, wherein the particular subset of operations are skipped.

[0172] Example 12 may include at least a portion of the subject matter of Example 11, wherein the neural network comprises a convolutional neural network (CNN).

[0173] Example 13 may include at least a portion of the subject matter of any of Examples 11-12, wherein the specific subset of operations includes arithmetic operations of the second level.

[0174] Example 14 may include at least a portion of the subject matter of Example 13, wherein the arithmetic operation comprises a three-dimensional (3D) matrix multiplication operation.

[0175] Example 15 may include at least a portion of the subject matter of Example 14, wherein the 3D multiplication operation includes multiplying elements in a particular 3D matrix with another operand, and the particular subset of operations includes multiplication involving a subset of elements in the particular 3D matrix, wherein the subset of elements has a value of 0.

[0176] Example 16 may include at least a portion of the subject matter of Example 15, wherein the particular subset of operations further comprises a memory fetch of elements of another operand to be multiplied by the subset of elements.

[0177] Example 17 may include at least part of the subject matter of any of Examples 15-16, wherein the instructions, when executed, further cause the machine to: determine a mapping between values ​​in the bitmap and corresponding elements in a particular 3D matrix.

[0178] Example 18 may include at least a portion of the subject matter of any of Examples 11-17, wherein the first layer comprises a rectified linear unit (ReLU) layer.

[0179] Example 19 may include at least a portion of the subject matter of Example 18, wherein the second layer comprises one of a convolutional layer and a pooling layer.

[0180] Example 20 may include at least a portion of the subject matter of any of Examples 18-19, wherein identifying the output of the ReLU layer includes: determining from the input to the ReLU layer that the output of the ReLU layer will include multiple zeros, and the bitmap is generated based on the multiple zeros.

[0181] Example 21 may include at least a portion of the subject matter of any of Examples 18-19, wherein identifying the output of the ReLU layer comprises predicting that the output of the ReLU layer will include a set of zeros based on signs of operands in a preceding layer of the neural network.

[0182] Example 22 may include at least a portion of the subject matter of Example 21, wherein the preceding layer comprises one of a convolutional layer and a pooling layer.

[0183] Example 23 may include at least a portion of the subject matter of any of Examples 11-22, wherein the bitmap comprises a 3D bitmap.

[0184] Example 24 may include at least a portion of the subject matter of any of Examples 11-23, wherein a "1" in the bitmap indicates an operation to be performed, and a "0" in the bitmap indicates an operation in the subset to be skipped.

[0185] Example 25 may include at least a portion of the subject matter of any of Examples 11-24, wherein the output includes a bitmap.

[0186] Example 26 may include at least a portion of the subject matter of any of Examples 11-25, wherein the bitmap includes a volume data structure that represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

[0187] Example 27 is a device comprising: a data processing device; a bitmap generation circuit system, the bitmap generation circuit system being used to: obtain values ​​of a matrix, the matrix being provided as input to a first layer of multiple layers within an artificial neural network; determine from the values ​​that the output of a particular layer will include a set of output values, the set of output values ​​including at least some zeros; and generate a bitmap matrix including binary values, the binary values ​​being used to indicate whether corresponding values ​​in the output of the particular layer include a zero value.

[0188] Example 28 may include at least a portion of the subject matter of Example 27, further comprising a scheduler to: retrieve from a memory a bitmap matrix associated with a multiplication to be performed using an output of another layer in a plurality of layers in a neural network; and skip a subset of the multiplication operations to be performed on the multiplication involving zero values ​​in the matrix based on the bitmap matrix.

[0189] Example 29 may include at least a portion of the subject matter of Example 28, wherein the neural network comprises a convolutional neural network, and the another layer comprises one of a convolutional layer or a pooling layer.

[0190] Example 30 may include at least a portion of the subject matter of any of Examples 27-29, wherein the specific layer includes a first layer, and the specific layer includes a rectified linear unit (ReLU) layer.

[0191] Example 31 may include at least a portion of the subject matter of Example 30, wherein the particular layer includes a first layer, and the bitmap generation circuit system includes a comparator that is used to determine whether the input value is negative, wherein the bitmap matrix is ​​used to be filled with binary values ​​generated by the comparator from the input value.

[0192] Example 32 may include at least a portion of the subject matter of Example 30, wherein the first layer comprises a layer prior to a ReLU layer, the output of the first layer is used to be provided as an input to the ReLU layer, the bitmap generation circuit system includes an exclusive OR (XOR) gate that is used to operate on at least a portion of each of two operands used to multiply in the first layer to generate the input to the ReLU layer, and the bitmap matrix is ​​used to be filled with binary values ​​generated by the XOR gate.

[0193] Example 33 may include at least a portion of the subject matter of any of Examples 27-32, wherein the matrix comprises a matrix having three or more dimensions.

[0194] Example 34 is a system comprising: one or more optical sensors for generating optical data; a volume data generator for generating a volume data structure from the optical data, wherein the volume data structure represents a geometric structure captured by the optical sensor within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

[0195] Example 35 may include at least a portion of the subject matter of Example 34, further comprising a scheduler for: obtaining a volume data structure from a memory; and using the volume data structure as a bitmap to identify a subset of multiplication operations to be skipped based on a subset of binary values ​​in the volume data structure that includes a 0 value.

[0196] Example 36 may include at least a portion of the subject matter of any of Examples 34-35, further comprising a matrix multiplication processor for performing a matrix multiplication using the volume data structure.

[0197] Example 37 may include at least a portion of the subject matter of Example 36, wherein the matrix multiplication is performed to correspond to one of a convolutional layer or a pooling layer in a convolutional neural network.

[0198] Example 38 may include at least a portion of the subject matter of any of Examples 34-37, wherein the volume data generator comprises a simultaneous localization and mapping (SLAM) pipeline for generating a depth map, and the volume data structure is generated from the depth map.

[0199] Example 39 is a method comprising: retrieving a volume data structure from a memory, wherein the volume data structure represents a geometric structure within a specific volume at multiple levels of detail, the specific volume including multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels in the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels in the first level of detail, and the binary value indicates whether the geometric structure exists in a corresponding subvolume of the specific volume defined by the corresponding voxel; processing the volume data structure to identify a portion of an operation set to be skipped based at least in part on the binary value of the first entry, wherein the operation set includes operations associated with the volume data structure; and causing operations of the operation set other than the portion to be performed, wherein the operations in the portion are to be skipped.

[0200] Example 40 may include at least a portion of the subject matter of Example 39, wherein the set of operations includes a three-dimensional (3D) matrix multiplication operation and the volume data structure includes a 3D matrix.

[0201] Example 41 may include at least a portion of the subject matter of Example 40, wherein the 3D matrix multiplication operation is performed in the convolution operation using the volume data structure.

[0202] Example 42 may include at least a portion of the subject matter of any of Examples 39-41, wherein the set of operations includes a memory fetch.

[0203] Example 43 may include at least a portion of the subject matter of Example 42, wherein the memory fetch comprises fetching weight values ​​in a 3D weight matrix to be multiplied with the volumetric data structure.

[0204] Example 44 may include at least a portion of the subject matter of any of Examples 39-43, wherein the volume data structure further includes a set of second entries, each of the second entries representing a voxel at a second level of detail in a plurality of levels of detail, each of the second entries corresponding to a corresponding one of the voxels at the first level of detail identified by a corresponding one of the binary values ​​as containing a geometric structure, and each of the second entries includes a binary value for each of the plurality of voxels at the second level of detail indicating whether a geometric structure exists within the corresponding voxel at the second level of detail.

[0205] Example 45 may include at least a portion of the subject matter of Example 44, wherein the binary value in the first entry indicates that a first number of voxels at a first level of detail are occupied by geometry and a second number of voxels at the first level of detail are blank, and the set of second entries consists of the first number of second entries, and in the set of second entries, entries are omitted for voxels at the second level of detail that represent sub-volumes of voxels at the first level of detail that are indicated as blank.

[0206] Example 46 may include at least a portion of the subject matter of any of Examples 44-45, wherein the voxels at the first level of detail include 64 voxels, the binary value in the first entry includes 64 binary values, and the voxels at the second level of detail represent 64 subvolumes for each of the 64 voxels at the first level of detail.

[0207] Example 47 may include at least a portion of the subject matter of any of Examples 39-46, wherein skipping the first portion of the operation comprises disabling at least a portion of circuitry for performing the portion of the operation.

[0208] Example 48 may include at least a portion of the subject matter of any of Examples 39-47, wherein each voxel in the volume data structure is represented by at least two bits, and one of the at least two bits includes a binary value indicating whether the corresponding voxel includes a geometric structure.

[0209] Example 49 is a system comprising means for performing the method of any of Examples 39-48.

[0218] Example 50 is a method comprising: identifying an output of a first layer among multiple layers within a neural network; determining a bitmap comprising a binary matrix from the output; determining a specific subset of operations to be skipped for a second layer among the multiple layers based on the bitmap; and causing operations other than the specific subset of operations to be performed for the second layer, wherein the specific subset of operations are skipped.

[0211] Example 51 may include at least a portion of the subject matter of Example 50, wherein the neural network comprises a convolutional neural network (CNN).

[0212] Example 52 may include at least a portion of the subject matter of any of Examples 50-51, wherein the specific subset of operations includes arithmetic operations of the second level.

[0213] Example 53 may include at least a portion of the subject matter of Example 52, wherein the arithmetic operation comprises a three-dimensional (3D) matrix multiplication operation.

[0214] Example 54 may include at least a portion of the subject matter of Example 53, wherein the 3D multiplication operation comprises multiplying elements in a particular 3D matrix with another operand, and the particular subset of operations comprises multiplication involving a subset of elements in the particular 3D matrix, wherein the subset of elements has a value of 0.

[0215] Example 55 may include at least a portion of the subject matter of Example 54, wherein the particular subset of operations further comprises a memory fetch of elements of another operand to be multiplied by the subset of elements.

[0216] Example 56 may include at least part of the subject matter of any of Examples 54-55, wherein the instructions, when executed, further cause the machine to: determine a mapping between values ​​in the bitmap and corresponding elements in a particular 3D matrix.

[0217] Example 57 may include at least a portion of the subject matter of any of Examples 50-56, wherein the first layer comprises a rectified linear unit (ReLU) layer.

[0218] Example 58 may include at least a portion of the subject matter of Example 57, wherein the second layer comprises one of a convolutional layer and a pooling layer.

[0219] Example 59 may include at least a portion of the subject matter of any of Examples 57-58, wherein identifying the output of the ReLU layer includes: determining from the input to the ReLU layer that the output of the ReLU layer will include multiple zeros, and the bitmap is generated based on the multiple zeros.

[0220] Example 60 may include at least a portion of the subject matter of any of Examples 57-58, wherein identifying the output of the ReLU layer comprises predicting that the output of the ReLU layer will include a set of 0s based on signs of operands in a preceding layer of the neural network.

[0221] Example 61 may include at least a portion of the subject matter of Example 60, wherein the preceding layer comprises one of a convolutional layer and a pooling layer.

[0222] Example 62 may include at least a portion of the subject matter of any of Examples 50-61, wherein the bitmap comprises a 3D bitmap.

[0223] Example 63 may include at least a portion of the subject matter of any of Examples 50-62, wherein a "1" in the bitmap indicates an operation to be performed, and a "0" in the bitmap indicates an operation to be skipped in the subset.

[0224] Example 64 may include at least a portion of the subject matter of any of Examples 50-63, wherein the output comprises a bitmap.

[0225] Example 65 may include at least a portion of the subject matter of any of Examples 50-64, wherein the bitmap includes a volume data structure that represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

[0226] Example 66 is a system comprising means for performing the method of any of Examples 50-65.

[0227] Example 67 is a device comprising a data processor, a memory, and a circuit system. The memory is used to store a volume data structure from the memory, wherein the volume data structure represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists in a corresponding subvolume of the specific volume defined by the corresponding voxel. The circuit system is used to: identify a first portion of an operation set to be skipped from the volume data structure based at least in part on the binary value of the first entry, wherein the operation set includes operations associated with the volume data structure; and cause operations other than the portion of the operation set to be performed, wherein the operations in the portion are to be skipped

[0228] Example 68 may include at least a portion of the subject matter of Example 67, wherein the set of operations comprises a three-dimensional (3D) matrix multiplication operation, and the volume data structure comprises a 3D matrix.

[0229] Example 69 may include at least a portion of the subject matter of Example 68, wherein a 3D matrix multiplication operation is performed in a convolution operation using a volume data structure.

[0230] Example 70 may include at least a portion of the subject matter of any of Examples 67-69, wherein the set of operations includes a memory fetch.

[0231] Example 71 may include at least a portion of the subject matter of Example 70, wherein the memory retrieval comprises retrieval of weight values ​​in a 3D weight matrix to be multiplied with the volumetric data structure.

[0232] Example 72 may include at least a portion of the subject matter of any of Examples 67-71, wherein the volume data structure further includes a set of second entries, each of the second entries representing a voxel at a second level of detail in a plurality of levels of detail, each of the second entries corresponding to a corresponding one of the voxels at the first level of detail identified by a corresponding one of the binary values ​​as containing a geometric structure, and each of the second entries includes a binary value for each of the plurality of voxels at the second level of detail indicating whether a geometric structure exists within the corresponding voxel at the second level of detail.

[0233] Example 73 may include at least a portion of the subject matter of Example 72, wherein the binary value in the first entry indicates that a first number of voxels at a first level of detail are occupied by geometry and a second number of voxels at the first level of detail are blank, and the set of second entries consists of the first number of second entries, and in the set of second entries, entries are omitted for voxels at the second level of detail that represent sub-volumes of voxels at the first level of detail that are indicated as blank.

[0234] Example 74 may include at least a portion of the subject matter of any of Examples 72-73, wherein the voxels at the first level of detail include 64 voxels, the binary value in the first entry includes 64 binary values, and the voxels at the second level of detail represent 64 subvolumes for each of the 64 voxels at the first level of detail.

[0235] Example 75 may include at least a portion of the subject matter of any of Examples 67-74, wherein skipping the first portion of the operation comprises disabling at least a portion of circuitry for performing the portion of the operation.

[0236] Example 76 may include at least a portion of the subject matter of any of Examples 67-75, wherein each voxel in the volume data structure is represented by at least two bits, and one of the at least two bits includes a binary value indicating whether the corresponding voxel includes a geometric structure.

[0237] Example 77 is an apparatus comprising: a data processor; a bitmap generation circuit system, the bitmap generation circuit system is used to: identify the output of a first layer of multiple layers in a neural network; and determine a bitmap including a binary matrix from the output; and a controller is used to: determine a specific subset of operations to be skipped for a second layer of the multiple layers based on the bitmap; and cause operations other than the specific subset of operations to be performed for the second layer, wherein the specific subset of operations are skipped

[0238] Example 78 may include at least a portion of the subject matter of Example 77, wherein the neural network comprises a convolutional neural network (CNN).

[0239] Example 79 may include at least a portion of the subject matter of any of Examples 77-78, wherein the specific subset of operations includes arithmetic operations of the second level.

[0240] Example 80 may include at least a portion of the subject matter of Example 79, wherein the arithmetic operation comprises a three-dimensional (3D) matrix multiplication operation.

[0241] Example 81 may include at least a portion of the subject matter of Example 80, wherein the 3D multiplication operation comprises multiplying elements in a particular 3D matrix with another operand, and the particular subset of operations comprises multiplication involving a subset of elements in the particular 3D matrix, wherein the subset of elements has a value of 0.

[0242] Example 82 may include at least a portion of the subject matter of Example 81, wherein the particular subset of operations further comprises a memory fetch of elements of another operand to be multiplied by the subset of elements.

[0253] Example 83 may include at least a portion of the subject matter of any of Examples 81-82, wherein the controller is further to: determine a mapping between values ​​in the bitmap and corresponding elements in a particular 3D matrix.

[0244] Example 84 may include at least a portion of the subject matter of any of Examples 77-83, wherein the first layer includes a rectified linear unit (ReLU) layer.

[0245] Example 85 may include at least a portion of the subject matter of Example 84, wherein the second layer comprises one of a convolutional layer and a pooling layer.

[0246] Example 86 may include at least a portion of the subject matter of any of Examples 84-85, wherein identifying the output of the ReLU layer includes: determining from the input to the ReLU layer that the output of the ReLU layer will include multiple zeros, and the bitmap is generated based on the multiple zeros.

[0247] Example 87 may include at least a portion of the subject matter of any of Examples 84-85, wherein identifying the output of the ReLU layer comprises predicting that the output of the ReLU layer will include a set of 0s based on the sign of an operand in a previous layer of the neural network.

[0248] Example 88 may include at least a portion of the subject matter of Example 87, wherein the preceding layer comprises one of a convolutional layer and a pooling layer.

[0249] Example 89 may include at least a portion of the subject matter of any of Examples 77-88, wherein the bitmap comprises a 3D bitmap.

[0250] Example 90 may include at least a portion of the subject matter of any of Examples 77-89, wherein a "1" in the bitmap indicates an operation to be performed, and a "0" in the bitmap indicates an operation to be skipped in the subset.

[0251] Example 91 may include at least a portion of the subject matter of any of Examples 77-90, wherein the output includes a bitmap.

[0252] Example 92 may include at least a portion of the subject matter of any of Examples 77-91, wherein the bitmap includes a volume data structure that represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel in the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel in the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

[0253] Thus, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results.

Claims

1. A device comprising: data processing device, A bitmap generation circuit system for: obtaining values ​​of a matrix for being provided as input to a first layer of a plurality of layers within an artificial neural network; determining from the values ​​that an output of the particular layer will include a set of output values, the set of output values ​​including at least some zeros; and A bitmap matrix is ​​generated including binary values ​​indicating whether corresponding values ​​in the output of the particular layer include a zero value.

2. The apparatus of claim 1, further comprising a scheduler for: Retrieving from a memory the bitmap matrix associated with a multiplication to be performed using an output of another layer of the plurality of layers in the neural network; and A subset of the multiplication operations for performing in multiplications involving zero values ​​in the matrix are skipped based on the bitmap matrix.

3. The device of claim 1, wherein: The specific layer comprises a first layer in the artificial neural network.

4. The device of claim 3, wherein: The bitmap generation circuitry includes a comparator for determining whether an input value is negative, wherein the bitmap matrix is ​​for populating with a binary value generated by the comparator from the input value.

5. The device of claim 3, wherein: The first layer comprises a layer before a rectified linear unit (ReLU) layer, the output of the first layer being used to be provided as an input to the ReLU layer, the bitmap generation circuit system comprising an exclusive OR (XOR) gate, the XOR gate being used to operate on at least a portion of each of two operands used for multiplication in the first layer to generate the input to the ReLU layer, and the bitmap matrix being used to be filled with binary values ​​generated by the XOR gate.

6. The device of claim 1, wherein: The matrix includes a volume data structure from a memory, wherein the volume data structure represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel of the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel of the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

7. At least one non-transitory machine-accessible storage medium having stored thereon instructions that, when executed on a machine, cause the machine to: obtaining values ​​of a matrix for being provided as input to a first layer of a plurality of layers within an artificial neural network; determining from the values ​​that an output of the particular layer will include a set of output values, the set of output values ​​including at least some zeros; and A bitmap matrix is ​​generated including binary values ​​indicating whether corresponding values ​​in the output of the particular layer include a zero value.

8. The storage medium according to claim 7, wherein: The instructions, when executed, further cause the machine to: Retrieving from a memory the bitmap matrix associated with a multiplication to be performed using an output of another layer of the plurality of layers in the neural network; as well as A subset of the multiplication operations for performing in multiplications involving zero values ​​in the matrix are skipped based on the bitmap matrix.

9. The storage medium according to claim 7, wherein: The specific layer comprises a first layer in the artificial neural network.

10. The storage medium according to claim 9, wherein: The bitmap generation circuitry includes a comparator for determining whether an input value is negative, wherein the bitmap matrix is ​​for populating with a binary value generated by the comparator from the input value.

11. The storage medium according to claim 9, wherein: The first layer comprises a layer before a rectified linear unit (ReLU) layer, the output of the first layer being used to be provided as an input to the ReLU layer, the bitmap generation circuit system comprising an exclusive OR (XOR) gate, the XOR gate being used to operate on at least a portion of each of two operands used for multiplication in the first layer to generate the input to the ReLU layer, and the bitmap matrix being used to be filled with binary values ​​generated by the XOR gate.

12. The storage medium according to claim 7, wherein: The matrix includes a volume data structure from a memory, wherein the volume data structure represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel of the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel of the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

13. A system comprising: processor; Memory; as well as A bitmap generator, executable by the processor to: obtaining values ​​of a matrix for being provided as input to a first layer of a plurality of layers within an artificial neural network; Determining from the values ​​that an output of a particular layer will include a set of output values, the set of output values ​​including at least some zeros; as well as A bitmap matrix is ​​generated including binary values ​​indicating whether corresponding values ​​in the output of the particular layer include a zero value.

14. The system of claim 13, further comprising a scheduler for: Retrieving from a memory the bitmap matrix associated with a multiplication to be performed using an output of another layer of the plurality of layers in the neural network; and A subset of the multiplication operations for performing in multiplications involving zero values ​​in the matrix are skipped based on the bitmap matrix.

15. The system of claim 13, wherein: The specific layer comprises a first layer in the artificial neural network.

16. The system of claim 15, wherein: The bitmap generation circuitry includes a comparator for determining whether an input value is negative, wherein the bitmap matrix is ​​for populating with a binary value generated by the comparator from the input value.

17. The system of claim 15, wherein: The first layer comprises a layer before a rectified linear unit (ReLU) layer, the output of the first layer being used to be provided as an input to the ReLU layer, the bitmap generation circuit system comprising an exclusive OR (XOR) gate, the XOR gate being used to operate on at least a portion of each of two operands used for multiplication in the first layer to generate the input to the ReLU layer, and the bitmap matrix being used to be filled with binary values ​​generated by the XOR gate.

18. The system of claim 13, wherein: The matrix includes a volume data structure from a memory, wherein the volume data structure represents a geometric structure within a specific volume at multiple levels of detail, the specific volume includes multiple voxels at a first lowest level of detail in the multiple levels of detail, a first entry in the volume data structure is used to represent the geometric structure in each voxel of the multiple voxels at the first level of detail, the first entry includes a binary value for each voxel of the multiple voxels at the first level of detail, and the binary value indicates whether the geometric structure exists within a corresponding subvolume of the specific volume defined by the corresponding voxel.

Images