Built-in self-test for system-on-chip programmable vision accelerators

Abstract

To provide a VPU and associated components that are optimized to improve VPU performance and throughput.SOLUTION: A VPU includes a min / max collector, an automatic store predication function, a SIMD data path configuration allowing inter-lane sharing, a transposed load / store including a stride parameter function, a load including a permute and zero insertion function, a hardware, logic device, and memory layout function allowing two point and two by two point lookups, and per memory bank load caching ability. Decoupled accelerators are used to offload VPU processing tasks, and a hardware sequencer is included in a DMA system. The DMA and VPU execute a VPU configuration mode allowing the VPU and DMA to operate without a processing controller for executing dynamic region-based data movement operations.SELECTED DRAWING: Figure 9A

Description

[Background technology]

[0001] A vector processing unit (VPU) is used to perform single instruction multiple data (SIMD) operations in parallel. Common uses of VPUs include operations such as image processing, computer vision, signal processing, deep learning (e.g., for convolution operations), and / or similar.

[0002] In some computer vision applications, for example, the dynamic range of intermediate values ​​is well understood. Therefore, calculated values ​​can be compared to this dynamic range to detect anomalies. However, traditional solutions to detecting and analyzing these minimum and maximum values ​​involve writing all values ​​to memory and then performing analysis of the values ​​in memory, requiring additional processing cycles. Furthermore, to achieve high throughput despite the load-to-use latency, processors with high clock rates may perform software pipelining and / or loop unrolling. However, if the original number of iterations is not divisible by the unrolling factor, some iterations may remain after the unrolled loop completes, requiring an additional extra loop to calculate the value for the final iteration. This extra loop, for example, cannot be unrolled to achieve optimal performance, increasing the system's code size and latency. In traditional single-instruction, multiple-data (SIMD) operations, each SIMD unit can operate independently of each other in parallel on its own data lanes. While some architectures may allow sharing between adjacent neighbors, this limited sharing is restrictive, and implementations of operations often require copying the same operand to each data lane for processing. Additionally, vector SIMD processors may require all memory read operations to use standard or consistent units, such as the vector processing width, which can be inefficient when memory banks are wide. For example, reading elements 4 through 67 in a 64-byte memory width may require two memory reads, one from 0 through 63 and another from 64 through 67. However, this means that many additional values, such as values ​​0 through 3 and values ​​68 through 127, are read even when these values ​​are not needed for the current operation. In traditional instruction sets, if additional data manipulation is required, the data is read and stored in registers, and then additional instructions can be used to operate on the memory data in the registers.For example, this may require loading data, performing a substitution on the data, and then performing the operation using the reconstructed data. Therefore, additional cycles are required for data operations, increasing latency. When performing table lookups using existing VPUs, the table may be duplicated so that only one value can be pulled from the duplicated table, or additional read ports may be added to each memory bank so that multiple values ​​can be read from the same table in the same bank. However, duplicating the table for each value requires additional memory and processing, and adding additional read ports requires additional space on the chip. In traditional VPUs, data caches may not be implemented because the VPU is programmed to execute a smaller set of highly optimized code, allowing the programmer to manage the contents of local data memory. However, doing so requires reading values ​​from each memory bank on each access, even if the data for the next iteration contains overlap with one or more previously read operations.

[0003] To optimize the performance of a processor such as a VPU, the instruction set architecture (ISA) can be extended to create custom instructions that speed up commonly performed operations such as table lookups, convolutions, and / or the like. However, using the ISA in this way requires the processor itself to also perform these operations, which means that the processor will be busy while the extended instructions are executing.

[0004] Additionally, the VPU can use a direct memory access (DMA) system to obtain data for processing by the VPU. Thus, the DMA system can act as a data movement engine, but it can also perform additional operations such as image padding, address manipulation, overlapping data management, scan order management, frame size management, and / or the like. However, the programming complexity for programming the DMA system and VPU increases as the number of DMA resources, e.g., descriptors, channels, triggers, etc., increases. When frame tiles contain spatial or temporal dependencies, dynamically updating DMA resources imposes a processing burden on the system. In traditional DMA systems, fetching unknown or data-dependent data requires a processing controller (e.g., an R5 or ARM processing core) to intervene in the processing cycle to determine updated information to direct the next processing iteration. For example, in object or feature tracking, the VPU can calculate the next location of the object or feature, and then the processing controller intervenes to update the memory addressing information and then triggers the DMA system to use the updated information. However, the intervention of the process controller increases latency and requires more complex programming for operations with domain-dependent data movement algorithms.

[0005] Furthermore, safety-critical applications, such as autonomous and semi-autonomous machine applications, have stringent requirements for detecting and isolating permanent faults. For example, when deep learning, computer vision, sensor processing, and / or other applications are performed on a machine, permanent fault detection must be performed periodically and within an allotted time budget, not only to enable accurate testing but also to enable the application to run properly, e.g., with low latency. To do this, 100 percent coverage may be required while meeting the time budget for each specific application to run with low latency. Traditional approaches use built-in self-test (BIST) to identify faults, but these BIST techniques may not provide sufficient coverage, introduce too much latency into the system, and / or not meet the time budget for running a specific application. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Application No. 15 / 141,703 [Patent Document 2] U.S. Patent Application No. 16 / 101,232 Summary of the Invention [Means for solving the problem]

[0007] Embodiments of the present disclosure relate to improvements to a vector processing unit (VPU), a decoupled accelerator that can be used to handle processing offloaded from the VPU, and a direct memory access (DMA) system that supports data movement between memory and the VPU. Considering various drawbacks of conventional or existing solutions, the VPU of the present disclosure may include a min / max hardware collector in the data path from the VPU to memory, which allows min / max values ​​to be stored before they are stored in memory. In this way, the min / max values ​​are available as soon as the memory write operation is completed, thereby reducing the latency associated with determining min / max values ​​after they are stored in memory. Additionally, the VPU may include an automatic predication feature that can use a predication flag by setting a predication bit for each value calculated in iterations beyond the final iteration. As a result, each set of iterations may include the same number of iterations executed, but due to the predication flag, one or more values ​​from the final set of iterations may not be written to memory. Considering the limitations of existing solutions in sharing data between lanes, the disclosed SIMD architecture can define slices within a processor, each containing multiple lanes, each of which can be configured to communicate with each other. Thus, operands from one lane can be used by other lanes, eliminating the need to copy each operand to each lane for processing. Considering the inefficiencies of loading from a single wide memory bank, the VPU can include multiple smaller memory banks that allow for smaller bit alignment—for example, 16-bit alignment if each memory bank is 16-bit. In this way, an example of reading values ​​4 through 67 can be performed with just one memory read, rather than two, one for 0 through 63 and one for 64 through 127.In addition to this memory bank configuration, the VPU may include a transposed load and / or store function that can offset values ​​stored within a memory bank to prevent bank conflicts and allow more data to be read or written per cycle. To account for the shortcomings of traditional instruction set data manipulation, a load containing a permutation instruction can be used to send the permutation pattern along with the memory address to local memory to retrieve data from memory according to a permutation or data manipulation pattern. Thus, data manipulation and data loads can be performed in the same cycle, thereby reducing latency. To address the shortcomings of duplicating a table for each value or adding a read port for table lookups, a 2-point or 2x2-point lookup can be performed per table per cycle, such that 2 or 4 points can be looked up, respectively. To achieve this, a 2-point or 4-point parallel lookup can be possible using the table's offset storage pattern, a per-memory bank address bus, and associated logic and routing. In an embodiment, each memory bank may include an associated data cache that can be enabled or disabled depending on the given operation. For example, in a filtering operation where there is a lot of data overlap between iterations, the data cache can be used to store values ​​from one or more previous lookups so that only minimal reads are required from each bank, thereby saving energy and power for the system.

[0008] To address the shortcomings of traditional ISAs in VPUs or other processor types, the disclosed systems and methods can use decoupled accelerators that are configured by and communicate with the VPU via shared memory, but can perform certain tasks independently of the VPU, allowing the VPU to continue processing other tasks in parallel with the accelerator. For example, a decoupled lookup table (DLUT) accelerator can be used to improve system performance when performing table lookups. In this way, rather than the VPU performing in-line conflict detection and resolution of memory banks, the DLUT accelerator can identify and resolve conflicts, improving system throughput.

[0009] In consideration of the shortcomings of conventional DMA systems, the disclosed systems and methods may include a hardware sequencer that operates on frame data, including a sequence of commands for the hardware sequencer. For example, the hardware sequencer may operate at the frame level rather than the tile level, eliminating the programming complexity of executing a sequence for the DMA engine and programming the DMA engine to perform the same operations, such as padding and address manipulation. In some embodiments, the DMA system may include a DMA trigger mode in which the DMA engine controls the movement of tiles to vector memory (VMEM) rather than requesting the VPU to trigger the DMA to load the next tile. Thus, the command sequence is reversed, and the DMA triggers the VPU. In consideration of the shortcomings of domain-dependent data movement operations in DMA systems, the DMA system may operate in a tightly coupled loop using the DMA and VPU without requiring intervention from the processing controller. For example, the VPU may update position information in the VMEM for various features and / or objects being tracked, and the DMA may use this updated information to update descriptors in the descriptor memory so that the next data provided to the VPU for processing corresponds to the next position of the feature or object. This process is repeated until processing is complete, which may eliminate the need for processing controller intervention and reduce system latency.

[0010] Additionally, considering the deficiencies of conventional approaches to BIST, the present systems and methods may perform BIST on a multiple input signature register (MISR) to perform fault detection, for example, on a programmable vision accelerator (PVA) in a system on a chip (SoC). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs controlled using one or more processing controllers (or control processors), such as an R5 processor, an ARM processor, a CPU, and / or the like. Thus, each component of the PVA may require testing, and the present systems and methods perform BIST on the MISR to detect 100 percent permanent faults. In this manner, permanent fault detection may be performed to achieve 100 percent coverage of both the control logic and data logic blocks, reporting errors directly to a safety processor to reduce latency, and customizing specific applications to meet associated execution time budgets.

[0011] The present system and method for improving a vector processing unit (VPU) is described in detail below with reference to the following figures of the accompanying drawings: [Brief explanation of the drawings]

[0012] [Figure 1A] FIG. 1 is a diagram of an exemplary min / max collection system, according to some embodiments of the present disclosure. [Figure 1B] 1 is a flow diagram illustrating a min / max collection method according to some embodiments of the present disclosure. [Figure 2A] FIG. 1 is a diagram of an example system including a processor having an address generation unit with automatic predication functionality, in accordance with some embodiments of the present disclosure. [Figure 2B] 1 is a table illustrating a sequence of state changes over time according to some embodiments of the present disclosure. [Figure 2C] 1 is a flow diagram illustrating a method for automatic store predication according to some embodiments of the present disclosure. [Figure 3A] FIG. 1 is an illustration of an exemplary single instruction multiple data (SIMD) data path configuration, according to some embodiments of the present disclosure. [Figure 3B] FIG. 2 illustrates operand sharing between slices in a SIMD architecture for filtering operations, in accordance with some embodiments of the present disclosure. [Figure 3C] FIG. 2 illustrates operand sharing between slices in a SIMD architecture for a dot-product operation, in accordance with some embodiments of the present disclosure. [Figure 3D] FIG. 1 illustrates operand sharing between slices of a SIMD architecture for a payload sorting operation, in accordance with some embodiments of the present disclosure. [Figure 3E] 1 is a flow diagram of a method for computing an output using operands shared across lanes of a SIMD architecture, according to some embodiments of the present disclosure. [Figure 4A] 1A-1C illustrate a logical diagram of a transpose load for reading from and writing to memory, and a diagram of a memory bank of the transpose load corresponding to the logical diagram, in accordance with some embodiments of the present disclosure. [Figure 4B] 1A-1C are logical diagrams of transpose loads with various line pitch and stride parameters for reading from and writing to memory, and diagrams of memory banks of the transpose loads corresponding to the logical diagrams, in accordance with some embodiments of the present disclosure. [Figure 4C] 10 is a flow diagram illustrating a method for configuring a write operation of a transposed load with a stride parameter according to some embodiments of the present disclosure. [Figure 4D] 10 is a flow diagram illustrating a method for performing a write operation of a transposed load with a stride parameter according to some embodiments of the present disclosure. [Figure 5A]1 is a table of data and coefficient layouts for various functions in a SIMD architecture, according to some embodiments of the present disclosure. [Figure 5B] 1 is a table of data and coefficient layouts for various functions in a SIMD architecture, according to some embodiments of the present disclosure. [Figure 5C] FIG. 1 illustrates a hardware architecture for performing loads with reordering and zero insertion in accordance with some embodiments of the present disclosure. [Figure 5D] FIG. 5D illustrates an exemplary use of the hardware architecture of FIG. 5C according to some embodiments of the present disclosure. [Figure 5E] 1 is a flow diagram illustrating a method of loading with reordering according to some embodiments of the present disclosure. [Figure 6A] FIG. 10 illustrates a 16-way parallel table configuration for one-point lookup, according to some embodiments of the present disclosure. [Figure 6B] FIG. 10 illustrates an 8-way parallel table configuration for a two-point lookup, according to some embodiments of the present disclosure. [Figure 6C] FIG. 10 is a logical diagram of a two-way parallel word type table for a 2x2 point lookup, according to some embodiments of the present disclosure. [Figure 6D] FIG. 6D is a diagram of a memory of a two-way parallel word type table for the 2×2 point lookup of FIG. 6C in accordance with some embodiments of the present disclosure. [Figure 6E] FIG. 10 illustrates a layout for processing pairs of lanes using horizontal blending, including data interleaving, according to some embodiments of the present disclosure. [Figure 6F] 10A-10C illustrate intermediate and final results of horizontal blending including data interleaving, according to some embodiments of the present disclosure. [Figure 6G] 1 is a flow diagram of a method for performing a multi-point lookup according to some embodiments of the present disclosure. [Figure 7A]FIG. 2 illustrates elements of arrays of data and coefficients according to some embodiments of the present disclosure. [Figure 7B] 1 illustrates a read operation required for a data operand using a data cache for a memory bank, according to some embodiments of the present disclosure. [Figure 7C] FIG. 10 illustrates the read operations required for coefficient operands using a data cache for a memory bank, according to some embodiments of the present disclosure. [Figure 7D] FIG. 1 illustrates an arrangement of memory banks for use with a load cache, according to some embodiments of the present disclosure. [Figure 7E] FIG. 1 illustrates a hardware architecture for using a data cache in a memory bank, according to some embodiments of the present disclosure. [Figure 7F] 1 is a flow diagram of a method for using a data cache for a memory bank according to some embodiments of the present disclosure. [Figure 8A] FIG. 1 illustrates a system including one or more decoupled accelerators, according to some embodiments of the present disclosure. [Figure 8B] 1 is a flow diagram of a method for using a decoupled accelerator to perform one or more operations, according to some embodiments of the present disclosure. [Figure 9A] FIG. 1 illustrates a system including a separable lookup table accelerator, according to some embodiments of the present disclosure. [Figure 9B] 1 is a table illustrating the actions taken by various components of a separable lookup table accelerator when performing various operations, according to some embodiments of the present disclosure. [Figure 9C] 1 is a flow diagram of a method for using a separable lookup table accelerator to perform one or more operations, according to some embodiments of the present disclosure. [Figure 10A] FIG. 10 is a visualization diagram illustrating padding a frame with a padded value according to some embodiments of the present disclosure. [Figure 10B]FIG. 2 is a visualization diagram illustrating address manipulation for a frame's descriptors, according to some embodiments of the present disclosure. [Figure 10C] FIG. 1 is a visualization diagram showing overlapping data between tiles of a frame, according to some embodiments of the present disclosure. [Figure 10D] 1A-1C are visualization diagrams illustrating various raster scan orders according to some embodiments of the present disclosure. [Figure 10E] FIG. 1 is a visualization diagram showing a scanning order of a cube, according to some embodiments of the present disclosure. [Figure 10F] 1A-1C are visualization diagrams illustrating various vertical mining traversal orders according to some embodiments of the present disclosure. [Figure 10G] FIG. 1 is a visualization diagram showing various image sizes in a pyramidal setting, according to some embodiments of the present disclosure. [Figure 10H] 1 is a diagram of a direct memory access (DMA) system including a hardware sequencer, in accordance with some embodiments of the present disclosure. [Figure 10I] 10H is a diagram of a frame format for storing sequencing commands for controlling a hardware sequencer of the DMA system of FIG. 10H in accordance with some embodiments of the present disclosure. [Figure 10J] FIG. 10I is an illustration of an example frame format of FIG. 10I for a raster scan sequence, according to some embodiments of the present disclosure. [Figure 10K] 10J is a diagram of an example tile structure sequenced in hardware in a raster scan sequence and processing frame addresses using the example frame format of FIG. 10J, in accordance with some embodiments of the present disclosure. [Figure 10L] 1 is a flow diagram of a method for using a hardware sequencer in a DMA system according to some embodiments of the present disclosure. [Figure 11A]1 is a data flow diagram for a process of configuring a direct memory access (DMA) system using a vector processing unit (VPU), according to some embodiments of the present disclosure. [Figure 11B] 1 is a table illustrating a VPU configuration format written by a VPU to a vector memory (VMEM) and read by a DMA system, according to some embodiments of the present disclosure. [Figure 11C] 1 is a flow diagram of a method for configuring a DMA system using a VPU, according to some embodiments of the present disclosure. [Figure 12A] FIG. 1 is a diagram of a built-in self-test (BIST) system that performs cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) in accordance with some embodiments of the present disclosure. [Figure 12B] FIG. 1 is a diagram of a BIST system for parallel channel CRC calculation of a PVA, according to some embodiments of the present disclosure. [Figure 12C] 1 is a flow diagram of a method for performing BIST to detect permanent faults in a PVA, according to some embodiments of the present disclosure. [Figure 13A] FIG. 1 is a diagram of an example autonomous vehicle, according to some embodiments of the present disclosure. [Figure 13B] FIG. 13B is a diagram of example camera positions and fields of view for the example autonomous vehicle of FIG. 13A, according to some embodiments of the present disclosure. [Figure 13C] FIG. 13B is a block diagram of an example system architecture of the example autonomous vehicle of FIG. 13A, in accordance with some embodiments of the present disclosure. [Figure 13D] FIG. 13B is a diagram of a system for communicating between a cloud-based server and the example autonomous vehicle of FIG. 13A, according to some embodiments of the present disclosure. [Figure 14] FIG. 1 is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 15]FIG. 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Systems and methods are disclosed relating to various components of a system-on-chip (SoC), such as a vector processing unit (VPU), a direct memory access (DMA) controller, and a hardware accelerator (e.g., a programmable vision accelerator (PVA), such as a PVA including one or more pairs of a VPU and a DMA). For example, in various embodiments of the present disclosure, a PVA may include one or more VPUs controlled using one or more DMA systems and one or more processing controllers (or control processors), such as an R5 processor, an ARM processor, a CPU, and / or the like. The present disclosure, including various components of an SoC, may be described in connection with, but is not intended to be limiting of, an exemplary autonomous vehicle 1300 (also referred to herein as “vehicle 1300” or “ego-vehicle 1300,” examples of which are described in connection with FIGS. 13A-13D ). For example, but not limited to, the systems and methods described herein may be used in non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), piloted and non-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or e-bikes, aircraft, construction vehicles, underwater craft, drones, and / or other vehicle types.Additionally, while the present disclosure may be described in the context of computer vision, machine learning, artificial intelligence, image processing, and / or the like, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology space where a vector processing unit (VPU), direct memory access (DMA) system, instruction set architecture (ISA), programmable vision accelerator (PVA), discrete accelerator, discrete lookup table, hardware sequencer, single-input multiple data (SIMD) architecture, and / or other one or more components of an SoC may be used. Furthermore, while the components and associated processes described herein may be described in the context of an SoC, this is not intended to be limiting, and the components may be implemented as stand-alone components, as separate components of a system, and / or as integrated components on an SoC. In some embodiments, the systems, components, features, functions, and / or methods of the present disclosure may be integrated into the example autonomous vehicle 1300 of FIGS. 13A-13D, the example computer processing device 1400 of FIG. 14, and / or the example data center 1500 of FIG. 15.

[0014] Min / Max Hardware Collector for Anomaly Detection For example, in computer vision applications, and especially in safety-critical vision applications, calculating the dynamic range of intermediate results is an important task. For example, to detect noise or errors in intermediate calculations, a known or expected dynamic range of values ​​can be used to identify values ​​that fall outside this range. In such instances, if a value falls outside the known or expected dynamic range, the value can be flagged as corresponding to noise, error, and / or another problem. Therefore, it may be desirable to collect minimum (min) and maximum (max) values ​​of intermediate results to detect data anomalies. In practice, these anomalies may be caused by, but are not limited to, noise in the image sensor, corner cases in the algorithm, or data corruption in memory or interconnects. Given these issues, collecting min / max values ​​is an effective way to detect outliers in this data. Min / max values ​​are also used in certain algorithms.

[0015] In certain instances, in autonomous vehicle applications, runtime exceptions, such as infinity or not a number, can result in invalid values ​​or errors, causing failures or otherwise undesirable results. With this in mind, algorithms running as part of the autonomous vehicle platform can be evaluated to determine a range of possible values, i.e., intermediate or otherwise, obtained during processing. Once the range of values ​​is known, the actual calculated value can be compared to the known range, and values ​​outside of minimum or maximum thresholds can be flagged as errors. If an error is flagged, a processing change can be implemented, such as ignoring the data for a given iteration or identifying and correcting the problem. In this way, possible runtime exceptions are considered, not relied upon, by the autonomous vehicle, and runtime exceptions are not tolerated.

[0016] As another example, min / max collection can be used in certain algorithms to normalize intermediate results to a particular numeric range to allow for more accurate processing, such as block floating point. This normalization process can include a dynamic range collection step that collects the minimum and / or maximum values ​​of an array, and an adjustment step that applies a scaling factor to the array. However, in traditional processes, collecting min / max values ​​requires writing all values ​​to memory, then analyzing the values ​​for min / max and adjusting them to scale.

[0017] Thus, these traditional approaches to evaluating min / max are implemented in software and require additional processing cycles. For example, the algorithm itself can run to calculate values, and then software can run to determine the min / max and compare the min / max to a known range of values ​​to identify anomalies. The software must execute additional instructions to read elements of an array of intermediate results and then perform the min / max operation. This results in a longer execution time for the system to detect anomalies, as the algorithm runs to completion and then performs the additional process of calculating the min / max of the algorithm's output. This can delay downstream processing until the min / max is calculated and compared to a threshold, or a downstream task may begin calculating erroneous data while the min / max evaluation is taking place. This not only increases execution time, but also increases the processing requirements and energy consumption of the system, as additional cycles are performed to identify anomalous data.

[0018] Referring to FIG. 1A, FIG. 1A illustrates an example processor architecture 100 for min / max collection, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. The various functions described herein performed by the entities may be implemented by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, architecture 100 may include components, features, and / or functionality similar to those of example autonomous vehicle 1300 of FIGS. 13A-13D, example computing device 1400 of FIG. 14, and / or example data center 1500 of FIG. 15.

[0019] In view of the deficiencies of conventional min / max evaluation processes as described herein, the present disclosure includes systems and methods for min / max collection using hardware. For example, upon calculation, the calculated values ​​may be written to memory 106, e.g., local memory, for use in downstream calculations within the same or another algorithm. To reduce execution time and processing, min / max collection hardware (e.g., min / max collector 104) may be used to capture the min / max values ​​before or as they are written to memory 106, rather than, for example, waiting for the values ​​to be read from memory 106 and then analyzing the min / max values. For example, an enable bit may be used to enable the min / max collection function of min / max collector 104, which, when enabled, may update the min / max values ​​as they are calculated and written to memory 106 using processor 102 (e.g., before or simultaneously with storage in memory 106). In an embodiment, an enable bit may indicate the type of array being calculated, e.g., signed or unsigned, such that min / max collector 104 is configured to collect min / max values ​​for a particular type of array. For example, an enable bit or another type of control feature may be used to disable min / max collector 104 and / or to configure min / max collector 104 to collect unsigned min / max values ​​or signed min / max values. A data path that stores data may include min / max collection logic in min / max collector 104 that reads values ​​calculated using processor 102 and stored in a register file to update or maintain the min / max values.

[0020] For example, during calculation, the current minimum and / or current maximum values ​​may be maintained within the min / max collector 104, and the current minimum and / or current maximum may be updated when a new, smaller minimum and / or a new, larger maximum is written to memory 106. If the newly calculated value is greater than the minimum and / or less than the maximum, the current minimum and / or maximum may be maintained by the min / max collector 104. In this manner, the min / max collector 104 can maintain the current minimum and / or maximum values ​​as each value is fully calculated throughout the calculation. Once calculation over a given iteration is complete, the minimum / maximum values ​​are immediately available to the min / max collector 104, and software and / or hardware can be used to compare these stored values ​​with minimum and / or maximum thresholds associated with the particular algorithm or calculation being performed to determine whether an anomaly exists. For example, mechanisms may be included that allow the collected minimum / maximum values ​​to be read for evaluation. Thus, compared to previous approaches, after an algorithm has been fully executed, the minimum / maximum values ​​are immediately available, and another cycle to calculate the minimum / maximum is not required. Furthermore, in embodiments, min / max collector 104 (e.g., including hardware and / or logic) can be store predication aware, such that min / max collection can exclude a particular data item if that particular data item is prohibited from being stored to memory 106, for example, by per-lane store predication. For example, if an address from the address generator includes a store predication flag, the calculated value can be canceled not only from being stored in memory 106, but also from updating min / max collector 104.

[0021] In some embodiments, the min / max collector 104 may be implemented as a feature of a system that includes an address generator, such as the address generator described in U.S. Patent Application No. 15 / 141,703, filed April 28, 2016, which is incorporated herein by reference in its entirety. The address generator may be included in any type of processor or other processing unit, such as a vector processing unit (VPU), central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), data processing unit (DPU), and / or another processing unit type (such as those described with respect to FIGS. 13A-13D, 14, and / or 15). In some embodiments, one or more VPUs may be included in a programmable vision accelerator (PVA) and / or as part of a system-on-chip (SoC).

[0022] As a non-limiting example, the input of a particular sensor type or algorithm may be limited to 16-bit units. To determine the dynamic range of this particular sensor and / or algorithm, the operations associated with the algorithm processing the sensor input may be evaluated. In such an example, assume the first operation is to add two 16-bit numbers, resulting in a first intermediate result of a 17-bit number. The 17-bit number may then be multiplied by a 5-bit number to produce a 22-bit number. If this is the end of the algorithm, it may be determined that the output must not exceed 22 bits. Similarly, the minimum value may be evaluated. Thus, during deployment, if the minimum / maximum value falls outside this known range (e.g., 22 bits), the output may be flagged.

[0023] In some embodiments, the store-data path (e.g., between processor 102 and memory 106) may include saturation and / or rounding logic 108 that rounds values ​​stored in memory 106 to between certain upper and lower limits, thresholds, or based on some particular rule. Thus, in traditional approaches, min / max evaluation may occur after saturation and / or rounding. If an anomaly exists, the anomaly may not be detected by such traditional approaches because the saturation and / or rounding may mask the anomaly, e.g., low and / or high values ​​may saturate between the upper and lower limits set by the saturation logic.

[0024] However, in certain implementations, unsaturated, unrounded, or absolute minimum / maximum values ​​may be valuable or desirable, for example, in addition to or instead of saturated minimum / maximum values. Thus, the min / max collector 104 of the present disclosure may collect minimum / maximum values ​​from raw or unsaturated data (e.g., before the values ​​are manipulated using the saturation / rounding logic 108) for use in anomaly detection. In an embodiment, collection of an average value of the data or an average absolute value of the data may be performed. The average value may be calculated, for example, by summing the elements, reading the sum back from the address generator's configuration register, and performing a division by the number of data items stored (which may be known to the application). In this manner, the absolute minimum / maximum, sum of values, and / or sum of absolute values ​​may be added to the processor's store data path, and the resulting statistics may be configured and collected. For example, they may be added to the address generator's configuration feature set or managed separately. In some embodiments, min / max collector 104 may collect values ​​before and / or after saturation, rounding, or other calculations that use saturate / round logic 108 .

[0025] Referring now to FIG. 1B, each block of method 110 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 110 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 110 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although method 110 is described with respect to architecture 100 of FIG. 1A, it may be performed by any system or combination of systems, including, but not limited to, those described herein.

[0026] 1B is a flow diagram illustrating a min / max collection method 110 according to some embodiments of the present disclosure. The method 110 includes, at block B102, calculating one or more values. For example, the one or more values ​​may be calculated using the processor 102 when executing one or more algorithms, such as a neural network, a computer vision algorithm, a filtering algorithm, etc.

[0027] The method 110 includes, in block B104, comparing a value of the one or more values ​​to a currently stored minimum value and a currently stored maximum value. For example, the min / max collector 104 may compare each value of the number of values ​​to be stored in the memory 106, e.g., a value in a register file, to a currently stored minimum value and a currently stored maximum value (e.g., currently stored in the hardware min / max collector 104). In such an example, the min / max collector may compare the value to the currently stored minimum and / or maximum value before or at the same time that the value is calculated and stored in memory. In one or more embodiments, the min / max collector may be included in a data path between a hardware unit that calculates one or more values ​​and a memory unit that stores one or more values.

[0028] The method 110 includes determining whether the value is either greater than the currently stored maximum value or less than the currently stored minimum value in block B106. For example, the system (e.g., the hardware min / max collector 104) can determine whether each value to be stored in memory is greater than the currently stored maximum value or less than the currently stored minimum value based on the comparison in block B104.

[0029] The method 110 includes updating the currently stored minimum value to a value that is less than the currently stored minimum value, in block B108. For example, if the calculated value to be stored in memory is less than the minimum value currently stored in the hardware min / max collector, the hardware min / max collector can update the currently stored minimum value to the calculated value.

[0030] The method 110 includes, in block B110, updating the currently stored maximum value based on a value greater than the currently stored maximum value. For example, if the calculated value to be stored in memory is greater than the maximum value currently stored in the hardware min / max collector, the hardware min / max collector can update the currently stored maximum value to the calculated value.

[0031] In this way, the min / max can be dynamically updated while storing the values ​​so that once some number (e.g., all) of the values ​​have been stored, the min / max are immediately available by reading the min / max from the currently stored values ​​in the min / max collector.

[0032] Automatic Store Predication Even among high-clock-rate processors, a common implementation is to configure the processor with multiple pipeline stages. As a result, there can be latency, such as load-to-use latency, between the issuance of an instruction to load a register from local memory and the time the register becomes available to operate on another instruction. To achieve high throughput despite the load-to-use latency, processor compilers and application developers can use software pipelining and / or loop unrolling. For example, software pipelining can be used to overlap the execution of multiple iterations of a loop, and loop unrolling can be used to expand a loop body by repeating its contents multiple times. These techniques can also enable multiple iterations of a loop's contents to execute simultaneously, resulting in fewer, ideally zero, idle cycles in the schedule. When performing loop unrolling, a compiler can divide the number of loop iterations by an unrolling factor. For example, the compiler can assume that the original number of iterations is a multiple of the unrolling factor, allowing the unrolled loop to execute with equivalent functional behavior. In such an example, if the original number of iterations was 60 and the loop was to be unrolled by a factor of 6, the unrolled loop may execute for 10 iterations. However, if the original loop iterations were 64, and with normal integer division, 64 / 6 is also 10, the loop may not execute enough times (e.g., the remainder of 4 iterations may not execute), resulting in different code behavior after unrolling and potentially causing application failures. Some techniques add assertion statements to ensure that the number of iterations is indeed a multiple of the unroll factor.

[0033] The set of steps or operations in a loop body may have a narrow range of optimal or desired unrolling factors. For example, the unrolling factor may be bounded at the lower end by the minimum number of copies of the loop's code to schedule in order to bridge gaps due to various latencies and achieve the best performance, and at the upper end by the maximum number of copies to schedule due to limitations in register file capacity, which may result in excessive register spilling (saves to the stack and restores from the stack) and result in suboptimal scheduling. As a further example, unrolling at a power of two is acceptable for many applications because it is feasible to select combinations of tile width and tile height that allow for iterations of several powers of two (e.g., 2, 4, 8, etc.). However, in an embodiment, the loop body may also be optimally unrolled six or seven times, while unrolling four or eight times may not be as efficient. In either case, unrolling a loop to achieve optimal scheduling may impose undesirable limits on the number of iterations. Therefore, conventional techniques for addressing this problem may result in performance degradation and increased code size.

[0034] For example, because limiting the number of iterations is inconvenient, a programmer can write two types of loops where there should be no limit on the number of iterations: a "multiple" loop and a "remainder" loop. To illustrate, the following illustrative code segments show: Code 1 - A vector addition loop without loop unrolling. Code 2 - The same loop with loop unrolling by 6, which can only work if the number of iterations is a multiple of 6. Code 3 - A two-type loop solution that works for any number of iterations, but the remainder loop is not unrolled, making it less efficient and resulting in larger code size due to the additional loop and iteration calculation.

[0035] Code 1

number

[0036] Code 2

number

[0037] Code 3

number

[0038] Using the vector processing unit (VPU) of the present disclosure, Code 1 can achieve 6 cycles per iteration, Code 2 can achieve 1 cycle per iteration, and the performance of Code 3 may vary depending on the number of iterations: when the number of iterations (niter) is niter=60 (a multiple of 6, so no remainder is executed), Code 3 can achieve 1.0 cycle per iteration, and when niter=64 (the remainder loop is executed 4 times), Code 3 can achieve, on average, 1.3125 cycles per iteration (e.g., (60*1+4*6) / 64=84 / 64=1.3125).

[0039] Referring to FIG. 2A, FIG. 2A is an exemplary system 200 including a processor 202 (e.g., a VPU) including an address generation unit with auto-predication capabilities, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, processor 202 may be included in and / or may include components, features, and / or functionality similar to those of example autonomous vehicle 1300 of FIGS. 13A-13D, example computer processing device 1400 of FIG. 14, and / or example data center 1500 of FIG. 15.

[0040] In embodiments of the present disclosure, loads and stores within a code segment may use an address generator 204 of the processor 202 (e.g., a VPU). For example, for each load and store, an address generator (agen) argument (agen_a, agen_b, agen_c) may be provided to the load / store function. The argument may identify an address generator register that contains parameters that may be used to calculate the address for the particular load and / or store operation, such as an address pointer, an iteration count, a current loop variable value, etc. In some embodiments, the VPU may be designed such that each address generator register supports six (or other values) addressing dimensions and therefore contains six (or other values) iteration counts and six (or other values) loop variables.

[0041] To account for the limited number of iterations with loop unrolling, the disclosed systems and methods may include an address generator 204 with logic (e.g., a predication flag or bit 208) for automatic predication of stores by the address generator 204. For example, predication may be used to provide a conditional execution indication, such as whether to execute something (or not). The value of the predication bit 208 (e.g., 0 to store or 1 to prevent a store, or vice versa) may be used to indicate whether an instruction is executed. Execution may not refer to the actual execution of an iteration, but rather to whether a value resulting from the execution of an iteration is stored in memory. Thus, in embodiments, an instruction that is not executed due to a predication flag may refer to an instruction or iteration that is executed but is prevented or prevented from changing the state of the memory 206 due to the result of the execution. Predication may include instruction-level predication and lane-level predication. Instruction-level predication may be used to indicate whether an entire instruction needs to be executed, while lane-level predication may be used to indicate which lanes of data need to be executed.

[0042] In some embodiments, after the loop variable reduces the iteration count to zero, execution of any subsequent store instructions automatically turns off predication, inhibiting further writes to memory 206. In this manner, the automatic store predication feature may enable code to be written clearly, for example, by rounding up iteration counts that are not a multiple of 6 (or another unrolling factor) to the next multiple of 6 and leaving iteration counts that are not a multiple of 6 (or another unrolling factor) unchanged. While a factor of 6 is used, this is not intended to be limiting, and any unrolling factor may be used without departing from the scope of this disclosure. Code 4 below includes an example of vector addition using automatic store predication.

[0043] Code 4

number

[0044] Code 4 allows the unrolled loop to execute 11 times, with 1.03125 cycles per iteration (e.g., 11 x 6 / 64 = 1.03125), if the original iteration count (niter) is 64. Another way to account for the iteration count limit that is a multiple of the unroll factor is to calculate the required predication flags for the loop and provide the predication flags in the store instruction. For example, Code 5 below shows an exemplary implementation of predication flag calculation:

[0045] Code 5

number

[0046] Because Code 5 may be compiled to 1.5 cycles per iteration on a VPU of the present disclosure, automatic predication may include a performance advantage over loop-computed predication. In an embodiment, the VPU may include a 7-way very long instruction word (VLIW) instruction style and may include two scalar slots per cycle for the scalar operations required to compute predication. If a loop has more vector operations per iteration, sufficient scalar slots may be included so that the predication computation fits within the available slots and does not cause performance degradation. Even for computational loops where there is no performance impact to computing predication in real time, using automatic predication may still provide code size and energy consumption advantages.

[0047] Thus, software can be used to set a number of iterations (e.g., N1 through N6), and the software can cause the execution of address generator-based loads / stores, typically in a loop. The address generator hardware can maintain loop variables (e.g., variables I1 through I6) and advance the address pointer appropriately. When the address generator-based loads / stores have executed for a preset number of iterations, the address pointer can become stuck at the last valid address, and auto-predication can be turned off (e.g., by setting a predication flag) to prevent subsequent stores to memory. Thus, address generator 204 can include an internal Boolean state for “auto-predication off,” and the loop variable iteration logic can be configured to support auto-predication being turned off. For example, with reference to FIG. 2B , when the address generator is initialized, in addition to loop variables I1 through I6, the value of parameter auto-predication off (“auto_pred_off”) (e.g., predication bit 208) can be initialized or reset to “0.” After the loop variable reaches zero for the programmed number of iterations, auto_pred_off may be updated to '1'. Then, execution of any subsequent store instructions may automatically turn off predication as a result of the predication bit being '1', preventing further writes to memory.

[0048] In the example of Figure 2B, the iteration numbers of registers N1-N6 of the address generator may be programmed as follows: N1 = 4, N2 = 2, N3 = N4 = N5 = N6 = 1. Therefore, the total number of programmed iterations may be 4*2*1*1*1*1 = 8, resulting in the execution of the sequence shown in Figure 2B. As shown, the initial state and the following seven executions (e.g., the first eight iterations) may correspond to an auto_pred_off bit value of 0, and the eighth and ninth executions (e.g., the last two iterations) may correspond to an auto_pred_off bit value of 1, thus preventing the results of the ninth and tenth executions from being stored in memory.

[0049] In practice, a VPU may be configured to handle several vector units, e.g., 8, 16, etc., working simultaneously. Therefore, the VPU may require the array to be a multiple of the number of vector units. This setup works well when the array is a multiple of the number of vector units. However, in general, the array may not be a multiple of the vector units (e.g., because there is no guarantee that data is calculated according to arrays of the same size). As a result, the array is padded to ensure that processing is always performed on batches of the same size. For example, the remaining iterations may be padded with the value "0," but this still requires additional cycles in software to process the padded values. Therefore, padding may be inefficient because the added data results in wasted calculations and increases software complexity, a common problem in single instruction, multiple data (SIMD) software. Therefore, automatic store predication may be used to account for this issue.

[0050] In a non-limiting example, if 16 batches are used, many of the 16 may be generated from the array, and the remaining values ​​may be included in the final batch, with any remaining or remaining space in the 16 batches predicated using the predication flag. In a specific example, if the array size is 82, five complete sets of 16 may be generated, with the remaining two elements included in the final iteration, and the other 14 may be automatically predicated, minimizing the needless computation of padding the batch with 14 values ​​and performing unnecessary computations on the padded data. As a further example, if the vector processing granularity includes a width of 32 and the array has 100 elements, three complete 32-element vectors may be processed, with the remaining four elements processed in four of the 32 lanes (e.g., the predication flag may be turned on), and the other 28 lanes may have predication turned off. In this way, a programmer may be able to vectorize arrays that are not a multiple of the number of units in a sample. For example, the hardware can calculate the number of elements to actually write to memory for every store and communicate this information to the store unit. Thus, even if it is possible to perform math on the padded or extra elements and store them, this extra calculation and storage is inefficient. This eliminates the need for extra reads, and a predication flag can be set so that the write of the calculated value from the padded value to memory is not performed (e.g., stopped or prevented). This automatic predication can be done at the instruction level, and software can be added to perform lane-level predication.

[0051] Additionally, for automatic predication, additional information may not be necessary because the address generator can be programmed for several iterations, and therefore the address generator may have memory to support automatic predication, and software instructions may be added to automatically activate store predication and predication off. In this way, the hardware can determine when to store a full result or when to store less than a full result, for example, by turning predication off or otherwise signaling off in the last iteration, and this can be done at zero cost while maintaining performance. If only software were used, the process would require extra cycles and therefore be slower.

[0052] In some embodiments, predication may be used at a per-lane level, such that these implementations can use iteration counts that are not multiples of the loop unrolling factor, as well as efficiently handle any task size that is not a multiple of the vector width. In such embodiments, per-lane predication may be driven using vector registers, which can provide the advantage that the information is calculated in real time. Using vector registers can provide a shortcut: copying from the vector register to a scalar predication register and then using the scalar predication register, eliminating the need to use predication flags per lane. For example, per-lane predication may be performed by vector registers, which can be beneficial when per-lane predication information is calculated in a loop, and the calculation can be vectorized.

[0053] For example, if a replacement of a specific value in an array is to be performed, such as replacing any value greater than 100 with 999, the code can be written as follows:

number

[0054] While this code may be functionally correct, it may cause performance degradation. Therefore, the code may be vectorized with per-lane predication, for example, by incorporating the following code:

number

[0055] If the predication calculation were vectorized in this way and per-lane predication could only be conveyed via scalar predication registers, execution time would increase due to the need to copy predication information from vector registers to scalar predication registers.

[0056] However, using per-lane predication driven directly by the vector register features described herein, as shown in the code below, can achieve better performance than moving the predication mask from vector lane 0 to a scalar register, in this example by performing bit packing.

number

[0057] Referring now to FIG. 2C , each block of method 220 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 220 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 220 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. While method 220 is described with respect to system 200 of FIG. 2A , it may be performed by any system or combination of systems, including, but not limited to, those described herein.

[0058] 2C is a flow diagram illustrating a method 220 for automatic store predication according to some embodiments of the present disclosure. The method 220 includes, at block B202, determining a total number of iterations. For example, the address generator 204 may determine the total number of iterations to be performed for a given instruction.

[0059] The method 220 includes dividing the total number of iterations into sets of multiple iterations at block B 204. For example, the address generator 204 can divide the iterations by an unrolling factor to generate a loop body that includes the number of iterations of the loop.

[0060] In block B206, the method 220 includes determining that a set of iterations among the sets of iterations includes a first number of iterations that is less than a second number of iterations corresponding to another set of iterations among the sets of iterations. For example, the address generator 204 may determine that a set of iterations includes fewer iterations than another set after dividing the iterations by the spread factor. For example, if the spread factor is 6 and the number of iterations is 62, there may be 11 sets of iterations. That is, 10 sets include 6 iterations and one set includes 2 iterations. Therefore, the address generator 204 may determine that of the remaining set of iterations including 2 iterations, two iterations need to be performed and the other four need to have predication turned off.

[0061] During execution of the set of iterations, the method 220 includes generating a predication flag corresponding to at least one iteration of the set of iterations in block B208. For example, upon determining that the set of iterations does not contain the same complete set of iterations as the other sets of iterations, the address generator 204 may enable the predication flag (changing the value of the predication off bit 208) to indicate that the results of the remaining iterations should be stored or written to memory.

[0062] At block B210, the method 220 includes preventing a value corresponding to at least one iteration of the set of iterations from being written to memory based at least in part on the predication flag. For example, a calculated value may be prevented from being written to memory or calculated based on the predication flag being set.

[0063] Improved SIMD data path organization for vector processors In traditional single-instruction, multiple-data (SIMD) architectures, each SIMD processing unit operates independently of the others in parallel on its own data lane. On some machines, each SIMD processing unit can communicate directly with its nearby neighbors (e.g., its left and right neighbors in a linear array processing unit, or its north, south, east, and west neighbors in a two-dimensional (2D) array or processing unit). However, communication only between adjacent data paths is limited and expensive for implementing operations that require a significant number of input operands. As an example, convolution is a common operation in image processing, computer vision, machine learning, and so on. During convolution, various filters may be applied to adjacent pixels, such as, in a non-limiting example, a three-tap one-dimensional (1D) filter operation, which requires three data operands and three coefficient operands. If these operands could not be shared across the data lanes of the SIMD architecture, six operands would need to be brought into each data lane to produce the result for that particular lane. To address this, some common approaches implement multiple read ports on the register file, which requires not only additional surface area for SIMD architectures but also additional power to operate.

[0064] Considering the deficiencies of conventional SIMD architectures, the SIMD architecture of the present disclosure may enable communication between lanes by defining slices in processors, such as vector processing units (VPUs), which consist of multiple lanes as a group. In a non-limiting example, in a processor, a SIMD lane configuration may include a hierarchical configuration including a 384-bit data path, which may be divided into, for example, eight lanes of 48 bits (extended word), 16 lanes of 24 bits (extended halfword), or 32 lanes of 12 bits (extended byte). In such an example, each byte may be extended by four bits. The first layer of communication on individual lanes, sometimes called a SIMD slice, may be (for example, but not limited to) 96 bits wide and consist of two extended word lanes (e.g., two 48-bit lanes), four extended halfword lanes (e.g., four 24-bit lanes), or eight extended byte lanes (e.g., eight 12-bit lanes). In a non-limiting example, the entire processor data path may include four SIMD slices, and the second layer of communication may be global between all four (or other number) SIMD slices and all lanes. In this manner, operand sharing between lanes of each slice may be achieved, which may be useful for instructions such as filtering, dot products, and payload sorting. SIMD architectures may be included in VPUs or other processor types, such as the processor of the example autonomous vehicle 1300 of FIGS. 13A-13D, the example computing device 1400 of FIG. 14, and / or the example data center 1500 of FIG. 15.

[0065] A SIMD instruction set architecture (ISA) may allow sharing between several (e.g., eight) lanes within a slice due to the physical routing of the SIMD architecture. For example, as shown in FIG. 3A, within each slice, communication between 32-bit word data types, 16-bit halfword data types, and 8-bit byte data types may be possible. As a result, in an example such as the filtering operation shown in FIG. 3B, if there are four input operands and four coefficients, an 8-bit by 8-bit multiplication and accumulation may be performed on halfwords, and the coefficients may be shared with data from different lanes. To perform the same calculation that can be performed using only three input operands in the SIMD architecture of the present disclosure, in a conventional SIMD architecture, each lane would need to load all eight operands. As a result, and because each read port is associated with increased surface area and energy consumption, requiring only three read ports saves space and energy to execute such instructions. During operation, four accumulators, e.g., 0, 1, 2, and 3, may be populated with the results of the following calculations as they are shared between lanes within a slice: ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3] ACC[1]+=D[1]*C[0]+D[2]*C[1]+D[3]*C[2]+D[4]*C[3] ACC[2]+=D[2]*C[0]+D[3]*C[1]+D[4]*C[2]+D[5]*C[3] ACC[3]+=D[3]*C[0]+D[4]*C[1]+D[5]*C[2]+D[6]*C[3]

[0066] As illustrated, for example, ACC[0] can access other lanes of src1a, including D[1], D[2], and D[3], as well as other lanes of src2, including C[1], C[2], and C[3]. Similarly, other accumulators (ACCs) can access various lanes of src1 and src2. This type of calculation would not be possible in a traditional vector processor, where sharing between lanes is limited or minimal. These calculations may include, for example, a sliding window approach, where each accumulator contains the results of moving a sliding window relative to the previous accumulator. For example, the first accumulator operates on D[0], D[1], D[2], and D[3]; the second accumulator operates on D[1], D[2], D[3], and D[4]; and so on. Each accumulator uses the same coefficients C[0], C[1], C[2], and C[3]. This is possible thanks to the physical routing, which is shared between the lanes of the slice in the SIMD architecture.

[0067] As another exemplary implementation of the SIMD architecture of the present disclosure, with reference to FIG. 3C , the dot product in a vector multiplication operation may be performed using lane sharing. In this example, two indices, e.g., D[0][0], indicate which lane the data belongs to and which set of outputs the data belongs to. In the dot product calculation, each lane uses only the data operands from its own lane, but the coefficients are shared between lanes. Thus, the outputs from each lane may use all four coefficients at some point during the dot product operation. During the operation, because of the sharing between lanes in the slice, four accumulators, e.g., 0, 1, 2, and 3, may be input with the following calculation results: ACC[0]+=D[0][0]*C[0]+D[1][0]*C[1]+D[2][0]*C[2]+D[3][0]*C[3] ACC[1]+=D[0][1]*C[0]+D[1][1]*C[1]+D[2][1]*C[2]+D[3][1]*C[3] ACC[2]+=D[0][2]*C[0]+D[1][2]*C[1]+D[2][2]*C[2]+D[3][2]*C[3] ACC[3]+=D[0][3]*C[0]+D[1][3]*C[1]+D[2][3]*C[2]+D[3][3]*C[3]

[0068] Another exemplary operation that can benefit from the SIMD architecture of the present disclosure is the 2-point sort operation of FIG. 3D , which uses two values ​​to sort payloads. This 2-point sort utilizes communication between pairs of lanes within a slice and is useful, for example, in various computer vision applications. For example, lane 0 contains the key for item 0, and lane 1 contains the corresponding payload, and the payloads can be sorted according to a key comparison for each key / payload pair, as in the following code:

number

[0069] Referring now to FIG. 3E, each block of method 300 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 300 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 300 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. While method 300 is described with respect to the SIMD architecture of the present disclosure, it may be performed by any system or combination of systems, including, but not limited to, those described herein.

[0070] 3E includes a flowchart of a method 300 for computing an output using operands shared across lanes of a SIMD architecture, according to some embodiments of the present disclosure. The method 300 includes, at block B302, dividing a bit width of the processor into a plurality of data slices, each data slice including a second bit width smaller than the first bit width, and each data slice of the plurality of data slices including a plurality of lanes, each data slice including a third bit width smaller than the second bit width. For example, a vector processor may be divided into a number (e.g., four) of slices, and each slice may include a number of lanes.

[0071] At block B304, the method 300 includes loading a first vector into a first vector register such that a first lane of the plurality of lanes includes a first operand of the first vector and a second lane of the plurality of lanes includes a second operand of the first vector. For example, with reference to FIG. 3B, a first data operand D[0] of the first vector may be loaded into the first lane and a second data operand D[1] corresponding to the first vector may be loaded into the second lane.

[0072] The method 300 includes, at block B306, loading the second vector into a second vector register such that a first lane of the plurality of lanes includes a third operand of the second vector and a second lane of the plurality of lanes includes a fourth operand of the second vector. For example, with reference to FIG. 3B , a third first coefficient operand C[0] may be loaded into the first lane and a second coefficient operand C[1] corresponding to the third vector may be loaded into the second lane.

[0073] The method 300 includes, at block B308, using an instruction to calculate an output based at least in part on the first operand, the second operand, the third operand, and the fourth operand. For example, with reference to FIG. 3B , a first accumulator (ACC[0]) may receive the result of the calculation ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3], which includes, among other things, the values ​​of D[0], D[1], C[0], and C[1]. This calculation may be performed thanks to internal sharing and routing between lanes of each slice.

[0074] The method 300 includes storing the output in a register at block B310. For example, with reference to Figure 3B, the calculated output may be stored in an accumulator register ACC[0], which may then be stored in memory.

[0075] Transposed load and store operations with stride parameters In conventional vector single instruction multiple data (SIMD) processors, local data memory may be sized to match the vector processing width. For example, for a 256-bit vector SIMD processor capable of processing 32 lanes of 8 bits, 16 lanes of 16 bits, or 8 lanes of 32 bits, the local data memory may include, for example, a 256-bit wide memory or a 512-bit wide memory (e.g., twice the width of the processing bit width). In such an instance, the local data memory is configured as a single memory bank with full-width memory words. However, a wide vector SIMD processor with a single bank of full-width memory words may be inefficient, especially when memory accesses are misaligned. For example, to load an array for a 16-element 32-bit array with byte addresses 4 through 67, the processor may require two memory reads, e.g., one read address 0 through 63 (including addresses 0 through 3, which contain data not needed for the current operation) and a second read address 64 through 127 (including addresses 68 through 127, which contain data not needed for the current operation). Thus, without the banked memory architecture of the present disclosure, the access pattern may be achieved with multiple loads or stores, potentially resulting in slower computational kernels, reduced performance, and increased power consumption.

[0076] With this in mind, instead, a single wide memory bank can be configured as multiple memory banks, such as a 16-bit memory bank (e.g., 32 banks of 16-bit memory, providing 512 bits of memory bandwidth per clock cycle). In this way, read and / or write operations can occur within any 16-bit alignment, thereby reducing the amount of redundant read / write operations, as described in the example above. In such a memory configuration, reading addresses 4 through 67 may require only a single memory read. In addition to memory bank configurations that include smaller individual memory banks, transposed load and / or store functions can be implemented. For example, a lane offset parameter K can be used to define the offset in the row address applied to each subsequent lane of memory. The lane size can correspond to the size of the data element, e.g., 8 bits, 16 bits, 32 bits, etc. If the 2D array is stored in memory with a line pitch of W*K+1 elements, the staggered access pattern may be converted to a perpendicular pattern, where K is an offset parameter and W is 64 / lane size (or data element size). For example, for 32-bit data elements, the line pitch may be 16*K+1. In some embodiments, the SIMD processor may be included as one of and / or may include similar components, features, and / or functionality as those of the example autonomous vehicle 1300 of FIGS. 13A-13D , the example computing device 1400 of FIG. 14 , and / or the example data center 1500 of FIG. 15 .

[0077] As an illustration, with reference to FIG. 4A, table 400 may include a logical diagram of a transposed load and an illustrative memory bank diagram of a transposed load having a line pitch of 17 over 256 bits. While the memory bank is shown terminated with 18 individual 16-bit banks in the memory bank diagram, this is for illustrative purposes only. For example, the memory banks may total 256 bits, 512 bits, or some other number of total bits, where each memory bank may be 16 bits wide. In the memory bank diagram using a transposed load of 17 line pitch, only one load operation may be performed to obtain each of the highlighted values ​​in the array.

[0078] While transpose loads using this technique are beneficial for many operations, certain algorithms, such as some computer vision algorithms, may require access to data patterns that cannot be achieved with a single read and / or write using the transpose load technique described with respect to FIG. 4A. For example, rather than loading a vertical vector 16 high, there may be instances where it is necessary to load a submatrix 8 high by 2 elements wide, a matrix 4 high by 4 elements wide, or another matrix or submatrix size. For a dot-product operation, for example, a T16 transpose store option with an appropriate line pitch may be desirable, so that the accumulation can be directed at two rows of 16 elements, 16 bits at a time, and thus, when storing, both rows can be written out as a single memory write transaction. With this in mind, a stride parameter may be used in transpose loads and / or stores. In some embodiments, the stride parameter may include, but is not intended to be limiting, a power-of-two stride, such as a stride of 2, 4, 8, 32, etc., which may be referred to as T2, T4, T8, T32, etc. Various examples of transpose loads using stride parameters are illustrated in table 410 of FIG. 4B, which includes a logical diagram of a transpose load and a diagram of the memory banks. The example of FIG. 4A, reflected in FIG. 4B, includes a stride parameter of 1, while other stride parameters are multiples of 2. For example, in T2, which has a line pitch of 18, a two-element wide by eight-high matrix can be stored as a transpose load, such that each of its values ​​can be retrieved using only one load transaction. Similarly, in T4, which has a line pitch of 20 and a stride of four, a four-element wide by four-high matrix can be stored, such that each of its values ​​can be retrieved using only one load transaction, and so on. Although described as a load transaction, this type of format can equally be used for store transactions, storing data to memory according to the transpose and stride parameters.

[0079] In such an example, the line pitch constraint may be adjusted depending on the stride. For a transposed access of word type T, the line pitch may be 16K+1; for a transposed access of word type T2 (e.g., with a stride of 2), the line pitch may be 16K+2; for a transposed access of word type T4 (e.g., with a stride of 4), the line pitch may be 16K+4, and so on. Thus, the line pitch may be equal to 16K+the stride value, or 16K+1+(T-1), where T is the stride parameter.

[0080] During operation, the architecture of the VPU's VMEM and the VPU's instruction set architecture (ISA) may be configured to perform transposed load and / or store operations, with or without a stride parameter, to enable data reads or writes to be organized into logical columns with a single read operation. For example, the ISA may be configured to receive information indicating a starting address for reading data from or writing data to (e.g., for reading or writing data from a register file), a write type (e.g., a transposed write operation, with or without a stride parameter), a line pitch value (e.g., the value of K in 16*K+1), and / or a stride parameter value. Note that the value 16 corresponds to the number of data elements in a particular implementation, but the value 16 (or W) may vary in various embodiments. Thus, the ISA may receive a starting address, line pitch, and / or stride parameter to write to the VMEM when writing data to memory according to a transposed write operation. As a result, when writing values, rather than writing the values ​​to a single memory bank in just one column of data, the data may be written according to a transposition or offset, as illustrated in Figures 4A and 4B, for example. If a stride parameter is used, the first value of the stride, followed by the number of next elements corresponding to the stride, may be written to memory, and then a line pitch may be applied to write the next set of values ​​to the memory bank so that each of the values ​​may be written to memory in just one cycle. Similarly, during a read operation, if data is organized or written according to a transposed store, the ISA may receive a starting address, a load type (e.g., a transposed load with or without a stride parameter), a line pitch value (e.g., the value of K), and a stride parameter value (e.g., an indicator of the data type, such as byte, halfword, etc.).The ISA can then access data from various memory banks according to the transposed load instruction (and / or stride parameter) to retrieve one column (or multiple columns) of data in just one read cycle. In this way, a single vector can be returned with just one read operation by retrieving one element from each memory bank.

[0081] 4C-4D, each block of methods 420 and 430 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Methods 420 and 430 may also be embodied as computer-usable instructions stored on a computer storage medium. Methods 420 and 430 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Methods 420 and 430 are described with respect to the SIMD architecture of the present disclosure, but may be performed by any system or combination of systems, including, but not limited to, those described herein.

[0082] 4C includes a flowchart of a method 420 for configuring a transpose store operation using a stride parameter, according to some embodiments of the present disclosure. The method 420 includes, at block B402, determining a matrix size. For example, a matrix width may be determined.

[0083] The method 420 includes determining a stride parameter and a line pitch for storing the matrix based on the size at block B404. For example, the line pitch may be determined using 16K+ stride value, and the stride value may be determined based on the width of the matrix.

[0084] The method 420 includes storing the matrix values ​​in memory using the stride parameter and the line pitch at block B406. For example, once the line pitch and stride are determined, the matrix values ​​can be stored in memory such that the line pitch and stride parameter values ​​do not result in bank conflicts when reading the matrix values ​​from memory.

[0085] 4D, which includes a flowchart of a method 430 for configuring a transpose store operation using a stride parameter, according to some embodiments of the present disclosure. The method 430 includes, at block B408, receiving data representing a line pitch and a starting memory address within one of a plurality of memory banks, the starting memory address corresponding to one element of a plurality of elements corresponding to a column of an array.

[0086] The method 430 includes, at block B410, reading a plurality of elements from a plurality of memory banks in a single read operation, wherein each element of the plurality of elements is read from a respective memory bank of the plurality of memory banks based at least in part on a line pitch.

[0087] Load with permute and zero insertion in a single instruction In a traditional processor instruction set, a load instruction can form a memory address through some indexing calculation, read the requested memory data from local memory, and place the memory data in a register. If an application requires additional data manipulation, additional instructions can be used to operate on the memory data in the register. In some cases, the data operation can include a simple data reorganization. In a traditional processor, even this simple data manipulation in the register file requires additional instructions, which adds latency. For example, a traditional system might load data, perform a permutation on the loaded data, and then perform one or more operations using the reorganized data. Enhancing the load instruction with this ability to reorganize data can save some processing time, allowing the computational kernel to run with higher performance and lower power consumption.

[0088] In consideration of these drawbacks, the disclosed systems and methods add loads containing reordering instructions that transmit a reordering pattern along with a memory address to local memory. As a result, the permutation can be performed without a significant amount of additional logic using existing data routing and multiplexing used to handle misaligned loads. In addition to saving instructions that would otherwise be consumed, e.g., five instructions used to perform the permutation of a double vector input and a double vector output, the overall latency of the permutation operation can be reduced. For example, the only latency is the load-to-use latency, rather than the load-to-use latency and the computation latency (e.g., to perform the permutation). In some embodiments, the loads containing the permutations and / or zero insertions described herein may be included in or similar to the components, features, and / or functions of the example autonomous vehicle 1300 of FIGS. 13A-13D , the example computing device 1400 of FIG. 14 , and / or the example data center 1500 of FIG. 15 .

[0089] Thus, a load with a permutation feature can be useful for manipulating data loaded from memory into a desired format for operation. By way of example, the coefficient data required for various filtering and dot-product instructions may contain specific repeating patterns that can be implemented by loads and permutations. For a filtering operation such as that described with reference to FIG. 3C, coefficients 0, 1, 2, and 3 may be repeated across the vector width (e.g., 16 bits), as illustrated in FIG. 5A. In such an example, the first register write could start at D[0] through D

[15] , then, using a sliding window 4, the next register could start at D[0] through D

[19] , and so on. In this filtering example, because coefficients C[0] through C[3] may be repeated across the vector width, a load with permutation can be useful for writing the coefficients in this order directly from the load, rather than loading all of the data, then performing the permutation, and then writing the vector to registers. Thus, in this example, a permutation pattern for the coefficient data may include {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3). In this same example, a permutation pattern for the data operands may be {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}. Thus, the data and coefficient operands may be read in permutation order rather than being read in order and then reordered before being written to registers for computation. As another example, as illustrated in Figure 5B, a filter processing instruction may include double vector coefficient operands and thus may include a permutation pattern such as {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3,4,5,6,7,4,5,6,7,4,5,6,7,4,5,6,7}. The permutation pattern may be static or fixed, or may be algorithmically calculated on the fly, allowing the permutation pattern to be flexible and dynamic.If the pattern is a repeating pattern, the first instance of the repeating element may be loaded, then replicated, and in an embodiment, then written out to a SIMD lane of a SIMD unit.

[0090] In some cases, it may be preferable to mask certain portions of memory data with zero values. For example, zeros may be inserted into unused entries for easier visualization in software development or to consume less energy (e.g., compared to leaving random data values). In other instances, zeros may be inserted to demarcate chunks of data in a data structure, such as when each chunk of data does not have a fixed length. In such instances, zero values ​​may indicate a gap between two data chunks. When processing fixed-size image patches, for example, if some variable-length information (e.g., feature point locations) is extracted from each image patch, zeros may be used to pad the remainder of the data that does not correspond to the extracted information.

[0091] In practice, the permutation index may typically include 32 or 16 elements in the information being read, for example, in the ranges 0 to 31 or 0 to 15, respectively. To include a zero value in the information being read, a negative index value may be included in a load that includes a permutation operation to write a zero to the corresponding lane of the destination register. Thus, during a write operation, for example, a negative value rather than zero may be written to the corresponding lane of a SIMD architecture.

[0092] As an example, a 30-wide by 30-high image patch may be processed by vector operations using 16 consecutive entries at a time. Because the width of 30 is not divisible by 16, each row may be processed by two vector operations: one processing the full 16-entry vector width and a second processing the partial 14-entry vector width. In such an example, it may be beneficial for the second 14-entry vector load to be padded with zeros to fill the last two vector lanes in place of irregular data values ​​that may currently be present in memory.

[0093] In one or more embodiments, padding zeros may be inserted into a desired lane position of a SIMD architecture, e.g., to save processing time that would otherwise be required to write a zero to this lane position. If there are 16 lanes, a typical permutation pattern may consist of 16 lane indices, e.g., 0 through 15. In this example, if there are values ​​{100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115} and the indices given as the replacement pattern are {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, -1, -1}, the final values ​​loaded into the destination register will be {100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 0, 0}. Therefore, the two -1 values ​​are automatically converted to 0 in the destination register based on the replacement pattern containing negative values. In the previous approach, -1, -1 would contain 14, 15 respectively, and the memory values ​​of 14, 15 would be written out to the register. However, this can contain irregular values ​​which may require additional processing time compared to the case where it contains a value of 0.

[0094] To implement a load that includes a replacement feature, routing and multiplexing in the memory logic may be used, e.g., routing and logic similar to that used to perform unaligned memory loads. For example, to support full memory width (e.g., 32 x 16 bits) loads from any 16-bit address (or 16 x 32-bit lanes from any 32-bit address), the memory logic may include multiplexing logic that selects any of the 32 lanes of memory data for routing to any destination register lane. For example, for a misaligned memory load, the logic may be driven as follows:

number

[0095] In a load that includes a replace feature, this same logic may be essentially reused, but modified logic may be included to perform the replace operation. An example of modified logic is included below.

number

[0096] As an example, with reference to FIG. 5C , a hardware structure 500 (which may be included as part of a VPU, SIMD unit, SoC, or other device type) with a crossbar 510 (e.g., a crossbar switch) can be used to take data from any location in memory 512 and drive the data to any lane in the SIMD via respective multiplexers (muxes) 514A-514N. In this way, it may be possible to write any of 16 inputs (or other width memories or registers) to any of 16 output locations or lanes. This may be useful for misaligned accesses, allowing load operations to start from any address and then align downstream. For example, if data in memory is to be read from locations 2-18, the data may be read from 2-18 but aligned to lanes 0-16 (e.g., 2 goes to lane 0, 3 goes to lane 1, and so on). This was not possible in conventional systems, which required vector loads to start at a location that is a multiple of 16, such as 0, 16, 32, etc. 5C , permutations can also be completed when data from any memory index can be output to any lane of a SIMD unit of a VPU. Multiplexers 518 can be used to introduce or insert per-lane reordering controls to inform multiplexers 514 of crossbar 510 which memory location to read data from based on a starting location (which may be aligned or misaligned) and a reordering pattern. Thus, rather than simply pulling data from an aligned location, the reordering pattern can be used to update memory read locations such that each multiplexer 514 sends the appropriate data to each of the SIMD unit's lanes. Additionally, multiplexers 516 can be used to insert zeros into the reordering pattern, including negative values ​​or other values ​​indicating zero insertion (e.g., when a value other than a negative value is used to perform zero insertion).Thus, when a memory access location is sent from multiplexer 518 to crossbar 510 and the value from the memory access is sent to multiplexer 516 for zero insertion, values ​​corresponding to negative values ​​in the permutation pattern may be converted to zero values ​​to pad the values ​​for the respective SIMD lane. Although only four sets of lanes, multiplexers, and memory indices are shown in Figure 5C, this is not intended to be limiting and any number of sets may be included without departing from the scope of this disclosure.

[0097] 5D illustrates an example use of hardware architecture 500. For example, the illustration of FIG. 5D may be based on the following information:

number

[0098] In addition, the following C code can represent the logic circuitry of the hardware architecture of Figures 5C and 5D:

number

[0099] Referring now to FIG. 5E, each block of method 550 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 550 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 550 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 550 is illustratively described with respect to the hardware structure of FIG. 5C. However, method 550 may additionally or alternatively be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0100] 5E is a flow diagram illustrating a method 550 for performing a load including a reordering operation according to some embodiments of the present disclosure. The method 550 includes, at block B502, determining a reordering pattern for loading data from memory. For example, the reordering pattern may be static or dynamically calculated. The reordering pattern may be aligned (e.g., 0 to 16 or 0 to 32), offset (e.g., 2 to 18), repeating (e.g., 0, 1, 2, 3, 0, 1, 2, 3, ...), and / or another pattern type.

[0101] The method 550 includes, at block B504, determining a memory address location for each lane of the plurality of lanes based at least in part on the reordering pattern. For example, the reordering pattern may indicate from which memory address location data intended for a particular lane or register should be loaded. The reordering pattern may be implemented using multiplexers 518 such that the appropriate memory address is sent to the crossbar 512 according to the reordering pattern.

[0102] The method 550 includes, in block B506, loading a value into each lane of the plurality of lanes based at least in part on the memory address location. For example, based on the memory address location, a multiplexer 514 in the crossbar 512 can retrieve a corresponding value from memory for writing to one or more lanes in one or more vector registers. In some embodiments, the multiplexer 516 can also be used to convert values ​​associated with negative values ​​(or other values ​​indicating zero padding) in the reordering pattern to zero. Thus, if one or more negative values ​​are included in the reordering pattern, the values ​​loaded from memory can be converted to zero before being written to the vector register.

[0103] The method 550 includes, at block B508, performing one or more operations in each of the plurality of lanes using the value and the at least one instruction. For example, upon inputting a vector register or processing lane of a SIMD unit, one or more operations, such as arithmetic instructions, logical instructions, shift / rotate instructions, bitwise operations, comparison instructions, conversion instructions, constant generation instructions, and / or the like, may be executed using one or more processing units corresponding to the one or more processing lanes.

[0104] Perform table lookups, multi-point lookups including blending In conventional processors performing vector SIMD computations, local memory may contain bit widths that match the bit width of the vector SIMD. As a result, such processors can generally only support read and / or write alignment and granularity that corresponds to the bit width. However, table lookups are a common technique in digital signal processing (DSP) and computer vision in embedded environments, for example, to implement various nonlinear functions. For example, square root, logarithm, sine, and cosine functions may require table lookups. To perform these functions, the input space may be uniformly sampled in a one-dimensional (1D) grid, and the output may be recorded in a 1D table at these input points. However, when nonlinear functions are implemented with table lookups, there is often a trade-off between the size of the table (e.g., the number of table entries) and precision. To improve accuracy without requiring large table sizes, an interpolated lookup can be performed: linear interpolation, looking up two points around the decimal index, or quadratic interpolation, looking up three points around the fractional index.

[0105] As an example, if the sine function is implemented using a lookup table and the sine value is tabulated in integer degrees, then table[0] = sin(0 degrees), table[1] = sin(1 degree), table[2] = sin(2 degrees), and so on. In such an example, if the evaluation is sin(1.7 degrees), then linear interpolation can be performed between two integer degree entries using a decimal as table[1] * 0.3 + table[2] * 0.7. In this example, the second entry, table[2], receives the decimal as a weight and the first entry receives 1 minus the decimal; thus, the closer the decimal is to 1.0, i.e., the position to which the second entry corresponds, the more heavily the second entry is weighted.

[0106] As another example, an image or patch of an image may be resampled, which may require looking up available pixels around a fractional pixel coordinate and then performing an interpolation lookup. In such an example, the table may contain image patches and may be two-dimensional. In this case, bilinear interpolation may be performed to linearly interpolate in each of the two dimensions. As an example, a patch at location Y=5.1, X=7.6 may be interpolated according to the following calculation:

number

[0107] With this in mind, the systems and methods described herein use two-point and / or two-by-two (2x2) point lookup operations to conserve memory space while increasing throughput (or, for example, matching the throughput of 32-way parallel processing). For example, using a per-memory-bank address bus and associated logic and routing, a two-point or two-by-two (e.g., four-point) parallel lookup can be performed with less memory usage. Thus, a single lookup into the table can obtain two points in a two-point lookup or four points in a two-by-two lookup. This can be achieved based on the hardware setup, e.g., bank addresses, logic, routing, etc., and memory storage patterns that allow multiple data reads without bank conflicts. Without these features, for example, to achieve a 32-way parallel lookup, the table would need to be replicated 32 times. For example, this 32-way parallel lookup can be performed using the following C code:

number

[0108] In this example, the lookup portion of the loop can be interpolated over two cycles (lookup and blending are performed in memory and vector math slots, respectively, pipelined to two cycles per iteration), performing 32 lookups per cycle and producing 32 outputs. Thus, the overall lookup / interpolation is 16 outputs per cycle, requiring the table to be replicated 32 times.

[0109] As a further example, with reference to FIG. 6A , a 16-way parallel table configuration is shown for performing a one-point lookup with an index vector {0, 1, 2, 3, 4, 5, 4, 3, . . .}. In such an example, using conventional architecture and memory layout techniques, a first lookup and a second lookup would need to be performed sequentially to read two entries from each memory bank. For example, a first memory bank T0 contains values ​​T0[0] and T0[1] to be read in a lookup operation, but because both of these values ​​are in the same memory bank T0 (which may include only one read port), the first value T0[0] is read in the first pass and the second value T0[1] is read in the subsequent second pass. In such a memory layout, two reads from the same memory bank can result in bank contention, introducing processing delays and / or causing algorithms or other calculations to execute incorrectly.

[0110] However, using the architecture of the present disclosure, the same 32 lookups may require only 16 table replications for a 2-point lookup, or only 8 for a 2 x 2-point lookup. For example, for a 2-point lookup, only 16 table replications can achieve the same performance of 16 outputs per clock cycle, thus reducing the memory footprint by a factor of 2. A 16-way parallel variation of the instruction can return a double vector, with the first entry at the bottom of the single vector and the second entry at the top of the single vector. In C code, this 16-way parallel lookup and interpolation may be expressed as follows:

number

[0111] In this example, the lookup and interpolation portions of the loop require only one clock cycle (the lookup and blending are performed in memory and vector math slots, respectively, pipelined to one cycle per iteration) and can be interpolated to produce 16 outputs. Therefore, the lookup / interpolation is 16 outputs per cycle. As an example, with reference to FIG. 6B, an 8-way parallel table configuration is shown to perform a two-point lookup in the index vector {0, 1, 2, 3, 4, 5, 4, 3, . . .}. In this example, because each memory bank T0, T1, T2, etc., contains only one value to be read during the lookup operation, all 16 values ​​can be read in a single pass, unlike the example of FIG. 6A, where only eight values ​​can be read in each of two passes due to potential bank conflicts. To do this, in an embodiment, the instruction for the lookup may include an index and a pattern that includes getting not only the index but also one more location in addition to the index. Thus, the instruction may read two values ​​for a two-point lookup, which may be written to the lookup table in this format, allowing this single read to be performed without bank conflicts.

[0112] As an example, when performing vector operations, each lane of a VPU may process a set of pixel values ​​retrieved from memory. In some cases, a lane may process multiple values ​​from the same memory bank, which can result in bank conflicts because a memory bank may contain only one read port. Therefore, the disclosed method and system distributes values ​​across memory banks to prevent bank conflicts, e.g., each value in only one processing lane of a VPU can access each corresponding value in only one read cycle.

[0113] In conventional systems where 2D bilinear interpolation lookups are performed, four lookups (e.g., 2 x 2) are required per output, with 32 copies of the table enabling an optimal throughput of eight outputs per clock cycle. For a 2 x 2 point lookup, eight table copies achieve eight outputs per cycle (compared to 32), thus reducing the memory footprint required for parallel subtables by a factor of four. For example, a 2 x 2 point lookup may read two entries from one row of the 2D table, followed by two entries from the next row. To avoid memory bank contention within any memory bank, the line pitch of the 2D table may be limited to m*k+2, where m is the number of entries stored horizontally in each subtable, and k is any integer large enough to fit into one row of the table. For an 8-way parallel 16-bit table, m = 32 (16-bit memory words) / 8 (parallelism) = 4. For a two-way parallel 32-bit table, m = 16 (32-bit memory words) / 2 (parallelism) = 8.

[0114] As an example, referring to Figures 6C-6D, line pitch limitations can be used to avoid memory conflicts. In this example, a two-way parallel word type table for a 2x2 point lookup is shown with a line pitch of 10. The number of consecutive elements (m) in the subtable is 8, and A[0][0...7] are arranged consecutively in the subtable, conforming to the formula 8k+2, where k can be any integer. Therefore, no matter what index value you start with, the 2x2 points to be retrieved will be located in different banks, which is mathematically guaranteed. For example, the bank numbers for a 2x2 point subtable are outlined below:

number

[0115] While Figure 6C shows two 2D tables, e.g., Table A and Table B, each 10 elements wide by 3 high, this is not intended to be limiting and the tables may be of any width and / or height depending on the implementation. Similarly, while the memory layout in Figure 6D includes a layout that is 16 elements wide by 3 high, this is not intended to be limiting and the memory width and / or height may be any configuration depending on the implementation.

[0116] In some implementations, such as when sampling an image patch, interpolation between fractional pixels may be performed. In some examples, a Vector Horizontal Blending with Interleaving (VHBlend_I) instruction may be executed to interpolate looked-up values ​​without additional instructions to manipulate the data, which may include horizontal blending that interleaves data to blend among pairs of lanes. For example, this instruction may perform bilinear interpolation after lookup in the same loop. This instruction may process each pair of lanes according to the table layout of FIG. 6E. In this manner, the calculations for Y0 and Y1 may be calculated as follows: Y0=x*(1-alpha0)+y*alpha0 Y1=z*(1-alpha1)+w*alpha1

[0117] Thus, this instruction allows horizontal blending between lane pairs x and y, and z and w, and the outputs can be interleaved within the destination register. For example, the following C code segment can be used to achieve optimal performance for an 8-way parallel table with 2x2 point lookups:

number

[0118] In this 8-way parallel table configuration, with sub-tables designated A, B, . . . , H, a loop can perform lookups and interpolations, resulting in 16 outputs per iteration. In such an example, the inputs can be configured as follows:

number

[0119] Referring now to FIG. 6G, each block of method 600 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 600 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 600 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 600 may be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0120] FIG. 6G is a flow diagram illustrating a method 600 for performing a multi-point lookup (e.g., in a single clock cycle in an accelerator for a decoupled lookup table (DLUT), such as those described with reference to FIGS. 9A-9C ) in accordance with some embodiments of the present disclosure. The method 600 includes, at block B602, replicating the table in memory such that the table includes a first value at a first physical address in a first memory bank and a second value at a second physical address in a second memory bank, the first value and the second value being in the same column of the logical memory view of the table. For example, the table may be replicated multiple times in memory to take advantage of memory access parallelism in a system. The table may include a first value at a first logical address and a second value at a second logical address in the same column as the first value, which, when stored in memory in this configuration, may cause bank conflicts because the two values ​​may be stored in the same memory bank. Thus, when replicating the table in memory, a write instruction can write a first value to a first physical address adjacent to, say, another memory bank, as a second value, so that the two values ​​can be retrieved in the same cycle.

[0121] The method 600 includes determining a first index corresponding to a first physical address in the memory at block B604. For example, a read operation may use an index that indicates a first location in the memory from which to begin reading a value.

[0122] At block B606, the method 600 includes reading, in a single cycle, a first value located at a first physical address and a second value located at a second physical address based at least in part on a read instruction corresponding to a multi-point lookup. For example, when replicating a table in memory, the table may be replicated such that pairs of points in the same column or table (e.g., corresponding to pixels in the same pixel column) are stored in separate memory banks. Thus, using a two-point lookup read instruction that uses an index of the first point of the pair of points to read the first point and the adjacent second point stored in different memory banks, the first value and the second value may be read from the first memory bank storing the first value and the second memory bank storing the second value in a single cycle. This operation may be performed for each pair of values ​​from each replicated table to generate a high-order vector containing the first value from each table and a low-order vector containing the second value from each table, and these vectors may be used as vector registers in the VPU in addition to instructions that generate output (e.g., interpolation).

[0123] At block B608, the method 600 includes performing one or more operations using the first and second values. For example, the first and second values ​​may be loaded into one or more lanes of a VPU, and square root, logarithm, sine, and cosine functions may be performed, linear or bilinear interpolation may be performed, and / or another type of operation may be performed. By way of example, if interpolation is performed and the table is replicated 16 times, 16 two-point lookup operations may be performed to generate 32 values, i.e., two values ​​per vector lane of the VPU, and interpolation may be performed in each lane to output 16 results. Thus, 16 interpolated outputs may be generated per cycle using only 16 replications of the table. This may be a result of using a two-point lookup, which requires the table containing the values ​​to be replicated only half the number of times (e.g., 16 times instead of 32) as with a traditional single-point lookup operation, allowing for the same 32-value throughput with half the memory footprint.

[0124] Load cache per memory bank in vector memory In traditional processors, the width of a data cache may be, for example, 32 bytes per cache line. A cache line is a unit of data that the hardware keeps track of. For example, the hardware may keep track of cache line usage information in tag memory, including all system addresses, whether the cache line has been written, and the last time the cache line was read compared to other cache lines to determine when to evict the cache line. In some implementations, a data cache is local memory or a portion of local memory, and maps larger data structures temporarily stored in external memory into local memory, allowing data to be processed without incurring the long memory latencies associated with processing data directly from external memory. Traditional desktop or laptop computers often use this type of data cache.

[0125] As a non-limiting example, a programmable vision accelerator and / or VPU may include an embedded processor designed to execute a smaller set of highly optimized code. Such processor types may not implement a data cache, allowing the programmer to manage the contents of the local data memory. The systems and methods of the present disclosure may include local memory that is managed by the programmer rather than cached, but may also include additional data caching capabilities in one or more, e.g., respective, memory banks. This data cache may be narrow, e.g., but not limited to, 16 bits wide, compared to more conventional data caches, e.g., including 32 bytes. This data cache may be used primarily to reduce power consumption, as opposed to the primary goal of conventional data caches, which is to reduce latency.

[0126] In computer vision processing, for example, data access patterns often have some degree of locality (e.g., staying in one neighborhood for a while before moving to the next neighborhood). For example, when performing a 7x7 2D filter operation using the VFilt4HHW instruction described herein (which computes four taps at a time), the data read stream might read from a neighborhood for three memory reads, then move to another neighborhood and read three more times, and so on. The coefficient reads of the operation might use the same array of zero-padded values ​​(e.g., 7*2*4=56 halfwords), stepping forward four halfwords at a time until the last group of four halfwords is read, then going back again and starting at the beginning of the 56 halfword array until the filtering kernel is complete.

[0127] Therefore, to take advantage of these local access patterns and reduce power consumption due to memory accesses, two-way set-associative load data caching (e.g., holding a total of 64 halfwords) may be implemented in each memory bank. When the load cache is enabled, the most recently read set of read data (e.g., last, last two, last three, etc.) may be recorded in the cache along with the row address and the most recently read entry in the tag memory. As a result, when the same memory address is read again, there is a chance of a cache hit, and the cache can supply the data without having to reread the data from local memory. In an embodiment, the load cache may be located between the memory logging logic and the memory itself, so that whenever a cache hit occurs, a memory read of that particular address or value is halted or prevented to conserve power.

[0128] Using this cache structure, for the 7x7 2D filter example above, the load cache may allow the system to skip nearly two-thirds of the data reads and almost all of the coefficient reads in steady state. An example of data cache usage in each bank is shown in Figures 7A-7C. For example, the VFilt4HHW instruction may perform a potentially larger, 4-tap filtering task and may use two single halfword vectors of data, e.g., data[0-15] and data[4-19], and one single halfword vector of coefficients, e.g., coef[0-3], repeated four times to fill a single 16-element vector. In both vector math slots, a 7x7 2D filter implementation using the VFilt4HHW instruction may use the data element and coefficient arrays of Figure 7A. The VPU of the present disclosure may be configured to read double vectors, so that data[y][0-15] and data[y][16-31] may be read as double vectors. Similarly, data[y][4-19] and data[y][20-35], as well as data[y][8-23] and data[y][24-39] may be read as double vectors. Therefore, the read patterns of the data and coefficients may correspond to those of Figures 7B-7C, respectively, assuming that the line pitch of the data is 100 and the line pitch of the coefficients is 8.

[0129] Figure 7D illustrates an example memory bank configuration. For example, a two-entry fully associative cache holds two copies of data within any superbank, with data and coefficients located in different superbanks for efficient caching. During a coefficient read, banks 0-3 may initially hold coefficient elements 0-3, add elements 32-35, and then read elements 64-67, deleting elements 0-3. The pattern repeats, so elements 0-3 must be reread during the next coefficient read. In steady state with the load cache enabled, only four banks may be read per sweep of the coefficient read pattern. As a result, by using the load cache for data reads from memory banks, there can be a savings of (3*32-(32+4+4)) / (3*32)=58.3%, which is a factor of (14*16-4) / (14*16)=98.2%.

[0130] Therefore, for certain algorithms, such as computer vision algorithms that use sliding windows, a load cache can be useful to avoid wasting power. For example, without a load cache, each bank would need to be read every cycle, even though much of the data is the same. In an example where 512 bits are read each iteration, the first 512 bits can be read, then another 512 bits, and so on. For example, if the sliding window is only 8 bytes, only 64 bits are new each iteration, and the remaining 448 bits remain the same. Without a data cache, these 448 bits would need to be read again from the data bank. However, with a data cache per memory bank, these 448 bits can be pulled from the load cache, and only the new 64 bits would need to be read from the other memory bank. Therefore, the power required to read 448 bits from the memory bank is saved. Examples of algorithms that may benefit from the use of a load cache are spatial filtering operations, deep learning inference operations (such as convolution operations), and / or similar.

[0131] Referring to FIG. 7E, a hardware architecture or logic for a memory bank with a load cache is illustrated. For example, to support misaligned access of memory (e.g., vector memory (VMEM)), data access can be accelerated in a sliding window, which is an important memory access pattern for many computer vision algorithms, including filtering and convolution. In a sliding window vector load, most of the data from the random access memory (RAM) bank 702 remains the same. In this example, if a 64B vector load slides by 4B, only 4B of the data changes, so only 4B of new data is read from the RAM bank 702. To optimize VMEM RAM power, a small cache called a "load cache" can be attached to each bank, per superbank, for a total of 3 superbanks x 32 banks = 96 load caches per VMEM. Each load cache configuration may include, in a non-limiting example, a 2-line (2 x 2B = 4B) capacity, a fully associative scheme, and a pseudo least recently used (pLRU) replacement policy.

[0132] The location where the data cache stores the most recent accesses is divided into two parts: tag store 706 and data store 704. Tag store 706 may store cached addresses and control information corresponding to previous accesses, while data store 704 may store data from previous accesses. The control information in tag store 706 may include a valid flag (e.g., whether the entry is valid), a dirty flag (e.g., whether the entry has been modified and needs to be written back to memory), and / or a last-used flag (e.g., indicating which entry to replace if an entry should be replaced with a least-recently-used policy). Because the cache is a load cache, write data may not update the cache, but valid and last-used flags may be included in tag store 706. The valid flag or bit can be used to condition an address match; any write should invalidate the entry. The last-used flag may be updated with each access.

[0133] As described herein, for an effective caching scheme, to reduce access time and save power, the storage capacity of the load cache is much lower than the storage capacity of the memory or RAM bank 702. In one embodiment, each load cache may correspond to only one RAM bank 702, which may each be a 2048 x 16-bit memory, and the load caches may each be 2 x 16-bit data stores 704 with 23-bit tag stores 706 (e.g., 2 entries x (11 bits address + 1 bit valid) + 1 bit last used)).

[0134] During operation, the offset 722, line address 724, and stride size 726 may be used to generate a memory address for the memory access. This memory address may be retrieved for comparison to tag store 706, for example, for comparison to several previously accessed addresses (e.g., two previous accesses). An arrow to the top of tag store 706 may represent a memory address. In some embodiments, tag store 706 may use the entire memory address to compare to memory addresses stored by previous accesses. In other embodiments, a subset of the tags may be addressed using a subset of address bits from the memory address, so that only a subset of the tags is compared to the memory address. For example, if more previous access tags are stored in tag store 706, only a subset of the tags may be compared using a subset of the memory address bits to reduce area and save power. In a load cache design with fewer tags, such as tags corresponding to two previous accesses, the entire tag of the previous entry may be compared to the entire memory address. The "==?" decision block 720 compares the current memory address of the RAM bank 702 with the address stored in the tag store 706. If there is a miss (e.g., the tag and memory address do not match), the read of the RAM bank 702 can be enabled using the read enable 708, and the RAM bank 702 can be selected and read via the read data multiplexer (rd data mux) 712 for sending to the staging flop 716. If there is a hit (e.g., the tag and memory address match), the data store 704 can be addressed with a 0 or 1 (in a two-entry embodiment) to indicate which previous access the hit corresponds to. The corresponding entry in the data memory can be sent to the staging flop 716 via the rd data mux 712.The staged flop 716 returns the readback data to the processor pipeline, where it may ultimately be routed to the destination scalar or vector register of a load instruction.

[0135] The multi-stage flop 714 can accommodate parity checking. For example, a memory large enough to have a parity bit (e.g., at the parity terminal 710) may be required to enable error detection and / or error correction. In the memory (e.g., VMEM), error detection and / or error correction logic may be used on the readback data.

[0136] Thus, the load cache may include tag bits from tag storage 706 in way 0 and way 1, each of which may include 11 bits of address and 1 valid bit. The load cache may further include a 1-bit pLRU and data bits from data storage 704 in way 0 and way 1, each of which may include 16 bits of data and 2 bits of parity. When enabled, the load cache may be looked up in the D1 stage. To minimize power consumption, only the load caches of the RAM banks 702 involved in the load may be enabled. For example, a single vector load may only look up 16 of the 32 load caches. On a load hit (e.g., if the load cache contains the data to be accessed), the read enable for the given RAM bank 702 is suppressed, thereby preventing the RAM bank 702 from powering up. The pLRU 720 may also be updated in the D1 stage. In the D2 stage, the data and parity bits are read from the hit way of the load cache and may be multiplexed with the RAM result.

[0137] In the case of a load cache miss, in the D1 stage, an existing entry in the victim's way that should be evicted to make room for the new entry may be determined based on the valid bit and pLRU. The tag of the victim's way may then be updated with the address of the miss, and the read enable 708 of the RAM bank 702 may not be suppressed. In the D2 stage, data / parity from the RAM bank 702 is not only sent to the read data crossbar, but also fills the data into the evicted cache line. Stores can also look up the load cache when enabled and engaged. A store hit may invalidate the hit way, and the store miss may be ignored.

[0138] A load cache hit saves the power of reading the RAM bank 702. On the other hand, a load cache miss not only incurs the power of reading the RAM bank 702, but also consumes the power of looking up the load cache to fill the victim way. Not all types of memory access patterns have a high hit rate in the load cache, especially when accessing superbanks in indexed addressing mode, so only vector linear loads can be looked up in the load cache.

[0139] All stores, when enabled, are looked up in the load cache, ensuring that the load cache does not get out of sync with data in, for example, VMEM RAM bank 702. In applications where a low load cache hit rate is expected for a given superbank, software can be used to disable the load cache for that superbank's RAM bank 702, as described in more detail below, to minimize power due to store lookups.

[0140] For example, in some embodiments, the use of a data cache may not be beneficial. For example, a data cache may not be useful for operations with non-repeated access patterns, and the extra task of checking the cache before a read may waste time and / or energy because a data bank may need to be read to access the appropriate data. Therefore, load caching can be enabled or disabled, which can reduce the power penalty of access patterns with high load cache miss rates, but the load cache can also be used for access patterns where the data cache can be used to save power. In some embodiments, the enablement or disablement can be programmed using application code, allowing a programmer to program the code to enable the data cache when desired and disable the data cache when undesirable. In other embodiments, the enablement or disablement can be performed by hardware that analyzes read patterns and detects overlapping patterns. For example, the hardware can enable the load cache for a threshold amount of overlap between consecutive read operations. However, if the overlap is less than the threshold, the load cache can be disabled. The threshold can be, by way of non-limiting example, 25%, 40%, 50%, 75%, or a different threshold amount of overlap between reads.

[0141] 7E, when the load cache is disabled, tag store 706 cannot be accessed and read enable 708 can be set on a read-by-read basis to enable reads to RAM bank 702. Similarly, data store 704 cannot be accessed and read data multiplexer 712 can always pass data from RAM bank 702 through to stager flop 716.

[0142] Furthermore, in some embodiments, the memory bank structure may include multiple superbanks, e.g., three superbanks, with each superbank capable of enabling or disabling its load cache depending on the particular access pattern within each superbank. For example, if three superbanks are used, each superbank may include 32 RAM memory banks, and the data cache per memory bank may include two entries, where each entry is one word, i.e., 16 bits. If more than one superbank is used, the superbanks may be of any size, varying sizes, the same size, or a combination thereof. For example, the first superbank may be 128 KB, the second superbank may be 256 KB, and the third superbank may be 512 KB.

[0143] Referring now to FIG. 7F, each block of method 750 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 750 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 750 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 750 may be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0144] FIG. 7F is a flow diagram illustrating a method 750 of using a data cache for a read operation according to some embodiments of the present disclosure. The method 750 includes, in block B702, receiving data representing a memory read address. For example, after a first read operation using several memory banks, a second read operation may be performed that includes one or more of the memory banks in addition to one or more additional or other memory banks. Because the first read operation may have included storing the read output in the data cache corresponding to the respective memory bank, this value may be reused rather than requiring another read of the memory bank. Thus, a memory read address corresponding to the next read operation may be received, and the load cache, if enabled, may be accessed to determine whether any of the data is stored in the load cache.

[0145] The method 750 includes comparing the memory read address with memory addresses stored in the load cache corresponding to previous memory reads at block B704. For example, after a previous memory read, data from the memory read may be stored in the load cache corresponding to a particular RAM bank 702. To store this information, the tag store 706 may include memory addresses corresponding to one or more previous reads from the RAM bank 702.

[0146] The method 750 includes determining, at block B706, that the memory read address at least partially overlaps with a memory address in the load cache. For example, the memory read address may be compared to a previous memory read address of a previous read stored in the tag store 706. If there is a hit, the load cache may be used to read at least a portion of the data corresponding to the memory read address of the current memory read.

[0147] The method 750 includes reading at least a portion of the data corresponding to the memory read address from the load cache at block B 708. For example, a load cache hit determined by the tag store 706 may cause a portion of the data from the overlapping memory address to be read from the load cache, and the remaining data, if any, may be read from the RAM bank 702.

[0148] Separate, configurable accelerator To optimize processor performance for specific applications, such as real-time applications, an instruction set architecture (ISA) can be extended to create custom instructions that accelerate commonly performed operations. This allows the processor to reduce the number of cycles required to execute a particular task. The process of customizing the ISA continues until the system's performance goals are achieved. However, these new instructions are added to the processor's register file or directly to memory as operands to operate on data, and are executed using the existing processor controller as well as existing memory addressing and access hardware. In such instances, it is desirable for the new instructions to fit within the total number of read / write operands in the processor's register file (e.g., by reusing existing ports), fit within the width of the register file (e.g., within the processor's data types), and fit within the processor's pipeline stages. These requirements for successfully adding instructions to an ISA limit the degree of flexibility for adding new instructions. Additionally, creating an ISA to handle a pipeline with many stages (e.g., 30, 40, or 50 stages) complicates the ISA configuration.

[0149] Furthermore, processors enable a high degree of flexibility at the expense of power dissipation, because every added instruction requires fetching, decoding / dispatching, and reading / writing from / to the register file and / or memory. Therefore, adding additional functional units to implement these custom instructions constrains the register file's read / write ports, resulting in increased area requirements (e.g., additional read / write ports may be required) and power requirements (e.g., extra loads to the register file may be performed). Furthermore, the processing pipeline of embedded applications typically has multiple stages, where the output from one stage provides the input to the next. Techniques such as running multiple threads (e.g., at various processing stages) on the processor can reduce the latency and thus reduce the overall processing time. However, multithreading comes at a hardware cost: multithreading requires the inclusion of control logic to handle the processor's multiple threads, as well as fetching / decoding / dispatching instructions from multiple threads and maintaining state information for each thread (e.g., in the register file). This increases the area and power requirements while making verification and programming of the processor more complex. Thus, while various methods exist for reducing the latency of a processing pipeline, existing methods require additional surface area for the processor's hardware, require additional power consumption due to the additional hardware, and increase the complexity of programming the processor to perform various tasks.

[0150] Considering the limitations of primary processor configuration and the deficiencies of multithreaded processors, the disclosed systems and methods use one or more units of a primary or main processor, such as a single-threaded processor like a VPU, in addition to a domain-specific accelerator or coprocessor that is separate from the main processor and communicates with the main processor through a shared memory, such as a vector memory (VMEM). Thus, the accelerator can operate as a subunit of the main processor, but once configured, the accelerator can execute independently of the main processor's instructions, rather than requiring the processor to execute instructions. For example, accelerator access instructions can be used to enable the main processor to configure and sequence the accelerator, and shared memory can enable inter-stage data structures to be shared between the main processor and the accelerator. Once the main processor kicks off or starts the accelerator (e.g., through the common accelerator interface and using one or more load / store instructions), the main processor is free to process another stage (thereby enabling the ability to work simultaneously on multiple stages of a processing pipeline while reducing execution time) or transition to a low-power or minimum-power state (e.g., minimizing power usage when not actively processing) while waiting for the accelerator to complete processing. Thus, once configured by the main processor, each of the one or more accelerators can operate independently and concurrently with the main processor. The main processor and accelerators can synchronize during processing via a handshake interface so that the main processor knows when an accelerator has finished processing and / or is ready for a new task, and vice versa.The shared memory can store configuration messages (e.g., to configure the accelerator when configuration instructions cannot be efficiently sent over the accelerator interface due to size constraints), input buffers (e.g., to store data for processing by the accelerator), and / or accelerator output results (e.g., after processing is complete, data from, e.g., a register file of the accelerator, can be stored back into the shared memory at a location indicated by a configuration instruction from the main processor). Thus, when triggered, the accelerator can read configuration parameters and / or input data structures from the shared memory and write output result data structures to the shared memory.

[0151] As a result, this combined system of main processor, shared memory, and separate accelerator enables a flexible, programmable main processor while achieving the power consumption levels of fixed-function hardware (e.g., because a computationally intensive processing stage of a processing pipeline may be implemented as an accelerator) without having to significantly increase the complexity of the main processor (e.g., because the main processor may only require additional accelerator configuration or access instructions to program the accelerator). For example, the accelerator's pipeline and data type (e.g., data width) may be independent of the main processor's pipeline and data type, allowing for further customization and optimization that may not be possible with the main processor alone, which requires the processor's register file's total number of read / write operands, register file width, and instructions that fit within the pipeline stages.

[0152] In some embodiments, the accelerator and main processor may be coupled during instruction execution to achieve some power savings for the accelerator while coupling execution to the main processor's pipeline. However, in such embodiments, instructions would be interleaved between the accelerator and main processor, reducing the ability to simultaneously process different stages of the pipeline. In one or more embodiments, the accelerator and main processor may be coupled via a higher-level, second-level (L2) memory rather than connected via shared memory. However, in such embodiments, decoupling at the higher level (e.g., decoupling from the higher level via shared memory) may increase communication overhead with the main processor.

[0153] The segregated accelerator may be used for any task in any domain, such as, by way of non-limiting example, for computer vision algorithms such as feature tracking, object tracking, image warping, pyramiding, and / or the like, for sensor processing such as matrix multiplication or other operations on lidar data, radar data, and / or the like, and / or for machine learning or deep learning applications, as an accelerator for segregated lookup tables, for detecting and resolving bank conflicts, performing 1D, 2D, etc. lookups, performing 1D / 2D interpolation, and / or the like. Thus, the aspects described herein may be applied to any processing pipeline in which portions of the processing may be offloaded to an accelerator.

[0154] Depending on the implementation, there may be any number of separate accelerators on a chip that communicate with a main processor via shared memory. For example, a system-on-chip (SoC) or other integrated circuit (IC) may include a main processor and one or more accelerators, and a programmer may write instructions or code that recognizes the various accelerators and uses the accelerators to enhance performance of any of the system's various tasks. While the main processor is primarily described as a VPU, this is not intended to be limiting, and the main processor may include any processor type, such as a CPU, GPU, DPU, or other processor, without departing from the scope of this disclosure.

[0155] Referring now to FIG. 8A, FIG. 8A illustrates a system 800 including one or more decoupled accelerators, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, system 800 may be included in and / or may include components, features, and / or functionality similar to those of example autonomous vehicle 1300 of FIGS. 13A-13D, example computing device 1400 of FIG. 14, and / or example data center 1500 of FIG. 15.

[0156] The system 800 may include a processor 802 (e.g., a main processor), such as a VPU, CPU, GPU, or DPU; a separate accelerator 804; and / or a shared memory 806 (e.g., a vector memory or VMEM). The processor 802 may be coupled to an instruction cache (I-cache) 810, which may cache instructions executed by the processor 802. The processor 802 may include general-purpose input / output (GPIO) 808 (e.g., digital signal pins on an IC that may be used as input, output, or both and may be controllable at runtime), and an IC configurator 812. In some embodiments, as shown, the processor 802 may communicate on-chip using an AXI interface, such as, but not limited to, a 256-bit advanced extensible interface (AXI). The IC configurator 812 may be used to configure the system 800.

[0157] The processor 802 can communicate directly with the separate accelerator 804 via a coprocessor or accelerator interface, such as an advanced peripheral bus (APB) interface, and / or a handshake, programming, or event interface. For example, the processor 802 can use the accelerator interface (or configuration bus) to configure the accelerator 804, the event interface to kick off or trigger accelerator 804 processing, and the handshake or event interface to synchronize with the accelerator 804. Thus, each accelerator 804 can include a mechanism to be configured to communicate with the processor 802 via its respective accelerator interface or configuration bus. For example, the accelerator 804 can indicate completion of processing to the processor 802 via a handshake mechanism when processing is complete, or the processor 802 can periodically poll the accelerator 804 to request status or completion time when the processor 802 is waiting for the accelerator 804 to finish processing. In some embodiments, the accelerator interface may include a 32-bit interface (or other smaller sized interface) so that configuration commands can be transmitted to the accelerator 804. However, in some embodiments, the configuration messages may be larger (e.g., greater than 32 bits or some multiple of 32 bits), and instead, the configuration messages may be stored in shared memory 806, and the location of the configuration information in memory 806 may be sent to the accelerator 804 via the accelerator interface to indicate where to obtain the configuration information.

[0158] Thus, the configuration bus can configure the accelerator 804 and, using events (or a programming interface), can allow the processor 802 to trigger or initiate processing by the accelerator 804. Once triggered or kicked off, the accelerator 804 can operate independently, while the processor 802 waits for processing to complete and / or to perform another processing task or stage. For example, an application programmer can program the processor 802 and the accelerator 804 with knowledge of what each can do, such that the application's program can be divided into portions, one portion for the processor 802 and one portion for the accelerator 804. Thus, processing can be performed in parallel between the processor 802 and the accelerator 804, in embodiments, reducing execution time and increasing efficiency. Configuration messages shared via the accelerator interface and / or via the shared memory 806 may be generated by the processor 802 and used to indicate to the accelerator 804 where in the shared memory 806 the data to process begins, how much data to process, and where to write the results back to the shared memory 806. The processor 802 may create an input buffer in the shared memory 806 at a specified location containing data on which the accelerator 804 will subsequently operate. Once the configuration message is transmitted and the input buffer is stored in the shared memory 806, the accelerator 804 may receive a trigger signal from the processor 802 via an event interface (e.g., a programming interface), and the accelerator 804 may begin processing the data. Once the accelerator 804 is triggered, the processor 802 may then perform other work or enter a low-power state; once the accelerator 804 has finished processing, the accelerator 804 may indicate to the processor 802 that it has finished processing and may wait for additional work.

[0159] The processor 802 may set up input buffers or input data structures for the accelerator 804 to process and store the input buffers or input data structures in memory 806. The processor 802 may configure the accelerator 804 using load / store operations, among other things, to configure and communicate with the accelerator 804. Configuration messages may configure various registers (e.g., in an embodiment, 256 32-bit registers) of the accelerator 804. For example, in a separable lookup table accelerator (described in more detail herein), the configuration information may indicate whether the lookup is a 1D lookup with interpolation, a 2D lookup with bilinear interpolation, and / or another type of lookup. When the accelerator 804 recognizes a particular mode or function, it may configure the registers to appropriately read data from memory 806, process it, and write the data back to memory 806.

[0160] In some embodiments, the processor 802 can configure the accelerator 804 to perform multiple tasks at once to increase efficiency. For example, if the accelerator 804 is to perform various smaller tasks, configuring the accelerator 804 individually may increase execution time as each task may be completed quickly, thereby requiring the processor 802 to stop processing, configure the accelerator 804 for another task, and so on. To do this, a first task message may include an address of a second task message that enables self-chaining of multiple tasks. Thus, the processor 802 may immediately generate configuration messages for multiple tasks, generating configuration information and input buffers for each task, so that the accelerator 804 can execute various tasks in succession before indicating to the processor 802 that it has completed processing and is ready to accept more work. Additionally, the accelerator 804 may be configured to overlap tasks to increase efficiency, such that the accelerator 804 can begin decoding and setting up registers for the next task as the first task nears completion. Finally, the accelerator 804 may be able to operate on different data formats or types than the processor 802 normally supports by including separate instructions for the processor 802 and the accelerator 804. This may be the result of the architecture and layout of the accelerator 804's registers, which are different and specialized for particular processing tasks.

[0161] In an embodiment, processor 802 may communicate with shared memory 806 via any number of memory interfaces (e.g., a 512-bit static random access memory (SRAM) interface). Similarly, as shown, accelerator 804 may communicate with shared memory 806 via any number of memory interfaces (e.g., a 512-bit SRAM interface). Arbiter 814 may determine on a cycle-by-cycle basis which of processor 802 and / or accelerator 804 is permitted to access shared memory 806.

[0162] Referring now to FIG. 8B, each block of method 850 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 850 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 850 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, while method 850 is described with respect to system 800 of FIG. 8A, method 850 may be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0163] 8B is a flow diagram illustrating a method 850 for using a decoupled accelerator according to some embodiments of the present disclosure. The method 850 includes, at block B802, receiving configuration information for one or more first processing tasks of a processing pipeline. For example, the accelerator 804 may receive the configuration information from the processor 802 (e.g., via a configuration message via an accelerator interface).

[0164] The method 850 includes configuring one or more registers of the accelerator based at least in part on the configuration information at block B 804. For example, the accelerator 804 may configure one or more registers based on the configuration information.

[0165] The method 850 includes, in block B806, reading data from an input buffer in memory based at least in part on an indication of a starting location of the input buffer included in the configuration information. For example, the configuration information may include an indication of where in memory 806 the input buffer is stored, and the accelerator 804 may read data from the input buffer into a register.

[0166] The method 850 includes processing the data from the input buffer to compute output data at block B 808. For example, the accelerator 804 may process the data from the input buffer to generate or compute output.

[0167] The method 850 includes, at block B810, writing the output data to a memory at a location determined based at least in part on the configuration information. For example, the accelerator 804 may write the results of the calculation to the memory 806 and may indicate to the processor 802 that processing is complete. The processor 802 may then use the output data to perform one or more second processing tasks in the processing pipeline.

[0168] Separate Lookup Table Accelerator Parallel processing is used to speed up many computational tasks, including, but not limited to, computer vision applications, deep learning applications, sensor processing applications, and / or other applications that benefit from parallel processing (e.g., when a processing task is unrelated to other processing tasks). For example, vector processors can operate on multiple elements in the same operation, achieving the efficiency needed to run these types of parallel processing algorithms in real time while consuming low power. For example, a common operation in computer vision or deep learning tasks is to perform a lookup from a lookup table, image patch, or surface based on an index or coordinate location. To do this, data from multiple elements may be accessed using a single vector load or store operation. If the index being looked up is not regular (e.g., consecutive or fixed integer strides horizontally, vertically, or depthwise), this results in an irregular indexed access from memory.

[0169] A processor can construct a vector memory using smaller banks of RAM to support regular, but unaligned, vector accesses from memory. In this way, the hardware can create interesting addressing patterns for the vector memory by generating unique addresses for each RAM bank separately. In a load operation of an irregularly indexed vector from memory, the indexes of different vector elements may be unrelated to each other, which can cause bank conflicts in one or more banks of RAM. Because bank conflicts are data-dependent, they may not be determined statically, which prevents the compiler from scheduling around the bank conflicts.

[0170] In some conventional systems, various architectural designs can be implemented to support irregularly indexed vector load operations. For example, multiple read ports can be added to RAM banks. In such an example, if the hardware can handle 32 vectors, 32 read ports are required for each bank, which not only increases cost, area, and power, but also increases location and routing congestion around the RAM banks. Another example includes lowering the indexed lookup throughput and performing only a single scalar lookup per load. However, this creates a bottleneck for vector execution and becomes the limiting factor for execution time. A further example includes creating multiple copies of the data structure in memory so that each vector lane can access data from only one bank. While this example may solve some of the throughput issues of other approaches, memory capacity is limited by occupying N times the space for the data structure (where N is the number of entries to be accessed), which, in addition to the overhead of creating copies, can cause overall performance degradation of the associated algorithm. However, this method is preferable when the data structures are small. In some instances, conflicts can be dynamically detected and resolved by ordering the conflicting lookups. However, this may result in increased hardware complexity due to the need to dynamically detect and resolve bank conflicts. Furthermore, these additional stages increase the load-to-use latency of such operations, thereby impacting the compiler's ability to efficiently schedule the code. In addition, data-dependent execution latency may be introduced, which is problematic for the compiler to efficiently schedule. In some instances, a combination of these methods may be implemented.

[0171] In consideration of these shortcomings of other architectures, the disclosed system and method includes a decoupled lookup table accelerator configured to support irregularly indexed vector load operations. The decoupled lookup table accelerator may be included as an accelerator 804 in the system 800 and may communicate with a processor 802, such as a VPU, via shared memory 806. The decoupled lookup table (DLUT) may support multiple modes for performing table lookups, such as a 1D lookup mode, a 2D lookup mode, a 2D conflict-free lookup mode, a 1D lookup with interpolation mode, a 2D lookup with interpolation mode, a table reformatting mode, and / or other modes. In any lookup mode, the DLUT may receive an array of VMEM indices, which may be in 1D(x) format or 2D(x,y) format. Each element may include, for example, 16 or 32 bits, which may be unsigned. The DLUT can then perform predefined index calculations, which may include, by way of non-limiting example, 2D to 1D mapping, truncation / rounding, integer / fractional splitting, and / or valid range detection. For example, the DLUT can detect or consolidate duplicate reads, detect bank conflicts in the index, and issue read requests to the VMEM to look up the requested table entry. Each element may include 8, 16, or 32 bits, which may be signed or unsigned. The DLUT can then perform post-interpolation processing as configured and write the output back to the VMEM. Each of these processing operations may be pipelined to increase throughput, reduce latency, and reduce power consumption.

[0172] As a result, the DLUT accelerator overcomes the deficiencies of implementing dynamic conflict detection and resolution in a processor's pipeline, allowing compilers to efficiently schedule all memory operations for deterministic execution latency while avoiding the complexity of line conflict detection. Because the accelerator operates as a tightly coupled accelerator, e.g., via a VMEM shared with the VPU, the processor can configure and kick off the accelerator while continuing to process other separate portions or stages of the processing pipeline or algorithm. In some embodiments, the accelerator may include additional features to further reduce the load on the main processor, such as offloading index generation for patches with specific lookup patterns, performing optional 1D blending and 2D interpolation on lookup data, and / or providing support for table reformatting without lookups or interpolation. In fact, the entire system, including the processor 802 and accelerator 804 for performing lookups, has been demonstrated to speed up the processing of various computer vision algorithms (e.g., feature tracking, object tracking, image warping, pyramid creation, etc.) by a factor of two, while achieving over 50% reduction in energy consumption compared to running the entire algorithm solely on the main processor.

[0173] Referring now to FIG. 9A, FIG. 9A illustrates a system 900 including an accelerator for a disaggregated lookup table (DLUT), according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, system 900 may be included in and / or may include components, features, and / or functionality similar to those of system 800 of FIG. 8A , the example autonomous vehicle 1300 of FIGS. 13A-13D , the example computing device 1400 of FIG. 14 , and / or the example data center 1500 of FIG. 15 .

[0174] The system 900 may include one or more processors 902 (which may correspond to processor 802 in FIG. 8A ), a memory 904 (which may correspond to shared memory 806 in FIG. 8A ), and a disaggregated lookup table (DLUT) accelerator 906 (which may be included as one of accelerators 804 in FIG. 8A ). The processor 902 may, in an embodiment, include a VPU, and the memory 904 may include a VMEM. The DLUT accelerator 906 (or “DLUT 906”) may include a processing unit (PU) interface (I / F) 908 for communicating with the processor 902, a controller 912 for communicating with the processor 902, and a configurator 910 for configuring the DLUT 906 based on information shared from the processor 902 via the PU interface 908 and / or based on instructions from the memory 904 by the processor 902 regarding a location in the memory 904 where information is stored, such as a configuration message. For example, the PU interface 908 and the controller 912 may correspond to the advanced peripheral bus (APB) and event or programming interface, respectively, of the system 800. To help synchronize the processor 902 with the DLUT 906, the controller 912 may receive kick-off or trigger commands or signals (e.g., via an arrow labeled "Start") from the processor 902 indicating that the DLUT 906 can begin processing and / or receive polling signals from the processor 902. Additionally, when the DLUT 906 finishes processing one or more assigned tasks, the controller 912 may generate a signal to the processor 902 (e.g., via an arrow labeled "Complete") so that the processor 902 can begin setting up the DLUT 906 for its next task.

[0175] During configuration, the processor 902 can configure the DLUT 906 directly via the PU interface 908 and / or indirectly by indicating the location of configuration information in memory 904 via the PU interface 908. In the latter instance, the DLUT 906 can retrieve configuration information from memory, for example, via the shared read port strm1_dm_rd, and use the stored configuration information to configure the DLUT 906 (e.g., configure subunits (e.g., IAU, CDRU, PPU, etc.) and / or other components of the DLUT 906) to perform one or more tasks. For example, the processor 902 can set up data structures in memory 904 that the DLUT 906 needs to perform one or more tasks. For example, for a lookup of 1000 coordinates, the processor 902 can set up data structures in memory 904 for each of the 1000 coordinates and can further allocate buffers in memory 904 to which the DLUT 906 should write output. The processor 902 can also indicate which operations should be performed by the DLUT 906, such as 1D or 2D lookup, with or without interpolation, table reformatting, etc., and the DLUT 906 can use this information to configure its subunits. The configuration information set up by the processor 902 can also include instructions such as the bit-width of the coordinate index and the bit-width of the table entries. Thus, once the input and output buffers are set up in memory 904 and configuration information such as bit-width and operation type is sent to the DLUT 906, the processor 902 can kick off or trigger the DLUT 906 to begin processing. As a result, in contrast to systems that rely solely on the processor 902, the processor 902 can perform other tasks while the DLUT 906 is performing lookup, interpolation, table reformatting, and / or the like, thereby reducing execution time and improving efficiency.

[0176] During operation, the DLUT 906 may receive a list of indices corresponding to coordinates from memory 904, and the DLUT 906 may retrieve values ​​from a table corresponding to the index (e.g., if the values ​​are integer values) and / or may retrieve surrounding values ​​of decimal values ​​(e.g., left and right values ​​for a 1D lookup, or top-left, bottom-left, top-right, and bottom-right values ​​for a 2D lookup) to perform interpolation or other operations on the surrounding values. Once the final value is determined (e.g., directly by lookup if no post-processing is performed, or after processing by a post-processing unit (PPU) 930), the value may be written to an output buffer in memory 904 that corresponds one-to-one to the indices from the input buffer. To efficiently perform these tasks, embodiments may use an index address unit (IAU) 922, a conflict detection and resolution unit (CDRU) 924, a control (CTL) first in first out (FIFO) 928, a fractional (FRAC) FIFO 926, a post-processing unit (PPU) 930, a data coalesce unit (DCU) 932, and / or other components.

[0177] For example, index (IDX) stream 916 may include a stream of indices read from memory 904 (e.g., via a read port, strm1_dm_rd) to be looked up in one or more lookup tables, and values ​​corresponding to the indices may be read from memory 904 (e.g., via a read port, strm0_dm_rd) through lookup table (LUT) stream 918. Output (OUT) stream 920 may be values ​​written back to memory 904 (e.g., via a write port, strm0_dm_wr) after processing using DLUT 906.

[0178] When configured, the processor 902 can indicate to the IDX stream 916 how to access the data structure for the index. For example, for a 1D lookup where the interface to the memory 904 is 64 bytes wide, 64 bytes can be read each cycle. When a 1D lookup is performed, only one coordinate (e.g., one (x) value) can be read per index value, and for a 2D lookup, two coordinate indices (e.g., one (x, y) value) can be read per index. In a non-limiting example, each index can be 16 or 32 bits, so that each 64-byte read can potentially yield 8, 16, or 32 coordinates from the IDX stream 916.

[0179] The data in the IDX stream 916 may be sent to the IAU 922 in raw format as raw indices, where each coordinate may be an integer or fractional value. If the indices are fractional values, the IAU 922 may divide the fractional values ​​and provide the fractional bits to the FRAC FIFO 926 to aid in blending the surrounding values ​​looked up in the table using the PPU 930. The IAU 922 may then determine a set of indices to send to the CDRU 924, where the number of indices sent may correspond to the number of lookups the LUT stream 918 can perform in a single cycle. For example, if the LUT stream 918 can perform, for example, 32 lookups per cycle (based on the bit width of each value in the lookup table), the IAU 922 may send 32 indices to the CDRU 924 in each iteration. In some instances, such as when the values ​​from the IDX stream 916 to the IAU 922 are integer values, the IAU 922 may send each set of indices without any processing. However, if a value from the IDX stream 916 is a fractional value, the IAU 922 can determine which indices need to be looked up (e.g., two indices for 1D interpolation, or four indices for 2D interpolation) to obtain each of the surrounding values ​​needed to perform an interpolation or other operation to obtain a blended value corresponding to the fractional value. For example, if the fractional value corresponding to an (x,y) coordinate for a 2D lookup and interpolation is (5.3,6.2), the IAU 922 can determine that lookups should be performed at (5,6), (5,7), (6,6), and (6,7), and the PPU 930 can then blend the values ​​to generate a final value corresponding to index (5.3,6.2). For example, the values ​​may be blended equally weighted, or may be blended using bilinear interpolation, such that values ​​closer to (5,6) rather than (6,7) are more heavily weighted to calculate the final value of (5.3,6.2).

[0180] The sets of lookups (e.g., 32 lookup indices if LUT stream 918 can read 32 values ​​per read cycle) may be sent to CDRU 924 in the appropriate order corresponding to the order of the indices in the input buffer of memory 904 that were read using IDX stream 916. CDRU 924 then performs conflict detection and resolution by identifying bank conflicts that would occur if the lookup tables in LUT stream 918 were read in the order received from IAU 922 and reordering the indices to avoid the bank conflicts, thereby resolving the bank conflicts. For example, if looking up a set of indices causes a bank conflict and another (e.g., later or earlier) set of indices is available for another lookup cycle, CDRU 924 can find a non-conflicting lookup from the other lookup cycle and replace the non-conflicting lookup with the conflicting lookup in that cycle. As a result, one or more bank conflicts are avoided, thereby increasing throughput. In an illustrative example, if the IAU sends 32 indices each cycle and there are bank conflicts for 6 of the indices in a given cycle, the CDRU 924 can determine up to 6 indices from other lookups that do not cause a conflict with the current lookup and can perform these 32 lookups, e.g., 26 lookups from the original 32 and 6 lookups from another set sent by the IAU 922. Once the set of lookups is determined (e.g., with or without substitution due to conflicts), the set of lookups can be read from memory 904 using the LUT stream 918.

[0181] When substitutions are made to account for out-of-order lookups, the CDRU 924 can use the CTL FIFO 928 to indicate to the data coalescing unit the lookup order for each set of lookups from the IAU 922. For example, the DCU can determine that an initial set of 32 lookups was performed 8 times in the first cycle, then 8 times in another cycle, then 16 times in another cycle, then determine that the entire set of 32 has been processed, and can then push the 32 lookups to the PPU 930 for post-processing, if applicable, or directly to the OUT stream 920 for writing to an output buffer in memory 904. This additional information indicating the actual order of the lookups, as determined by the CDRU 924 and read into the LUT stream 918, can be communicated to the DCU 932 via the CTL FIFO 928. Thus, any changes that the CDRU 924 makes to the order of the indices received from the IAU 922 can be taken into account by the DCU 932. CTL FIFO 928 can be useful because the number of cycles through IAU 922, CDRU 924, etc. are not deterministic but rather data dependent. For example, since conflicts are not known in advance (e.g., because the data cannot be deterministic) and are a result of programming, there is no solution that completely avoids conflicts, and CTL FIFO 928 helps indicate to DCU 932 the configuration of lookups as a result of conflict resolution.

[0182] The PPU 930 can calculate final values ​​for each index that can be read into memory 904 as needed, such as when additional operations need to be performed on the lookup table values. If no post-processing is required, the PPU 930 may be unnecessary other than collecting the results. For example, if a regular 1D or 2D lookup is performed on an index that is an integer value that maps directly to a location in the lookup table, the PPU 930 and FRAC FIFO 926 are not used to perform additional processing. For example, if linear interpolation in a 1D lookup or bilinear interpolation in a 2D lookup and / or other operations are performed, the PPU 930 and FRAC FIFO 926 can be used to convert the collected results into updated results or values ​​for writing to memory 904.

[0183] In some embodiments, the DLUT 906 can be used in a table reformatting mode. For example, the IDX stream 916 and the OUT stream 920 can be used to update addresses for access and / or transposition. In such an example, if there is a buffer in memory 904 and the indices in the buffer are to be transposed, this operation can be offloaded to the DLUT 906 (rather than having the address generation unit of the processor 902 perform the transposition). Configuration information from the processor 902, for example, from the address generation unit, can indicate a read pattern for reading from the buffer in memory 904 and a write pattern for writing addresses back to memory 904 in a different pattern. For example, if a programmer knows that a particular access pattern results in many collisions, the programmer can program the processor 902 to configure the DLUT 906 to reformat the table and shuffle the data so that collisions are less likely or not likely to occur at all.

[0184] As a further example, the DLUT 906 may be used to return a sentinel value upon out-of-range detection, or to write output with out-of-range predication off. Thus, for example, if a coordinate in the IDX stream 916 is outside a given image patch and the corresponding value should not be written, the DLUT 906 may instead write a sentinel value, which may indicate to the processor 902 that the sentinel value should not be trusted or used in processing when processing the information in the output buffer. In some embodiments, this sentinel value may indicate to the processor 902 that the value should not be written to memory, and values ​​identified as error values ​​may not be stored.

[0185] Thus, DLUT 906 may be implemented as a pipeline of subunits that work together to perform a particular task or operation. Each subunit may operate independently and communicate with other subunits through a shared interface. With reference to Figure 9B, table 940 illustrates the tasks of the various subunits of DLUT 906 while processing a particular operation.

[0186] As a result of the DLUT accelerator described herein, the processor pipeline remains deterministic by offloading dynamic conflict detection and resolution to a separate accelerator. Additionally, the accelerator can operate independently of and concurrently with the primary processor (e.g., VPU), thereby reducing execution time. The DLUT accelerator may also enable 1D and / or 2D lookups from a common table with conflict detection / resolution. The accelerator can perform various post-processing operations, such as 1D lookups with linear interpolation, 2D lookups with bilinear interpolation, sentinel returns with out-of-range detection (both 1D and 2D), and / or output writes with out-of-range predication (both 1D and 2D). The DLUT accelerator can be configured to perform interpolation with a configurable number of fractional bits and can support a variety of index and data formats, including 8-, 16-, and 32-bit signed and unsigned data formats, and 16- and 32-bit 1D and 2D coordinate to index formats. The DLUT accelerator can also convert between global and local coordinates using configurable X / Y offsets. The DLUT accelerator also supports a data stream unit, which can read the index buffer from VMEM, perform lookups from VMEM, and write the results (or lookups or interpolations) to VMEM. The data stream unit can support up to 2D addressing for linear and transposed accesses. The lookup indices may be out of order to minimize bank contention to optimize the number of cycles required for lookup / interpolation; for example, if the VMEM supports N lookups, the accelerator may use M×N indices to maximize the number of indices that can survive conflict detection; and duplicate detection may be performed to eliminate duplicate indices that would reliably create conflicts.Furthermore, the 2D lookup and interpolation mode of the accelerator for DLUT can include an index that is automatically generated from some parameters within the accelerator (called auto-index mode), rather than the programmer providing a set of index data, which offloads the preparation of the index from the main processor to the accelerator.

[0187] Referring now to FIG. 9C , each block of method 950 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 950 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 950 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, while method 950 is described with respect to system 900 of FIG. 9A , method 950 may be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0188] 9C is a flow diagram illustrating a method 950 for using a separate lookup table accelerator in accordance with some embodiments of the present disclosure. The method 950 includes, at block B902, configuring one or more subunits of the DLUT accelerator based at least in part on configuration information generated using a processor. For example, the DLUT 906 may configure its subunits using information received from the processor 902 and / or retrieved from the memory 904.

[0189] The method 950 includes determining a first subset of indexes from a first set of indexes of the stream of indexes read from memory that are free of bank conflicts, at block B 904. For example, the IAU 922 can generate a set of indexes for the CDRU 924 to handle conflicts, and the CDRU 924 can determine a subset of indexes from the set that are free of bank conflicts.

[0190] The method 950 includes determining, from a second set of indexes of the stream of indexes read from memory, a second subset of indexes that do not have a bank conflict with the first subset of indexes, at block B906. For example, the IAU 922 can generate another set of indexes for the CDRU 924 to handle the conflict, and the CDRU 924 can determine to replace one or more of the conflicting indexes from the first set with one or more indexes from the second set of indexes that do not cause a conflict with the first set of indexes.

[0191] The method 950 includes, at block B 908, performing lookups of one or more lookup tables to retrieve multiple values ​​from memory in a single read cycle using the first subset of indexes and the second subset of indexes. For example, the DLUT 906 can read values ​​from memory 904 into the LUT stream 918 using a subset of values ​​from the set of indexes and values ​​from the second set of indexes that are determined not to conflict with the subset of values ​​from the first set of indexes.

[0192] The method 950 includes writing a plurality of values ​​to memory in block B910. For example, values ​​from the LUT stream 918 may be written to the memory 904 in the output stream 920. The DCU 932 may reorganize the data before writing out so that the data is in one-to-one order with the indices read from the input buffer in the IDX stream 916. In some embodiments, the PPU 930 may perform one or more operations, such as interpolation, on the obtained values ​​before writing the final values ​​to the memory 904 in the OUT stream 920.

[0193] Hardware Sequencer for Direct Memory Access Systems A direct memory access (DMA) system can be used to move data from different memory locations without the need for a central processing unit (CPU). For example, a DMA can act as a data movement engine, moving data from a source, such as external memory (e.g., DRAM) or internal memory, such as an L2 buffer or vector memory (VMEM) of a vector processing unit (VPU), to a destination, such as the VPU. The DMA system can actually perform additional operations, such as, but not limited to, padding frame data, manipulating addresses, managing overlapping data, managing different scan orders, and accounting for different frame sizes.

[0194] In digital signal processing, multiple DMA resources can be used to describe the movement of structured tiled data between external memory and a processor, such as a VPU. For example, the DMA resources can include descriptors, channels, triggers, and / or registers. The descriptors can describe the movement of tiles, such as source location, destination location, line pitch, tile width, tile height, circular buffer placement, and / or the like. However, moving tiled data across an image plane with spatial and temporal dependencies presents users with additional programming model challenges and requires many different DMA configuration resources. Such tiled data dependencies can also complicate the control code and control sequences of the processor (e.g., VPU) code. For example, a typical processing operation can include a filter operation, such as a 3x3 filter operation. This type of operation introduces spatial dependencies because every output pixel depends on the corresponding values ​​of the 3x3 pixels surrounding the output pixel. In such an operation, a filter can be performed using a 3x3 matrix of values, and this operation can be referred to as a spatially dependent operation. In practice, to ease the programming challenge, all tiles in a frame can be the same size, e.g., 64x64. However, if a 3x3 filter is used with a 64x64 tile, additional pixels would be needed above and below from adjacent tiles, as shown, for example, in the shaded area of ​​Figure 10C. Therefore, to be able to properly fetch data across tiles, this information needs to be encoded in the DMA resources, which incurs extra programming overhead to complete.

[0195] 10A-10G, which illustrate various challenges related to data movement when using a DMA system. For example, the visualization 1000 of FIG. 10A can accommodate padding of frame data. In the visualization 1000, there may be nine sections: an upper-left section, an upper section, an upper-right section, a left section, a center section, a right section, a lower-left section, a lower section, and a lower-right section. In this example, each section may include one or more tiles. For example, the upper-left section may include one tile, while the upper section may include, for example, four tiles. Therefore, to precisely define this sectioning, existing techniques may describe this frame using nine descriptors (e.g., one per section), three channels (e.g., one in the left column, one in the center column, and one in the right column), and three triggers (e.g., one per channel).

[0196] Regarding padding, when performing operations on data near the boundaries of a tile or frame section, for example, due to spatial dependencies, the DMA system may pad or fabricate values ​​for pixels outside the image boundaries. This may be because, in certain implementations, requesting data outside the image's memory area can cause a failure. Therefore, padding or fabricating values ​​can be performed after fetching image data from the corresponding memory area using DMA to avoid the failure. Without padding, for example, when performing a filtering operation, the data structure may not match the kernel size. The fetched data, including the additional padded values, can then be sent to the destination, e.g., a VPU, allowing the VPU to process the data according to the settings in the fetched data and process the data in the same manner across the entire (padded) frame. When padding, zero padding may be used (e.g., where each new data point contains a value of zero), duplicated values ​​may be used (e.g., duplicating pixel values ​​of adjacent pixels from the fetched data), and / or another padding mechanism may be used. Additionally, padding can be added to any side of a frame, and may be added differently to different sides. For example, in Figure 10A, the padded area 1002 may be larger on the right side of the frame than on the left, top, or bottom. Padding increases the complexity of DMA programming when moving data from a source to a destination, such as from memory to VMEM, and also increases the complexity of VPU programming when processing larger padded frames.

[0197] Referring now to FIG. 10B, the visualization 1010 in FIG. 10B corresponds to address manipulation by the DMA system. For example, to fetch consecutive frame data, the addresses of various descriptors can be manipulated and programmed. To effectively execute DMA, the addresses of the data movement can be consecutive. Therefore, the address of each descriptor can be manipulated, and this manipulation must be carried over from one descriptor to another. For example, as shown, when padding values, the starting address for each descriptor can be manipulated so that the fetched data includes the padded values. To do this, the programmer uses the starting address, tile width, and number of tiles in each section and uses this information to generate the address of the next descriptor. For example, the first descriptor can fetch data starting at the top left, then the top, then the top right, then the left, then the center, etc., as indicated by the arrows in FIG. 10B. However, the starting addresses of the descriptors add complexity to DMA programming when moving data to a destination such as VMEM.

[0198] As another example, with respect to FIG. 10C , the DMA system may need to read vertically and horizontally overlapping data from adjacent tiles to ensure continuous data processing. For example, as shown by the shaded area in FIG. 10C , overlapping data from a tile in the top-left section and a tile in the adjacent top section may need to be read in the same operation. Similarly, overlapping data from a tile in the top-left section and a tile in the adjacent left section may need to be read in the same operation. To achieve this, descriptors must be updated or moved to include the overlapping portions. For example, a base descriptor may contain the address of the start of the top section, but to include data from a tile in the adjacent top-left section, the descriptor for the top section must be updated (e.g., moved left) to include the data from the top-left tile. This updating requires more complex programming, especially as the number of descriptors increases.

[0199] Additionally, with respect to Figures 10D-10F, the DMA system may need to support various scan orders for sequentially reading data from memory. For example, whether performing filtering, convolution, matrix multiplication, and / or another operation, the associated scan order may differ. With this in mind, various scan orders may be supported, such as those shown in Figure 10D, including a raster scan order starting from the upper left (visualization 1030), a raster scan order starting from the upper right (visualization 1032), a raster scan order starting from the lower left (visualization 1034), and / or a raster scan order starting from the lower right (visualization 1036). Similarly, with respect to visualization 1038 of Figure 10E, various cubic scan orders may be supported by the DMA system for a cubic image. 10F illustrates various vertical mining scan orders that may be supported by a DMA system, such as a vertical mining scan order starting from the top left (visualization 1040), a vertical mining scan order starting from the top right (visualization 1042), a vertical mining scan order starting from the bottom left (visualization 1046), and / or a vertical mining scan order starting from the bottom right (visualization 1048). Supporting each of these various scan orders increases the complexity of the DMA programming to move data into memory (e.g., VMEM).

[0200] With reference to FIG. 10G, the DMA system may also need to support various frame sizes, such as the transfer of multiple frames (e.g., luma / chroma composites or various pyramid levels) having various sizes. For example, a processor such as a VPU may process frames of various sizes to generate a final desired output. FIG. 10A shows an example visualization 1048 corresponding to pyramidal processing of frames for optical flow estimation computation. In such an example, pixel transfer may be calculated first for smaller frame sizes, then for larger frame sizes using cues from the output of the smaller frame sizes, then for even larger frame sizes using cues from the larger frame sizes, and so on. Thus, while the DMA system can support fetching frame data for various frame sizes, this functionality requires more complex programming of the DMA system. For example, descriptors need to be programmed or updated for each different frame size.

[0201] To simplify the programming of these various operations supported by the DMA system, the DMA system and method of the present disclosure can use a hardware sequencer in combination with a DMA engine to account for data movement. For example, data movement across an entire image can be explicitly and completely described in hardware sequencing mode using a simplified programming model (e.g., a frame image structure, as shown in FIG. 10I) that handles tile sequencing (triggers), padding, overlap (offsets), scan order, and different frame sizes. The hardware sequencer can reduce DMA resource usage (e.g., reducing the number of required descriptors, triggers, channels, etc.), offload control from the VPU for VPU control processing, and reduce the complexity of DMA programming. This can be achieved by loading an image or frame descriptor diagram (e.g., as shown in FIG. 10I) in the form of a series of commands from programmable local memory. The hardware sequence commands can incorporate each of the operations described herein that increase programming complexity, including image padding, tile overlap or offset, frame offset, image scan order, and image size at tile granularity. The hardware sequencer can read the image commands from memory, along with descriptor information (e.g., from the image commands or from a separate descriptor memory or SRAM), and sequence the tile movements to scan and fill the entire frame.

[0202] Referring now to FIG. 10H, FIG. 10H illustrates a DMA system 1050 including a hardware sequencer, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, system 1050 may be included in and / or may include components, features, and / or functionality similar to those of example autonomous vehicle 1300 of FIGS. 13A-13D, example computing device 1400 of FIG. 14, and / or example data center 1500 of FIG. 15.

[0203] The system 1050 may include a DMA engine 1056, a register control unit 1058, a hardware (HW) sequencer controller 1060, a descriptor SRAM 1052, and / or a hardware (HW) sequencer command SRAM 1054. Existing systems may only include the DMA engine 1056 and a descriptor SRAM 1052 that stores frame descriptors. Thus, as described herein, the DMA engine 1056 previously had to perform all of the operations, such as padding and address manipulation, when sending data from a source to a destination, and the VPU or other source had to perform the sequencing handshake with the DMA system (e.g., with the VPU as the primary node and the DMA as the secondary node). In such an example, the DMA engine 1056 would operate at the tile level, using descriptors of various sections of the frame, each containing one or more tiles, to retrieve tiles one at a time and send them to the destination, with subsequent tiles being retrieved according to the descriptor based on instructions from the VPU to retrieve the next tile.

[0204] However, using the system 1050 of FIG. 10H, frames can be processed at the frame level. For example, only one descriptor can be used for the frame shown in FIG. 10C, which previously required nine descriptors. Thus, in effect, when the DMA engine 1056 attempts to load a descriptor from the descriptor SRAM 1052 (or, more generally, the descriptor memory 1052), the HW sequencer control unit 1060 can intercept the descriptor load and use a command sequence processing structure to handle multiple frames, tile rows / columns, and multiple descriptors. To do this, a frame format 1070 (FIG. 10I) can be used, which describes frames at a higher level by processing tile rows / columns (according to scan order) in hardware rather than at the tile level. For example, instead of padding tiles, the frame format 1070 can be used to pad an entire frame, thereby padding many frames with just one padding command. Thus, the entire frame can be understood, including where to pad, where to overlap, how to automatically manipulate addresses, etc. Additionally, because the DMA engine 1056 can pull descriptors directly from the descriptor SRAM 1052 without intervention from the HW sequencer control unit 1060, traditional formats can still be supported for operations that may not be able to benefit from the HW sequencer control unit 1060.

[0205] The HW sequencer control unit 1060 can operate as a state machine, for example, reading the HW sequencer command SRAM 1054 (or more generally, the HW sequencer command memory 1054), which stores frame formats 1070 containing sequence processing commands. A processing controller, such as an R5 processor, CPU, or ARM processor, can program or configure the HW sequencer command SRAM 1054 and descriptor SRAM 1052 using programmed code and / or settings from higher-level engines.

[0206] The descriptor SRAM 1054 may include one or more descriptors that may specify tile dimensions (e.g., tile width dx and tile height dy), the start point of the image or frame (e.g., top left, bottom right, etc.), the type of trigger, and / or other micro information related to the scan type of the descriptor.

[0207] The HW sequencer command SRAM 1054 can store a frame format 1070 that defines the entire frame, the frame size, frame padding, etc. For example, the frame format 1070 can include a frame header for header control, offset control, and padding control, as well as column and / or row headers for the columns or rows of the frame (e.g., column headers for a vertical scan pattern and row headers for a raster scan pattern). The frame header control can include the number of descriptor rows and / or descriptor columns as well as a frame repetition factor to identify how many times a particular frame should be repeated. The frame header offset control can include a frame tile offset (e.g., tile-to-tile offset) and a frame offset (e.g., the offset between two or more frames that can be read using only one channel, such as when a YUV frame containing three separate planes is being processed). The frame padding header can indicate the number of lines or pixels of padding to add at the frame level (rather than at the per-tile level as in previous approaches), such as padding on the left side of the frame, the top of the frame, the right side of the frame, and / or the bottom of the frame, so that the entire frame is padded rather than padding each tile within each section of the frame at the tile level.

[0208] Column headers may be used when the scan order is vertical, and row headers may be used when the scan order is raster or horizontal. The column and / or row headers may include a column or row offset (e.g., the magnitude of the offset between each column or row), a column or row repetition factor (e.g., the number of times to repeat the same column or row processing type across a frame, such as N-1 times, where N is the number of times the column or row is processed), and the number of descriptors used per column or row (e.g., only one descriptor may be used to repeat the same tile across a row or column, or a first descriptor may be used to scan part of a row and a second descriptor may be used to scan another part of the row, etc.). The descriptor ID may be described, for example, so that a descriptor stored in descriptor SRAM 1052 may be retrieved and used to describe the row or column. For example, the descriptor ID may indicate which descriptor to use for a particular column and / or row, as well as how many times the descriptor is repeated (e.g., N-1 times, where N is the total number of times the descriptor is used). In an embodiment, there may be a set number of descriptors (e.g., 64), and the descriptor ID may be used to determine which descriptor should be used for a particular column and / or row. In this manner, the hardware sequencer controller 1060 examines the higher-level structure of the frame on top of the basic descriptors from the descriptor SRAM 1052, thereby simplifying the resources required by the DMA engine 1056 to perform the same data transfer. Additionally, the HW sequencer control unit 1060 may prefetch tiles in advance, e.g., using the register control unit 1058, to reduce latency, so that the tile's data may be immediately available upon request from the DMA engine 1056.

[0209] During operation, the HW sequencer control unit 1060 can read the image structure (e.g., frame format 1070) from the HW sequencer command SRAM 1054 along with the descriptor information from the descriptor SRAM 1052 and combine the information to sequence across frames for the DMA engine 1056. Thus, rather than requiring separate coding for the DMA engine 1056 for each descriptor, trigger, channel, etc., the HW sequencer control unit 1060 can read the image structure, fetch the descriptors, and sequence across frames in the appropriate descriptor format for the DMA engine 1056. The register control unit 1058, in embodiments, can help control scan order, prefetching, and / or other frame addressing control. The HW sequencer control unit 1060 further simplifies the VPU's code so that the VPU does not need to consider multiple channels. Instead, the VPU can request a tile, then the next tile, then the next tile, etc. Since the HW sequencer control 1060 knows the current position in the frame, it knows the next tile to be fetched for the DMA engine 1056, and the DMA engine 1056 does not have to keep track of this information internally.

[0210] Thus, system 1050 can be backward compatible with previous approaches, since the system can still support the use of various descriptors, triggers, channels, etc., but can also understand them at the frame level to reduce complexity. System 1050 can support image padding at all corners of a frame with various pixel padding sizes, vertical and / or horizontal tile overlap to allow VPUs to access adjacent tiles for processing along tile boundaries, and frame scanning in different scan orders. Additionally, system 1050 can support automatic tile offset adjustment by HW sequencer controller 1060 in the destination VMEM. Because frame descriptors are linked via hardware, users do not need to link or stitch descriptors together. HW sequencer controller 1060 can manage the sequencing of addresses across descriptors / tiles across a frame without additional programming complexity, and HW sequencer controller 1060 can prefetch tiles to improve performance.

[0211] In some embodiments, the descriptors may be included in the image or frame structure rather than being stored separately in the descriptor SRAM 1052. For example, if legacy compatibility is not implemented, the entire sequencing structure and tile structure may be described within the frame structure. In such an example, the frame format of FIG. 10I may include additional information for the descriptor, such as tile width, trigger type, etc., so that the HW sequencer control unit 1060 can use the same information as if the descriptors were stored separately in the descriptor SRAM 1052.

[0212] Referring to FIG. 10J, FIG. 10J is an example of the frame format 1070 of FIG. 10I when implemented for a raster scan sequence, according to some embodiments of the present disclosure. For example, frame format 1070A is an example of a frame format using only one channel, only one trigger, and only one descriptor and frame addressing in raster mode. In this example, the tile structure may be 16x8. FIG. 10K is an example of this tile structure sequenced in hardware in a raster scan sequence and frame addressing using the example frame format 1070A, according to some embodiments of the present disclosure. For example, the same descriptor (e.g., tile dimension) may be used for each row of tiles (denoted by "D1" in the visualization 1072), such that the same tile is used 16 times along each row (C1 through C16) and repeated for eight rows (R1 through R8) from top to bottom. The sequence may include 20 bytes, as shown in frame format 1070A, and each row may have N*2+ bytes, where N represents the number of entries per row (as shown in FIG. 10J). Thus, to sequence frames as shown in visualization 1072, frame format 1070A may include no frame repetition, the number of descriptor rows may be zero, no tile offset, no frame offset, three rows for left pixel (PL), right pixel (PR), top pixel (PT), and bottom pixel (PB) as frame padding, rows may be repeated seven times (for a total of eight rows), each row offset is the tile height (Ty) (so that each row is offset by the tile height), one descriptor with descriptor ID D1 may be used, and the descriptor may be repeated 15 times in each row (16 times total).Thus, in effect, the HW sequencer controller can use the descriptor corresponding to D1 (including tile height and tile width) from descriptor SRAM 1052 and sequence the image tile by tile (16 tiles per row), row by row (R1 through R8), to the destination processor (e.g., a VPU) using the image structure in frame format 1072 stored in HW sequencer control SRAM 1054. In this way, only one descriptor, one trigger, and one channel can be used, thereby reducing programming complexity while also allowing DMA system 1050 to be the primary or controlling component in the interaction between DMA system 1050 and the VPU.

[0213] In some embodiments, as an extension to the HW sequencer controller 1060, a DMA trigger mode can be used to reduce software intervention in programming the VPU by having the DMA system 1050 command a sequence of descriptors. For example, the DMA system 1050 can read an image from external memory, tile the image, and sequence the tiles for the VPU. The VPU may make start and completion signals accessible to facilitate this. VPU initiation can be driven by the DMA system 1050, and when the VPU finishes processing a block of instructions, the VPU can send a completion signal to the DMA system 1050. Thus, the DMA system 1050 (e.g., the HW sequencer controller 1060) and the VPU can engage in a handshake mechanism where the DMA system 1050 is the primary node and the VPU is the secondary node. This DMA trigger mode can minimize the overhead of VPU tile control and simplify the programming model for the DMA engine 1056. For example, specific code to double-buffer DMA data movements may not be necessary, and the DMA kernel code may be unrelated to the VPU kernel code. Therefore, the DMA trigger mode simplifies the VPU code because tile sequencing is handled by the DMA system using the HW sequencer controller 1060. The example code below shows the VPU code before and after adding the DMA trigger.

[0214] before

number

[0215] rear

number

[0216] As a result, whereas previously the VPU would request the movement of tiles to VMEM, now that HW sequencer control unit 1060 controls the sequencing, DMA system 1050 can trigger the movement of tiles to VMEM, targeting the VPU. In this way, DMA system 1050 can pre-fetch data to be processed by the VPU, and when the VPU indicates that processing is complete, DMA system 1050 can immediately make the next data to be processed available (e.g., in VMEM) and indicate this to the VPU as well.

[0217] When performing processing of a frame, the HW sequencer control unit 1060 can obtain descriptors (which may indicate tile dimensions, trigger type, etc.) from the descriptor SRAM 1052 and can obtain image structures from the HW sequencer command SRAM 1054. The HW sequencer command unit 1060, in conjunction with the register control unit 1058, can then begin scanning the first row or column according to scan order and using the first (and in this embodiment, only) descriptor, then, if more than one descriptor is used, move to the second descriptor based on the number of iterations (e.g., 1-N) that are satisfied, and so on. Once each tile is determined, the DMA engine 1056 can obtain the tile's data from the source data and write the tile's data to the destination data (e.g., in VMEM). Once the data is written to the data's destination, the processor (e.g., the VPU) can be notified by the HW sequencer control unit 1060 that the data is available for the processor to begin processing. The DMA system 1050 can then, during processing, fetch the next tile of data based on the sequence from the HW sequencer control unit 1060 and write the data to the data's destination, so that when the processor indicates that processing is complete, the HW sequencer control unit 1060 can indicate to the VPU (via a handshake mechanism) that the next data to be processed is available, and so on until processing is complete.

[0218] Referring now to FIG. 10L, each block of method 1080 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 1080 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 1080 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, while method 1080 is described with respect to the system of FIG. 10H, method 1080 may be performed by any one system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0219] 10L is a flow diagram of a method 1080 for a DMA system including a hardware sequencer according to some embodiments of the present disclosure. The method 1080 includes, in block B1002, retrieving a tile structure from a descriptor memory and a frame structure corresponding to a frame from a HW sequencer command memory. For example, the HW sequencer control unit 1060 can retrieve a descriptor from the descriptor SRAM 1052.

[0220] The method 1080 includes sequencing the retrieval of tiles of a frame from a source memory in block B 1004. For example, in an embodiment, the hardware sequencer control unit 1060, in conjunction with the register control unit 1058, can sequence the retrieval of tiles from the source memory by the DMA engine 1056 according to descriptions of the tiles from the frame (or image) structure and descriptors.

[0221] The method 1080 includes writing the data corresponding to the retrieved tile to a destination memory in block B 1006. For example, the DMA engine 1056 can write the data corresponding to the retrieved tile to a destination memory (e.g., VMEM) for processing by a destination processor (e.g., VPU).

[0222] The method 1080 includes, in block B1008, providing an indication to a processor associated with the destination memory that the retrieved data has been stored in the destination memory. For example, the HW sequencer control unit 1060 can indicate to the processor that the next tile of data is ready for processing.

[0223] Method 1080 includes receiving an indication that processing of the acquired data is complete in block B1010. For example, once processing is complete, the processor can indicate to DMA system 1050 that processing is complete, at which point the next tile of data can be loaded (or may have been preloaded) into the destination memory, and DMA system 1050 can indicate this to the processor.

[0224] Use the VPU to configure the DMA system for region-dependent data movement When fetching a known data pattern, a processing controller can configure the direct memory access (DMA) system, and a processor (e.g., a vector processing unit (VPU)) can trigger and sequence the DMA. However, when processing various data points or features for irregular or unknown data patterns, reconfiguring data movement can be challenging because feature or object locations are dynamically calculated. For example, object tracking algorithms, feature tracking algorithms, object detection algorithms, deep learning algorithms using variable-sized regions of interest (ROIs), and / or other region-dependent data movement algorithms require the DMA system to dynamically adjust address and data pairs so that a processor, such as a VPU, has the appropriate information to process. In conventional systems, when fetching an unknown data pattern, such as in object tracking, the processing controller (e.g., an R5 processor core controlling a programmable vision accelerator (PVA)) may require an interrupt to intervene in the processing cycle to determine the updated information calculated by the processor (e.g., the VPU) and reconfigure the DMA for the next iteration. Thus, the processing controller introduces additional latency into, for example, tracking algorithms that require short response times.

[0225] Considering the shortcomings of conventional systems requiring intervention by a processing controller, the disclosed system and method can use a DMA and a processor (e.g., a VPU) to set up a tightly coupled processing loop that allows the DMA to reconfigure the DMA's descriptors based on the processor's output. Thus, the DMA can be dynamically reprogrammed at runtime to handle certain algorithms that require region-dependent data movement. Using this VPU configuration mode, the DMA's descriptors can be updated to track feature data, including location, based on the VPU's calculations at runtime. To this end, the VPU can specify a list of address and data pairs in memory, such as VMEM, and then trigger the DMA to update its own descriptors and collect data from the region with the newly calculated address. By relying on an interface between the VPU and DMA, processing controller intervention may be unnecessary once the processing controller (e.g., an R5 or ARM processing core) initially configures the VPU and DMA to begin processing. Therefore, this bulk, fast, synchronous MMIO access to update feature descriptors reduces latency for object tracking, feature tracking, object detection, deep learning, and / or other algorithms involving region-dependent data movement.

[0226] Referring now to FIG. 11A, FIG. 11A illustrates a data flow diagram 1100 of a process for configuring a direct memory access (DMA) system using a vector processing unit (VPU), according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, process 1100 may be performed by a system including components, features, and / or functionality similar to those of the example autonomous vehicle 1300 of FIGS. 13A-13D, the example computer processing device 1400 of FIG. 14, and / or the example data center 1500 of FIG. 15.

[0227] A system executing process 1100 may include a processing controller 1102 (e.g., an R5 processor, an ARM processing core, an instruction set architecture (ISA), an X86 architecture, etc.), a direct memory access (DMA) system 1104, a vector processing unit (VPU) 1108 (or another processor type), a vector memory (VMEM) 1110 (or another memory type), and a descriptor RAM 1106. In practice, a VPU configuration mode may configure a DMA descriptor by writing a sequence of non-contiguous address / data pairs to the DMA descriptor SRAM. Process 1100 may be described with reference to an exemplary feature or object tracking algorithm. However, this is not intended to be limiting, and process 1100 and the underlying system may be used to perform any type of algorithm, such as one involving region-dependent data movement.

[0228] For example, a first operation may include the processing controller 1102 setting up the DMA 1104 and the VPU 1108 to perform processing on some data, and then triggering both the DMA 1104 and the VPU 1108 to perform the processing. For example, the processing controller 1102 may load a starting point in memory into the descriptor RAM 1106 for processing and may also set registers in the VPU 1108 for the particular type of operation that the VPU 1108 will perform on the data.

[0229] In a second operation, the VPU 1108 can trigger the DMA 1104 to read an initial feature data point in the VMEM 1110. For example, the VPU 1108 needs data from the DMA 1104 to begin work, so the VPU 1108 configures the DMA 1104 to load a data point into a location in the VMEM 1110 that the VPU 1108 knows about to obtain the data for processing.

[0230] In a third operation, the VPU 1108 may process the current feature data set and then calculate the position of the tracked object or feature, so that the VPU 1108 now has a calculated new, or updated, position of the tracked feature or object.

[0231] In a fourth operation, the VPU 1108 can update the VMEM 1110 with the updated location using a VPU configuration format (described in connection with FIG. 11B), which can then trigger the DMA 1104 to update the DMA descriptor in the descriptor RAM 1106. For example, FIG. 11B is a table 1120 illustrating a VPU configuration format written by the VPU to the vector memory (VMEM) and read by the DMA system, according to some embodiments of the present disclosure. For example, the format can include four bytes for the address and four bytes for the data per address / data pair.

[0232] In a fifth operation, the DMA 1104 can update the descriptors in the descriptor RAM 1106 to obtain the appropriate data for the next iteration of processing by the VPU 1108. For example, the DMA 1104 can read the address / data pair VPU configuration format and patch the operation descriptor with the updated locations. In an embodiment, there can be a one-to-one correspondence between feature points and descriptors, and thus each tracked feature, object, or point can include an associated descriptor. In this way, the address / data pair for each tracked feature, object, or point can be updated over time using the individual descriptors.

[0233] In a sixth operation, the DMA 1104 can fetch new feature data for the location using the newly updated descriptor in the descriptor RAM 1106. For example, the DMA 1104 can indicate to the VPU 1108 that the descriptor has been updated, and the VPU 1108 can trigger the DMA 1104 to read the new data from the VMEM 1110, and so on.

[0234] As a result, after the first setup operation by the processing controller, operations 2 through 6 are repeated, forming a tightly synchronized VPU setup loop that does not require processing controller intervention, thereby reducing latency to account for the short response times required by tracking or detection algorithms. Additionally, as DMA 1104 overwrites the address in memory with the new, updated address, DMA 1104 is also updating the code that DMA 1104 needs to consult to understand what to fetch next. This improves throughput, unlike conventional systems that relied on a control bus to update registers according to the address and data. Thus, the benefits of defining an address / data protocol are realized when a variable amount of address locations can be updated with a variable amount of data, along with how address / data pairs are updated. This allows DMA 1104, which may have a width greater than that of the control bus (e.g., 512 bits versus 32 bits, respectively), to update up to (e.g., but not limited to) eight address / data pairs at a time (if each address / data pair is defined using eight bytes, as in FIG. 11B).

[0235] Additionally, while the DMA is shown updated using the VPU configuration mode of process 1100, additional or alternative elements or components of the system may be updated. For example, the instruction cache of VPU 1108 may be updated using a similar technique using the VPU. As another example, a hardware sequencer program to be updated may be written to update the hardware sequencer memory by providing address data. This would essentially involve writing the new program to a hardware sequencer RAM, such as hardware sequencer RAM 1054 for hardware sequence controller 1060 in FIG. 10H.

[0236] Referring now to FIG. 11C , each block of method 1150 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. Method 1150 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 1150 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, while method 1150 is described with respect to the system of FIG. 11A , method 1150 may be performed by any system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0237] 11C is a flowchart of a method 1150 for configuring a DMA system using a VPU, according to some embodiments of the present disclosure. The method 1150 includes, at block B1102, calculating, using a processor, a first output corresponding to one or more first updated positions of the tracked features based at least in part on first data written to memory using the DMA system. For example, the VPU 1108 can access data from VMEM 1110 written to VMEM 1110 using DMA 1104 and can process the data to calculate positions of one or more objects corresponding to tracked features, objects, points, etc.

[0238] The method 1150 includes, in block B1104, using a processor to update the memory to include second data representing one or more address / data pairs corresponding to the one or more first updated locations. For example, after calculating the one or more locations, the VPU 1108 can update the address / data pairs in the VMEM 1110 in a format such as the format described in connection with FIG. 11B .

[0239] The method 1150 includes, in block B1106, updating one or more descriptors corresponding to the tracked features using the DMA system and based at least in part on the one or more address / data pairs. For example, the DMA 1104 can access the address / data pairs in the VMEM 1110 and use the address / data pairs to update the descriptors in the descriptor RAM 1106 for the next read operation.

[0240] The method 1150 includes, in block B1108, writing third data to memory using the DMA system and based at least in part on the one or more descriptors. For example, the DMA 1104 can write updated data corresponding to the address / data pair identified using the descriptor to the VMEM 1110.

[0241] The method 1150 includes, in block B1110, calculating, using the processor and based at least in part on the third data, a second output corresponding to a second, updated position of one or more of the tracked features. For example, the VPU 1108 may calculate a next set of updated address / data pairs corresponding to the tracked features, objects, points, etc. as the updated data enters the VMEM 1110, and this process may be repeated until processing is complete.

[0242] Detecting Permanent Faults in Programmable Vision Accelerators (PVAs) Safety-critical applications, such as autonomous and semi-autonomous machine applications, have stringent requirements for detecting and isolating permanent faults. For example, when running deep learning, computer vision, sensor processing, and / or other machine-based applications, permanent fault detection must be performed periodically and within an allotted time budget, not only to enable accurate testing but also to enable proper execution of the application, e.g., with low latency. For automotive safety integrity level (ASIL) D, applications running on autonomous or semi-autonomous machines may require 90% or greater coverage of permanent faults. To achieve this, 100% coverage may be required while still meeting the time budget for each specific application to run with low latency. Traditional approaches use built-in self-test (BIST) to identify faults, but these BIST techniques do not provide sufficient coverage, introduce excessive latency into the system, and / or do not meet the time budget for running a specific application.

[0243] In view of these deficiencies of conventional approaches, the present system and method can perform BIST on a multiple-input signature register (MISR) to perform fault detection on, for example, a programmable vision accelerator (PVA) in a system-on-chip (SoC). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs controlled using one or more processing controllers (or control processors), such as an R5 processor, an ARM processor, a CPU, and / or the like. As such, each component of the PVA may require testing, and the present system and method performs BIST on the MISR to detect 100 percent of permanent faults. In this manner, permanent fault detection can be performed with 100 percent coverage of both control logic and data logic blocks, reporting errors directly to a safety processor to reduce latency, and customizing specific applications to meet associated execution time budgets.

[0244] In various embodiments, a MISR may be used in a PVA to perform BIST of software logic for permanent fault detection. The MISR hardware described herein with respect to FIG. 12A and / or FIG. 12B may include cyclic redundancy check (CRC) hardware that is initialized, for example, with a known seed value using a processing controller. When executing a PVA application, the processing controller may allocate a portion of the timing budget, for example, approximately 10% or less of the timing budget, to run a known software MISR test using known inputs with deterministic pre-computed outputs of the correct signature or golden value. For example, if the timing budget corresponds to 30 frames per second, a timing budget equivalent to 3 frames or less may be allocated to the MISR test. The processing controller may start the MISR test at the allocated time, wait for the test to complete, and terminate the MISR CRC calculation. Upon completion of the test, the MISR hardware may read back the final CRC value and check it against the pre-computed golden value. In the event of a mismatch, the MISR hardware reports the error directly to the SoC's safety processor, which can take further action to handle the safety error, such as causing the application's output to be ignored, or addressing or avoiding a permanent fault.

[0245] Thus, the MISR hardware in the DMA block can monitor one or more (e.g., in an embodiment, all) transactions on one or more (e.g., in an embodiment, all) of the PVA's Advanced Extensible Interface (AXI) master ports. In an embodiment, by inspecting all output stages from the PVA, the safety integrity of the PVA can be checked for permanent faults that could corrupt the output stage, e.g., output information, which could potentially be consumed by the PVA and / or another engine during application execution. Thus, the MISR hardware can detect errors across various blocks of the PVA (e.g., the processing controller, the VPU, and the DMA system) because all of these components cooperate and interact with respect to producing the output stage. Signatures computed by the MISR hardware can represent the state of these various blocks of the PVA throughout the duration of the MISR test.

[0246] In an embodiment, the MISR method may include a CRC check on both the write address (e.g., 40-bit control) and write data (e.g., 512-bit data) leaving the AXI master port. This feature may allow for control path failures (e.g., addressing errors) to be separated from data path failures (e.g., calculation errors). The MISR hardware configuration described herein may allow for checking of each DMA AXI port. In an embodiment, a control bit may be used to disable address and data output writes for all channels involved in the MISR calculation to conserve memory subsystem bandwidth consumption during memory allocation. Additionally, the MISR method may include a control register bit per channel to exclude or mask specific channels from the MISR calculation, e.g., to isolate non-secure channels. The DMA, in an embodiment, performs the MISR CRC:X using the IEEE802 and MPEG CRC-32 primitive polynomials. 32 +X26 +X 23 +X 22 +X 16 +X 12 +X 11 +X 10 +X 8 +X 7 +X 5 +X 4 +X 2 +X+1 can be calculated. The MISR SET register can be used to set the CRC initial value (e.g., seed value) for both address and data CRC calculations. The MISR REF register can be used to compare CRC values ​​for both address and data CRC calculations.

[0247] To support a 512-bit MISR, 8:1 bit data compression can be applied. For example, each data byte can be compressed to one data bit through an 8>1 exclusive OR (XOR) operation to form 2 x 32-bit message data. To support a 40-bit MISR address, the nine most significant bits can be compressed. For example, the nine most significant address bits can be compressed through a 9>1 XOR operation to form a 32-bit message address. Test pattern and instruction diversity can be used to cover aliasing related to compression. If there are even numbers of errors in a byte of the output image, the probability of aliasing occurring can be low because the error failure will not result in an address CRC error. Additionally, the likelihood of aliasing can be low because a reference CRC can be calculated for output images with the same pattern in the same even error bit location throughout the entire MISR test. Aliasing was shown to cause an average loss of 0.25% in coverage during testing. Data compression with such low aliasing is valuable because the width of the bus is, for example, 512 bits in the illustrative embodiment; without compression, MISR testing may not meet the latency or execution time budget of the system.

[0248] The MISR timer register can be used to time out the MISR calculation, and the MISR timer register can be decremented every AXI clock. The time-out feature can be useful in the event of a fault that could cause the MISR test to hang and prevent the MISR hardware from reporting an error. The processing controller can use a software event to stop the MISR calculation when the MISR test is finished. The DMA system can compare the MISR REF value with the MISR VAL value for both the data and address outputs of the MISR test, and the DMA hardware can update the MISR status register based on the results of the comparison. For example, the MISR status register can contain one of the following values: 0: idle, 1: complete, data failed, 3: busy, 4: complete, both address and data failed, 5: complete, failed due to time-out, 6: reserved, and 7: complete, passed. In case of a MISR time-out error, the DMA can generate a time-out signal to the secure processor, and in case of a data and / or address CRC check error, the DMA can assert a secure error to the secure processor.

[0249] Referring to FIG. 12A, FIG. 12A is a diagram of a built-in self-test (BIST) system performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, MISR hardware 1250 may include components, features, and / or functionality similar to those of DMA system 1050 of FIG. 10H , example autonomous vehicle 1300 of FIGS. 13A-13D , example computing device 1400 of FIG. 14 , and / or example data center 1500 of FIG. 15 . For example, MISR hardware 1200 may be included in a DMA block of a PVA, such as that shown in FIG. 10H . Thus, MISR hardware 1200 may operate in the output stage of data movement, and addressing (or control) may tap into the output of DMA engine 1056. In the example of FIG. 12A , there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used, depending on the embodiment.

[0250] In operation, the processing controller can control one or more processing components of the system, such as DMA system 1050 and MISR hardware 1200, as well as the VPU. When performing MISR testing of DMA system 1050, in an embodiment, the test code can include all zeros, all ones, alternating zeros and ones, and / or irregular code sequences. In this manner, high coverage can be achieved for DMA system 1050. For example, when testing a VPU, the test code can include application-specific or customized code. For example, components or portions of the VPU (e.g., registers, logic, etc.) that are used during coverage testing for a particular application can be determined, and test code can be generated to include those specific components or portions of the VPU when executing the test code. For example, this irregular data can be included in the test code in response to different instructions, such that tests are sequenced with different instructions to use different areas of the VPU's logic. In this manner, coverage can be increased for the VPU in general and for the particular application running on the VPU in particular. By performing DMA and VPU testing in this manner, and because the processing controller is involved in the control and interaction between various components (e.g., DMA system 1050 and the VPU), the processing controller can have high coverage because data movement and addressing output are affected by the processing controller's interactions.

[0251] When various code patterns are used during testing, the code patterns may be used in an alternating pattern, or one code may be used in a first time period (e.g., a time corresponding to 30 fps), another code may be used in a second time period (e.g., a time corresponding to 30 fps), another code may be used in a third time period (e.g., a time corresponding to 30 fps), etc. For example, in the DMA code example, a code of 0 may be used in the first time period, then a code of 1 may be used in the second time period, then a code of alternating 0s and 1s (e.g., 0101010101...) may be used in the third time period, then an irregular code (e.g., 011100100010...) may be used in the fourth time period, then these four codes may be repeated, and so on.

[0252] In practice, for example, when testing DMA, the processing controller can interact with the MISR control unit 1206 to write set reference values ​​to the MISR data set register 1210 and the MISR address set register 1216. These values ​​can be different for data and addresses and can be referred to as seed values ​​for the CRC calculation. The processing controller can then initialize the channels performing the data movement within the DMA engine 1056, and since the processing controller knows the location of the test code in memory, it can use a descriptor (e.g., set by the processing controller in descriptor SRAM 1052) to sequence the DMA engine 1056 with the data for the MISR test. The processing controller can set a timer 1226 for the MISR hardware 1200 to enable the MISR test and then trigger the channels of the DMA engine 1056 to begin reading test data from the source destination and output the data to the MISR hardware 1200 for the MISR test. Thus, when testing DMA, MISR hardware 1200 can utilize the output of DMA engine 1056 when performing the data movement of the test code, so that data movement is tested (e.g., proper addressing and accurate data at the address locations). The utilization of this output stage may be shown in FIG. 12A as external memory, which, as sequenced by the processing controller, may be fed into one data channel at a time and one address channel at a time. For example, in the data channel, the processing controller may sequence through, e.g., each of 16 data channels, and the corresponding AXI write data (wdata) for each channel may be fed, e.g., serially, through data CRC calculator 1202 for CH0 through CH16. For example, the processing controller may configure channel output register 1220 to sequence through the channels one at a time according to the sequence set by the processing controller.In an embodiment, channel mask register 1208 (programmed by MISR control unit 1206, e.g., based on interaction with the processing controller) may be set by the processing controller to mask out or exclude various channels, e.g., channels not being tested, from the CRC calculation. This masking may be performed using an AND gate in an embodiment. If one or more channels are masked out, the golden value in MISR data reference register 1222 (which may be provided to MISR control unit 1206 by the processing controller) may only correspond to the CRC calculation of the unmasked channels. For each unmasked channel, the channel's data generated using the test code read from memory (e.g., with or without compression) may be applied to the polynomial in CRC data calculation unit 1202 to generate the MISR data value 1214 for that channel. Once a channel has been calculated, the processing controller can receive instructions to send the next channel of data to the CRC calculation unit 1202 to calculate the next MISR data value 1214, and so on until each unmasked channel has a corresponding MISR data value 1214. Once each of the MISR data values ​​1214 for a particular iteration has been calculated, these values ​​1214 can be combined to produce a final MISR data value that can be compared to a golden value in the MISR data reference register 1222 to produce a determined MISR data status (which can include, for example, a status corresponding to the values ​​0-7 listed above).

[0253] As another example, for address channels, the processing controller can sequence through, for example, each of 16 address or control channels, and the corresponding AXI write address (waddress) for each channel can be fed, e.g., sequentially, through address CRC calculation unit 1204 for CH0 through CH16. In an embodiment, channel mask register 1208 can be set by the processing controller to mask out or exclude various channels, e.g., channels not being tested, from the CRC calculation. This masking can be performed, in an embodiment, using AND gates. If one or more channels are masked out, the golden value in MISR data reference register 1224 can only correspond to the CRC calculation for the unmasked channels. For each unmasked channel, the channel's address, generated using test code read from memory (e.g., with or without compression), can be applied to the polynomial in CRC address calculation unit 1204 to generate the MISR address value 1218 for that channel. Once a channel has been calculated, the processing controller can receive instructions to send the next channel of address data to the CRC calculation unit 1204 to calculate the next MISR address value 1218, and so on until each unmasked channel has a corresponding MISR address value 1218. Once each of the MISR address values ​​1218 for a particular iteration has been calculated, these values ​​1218 can be combined to generate a final MISR address value that can be compared to a golden value in the MISR address reference register 1224 to generate a determined MISR address status (which may include, for example, a status corresponding to the values ​​0-7 listed above).

[0254] In some embodiments, MISR testing may be iterative, such that initial code may be processed, the outputs may be tested, and then the outputs may be used in the next iteration to be tested, etc. In such embodiments, MISR testing may include multiple stages, and completing the MISR test may include executing each stage.

[0255] If MISR hardware 1200 is used specifically to test a VPU, for example, DMA system 1050 can move test code to VMEM, the VPU can process the test code and write the results back to VMEM, and DMA engine 1056 can read the results from VMEM and return them to a destination location. When writing the results back to the destination location, MISR hardware 1200 can utilize the DMA output to perform a MISR on the data (e.g., including data and addresses) and perform a MISR similar to those discussed herein. In this manner, the interaction of the VPU and test code can be tested using MISR hardware 1200.

[0256] After the MISR test is completed, the processing controller may receive an interrupt. For example, the processing controller may receive a completion interrupt and, if there is no error, may wait for the next MISR test cycle. If the interrupt is an error interrupt, the type of error (e.g., data fail, address fail, both fail, etc.) may be determined and a safety error may be asserted to the safety processor. In some embodiments, if the MISR hardware 1200 hangs or goes into an idle state (e.g., a time-out error), for example, the DMA may generate a time-out signal to the safety processor of the SoC.

[0257] In some embodiments, to speed up MISR calculations and calculate the CRC on one or more (e.g., all 16, in an embodiment) channels without serializing or staging the channel-MISR calculations, the channels can be demultiplexed based on the channel ID present in the AXI ID field, parallelizing the channel calculations. For example, the approach of FIG. 12A involved sequential processing of the channels one after the other because the CRC calculations completed at different rates. However, using the system of FIG. 12B described below, such calculations can be completed in parallel. For example, once the processing controller finishes the MISR calculations, the MISR controller can sequence through all channel outputs to calculate a final signature that can be compared to a reference or gold standard for both the address and data outputs. This feature can speed up the detection of permanent faults, for example, because the same control registers can be used for all channels, without the need for an additional programmer register interface.

[0258] Accordingly, reference is made to FIG. 12B, which is a diagram of a built-in self-test (BIST) system for parallel channel cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA), in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, MISR hardware 1250 may include components, features, and / or functionality similar to those of DMA system 1050 of FIG. 10H , example autonomous vehicle 1300 of FIGS. 13A-13D , example computing device 1400 of FIG. 14 , and / or example data center 1500 of FIG. 15 . For example, MISR hardware 1250 may be included in a DMA block of a PVA, such as that shown in FIG. 10H . Thus, MISR hardware 1250 may operate at the output stage of data movement, and addressing (or control) may utilize the output of DMA engine 1056. In the example of FIG. 12A , there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used, depending on the embodiment.

[0259] MISR hardware 1250 can operate similarly to MISR hardware 1200 of FIG. 12A, except that MISR hardware 1250 can be configured for CRC calculations on parallel data channels and parallel address channels. For example, a processing controller can configure MISR data set register 1256 to set a seed or reference value for each data CRC calculation portion 1260A-1260N (corresponding to AXI data channels 0-15, respectively) and can configure MISR address set register 1258 to set a seed or reference value for each address CRC calculation portion 1262A-1262N (corresponding to AXI address channels 0-15, respectively). A processing controller similar to that described with respect to FIG. 12A can then trigger data movement (e.g., for DMA testing) and / or VPU processing (e.g., for VPU-specific testing) for DMA system 1050 to move data around, and MISR hardware 1250 can utilize the output stage to be tested.

[0260] Thus, the processing controller can cause 16 channels of data to be sent to multiplexer (mux) 1252 and 16 channels of address data to be sent to multiplexer (mux) 1254. Mux 1252 can then provide the corresponding channel of data to the corresponding CRC calculator 1260A-1260N (e.g., channel 0 AXI data to channel 0 CRC calculator 1260, channel 1 data to channel 1 CRC calculator 1260B, etc.), and each CRC calculator 1260 can calculate a MISR data value 1284A-1284N using a CRC polynomial with the data and the reference value (e.g., channel 0 CRC calculator 1260A can calculate MISR data 0 value 1284A, channel 1 CRC calculator 1260B can calculate MISR data 1 value 1284B, etc.). The MISR data values ​​1284A-1284N can then be sequenced out of the multiplexer (mux) 1264 according to the MISR sequence from the MISR control unit 1270 set by the processing controller. In an embodiment such as that described with reference to FIG. 12A , because one or more channels may not be included in a particular MISR test, the channel mask register 1268 can be set by the processing controller to update the MISR sequence, so that the MISR data values ​​1284 corresponding to one or more masked channels are not provided to the data CRC calculation unit 1274 for channels 0-16 for calculating the final CRC value. For unmasked channels, the MISR data values ​​1284 can be output from the mux 1264 according to the MISR sequence. In this way, the MISR data values ​​1284 are forced out according to the MISR sequence rather than being sent to the CRC calculation unit 1274 depending on when the CRC calculation is completed, thereby taking into account the different calculation times of the different channels and the CRC calculation unit 1260. When the MISR sequence of MISR data values ​​1284 is output from mux 1264 to CRC calculator 1274 , CRC calculator 1274 can calculate a final CRC value and store the final CRC value in VAL register 1276 .The final CRC value in the VAL register 1276 can then be compared with the golden value in the MISR data reference register 1272 (set by the MISR control unit 1270 from the processing controller) to determine the status of the MISR data.

[0261] Similarly, the processing controller can cause 16 channels of addresses to be sent to multiplexer (mux) 1254, which can then provide the corresponding channel of addresses to the corresponding CRC calculation units 1262A-1262N (e.g., the AXI address for channel 0 to channel 0 CRC calculation unit 1262, the address for channel 1 to channel 1 CRC calculation unit 1262B, etc.), and each CRC calculation unit 1262 can calculate a MISR address value 1286A-1286N using a CRC polynomial with the address and reference value (e.g., channel 0 CRC calculation unit 1262A can calculate MISR address 0 value 1286A, channel 1 CRC calculation unit 1262B can calculate MISR address 1 value 1286B, etc.). The MISR address values ​​1286A-1286N can then be sequenced out of the multiplexer (mux) 1266 according to the MISR sequence from the MISR control unit 1270 set by the processing controller. In the embodiment described with reference to FIG. 12A , because one or more channels may not be included in a particular MISR test, the channel mask register 1268 can be set by the processing controller to update the MISR sequence, so that the MISR address values ​​1286 corresponding to one or more masked channels are not provided to the channel 0-16 address CRC calculation unit 1280 for calculating the final CRC value. For unmasked channels, the MISR address values ​​1286 can be output from the multiplexer 1266 according to the MISR sequence. In this way, the MISR address values ​​1286 are forced out according to the MISR sequence rather than being sent to the CRC calculation unit 1280 depending on when the CRC calculation is completed, thereby taking into account the different calculation times of the different channels and CRC calculation unit 1262. When the MISR sequence of the MISR address value 1286 is output from the multiplexer 1266 to the CRC calculator 1280 , the CRC calculator 1280 can calculate the final CRC value and store the final CRC value in the VAL register 1282 .The final CRC value in the VAL register 1282 can then be compared with the golden value in the MISR address reference register 1278 (set by the MISR control unit 1270 from the processing controller) to determine the status of the MISR address.

[0262] The status of the MISR data and the status of the MISR address may be checked and used in the same manner as described above with respect to Figure 12A.

[0263] Referring now to FIG. 12C , each block of method 1290 described herein includes a computerized process that may be implemented using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 1290 may also be embodied as computer-usable instructions stored on a computer storage medium. Method 1290 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, while method 1290 is described with respect to the system of FIG. 12A , method 1290 may be performed by any one system, structure, or component, or any combination of systems, structures, or components, including, but not limited to, those described herein.

[0264] 12C is a flow diagram of a build-in self-test (BIST) method 1290 for detecting permanent faults in a PVA according to some embodiments of the present disclosure. Method 1290 includes, at block B1202, receiving multiple channels of data from a DMA system, one channel at a time, based on a sequence determined by a processing controller. For example, MISR hardware 1200 may receive one channel of data (or one channel of address data) at a time according to a sequence determined by the processing controller.

[0265] The method 1290 includes calculating, in block B 1204, a plurality of MISR values ​​by performing a CRC calculation for each channel using a CRC calculation polynomial and the respective data corresponding to the channel to calculate an MISR value. For example, for each channel, the CRC calculation unit 1202 (or 1204, if an address) can calculate an MISR data value 1214 (or MISR address value 1216, if an address) using the data (or address) from the channel and the CRC calculation unit 1202 polynomial (starting with a seed value from the CRC MISR data set register 1210 or the MISR address set register 1216).

[0266] The method 1290 includes calculating a final MISR value using the multiple MISR values ​​in block B 1206. For example, the MISR data values ​​1214 from each channel (or the MISR address values ​​from each channel) may be combined to generate the final MISR value.

[0267] The method 1290 includes comparing the final MISR value to a signature value in block B 1208. For example, the final MISR value generated from the individual MISR values ​​1214 (or address values ​​1216) may be compared to a signature or golden value in the MISR data reference register 1222 (or in the case of an address, the MISR address reference register 1224).

[0268] The method 1290 includes outputting a MISR status based at least in part on the comparison in block B1210. For example, based on the comparison in block B1208, a status, e.g., data fail, address fail, both fail, complete, etc., may be determined and used to notify a safety processor of the SoC of where the error status was generated.

[0269] Exemplary Autonomous Vehicle 13A is a diagram of an example autonomous vehicle 1300 according to some embodiments of the present disclosure. The autonomous vehicle 1300 (alternatively referred to herein as "vehicle 1300") may include passenger vehicles such as, but not limited to, cars, trucks, buses, emergency response vehicles, shuttles, electric or e-bikes, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, underwater vehicles, drones, vehicles coupled to trailers, and / or other types of vehicles (e.g., unmanned and / or accommodating one or more passengers). Autonomous vehicles are generally described in terms of levels of automation in the "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" standard established by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 1300 may be capable of functioning according to one or more of Levels 3 through 5 of autonomous driving. For example, vehicle 1300 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0270] Vehicle 1300 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid generator, an all-electric engine, and / or another propulsion system type. Propulsion system 1350 may be coupled to a drivetrain of vehicle 1300, which may include a transmission, to enable propulsion of vehicle 1300. Propulsion system 1350 may be controlled in response to receiving a signal from a throttle / accelerator 1352.

[0271] A steering system 1354, which may include a steering wheel, may be used to move the vehicle 1300 (e.g., along a desired path or route) when the propulsion system 1350 is operating (e.g., when the vehicle is moving). The steering system 1354 may receive signals from a steering actuator 1356. For fully automated (Level 5) functionality, a steering wheel may be optional.

[0272] A brake sensor system 1346 may be used to activate the vehicle brakes in response to receiving signals from a brake actuator 1348 and / or brake sensor.

[0273] A controller 1336, which may include one or more systems-on-chip (SoCs) 1304 ( FIG. 13C ) and / or a GPU, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1300. For example, the controller can send signals to actuate vehicle brakes using one or more brake actuators 1348, to actuate a steering system 1354 using one or more steering actuators 1356, and to actuate a propulsion system 1350 using one or more throttle / accelerators 1352. The controller 1336 can include one or more on-board (e.g., integrated) computer processing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in operating the vehicle 1300. The controllers 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for practical safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some instances, a single controller 1336 may handle two or more of the above functions, two or more controllers 1336 may handle only one function, and / or any combination of these functions.

[0274] Controller 1336 may provide signals to control one or more components and / or systems of vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, global navigation satellite system sensors 1358 (e.g., global positioning system sensors), radar sensors 1360, ultrasonic sensors 1362, lidar sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide field of view cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360-degree cameras), long-range and / or mid-range cameras 1398, speed sensors 1344 (e.g., for measuring the speed of the vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0275] The one or more controllers 1336 may receive input (e.g., represented by input data) from the instrument cluster 1332 of the vehicle 1300 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1334, an audible annunciator, a loudspeaker, and / or via other components of the vehicle 1300. The output may include information such as vehicle speed, speed, time, map data (e.g., the HD map 1322 of FIG. 13C ), location data (e.g., the location of the vehicle 1300 on a map, etc.), direction, the locations of other vehicles (e.g., occupancy grids), information about objects and object statuses perceived by the controllers 1336, etc. For example, the HMI display 1334 may display information regarding the presence of one or more objects (e.g., road signs, caution signs, traffic lights changing, etc.) and / or information regarding a driving maneuver that the vehicle has performed, is performing, or is expected to perform (e.g., changing lanes here, taking exit 34B in 2 miles, etc.).

[0276] Vehicle 1300 further includes a network interface 1324 that can use one or more wireless antennas 1326 and / or a modem to communicate over one or more networks. For example, network interface 1324 can be capable of communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. Wireless antenna 1326 can also enable communication between objects in the environment (e.g., vehicles, portable devices, etc.) using local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or low power wide-area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0277] 13B is an illustration of camera positions and fields of view for the example autonomous vehicle 1300 of FIG. 13A, according to some embodiments of the present disclosure. The cameras and their respective fields of view shown are one illustrative example and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or cameras may be located at different locations on the vehicle 1300.

[0278] The camera type may include, but is not limited to, a digital camera that may be adapted for use with components and / or systems of vehicle 1300. The camera may be capable of operating at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some instances, the color filter array may include a red, clear, clear, clear (RCCC) color filter array, a red, clear, clear, blue (RCCB) color filter array, a red, blue, green, clear (RBGC) color filter array, a FoveonX3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a transparent pixel camera, such as a camera with an RCCC, RCCB, and / or RBGC color filter array, may be used to enhance light sensitivity.

[0279] In some instances, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono camera may be installed with functions including lane departure warning, traffic sign assistance, and intelligent headlamp control. One or more of the cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0280] To block stray light and reflections from the interior of the vehicle (e.g., reflections from the instrument panel reflected by the windshield mirror) that may interfere with the camera's image data capture function, one or more of the cameras may be attached to a mounting assembly, such as a custom-designed (3D printed) assembly. For door mirror mounting assemblies, the door mirror assembly may be custom 3D printed so that the camera mounting plate matches the shape of the door mirror. In some instances, the camera may be integrated into the door mirror. For side-view cameras, the camera may also be integrated into four support columns at each corner of the cabin.

[0281] A camera with a field of view that includes a portion of the environment in front of the vehicle 1300 (e.g., a front-facing camera) can be used to obtain a surround view that helps identify the path and obstacles ahead and, with the assistance of one or more controllers 1336 and / or control SoCs, generate an occupancy grid and / or provide important information in determining a preferred vehicle path. Front-facing cameras can be used to perform many of the same ADAS functions as lidar, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including other functions such as lane departure warnings (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0282] Various cameras, including monocular camera platforms, including, for example, CMOS (complementary metal oxide semiconductor) color imagers, can be used in a front-facing configuration. Another example can be a wide-field camera 1370, which can be used to perceive objects (e.g., pedestrians, crosswalks, or bicycles) coming into view from the periphery. While only one wide-field camera is shown in FIG. 13B, any number of wide-field cameras 1370 can be present in the vehicle 1300. Additionally, a long-range camera 1398 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, particularly for objects for which a neural network has not yet been trained. The long-range camera 1398 can also be used for object detection and classification, as well as basic object tracking.

[0283] One or more stereo cameras 1368 may also be included in the front configuration. The stereo camera 1368 may include an integrated control unit with a scalable processing unit, which may include a multi-core microprocessor with a programmable gate array (FPGA) and a CAN or Ethernet interface integrated on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 1368 may include a compact stereo vision sensor, which may include two camera lenses (one on each side), and an image processing chip that can measure the distance from the vehicle to target objects and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning features. Other types of stereo cameras 1368 may be used in addition to or instead of those described herein.

[0284] Cameras with fields of view that include portions of the environment on the sides of the vehicle 1300 (e.g., side-view cameras) may be used for surround view, providing information used to create and update the occupancy grid as well as generate side collision warnings. For example, surround cameras 1374 (e.g., four surround cameras 1374 shown in FIG. 13B ) may be disposed on the vehicle 1300. The surround cameras 1374 may include wide-field cameras 1370, fisheye cameras, 360-degree cameras, and / or the like. In a four-in-one example, four fisheye cameras may be disposed at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle may use three surround cameras 1374 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., a front camera) as a fourth surround view camera.

[0285] A camera having a field of view that includes a portion of the environment behind the vehicle 1300 (e.g., a rear-view camera) may be used for parking assistance, surround vision, collision warning, and creating and updating occupancy grids. A wide variety of cameras may be used, including, but not limited to, cameras described herein that are also suitable as forward-facing cameras (e.g., long-range and / or mid-range camera 1398, stereo camera 1368, infrared camera 1372, etc.).

[0286] FIG. 13C is a block diagram of an example system architecture for the example autonomous vehicle 1300 of FIG. 13A , in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., mechanisms, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in combination with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0287] Each of the components, features, and systems of the vehicle 1300 in FIG. 13C is shown connected via a bus 1302. The bus 1302 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a “CAN bus”). The CAN may be a network internal to the vehicle 1300 used to help control various features and functions of the vehicle 1300, such as brake application, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to determine steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0288] Although the bus 1302 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or instead of a CAN bus. Furthermore, although only a single line is used to represent the bus 1302, this is not intended to be limiting. For example, there may be any number of buses 1302, including one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using other protocols. In some instances, two or more buses 1302 may be used to perform different functions and / or for redundancy. For example, a first bus 1302 may be used for collision avoidance functions, and a second bus 1302 may be used for operational control. In any instance, each bus 1302 may communicate with any of the components of the vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304, each controller 1336, and / or each computer in the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1300) and may be connected to a common bus, such as a CAN bus.

[0289] Vehicle 1300 may include one or more controllers 1336, such as those described herein with respect to FIG. 13A. Controller 1336 may be used for a variety of functions. Controller 1336 may be coupled to any of various other components and systems of vehicle 1300 and may be used for control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment for vehicle 1300, and / or the like.

[0290] Vehicle 1300 may include a system-on-chip (SoC) 1304. SoC 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. In various platforms and systems, SoC 1304 may be used to control vehicle 1300. For example, SoC 1304 may be combined in a system (e.g., that of vehicle 1300) with an HD map 1322 that can obtain map refreshes and / or updates via network interface 1324 from one or more servers (e.g., server 1378 of FIG. 13D ).

[0291] CPU 1306 may include a CPU cluster or CPU complex (also referred to herein as a "CCPLEX"). CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1306 may include four dual-core clusters, each with its own dedicated L2 cache (e.g., 2 MB of L2 cache). CPU 1306 (e.g., a CCPLEX) may be configured to support simultaneous cluster operation, allowing any combination of CPU 1306 clusters to be active at any given time.

[0292] The CPU 1306 may implement power management functions, including one or more of the following features: During idle, individual hardware blocks may be automatically clock gated for dynamic power savings; During idle, the clock of each core may be gated when the core is not actively executing instructions by executing a WFI / WFE instruction; Each core may be power gated separately; Each core cluster may be clock gated separately when all cores are clock gated or power gated; and / or each core cluster may be power gated separately when all cores are power gated. The CPU 1306 may further implement an extensive algorithm to manage power states, where allowed power states and expected wake-up times are specified and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. Processing cores may support a simplified power state entry sequence in software, where work is offloaded to microcode.

[0293] GPU 1308 may include an integrated GPU (also referred to herein as an “iGPU”). GPU 1308 may be programmable and efficient for parallel workloads. In some instances, GPU 1308 may use an extended tensor instruction set. GPU 1308 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache having at least 96 KB of storage), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache having 512 KB of storage). In some embodiments, GPU 1308 may include at least eight streaming microprocessors. GPU 1308 may use a computational application programming interface (API). Additionally, GPU 1308 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0294] The GPU 1308 may be power-optimized to deliver the highest performance for automotive and embedded applications. For example, the GPU 1308 may be fabricated based on Fin field-effect transistors (FinFETs). However, this is not intended to be limiting, and the GPU 1308 may be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor may incorporate several mixed-precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA TENSOR COREs for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. Additionally, streaming microprocessors may include separate parallel integer and floating-point data paths that combine computation with address calculations to enable efficient execution of workloads. Streaming microprocessors may include separate thread scheduling capabilities that allow for finer-grained synchronization and coordination between parallel threads. Streaming microprocessors may include a combination of an L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0295] The GPU 1308 may include a high bandwidth memory (HBM) and / or 16GB HBM2 memory subsystem, which in some instances enables a peak memory bandwidth of approximately 900 GB / s. In some instances, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used in addition to or instead of the HBM memory.

[0296] The GPU 1308 may include unified memory technology, including access counters, that enable more accurate transfer of memory pages to the processor that most frequently accesses them, thereby increasing the efficiency of memory ranges shared between processors. In some instances, address translation service (ATS) support may be used to enable the GPU 1308 to directly access the CPU 1306's page tables. In such instances, a miss in the GPU 1308's memory management unit (MMU) may transmit an address translation request to the CPU 1306. In response, the CPU 1306 can consult its page table, which maps virtual addresses to physical addresses, and send this translation back to the GPU 1308. The unified memory technology thus allows for a single unified virtual address space for both the CPU 1306 and the GPU 1308's memories, which may simplify programming the GPU 1308 and porting applications to the GPU 1308.

[0297] Additionally, the GPU 1308 may include access counters that can keep track of the frequency of accesses by the GPU 1308 to the memory of other processors. The access counters can help ensure that memory pages are moved to the physical memory of the processors that are accessing the pages most frequently.

[0298] The SoC 1304 may include any number of caches 1312, including those described herein. For example, the cache 1312 may include an L3 cache that is available to (e.g., connected to) both the CPU 1306 and the GPU 1308. The cache 1312 may include a write-back cache that can keep track of line state, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the implementation, although smaller cache sizes may also be used.

[0299] The SoC 1304 may include an arithmetic logic unit (ALU) that may be utilized in performing processing for any of various tasks or operations of the vehicle 1300, such as processing DNNs. Additionally, the SoC 1304 may include a floating point unit (FPU) or other math or numeric coprocessor type for performing mathematical operations within the system. For example, the SoC 1304 may include one or more FPUs integrated as execution units within the CPU 1306 and / or GPU 1308.

[0300] The SoC 1304 may include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 1304 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU 1308 and offload some of the GPU 1308's tasks (e.g., freeing up more cycles for the GPU 1308 to perform other tasks). As an example, the accelerator 1314 may be used for target workloads that are sufficiently stable and suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Faster RCNNs (e.g., used for object detection).

[0301] The accelerator 1314 (e.g., a hardware-accelerated cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs), which may be configured to enable up to 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured and optimized to perform image processing functions (e.g., for CNN, RCNN, etc.). The DLA may be further optimized for a specific neural network type and set of floating-point operations, as well as for inference. The design of the DLA may enable higher performance per millimeter than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including single-instance convolution functions as well as post-processor functions, supporting, for example, data types INT8, INT16, and FP16 for both features and weights.

[0302] The DLA may quickly and efficiently run neural networks, particularly CNNs, on the processed or raw data for any of a variety of functions, including, but not limited to, for example, CNNs for object identification and detection using data from camera sensors, CNNs for distance estimation using data from camera sensors, CNNs for emergency vehicle detection, identification, and location using data from microphones, CNNs for face recognition and vehicle owner identification using data from camera sensors, and / or CNNs for security and / or safety related events.

[0303] The DLA can perform any function of the GPU 1308, and for any function, the designer can target either the DLA or the GPU 1308, for example, by using an inference accelerator. For example, the designer can centralize CNN processing and floating-point operations in the DLA and offload other functions to the GPU 1308 and / or other accelerators 1314.

[0304] The accelerator 1314 (e.g., a hardware acceleration cluster) may include programmable vision accelerators (PVAs), which may alternatively be referred to herein as computer vision accelerators. The PVAs may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVAs may balance performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0305] The RISC cores may interact with an image sensor (e.g., an image sensor in any of the cameras described herein), an image signal processor, and / or the like. Each of the RISC cores may include any amount of memory. The RISC cores may use any of several protocols, depending on the embodiment. In some instances, the RISC cores may execute a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0306] The DMA may allow components of the PVA to access system memory independently of the CPU 1306. The DMA may support any number of features used to optimize the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some instances, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0307] A vector processor may be a programmable processor that can be designed to efficiently and flexibly program computer vision algorithms and implement signal processing functions. In some instances, a PVA may include a PVA core and two partitions of vector processing subsystems. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripheral devices. The vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single instruction, multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW may increase throughput and speed.

[0308] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some instances, each of the vector processors may be configured to execute independently of the other vector processors. In other instances, the vector processors included in a particular PVA may be configured to exploit data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of the image. In other instances, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on successive images or portions of an image. Notably, any number of PVAs may be included in a hardware-accelerated cluster, and any number of vector processors may be included in each PVA. Additionally, the PVAs may include additional error correcting code (ECC) memory to increase the overall security of the system.

[0309] The accelerator 1314 (e.g., a hardware-accelerated cluster) may include an on-chip computer vision network and SRAM, providing high-bandwidth, low-latency SRAM for the accelerator 1314. In some instances, the on-chip memory may include, for example, but not limited to, at least 4 MB of SRAM consisting of eight field-configurable memory blocks that may be accessible by both the PVA and DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory through a backbone that provides high-speed access to the memory for the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using the APB).

[0310] The on-chip computer vision network may include an interface that determines whether both the PVA and DLA provide a ready and valid signal before transmitting any control signals, addresses, or data. Such an interface may provide separate phases and separate channels for transmitting control signals, addresses, and data, as well as burst-type communication for continuous data transfer. This type of interface may conform to the ISO 26262 or IEC 61508 standards, although other standards and protocols may also be used.

[0311] In some examples, SoC 1304 may include a real-time ray tracing hardware accelerator, such as that described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) and generate real-time visualization simulations for radar signal interpretation, sound propagation synthesis and / or analysis, sonar system simulation, general wave propagation simulation, comparison with lidar data for localization and / or other functions, and / or other uses. In some examples, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0312] The accelerator 1314 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The PVA's capabilities are well suited to algorithm domains that require low-power, low-latency, and predictable processing. In other words, the PVA performs well on semi-dense or dense regular calculations, even on small data sets that require predictable execution times with low latency and low power. Therefore, for autonomous vehicle platforms, the PVA is designed to run typical computer vision algorithms because it is efficient at object detection and integer math operations.

[0313] For example, according to one embodiment of the technology, PVA is used to perform computational stereo vision. A semi-global matching-based algorithm may be used in some instances, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require motion estimation / stereo matching during operation (e.g., structure from motion, pedestrian recognition, lane detection, etc.). PVA can perform computational stereo vision functions with input from two monocular cameras.

[0314] In some instances, PVA can be used to perform dense optical flow (e.g., using a 4D Fast Fourier Transform) following processing of raw radar data to provide processed radar. In other instances, PVA is used to process time-of-flight depth, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0315] DLA can be used to implement any type of network for enhancing control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such a confidence value can be interpreted as indicating the probability or relative "weight" of each detection compared to other detections. This confidence value can further enable the system to make decisions regarding which detections should be considered true-positive rather than false-positive detections. For example, the system can set a confidence threshold and consider only detections above the threshold to be true-positive detections. In an automatic emergency braking (AEB) system, a false-positive detection could cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most reliable detections should be considered to trigger AEB. DLA can implement a neural network to regress the confidence value. The neural network may receive as inputs to the neural network at least a partial subset of the parameters, such as bounding box dimensions, ground plane estimates obtained (e.g., from another subsystem), and the output of an inertial measurement unit (IMU) sensor 1366 that correlates with estimates of vehicle 1300 orientation, distance, and object 3D position obtained from the neural network and / or other sensors (e.g., lidar sensor 1364 or radar sensor 1360), among others.

[0316] The SoC 1304 may include a data store 1316 (e.g., memory). The data store 1316 may be on-chip memory of the SoC 1304 that can store neural networks to be executed by the GPU and / or DLA. In some instances, the data store 1316 may be large enough to store multiple instances of a neural network for redundancy and safety. The data store 1312 may comprise an L2 or L3 cache 1312. References to the data store 1316 may include references to memory associated with the GPU, DLA, and / or other accelerators 1314, as described herein.

[0317] The SoC 1304 may include one or more processors 1310 (e.g., embedded processors). The processors 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem that handles boot power and management functions and associated security enforcement. The boot and power management processor may be part of the boot sequence of the SoC 1304 and may provide runtime power management services. The boot power and management processor may enable clock and voltage programming, assist in system low-power state transitions, thermal and temperature sensor management of the SoC 1304, and / or power state management of the SoC 1304. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 1304 may use the ring oscillator to detect the temperature of the CPU 1306, GPU 1308, and / or accelerator 1314. If it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine, place the SoC 1304 in a lower power state, and / or cause the vehicle 1300 to enter a safe shutdown mode (e.g., bring the vehicle 1300 to a safe shutdown).

[0318] The processor 1310 may further include a set of embedded processors that can function as an audio processing engine. The audio processing engine may be an audio subsystem that enables full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some instances, the audio processing engine is a dedicated processor core that includes a digital signal processor with dedicated RAM.

[0319] The processor 1310 may further include an always-on processor engine capable of implementing the hardware features necessary to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripheral devices (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0320] The processor 1310 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management for automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, two or more cores can operate in lockstep mode and function like a single core, with comparison logic detecting any differences between the cores' operations.

[0321] The processor 1310 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0322] The processor 1310 may further include a high dynamic range signal processor, which may include an image signal processor that is a hardware engine that is part of the camera processing pipeline.

[0323] The processor 1310 may include a video image compositor, which may be a processing block (e.g., implemented in a microprocessor) that performs video post-processing functions required by a video playback application to generate the final image in the player window. The video image compositor may perform lens distortion correction for the wide-field camera 1370, the surround camera 1374, and / or the in-cabin surveillance camera sensor. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on a separate instance of the advanced SoC that is configured to identify and respond to events within the cabin. The in-cabin system may perform lip reading, activate cellular services and make phone calls, dictate emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain features are only available to the driver when the vehicle is operating in autonomous mode and are disabled otherwise.

[0324] The video image combiner may include enhanced temporal noise reduction to reduce both spatial and temporal noise. For example, when motion is present in the video, noise reduction appropriately weights spatial information and reduces the weight of information provided by adjacent frames. When an image or portion of an image does not contain motion, the temporal noise reduction performed by the video image combiner can use information from previous images to reduce noise in the current image.

[0325] The video image compositor can also be configured to perform stereo rectification on the input stereo lens frames. The video image compositor can also be used for user interface compositing when a desktop operating system is being used, eliminating the need for the GPU 1308 to continually render new surfaces. The video image compositor can be used to offload the GPU 1308, improving performance and responsiveness, even when the GPU 1308 is powered on and actively performing 3D rendering.

[0326] The SoC1304 may further include a mobile industry processor interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block and related pixel input functions that may be used for a camera to receive video and input from a camera. The SoC1304 may further include an input / output controller that may be controlled by software and used to receive I / O signals that are not tied to a specific role.

[0327] The SoC 1304 may further include a wide range of peripheral interfaces, enabling communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC 1304 may be used to process data from cameras (e.g., connected via a gigabit multimedia serial link and Ethernet), sensors (e.g., lidar sensor 1364, radar sensor 1360, etc., which may be connected via Ethernet), data from bus 1302 (e.g., vehicle 1300 speed, steering wheel location, etc.), and data from GNSS sensor 1358 (e.g., connected via Ethernet or CAN bus). The SoC 1304 may include its own DMA engine and may further include a dedicated high-performance mass storage controller that may be used to offload routine data management tasks from the CPU 1306.

[0328] The SoC1304 is an end-to-end platform with a flexible architecture spanning levels 3-5 of automation, providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and can provide a flexible and reliable driving software stack platform along with deep learning tools. The SoC1304 is potentially faster, more reliable, and more energy- and space-efficient than traditional systems. For example, the Accelerator 1314, when combined with the CPU 1306, GPU 1308, and Data Store 1316, can provide a fast and efficient platform for levels 3-5 of autonomous vehicles.

[0329] This technology therefore offers capabilities and functionality not achievable with conventional systems. For example, computer vision algorithms can be executed on a CPU, configured using high-level programming languages ​​such as the C programming language, to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute complex object detection algorithms in real time, a requirement for in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0330] In contrast to conventional systems, the technology described herein includes a CPU complex, a GPU complex, and a hardware acceleration cluster, allowing multiple neural networks to run simultaneously and / or sequentially and combine the results to enable Level 3-5 autonomous driving capabilities. For example, a CNN running on the DLA or dGPU (e.g., GPU1320) can include text and word recognition, enabling the supercomputer to read and understand traffic signs, including signs for which the neural network was not specifically trained. The DLA can further include a neural network that can identify, interpret, and provide a semantic understanding of the signs and pass that semantic understanding to a path planning module running on the CPU complex.

[0331] As another example, when Level 3, 4, or 5 driving is required, multiple neural networks can be run simultaneously. For example, a warning sign with an electric light stating "Caution: Flashing lights indicate icy conditions" can be interpreted independently or collectively by several neural networks. The sign itself can be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a second deployed neural network, which notifies the vehicle's route planning software (preferably running on the CPU complex) that icy conditions exist if a flashing light is detected. The flashing light can be identified by running a third deployed neural network over multiple frames, and the third deployed neural network notifies the vehicle's route planning software of the presence (or absence) of a flashing light. All three neural networks can run simultaneously, such as within the DLA and / or on the GPU 1308.

[0332] In some instances, a CNN for facial recognition and vehicle owner identification can use data from a camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1300. An always-on sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and, when in security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1304 provides security against theft and / or carjacking.

[0333] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from the microphone 1396. In contrast to conventional systems that use general-purpose classifiers to detect sirens and manually extract features, the SoC 1304 uses a CNN to classify not only ambient sounds and urban sounds, but also visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative approaching speed of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by the GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, while in the United States, the CNN would attempt to identify only North American sirens. Once an emergency vehicle is detected, the control program can be used to execute emergency vehicle safety routines, slow down the vehicle, pull over to the side of the road, park the vehicle, and / or idle the vehicle, with the aid of the ultrasonic sensor 1362, until the emergency vehicle has passed.

[0334] The vehicle may include a CPU 1318 (e.g., a discrete CPU or dCPU), which may be coupled to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 may include, for example, an X86 processor. The CPU 1318 may be used to perform any of a variety of functions, including reconciling potentially conflicting results between the ADAS sensors and the SoC 1304 and / or monitoring the status and health of, for example, the controller 1336 and / or the infotainment SoC 1330.

[0335] Vehicle 1300 may include GPU 1320 (e.g., a discrete GPU or dGPU), which may be coupled to SoC 1304 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1320 may provide additional artificial intelligence functionality, such as by running redundant and / or separate neural networks, and may be used to train and / or update neural networks based on input (e.g., sensor data) from sensors of vehicle 1300.

[0336] Vehicle 1300 may further include a network interface 1324, which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for various communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). Network interface 1324 may be used to enable wireless connections to the cloud via the Internet (e.g., with server 1378 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between the two vehicles and / or...

Claims

1. a direct memory access (DMA) system; a processing controller; 1. A multiple input signature register (MISR) hardware component including a processing circuit, the processing circuit comprising: receiving data of a plurality of channels from the DMA system based on the sequence processing of the processing controller; calculating a plurality of MISR values ​​by performing a cyclic redundancy check (CRC) calculation to calculate a MISR value for each channel of the plurality of channels; calculating a final MISR value using the plurality of MISR values; comparing the final MISR value with the value of a signature; and outputting a MISR status based at least in part on said comparison; MISR hardware components; A system comprising:

2. 2. The system of claim 1, further comprising a memory, wherein the data from the DMA system includes at least one of data retrieved from the memory using the DMA system or address data corresponding to the data retrieved from the memory.

3. 2. The system of claim 1, wherein a seed value for each CRC calculation is programmed for each channel using a MISR control unit of the MISR hardware component, the MISR control unit being configured using the processing controller.

4. 2. The system of claim 1, further comprising a vector processing unit (VPU), wherein the data of the multiple channels from the DMA system is obtained from a vector memory (VMEM) after the VPU processes a MISR test.

5. 5. The system of claim 4, wherein the MISR test includes test data and a test instruction, and the VPU processes the test data according to the test instruction and writes an output of the MISR test to the VMEM.

6. The system of claim 1 , further comprising a safety processor, wherein if the MISR status indicates an error or a timeout, the MISR status is output to the safety processor.

7. 2. The system of claim 1, wherein the MISR hardware component further excludes one or more channels of the plurality of channels based at least in part on a configuration of a channel mask register of the MISR hardware component, the channel mask register being configured using the processing controller.

8. The system comprises: Control systems for autonomous or semi-autonomous machines, Perception systems for autonomous or semi-autonomous machines, a system for performing a simulation operation; a system for performing deep learning operations; System on Chip (SoC), A system including a programmable vision accelerator (PVA); a system including a vision processing unit; a system implemented using edge devices; Systems implemented using robots, a system incorporating one or more virtual machines (VMs); a system that is at least partially implemented within a data center; or Systems implemented at least in part using cloud computing resources The system of claim 1 , which is included in at least one of:

9. 1. A multiple input signature register (MISR) hardware component including a processing circuit, the processing circuit comprising: receiving multiple channels of data from a direct memory access (DMA) system based on sequencing by the processing controller; calculating a plurality of MISR values ​​by performing a cyclic redundancy check (CRC) calculation to calculate a MISR value for each channel of the plurality of channels; calculating a final MISR value using the plurality of MISR values; comparing the final MISR value with the value of a signature; and outputting a MISR status based at least in part on said comparison; MISR hardware components.

10. 10. The MISR hardware component of claim 9, wherein the data from the DMA system includes at least one of data retrieved from memory using the DMA system or address data corresponding to the data retrieved from memory.

11. 10. The MISR hardware component of claim 9, wherein a seed value for each CRC calculation is programmed for each channel using a MISR control unit of the MISR hardware component, the MISR control unit being configured using the processing controller.

12. 10. The MISR hardware component of claim 9, wherein the multiple channels of data from the DMA system are obtained from a vector memory (VMEM) after a vector processing unit (VPU) processes an MISR test.

13. 13. The MISR hardware component of claim 12, wherein the MISR test includes test data and test instructions, and the VPU processes the test data according to the test instructions and writes an output of the MISR test to the VMEM.

14. The MISR hardware component of claim 9 , wherein if the MISR status indicates an error or a timeout, the MISR status is output to a secure processor.

15. 10. The MISR hardware component of claim 9, further comprising functionality to exclude one or more channels of the plurality of channels based at least in part on a configuration of a channel mask register of the MISR hardware component, the channel mask register being configured using the processing controller.

16. the MISR hardware component comprises: Control systems for autonomous or semi-autonomous machines, Perception systems for autonomous or semi-autonomous machines, a system for performing a simulation operation; a system for performing deep learning operations; System on Chip (SoC), A system including a programmable vision accelerator (PVA); a system including a vision processing unit; a system implemented using edge devices; Systems implemented using robots, a system incorporating one or more virtual machines (VMs); a system that is at least partially implemented within a data center; or Systems implemented at least in part using cloud computing resources 10. The MISR hardware component of claim 9, included in at least one of:

17. A method performed by a processing circuit, comprising: receiving multiple channels of data, one channel at a time, from a direct memory access (DMA) system based on sequential processing by a processing controller; For each channel, calculating a plurality of multiple input signature register (MISR) values ​​by performing a cyclic redundancy check (CRC) calculation to calculate the MISR value; calculating a final MISR value using the plurality of MISR values; comparing the final MISR value with a signature value; outputting a MISR status based at least in part on said comparison; A method comprising:

18. 20. The method of claim 17, wherein the data from the DMA system includes at least one of data retrieved from memory using the DMA system or address data corresponding to the data retrieved from memory.

19. 18. The method of claim 17, wherein a seed value for each CRC calculation is programmed for each channel using a MISR control unit of a MISR hardware component, the MISR control unit being configured using the processing controller.

20. 18. The method of claim 17, wherein the data of the multiple channels from the DMA system is obtained from a vector memory (VMEM) after a vector processing unit (VPU) processes a MISR test.

Images