Direct memory access system configured for feature tracking operations in a system on chip using a vector processor

Abstract

The present disclosure relates to a direct memory access system configured for feature tracking operations in a system-on-a-chip using a vector processor. In various examples, the VPU and associated components can be optimized to improve VPU performance and throughput. For example, the VPU can include a min / max collector, an auto-store prediction function, a SIMD data path organization that allows inter-lane sharing, a transpose load / store with a stride parameter function, a load with permute and zero insertion function, a hardware, logic, and memory layout function to allow two-point and two-point-by-two-point lookups, and a per-memory bank load cache function. Further, a decoupled accelerator can be used to offload VPU processing tasks to improve throughput and performance, and a hardware sequencer can be included in the DMA system to reduce programming complexity of the VPU and DMA system. The DMA and VPU can perform a VPU configuration mode that allows the VPU and DMA to operate without a processing controller for performing dynamic region-based data movement operations.

Description

BACKGROUND

[0001] Vector processing units (VPUs) are used to perform single instruction multiple data (SIMD) operations in parallel. Popular uses of VPUs include operations such as image processing, computer vision, signal processing, deep learning (e.g., for convolution operations), and the like.

[0002] In some computer vision applications, for example, the dynamic range of intermediate values is well understood. Thus, to detect anomalies, computed values can be compared to these dynamic ranges. However, conventional solutions for detecting and analyzing these minima and maxima include writing all values to memory, and then analyzing the values in memory, which requires additional processing cycles. Furthermore, to achieve high throughput, a high clock rate processor can perform software pipelining and / or loop unrolling despite the latency of load usage. However, in cases where the original iteration count is not evenly divided by the unrolling factor, some number of iterations can remain after completing the unrolled loop, requiring an additional remainder loop to compute the values of one or more final iterations. This remainder loop increases the code size and latency of the system - for example, because the remainder loop cannot be unrolled to achieve optimal performance. In traditional single instruction multiple data (SIMD) operations, each SIMD lane can operate in parallel and independently of each other in its own data lane. Some architectures can allow sharing between adjacent neighbors, but this limited sharing is limiting and makes implementation of the operation generally require copying the same operand to each data lane for processing. Furthermore, vector SIMD processors can require each memory read operation to use a standard or uniform unit, for example, equal to the vector processing width, which can be inefficient in cases where the memory bank is wide. For example, when reading elements 4 through 67 from a memory with a width of 64 bytes, two memory reads can be required - for example, once from 0 through 63, and once from 64 through 67. However, this results in reading many additional values - for example, values 0-3 and values 68-127 - even though these values are not needed for the current operation. In traditional instruction sets that require additional data operations, additional instructions are available to operate on the memory data in the register after the data has been read out and stored in the register. For example, this can require loading the data, performing a permutation on the data, and then performing the operation using the reorganized data. Thus, the data operations require additional cycles and increase latency. When performing a table lookup using an existing VPU, the table can be replicated so that each individual value can be fetched from the replicated table, or an additional read port can be added to each memory bank to allow multiple values to be read from the same table in the same bank. However, replicating the table for each value requires additional memory and processing, and adding an additional read port requires additional space on the chip. In traditional VPUs, data caching can not be implemented because the VPU is programmed to execute on a small set of highly optimized code, and thus the programmer can manage the contents of the local data memory. However, by doing so, each access requires reading the value from each memory bank even if the data for the next iteration includes overlap with one or more previous read operations.

[0003] To optimize the performance of a processor (e.g., a VPU), the instruction set architecture (ISA) can be enhanced to create custom instructions to accelerate common operations - e.g., table lookups, convolution operations, etc. However, using the ISA in this way requires the processor itself to also perform these operations, which means that the processor is busy during the execution of these enhanced instructions.

[0004] Further, the VPU can use a direct memory access (DMA) system to retrieve data for processing by the VPU. As such, the DMA system can operate as a data movement engine, but can also perform additional operations such as image padding, address manipulation, overlapping data management, traversal order management, frame size management, etc. However, as the DMA resources (e.g., descriptors, channels, triggers, etc.) increase, the programming complexity of programming the DMA system and the VPU also increases. In cases where the tiles of a frame contain spatial or temporal dependencies, the dynamic updating of the DMA resources becomes a processing burden for the system. When acquiring unknown or data dependent data, a conventional DMA system requires a processing controller (e.g., an R5 or ARM processing core) to intervene in the processing cycle to determine the update information used to direct the next processing iteration. For example, in object or feature tracking, the VPU can calculate the next position of an object or feature, and then the processing controller will intervene to update the memory addressing information, and then trigger the DMA system to use the updated information. However, the intervention of the processing controller increases the latency and requires more complex programming to operate using region dependent data movement algorithms.

[0005] Further, in safety critical applications such as autonomous and semi-autonomous machine applications, there are strict requirements for permanent fault detection and isolation. For example, when performing deep learning, computer vision, sensor processing, and / or other applications in a machine, permanent fault detection must be performed periodically within an allocated time budget in order to perform accurate testing while also allowing the application to execute correctly - e.g., with low latency. To this end, end-to-end coverage can be required with low latency while also meeting the runtime budget of each particular application. Conventional approaches use built-in self-test (BIST) to identify faults, but these BIST techniques either do not include sufficient coverage, introduce too much latency in the system, and / or do not meet the runtime budget of certain applications. SUMMARY

[0006] Embodiments of the present disclosure relate to improvements to vector processing units (VPUs), decoupled accelerators that can be used to process offloaded processing from the VPU, and direct memory access (DMA) systems that support data movement between memory and the VPU. To address various shortcomings of conventional or existing solutions, the VPU of the present disclosure can include a minimum / maximum hardware gatherer that is included in the data path from the VPU to memory so that the minimum / maximum can be stored prior to storage to memory. In this way, the minimum / maximum can be available immediately after the memory write operation is complete, thereby reducing the latency of determining the minimum / maximum after the values are stored to memory. Further, the VPU can include an auto-prediction function that can apply a prediction flag by setting a prediction bit for each value calculated in iterations after the final iteration. As a result, each set of iterations can include the same number of executed iterations, but one or more values from the final iteration set can not be written out to memory due to the prediction flag. To address the limitation of sharing between data lanes of existing solutions, the SIMD architecture of the present disclosure can define slices in the processor, each slice including multiple lanes, and each lane can be configured to communicate between one another. In this way, operands from one lane can be used by other lanes, thereby eliminating the requirement to copy each operand to each lane for processing. To address the inefficiency of loading from a single wide memory bank, the VPU can include multiple smaller memory banks to allow for smaller bit alignment - e.g., 16-bit alignment, where the memory bank is every 16 bits. In this way, an example of reading values 4 through 67 can occur in one memory read, rather than two memory reads of 0-63 and 64-127. In addition to this memory bank organization, the VPU can include transpose load and / or store functions to allow the stored values to be offset in the memory bank so that no memory bank conflicts occur, and more data can be read or written per cycle. To address the data operation deficiencies of conventional instruction sets, a load with permute instruction can be used to send a permute pattern along with a memory address to local memory in order to retrieve data from memory according to the permute or data manipulation pattern. In this way, the data operation and data load can be performed in the same cycle, thereby reducing latency. To address the shortcomings of table replication per value or additional read ports for table lookup, a two-point or two-point-at-a-time lookup can be performed so that two or four points per table per cycle can be looked up, respectively. To implement this, an offset store pattern of tables and per memory bank address bus and related logic and routing can be used to allow for parallel lookup of two points or four points. In embodiments, each memory bank can include an associated data cache that can be enabled or disabled depending on a given operation.For example, for filtering operations where there is a large amount of data overlap between iterations, a data cache can be used to store values from one or more previous lookups, such that each memory bank only needs a minimum number of reads, thereby preserving energy and power for the system.

[0007] To address the shortcomings of conventional ISAs for VPUs or other processor types, the systems and methods of the present disclosure can use a decoupled accelerator that can be configured by a VPU and communicate with the VPU through shared memory, but can independently perform specific tasks VPU to allow the VPU to continue with other processing tasks in parallel with the accelerator. For example, a decoupled lookup table (DLUT) accelerator can be used to improve the performance of the system in performing lookup tables. In this way, the DLUT accelerator can identify conflicts, resolve conflicts, and increase the throughput of the system, rather than the VPU performing memory bank conflict detection and resolution online.

[0008] To address the shortcomings of conventional DMA systems, the systems and methods of the present disclosure can include a hardware sequencer that operates on frame data that includes a sequence of commands for the hardware sequencer. For example, the hardware sequencer can run at the frame level rather than the tile level, and can perform sequencing for a DMA engine to eliminate the programming complexity of programming the DMA engine to perform the same operations (e.g., padding, address manipulation, etc.). In some embodiments, the DMA system can include a DMA trigger mode, where the DMA engine controls tile movement to vector memory (VMEM) rather than requiring the VPU to trigger DMA to load the next tile. As a result, the sequence of commands is reversed, and DMA becomes a trigger for the VPU. To address the shortcomings of region-dependent data movement operations in DMA systems, the DMA system can use DMA and VPU to operate in a tightly coupled loop without the need for a processing controller intervention. For example, the VPU can update location information in the VMEM for various features and / or objects that are being tracked, and the DMA can use this updated information to update descriptors in a descriptor memory to provide the VPU with the next data for processing corresponding to the next location of the feature or object. This process can repeat until processing is complete, thereby eliminating the need for a processing controller intervention and reducing the latency of the system.

[0009] Furthermore, to address the deficiencies of conventional approaches to BIST, the present systems and methods may perform a Multiple Input Signature Register (MISR) BIST—for example, to perform fault detection on a programmable vision accelerator (PVA) of a system-on-chip (SoC). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) (e.g., an R5 processor and an ARM processor, CPU, and / or the like). Thus, each component of the PVA may require testing, and the present systems and methods perform a MISR BIST to detect permanent faults in an end-to-end manner. In this manner, permanent fault detection may be performed to cover end-to-end blocks of control and data logic, report errors directly to a safety processor to reduce latency, and be customized for a specific application to meet an associated runtime budget. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present system and method for improving a vector processing unit (VPU) are described in detail below with reference to the accompanying drawings, wherein:

[0011] FIG. 1A is an example min / max collection system according to some embodiments of the present disclosure;

[0012] FIG. 1B is a flow chart illustrating a method for min / max collection according to some embodiments of the present disclosure;

[0013] FIG. 2A is an example system including a processor having an address generation unit with automatic prediction capabilities according to some embodiments of the present disclosure;

[0014] FIG. 2B is a table showing a sequence of state changes over time according to some embodiments of the present disclosure;

[0015] FIG. 2C is a flow chart illustrating a method for automatic storage prediction according to some embodiments of the present disclosure;

[0016] FIG. 3A is a diagram of an example single instruction multiple data (SIMD) datapath organization according to some embodiments of the present disclosure;

[0017] FIGS. 3B-3D shows operand sharing between slices of a SIMD architecture for filter operations, dot product operations, and sort operations with payloads, respectively, according to some embodiments of the present disclosure;

[0018] FIG. 3E A flowchart of a method for computing output using shared operands across lanes of a SIMD architecture according to some embodiments of the present disclosure is included.

[0019] FIG. 4A is a logical view of a transpose load for reading and writing memory and a memory bank view of the transpose load corresponding to the logical view according to some embodiments of the present disclosure;

[0020] FIG. 4B is a logical view of a transpose load for reading and writing memory with various row pitches and stride parameters for the write operation and a memory bank view of the transpose load corresponding to the logical view according to some embodiments of the present disclosure;

[0021] FIG. 4C is a flowchart illustrating a method of configuring a transpose load for a write operation with stride parameters according to some embodiments of the present disclosure;

[0022] FIG. 4D is a flowchart illustrating a method of performing a write operation of a transpose load using stride parameters according to some embodiments of the present disclosure;

[0023] FIGS. 5A-5B illustrates a data and coefficient layout table in a SIMD architecture for different functions according to some embodiments of the present disclosure;

[0024] FIG. 5C illustrates a hardware architecture for performing a load with permutation and zero insertion according to some embodiments of the present disclosure;

[0025] FIG. 5D illustrates an example use of the hardware architecture of FIG. 5C according to some embodiments of the present disclosure;

[0026] FIG. 5E is a flowchart illustrating a method of utilizing a permuted load according to some embodiments of the present disclosure;

[0027] FIG. 6A illustrates a 16-way parallel table organization for single point lookup according to some embodiments of the present disclosure;

[0028] FIG. 6B illustrates an 8-way parallel table organization for two point lookup according to some embodiments of the present disclosure;

[0029] FIG. 6C illustrates a logical view of a 2-way parallel word type table for 2x2 point lookup according to some embodiments of the present disclosure;

[0030] FIG. 6D illustrates a memory view of a 2-way parallel word type table for 2x2 point lookup of FIG. 6C according to some embodiments of the present disclosure;

[0031] FIG. 6E A layout for processing a pair of lanes using horizontal blending with interleaved data operations is shown in accordance with some embodiments of the present disclosure;

[0032] FIG. 6F Intermediate and final results of horizontal blending with interleaved data operations are shown in accordance with some embodiments of the present disclosure;

[0033] FIG. 6G is a flowchart of a method for performing a multi-point lookup in accordance with some embodiments of the present disclosure;

[0034] FIG. 7A Elements of data and coefficient arrays are shown in accordance with some embodiments of the present disclosure;

[0035] FIGS. 7B-7C Read operations required to use a data cache for a memory bank separately for data operands and coefficient operands are shown in accordance with some embodiments of the present disclosure;

[0036] FIG. 7D Memory bank organization used with a load cache is shown in accordance with some embodiments of the present disclosure;

[0037] FIG. 7E Hardware architecture for using a data cache in a memory bank is shown in accordance with some embodiments of the present disclosure;

[0038] FIG. 7F is a flowchart of a method for using a data cache for a memory bank in accordance with some embodiments of the present disclosure;

[0039] FIG. 8A A system including one or more decoupled accelerators is shown in accordance with some embodiments of the present disclosure;

[0040] FIG. 8B is a flowchart of a method for performing one or more operations using a decoupled accelerator in accordance with some embodiments of the present disclosure;

[0041] FIG. 9A A system including a decoupled lookup table accelerator is shown in accordance with some embodiments of the present disclosure;

[0042] FIG. 9B is a table showing different component actions of a decoupled lookup table accelerator in performing various operations in accordance with some embodiments of the present disclosure;

[0043] FIG. 9C is a flowchart of a method for performing one or more operations using a decoupled lookup table accelerator in accordance with some embodiments of the present disclosure;

[0044] FIG. 10Ais a visualization illustrating filling a frame with a fill value according to some embodiments of the present disclosure;

[0045] FIG. 10B is a visualization illustrating address manipulation of a frame's descriptor according to some embodiments of the present disclosure;

[0046] FIG. 10C is a visualization showing overlapping data between tiles of a frame according to some embodiments of the present disclosure;

[0047] FIG. 10D including visualizations illustrating various raster traversal orders according to some embodiments of the present disclosure;

[0048] FIG. 10E is a visualization illustrating a three-way traversal order according to some embodiments of the present disclosure;

[0049] FIG. 10F including visualizations illustrating various vertical drill-down traversal orders according to some embodiments of the present disclosure;

[0050] FIG. 10G is a visualization illustrating various image sizes in a pyramid configuration according to some embodiments of the present disclosure;

[0051] FIG. 10H is a direct memory access (DMA) system including a hardware sequencer according to some embodiments of the present disclosure;

[0052] FIG. 10I According to some embodiments of the present disclosure, FIG. 10H The frame format of the sequencing command controlled by the hardware sequencer of the DMA system;

[0053] FIG. 10J is a method for raster scanning sequences according to some embodiments of the present disclosure FIG. 10I An example of a frame format;

[0054] FIG. 10K is an example tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, using FIG. 10J Perform frame address processing on the example frame format;

[0055] FIG. 10L is a flowchart of a method of using a hardware sequencer in a DMA system according to some embodiments of the present disclosure;

[0056] FIG. 11A A data flow diagram illustrating a process for configuring a direct memory access (DMA) system using a vector processing unit (VPU) according to some embodiments of the present disclosure;

[0057] FIG. 11B is a table illustrating a VPU configuration format written into a vector memory (VMEM) by a VPU and read by a DMA system, according to some embodiments of the present disclosure;

[0058] FIG. 11C is a flowchart of a method of using a VPU configuration DMA system, according to some embodiments of the present disclosure;

[0059] FIG. 12A is a built-in self-test (BIST) system diagram for performing cyclic redundancy check (CRC) computation of a programmable vision accelerator (PVA), according to some embodiments of the present disclosure;

[0060] FIG. 12B is a BIST system diagram for parallel channel CRC computation of a PVA, according to some embodiments of the present disclosure;

[0061] FIG. 12C is a flowchart of an execution (BIST) method for permanent fault detection in a PVA, according to some embodiments of the present disclosure;

[0062] FIG. 13A is an illustration of an example autonomous vehicle, according to some embodiments of the present disclosure;

[0063] FIG. 13B is an example of a camera location and field of view of the example autonomous vehicle of FIG. 13A , according to some embodiments of the present disclosure;

[0064] FIG. 13C is a block diagram of an example system architecture of the example autonomous vehicle of FIG. 13A , according to some embodiments of the present disclosure;

[0065] FIG. 13D is a system diagram for communication between one or more cloud-based servers and the example autonomous vehicle of FIG. 13A , according to some embodiments of the present disclosure;

[0066] FIG. 14 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0067] FIG. 15 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0068] Systems and methods related to various components of a system on a chip (SoC) are disclosed - e.g., a vector processing unit (VPU), a direct memory access (DMA) controller, and a hardware accelerator (e.g., a programmable vision accelerator (PVA), such as a PVA that includes one or more pairs of VPUs and DMAs). For example, in various embodiments of the present disclosure, a PVA can include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) (e.g., R5 processors and ARM processors, CPUs, and / or the like). Although the present disclosure (including various components of a SoC) can be described with respect to an example autonomous vehicle 1300 (also referred to herein as “vehicle 1300” or “myself vehicle 1300”), which is an example with respect to FIGS. 13A-13D ), this is not limiting. For example, the systems and methods described herein can be used by, without limitation, a non-autonomous vehicle, a semi-autonomous vehicle (e.g., in one or more advanced driver assistance systems (ADAS)), a driving and non-driving robot or robot use platform, a warehouse vehicle, an off-road vehicle, a vehicle coupled to one or more trailers, an aircraft, a boat, a shuttle, an emergency response vehicle, a motorcycle, an electric or motorized bicycle, an airplane, an engineering vehicle, an underwater vehicle, a drone, and / or other vehicle types. Moreover, although the present disclosure can be described with respect to computer vision, machine learning, artificial intelligence, image processing, and the like, this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology space in which a vector processing unit (VPU), a direct memory access (DMA) system, an instruction set architecture (ISA), a programmable vision accelerator (PVA), a decoupled accelerator, a decoupled lookup table, a hardware sequencer, a single input multiple data (SIMD) architecture, and / or one or more other components of a SoC can be used. Moreover, although the components and related processes described herein can be described with respect to a SoC, this is not meant to be limiting, and these components can be implemented as standalone components, discrete components of a system, and / or integrated components of a SoC. In some embodiments, the systems, components, features, functions, and / or methods of the present disclosure can be integrated into FIGS. 13A-13D an example autonomous vehicle 1300, FIG. 14 an example computing device 1400, and / or FIG. 15 an example data center 1500.

[0069] Min / max hardware gatherer for anomaly detection

[0070] For example, in computer vision applications, especially in safety-critical vision applications, the dynamic range of computed intermediate results is an important task. For example, to detect noise or errors in intermediate computations, known or expected dynamic value ranges can be used to identify values that fall outside of these ranges. In such examples, where values fall outside of the known or expected dynamic range, these values can be flagged as corresponding to noise, errors, and / or another problem. Thus, it can be desirable to collect minimum (min) and maximum (max) values of intermediate results to detect data anomalies. In practice, these anomalies can be caused by, but are not limited to, noise in image sensors, algorithmic corner cases, or data corruption in memory or interconnects. To address these issues, collecting min / max is an effective method to detect outliers in this data. Min / max is also used in certain algorithms.

[0071] For a particular example, in autonomous vehicle applications, runtime exceptions - such as infinity or non-numeric - can be invalid values or produce errors, cause malfunctions, or other undesirable results. With this in mind, algorithms executed as part of an autonomous vehicle platform can be evaluated to determine the range of values (intermediate or otherwise) that can be produced in processing. Once the range of values is known, actual computed values can be compared to the known range, and values that exceed a minimum or maximum threshold can be flagged as errors. In the case of flagged errors, changes in processing can be performed - such as ignoring data for a given iteration, identifying and repairing problems, and the like. In this way, runtime exceptions are not allowed and are not relied upon by the autonomous vehicle due to potential runtime exceptions being accounted for.

[0072] As another example, min / max collection can be used in certain algorithms to normalize intermediate results to a certain numerical range, thereby enabling higher accuracy in processing - such as block floating point. This normalization process can include a dynamic range collection step of collecting the min and / or max of an array, and an adjustment step of applying a scaling factor to the array. However, to collect min / max, conventional processes require writing all values to memory, then analyzing the min / max of these values and adjusting the scale.

[0073] Accordingly, these conventional methods for minimum / maximum evaluation are performed in software and require additional processing cycles. For example, the algorithm itself can be run to compute the values, and then software can be run to determine the minimum / maximum and compare the minimum / maximum to the values of the known range to identify anomalies. This software requires additional instructions to be executed to read the elements in the intermediate result array and then perform the minimum / maximum operation. As a result, the run time of the system for detecting anomalies is increased because the algorithm is executed to completion and then additional processes are performed to compute the minimum / maximum of the algorithm output. This can cause downstream processing to be delayed until the minimum / maximum is computed and compared to the threshold, or can cause downstream tasks to begin performing calculations on data containing errors while the minimum / maximum evaluation is being performed. This not only increases the run time, but also increases the processing requirements and energy consumption of the system as these additional loops are executed to identify anomalous data.

[0074] Reference FIG. 1A , FIG. 1A is an example processor architecture 100 for minimum / maximum collection according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be carried out by hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. In some embodiments, the architecture 100 can include components, features, and / or functionality similar to those of an example autonomous vehicle 1300 of FIGS. 13A-13D an example computing device 1400 of FIG. 14 and / or an example data center 1500 of FIG. 15 .

[0075] To address the deficiencies of conventional minimum / maximum evaluation processes such as described herein, the present disclosure includes systems and methods for minimum / maximum collection using hardware. For example, during computation, computed values can be written out to memory 106 (e.g., local memory) and used for downstream computation within the same algorithm or another algorithm. To reduce runtime and processing, minimum / maximum collection hardware (e.g., minimum / maximum collector 104) can be used to capture minimum / maximum values before or as they are written out to memory 106— e.g., rather than waiting for the values to be read out of memory 106 and then analyzing the minimum / maximum values. For example, an enable bit can be used to enable the minimum / maximum collection functionality of the minimum / maximum collector 104, and once enabled, the minimum / maximum collector 104 can update the minimum / maximum values as they are computed using the processor 102 and written out to memory 106 (e.g., before storage or concurrently with storage to memory 106). In embodiments, the enable bit can indicate the type of array being computed— e.g., signed or unsigned— such that the minimum / maximum collector 104 is configured to collect minimum / maximum values for a particular type of array. For example, the enable bit or another type of control feature can be used to disable the minimum / maximum collector 104 and / or configure the minimum / maximum collector 104 to collect unsigned minimum / maximum values or collect signed minimum / maximum values. In the data store data path, minimum / maximum collection logic of the minimum / maximum collector 104 can be included that reads values as they are computed using the processor 102 and stored in the register file to update or maintain the minimum / maximum values.

[0076] For example, during operation, the current minimum and / or current maximum can be held in the minimum / maximum collector 104, and the current minimum and / or current maximum can be updated to a new, lower minimum and / or a new, higher maximum, which is written out to the memory 106. In the case that the newly computed value is greater than the minimum and / or less than the maximum, the current minimum and / or maximum can be held by the minimum / maximum collector 104. In this way, the minimum / maximum collector 104 can hold the current minimum and / or maximum as each value is computed throughout the computation. Once the computation for a given iteration is complete, the minimum / maximum values are immediately available in the minimum / maximum collector 104, and software and / or hardware can be used to compare these stored values to minimum and / or maximum thresholds associated with the particular algorithm or computation being performed to determine whether an anomaly exists. For example, mechanisms can be included that allow the collected minimum / maximum values to be read for evaluation. Thus, as compared to previous approaches, no additional loop is needed to compute the minimum / maximum values after the algorithm is fully executed, as the minimum / maximum values are immediately available. Further, in embodiments, the minimum / maximum collector 104 (e.g., including hardware and / or logic) can be aware of store predictions, such that if a particular data item is prohibited from being stored to the memory 106 via, for example, per-channel store prediction, the minimum / maximum collection can exclude that particular data item. For example, in the case that an address from the address generator includes a store prediction flag, the computed value can be ignored for both storage to the memory 106 as well as updating the minimum / maximum collector 104.

[0077] In some embodiments, the minimum / maximum collector 104 can be implemented as a feature of a system including an address generator— such as one or more address generators described in U.S. Non-Provisional Application No. 15 / 141,703, filed April 28, 2016, the entire contents of which are incorporated by reference herein. The address generator can be included in any type of processor or other processing unit— such as a vector processing unit (VPU), a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a data processing unit (DPU), and / or another processing unit type (such as those described in relation to FIG. 1B , 14, and / or 15). In some embodiments, one or more VPUs can be included in a programmable vision accelerator (PVA) and / or as part of a system on a chip (SoC).

[0078] As a non-limiting example, the input for a particular sensor type or algorithm can be limited to 16-bit units. To determine the dynamic range for that particular sensor and / or algorithm, the operations associated with the algorithm that processes the sensor input can be evaluated. In such an example, assume that the first operation is the addition of two 16-bit numbers, then the first intermediate result is a 17-bit number. The 17-bit number can then be multiplied by a 5-bit number to produce a 22-bit number. If this is the end of the algorithm, then it can be determined that the output can not exceed 22 bits. Similarly, the minimum value can be evaluated. Thus, during deployment, if the minimum / maximum value exceeds this known range (e.g., 22 bits), then the output can be flagged.

[0079] In some embodiments, the store data path (e.g., between the processor 102 and the memory 106) can include saturation and / or rounding logic 108 to limit the values stored to the memory 106 between certain upper and lower limits or thresholds, or rounded according to certain specific conventions. Thus, in traditional approaches, the evaluation of the minimum / maximum value can be performed after saturation and / or rounding. In the presence of anomalies, these traditional approaches can fail to detect anomalies because the saturation and / or rounding can hide the anomalies - e.g., low and / or high values can saturate between the upper and lower limits configured for the saturation logic.

[0080] However, for certain implementations, it can be valuable or desirable to have unsaturated, unrounded, or absolute minimum / maximum values - e.g., in addition to or instead of the saturated minimum / maximum values. Thus, the minimum / maximum collector 104 of the present disclosure can collect minimum / maximum values from the raw or unsaturated data (e.g., before the values are manipulated using the saturation / rounding logic 108) for anomaly detection. In embodiments, the collection of the average of the data or the average absolute value of the data can be performed. The average can be computed, e.g., by summing the elements, reading back the sum from the address generator configuration register, and dividing by the number of data items stored (which the application can know). In this way, the minimum / maximum of the absolute values, the sum of the values, and / or the sum of the absolute values can be added to the processor store data path, and the configuration and collection of the results statistics can be performed - e.g., can be added to the address generator configuration function set, or can be managed separately. In some embodiments, the minimum / maximum collector 104 can collect the values before and / or after saturation, rounding, or other computations using the saturation / rounding logic 108.

[0081] Reference is now made to FIG. 1AEach block of the method 110 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 110 can also be embodied as computer-usable instructions stored on computer storage media. The method 110 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, just to name a few. Although described with respect to the architecture 100, the method 110 can be performed by any one system or any combination of systems, including but not limited to those described herein. FIG. 1B Although described with respect to the architecture 100, the method 110 can be performed by any one system or any combination of systems, including but not limited to those described herein.

[0082] FIG. 2A is a flowchart illustrating a method 110 for minimum / maximum collection according to some embodiments of the present disclosure. At block B102, the method 110 includes computing one or more values. For example, one or more values can be computed using a processor 102 when executing one or more algorithms, e.g., neural networks, computer vision algorithms, filtering algorithms, etc.

[0083] At block B104, the method 110 includes comparing a value of the one or more values to a currently stored minimum value and a currently stored maximum value. For example, the minimum / maximum collector 104 can compare each of any number of values to be stored to memory 106 (e.g., values in a register file) to the currently stored minimum value and the currently stored maximum value (e.g., currently stored by the hardware minimum / maximum collector 104). In such an example, as values are computed and before or as the values are stored to memory, the minimum / maximum collector can compare the values to the currently stored minimum value and / or maximum value. In one or more embodiments, the minimum / maximum collector can be included in a data path between hardware units that compute the one or more values and memory units that store the one or more values.

[0084] At block B106, the method 110 includes determining that the value is one of greater than the currently stored maximum value or less than the currently stored minimum value. For example, based on the comparison of block B104, the system (e.g., the hardware minimum / maximum collector 104) can determine that each value to be stored to memory is one of greater than the currently stored maximum value or less than the currently stored minimum value.

[0085] At block B108, the method 110 includes updating the currently stored minimum value to the value based on the value being less than the currently stored minimum value. For example, in the case that a computed value to be stored to memory is less than the minimum value currently stored by the hardware minimum / maximum collector, the hardware minimum / maximum collector can update the currently stored minimum value to the computed value.

[0086] At block B110, the method 110 includes updating the currently stored maximum value to the value based on the value being greater than the currently stored maximum value. For example, in the case that a computed value to be stored to the memory is greater than the currently stored maximum value of the hardware minimum / maximum collector, the hardware minimum / maximum collector can update the currently stored maximum value to the computed value.

[0087] In this way, the minimum / maximum values can be dynamically updated during the storing of the values such that, once some (e.g., all) of the values are stored, the minimum / maximum values are immediately available by reading them out of the currently stored values of the minimum / maximum collector.

[0088] Automatic storage prediction

[0089] In high clock rate processors, a popular implementation is to configure the processor into multiple pipeline stages. Thus, there can be a delay between the time an instruction is issued to load a register from a local memory to the time the register is available for another instruction operation - e.g., a load-to-use delay. To achieve high throughput in the presence of load-to-use delays, processor compilers and application development can use software pipelining and / or loop unrolling. For example, software pipelining can be used to overlap the execution of multiple iterations of a loop, and loop unrolling can be used to expand the loop body by repeating its contents multiple times. In sum, these techniques can allow multiple iterations of the loop contents to be executed concurrently, reducing idle periods in the schedule (ideally none). When performing loop unrolling, a compiler can divide the loop iteration count by the unroll factor. For example, the compiler can assume that the original iteration count is a multiple of the unroll factor, and thus the unrolled loop can be executed with equivalent functionality. In such an example, if the original iteration count is 60, and the loop is to be unrolled by a factor of 6, the unrolled loop can run 10 iterations. However, if the original loop iteration count is 64, 64 / 6 also results in 10 through normal integer division, so the loop will not execute enough times (e.g., can not execute the additional 4 iterations), resulting in different code behavior after unrolling, which can cause the application to fail. In certain techniques, an assert statement is added to ensure that the iteration count is indeed a multiple of the unroll factor.

[0090] The collection of steps or operations in a loop body can have a narrow range of optimal or desirable unrolling factors. For example, the lower bound of the unrolling factor can be the minimum number of copies of the looped code to schedule to fill in the gaps due to various latencies and achieve optimal performance, and the upper bound can be the maximum number of copies to schedule with the limited capacity in the register file - which can result in excessive register spilling (saving to and restoring from the stack) and suboptimal scheduling, for example. As another example, due to the feasibility of selecting tile width and tile height combinations that allow the iteration count to be a power of 2 (e.g., 2, 4, 8, etc.), unrolling by a power of 2 is acceptable for many applications. However, in embodiments, the loop body can also be optimally unrolled 6 or 7 times, and unrolling by 4 or 8 times can be inefficient. In any case, unrolling the loop to achieve optimal scheduling can impose an inconvenient restriction on the iteration count. Thus, conventional techniques to address this issue can result in performance degradation and increased code size.

[0091] For example, the restriction on the iteration count is inconvenient, so a programmer can write two loops - e.g., a "multiple" loop and a "remainder" loop - when such a restriction on the iteration count should not be imposed. As an example, the following illustrative example code segments show: code 1 - a vector addition loop without loop unrolling; code 2 - the same loop unrolled by 6, which is only valid when the iteration count is a multiple of 6; and code 3 - a double loop solution that works for any iteration count, but the remainder loop is not unrolled and thus is less efficient, and also results in larger code size due to the additional loop and iteration count computation.

[0092] Code 1:

[0093]

[0094] Code 2:

[0095]

[0096] Code 3:

[0097]

[0098] Using a vector processing unit (VPU) of the present disclosure, code 1 can achieve 6 cycles per iteration, code 2 can achieve 1 cycle per iteration, and the performance of code 3 can depend on the iteration count. For an iteration count (niter) of niter = 60 (a multiple of 6, so the remainder does not run), code 3 can achieve 1.0 cycles per iteration, and for niter = 64 (the remainder loop runs 4 times), code 3 can achieve an average of 1.3125 cycles per iteration (e.g., (60*1 + 4*6) / 64 = 84 / 64 = 1.3125).

[0099] Reference is made to FIG. 2A , FIGS. 13A-13D is an example system 200 including a processor 202 (e.g., VPU) including an address generation unit with automatic prediction capability, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be carried out by hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. In some embodiments, the processor 202 can be included in and / or can include components, features, and / or functionality similar to those of an example autonomous vehicle 1300 of FIG. 14 an example computing device 1400 of FIG. 15 and / or an example data center 1500 of FIG. 2B .

[0100] In embodiments of the present disclosure, loads and stores in code segments can use an address generator 204 in the processor 202 (e.g., VPU). For example, in each load and store, address generator (agen) parameters (agen_a, agen_b, agen_c) can be provided to the load / store function. The arguments can identify an address generator register that contains parameters that can be used for address calculation for the particular load and / or store operation— e.g., address pointer, number of iterations, current loop variable value, etc. In some embodiments, the VPU can be designed to support 6 (or other value) addressing dimensions per address generator register, thus including 6 (or other value) iteration counts and 6 (or other value) loop variables.

[0101] To address the limitation of loop unrolling on the number of iterations, the system and method of the present disclosure can include an address generator 204 having logic to automatically predict storage from the address generator 204 (e.g., a prediction flag or bit 208). For example, the prediction can be used to provide an indication of conditional execution, such as whether or not to do something. The value of the prediction bit 208 (e.g., 0 to indicate storage or 1 to prevent storage, or vice versa) can be used to indicate whether the instruction is to be executed. Execution can not refer to the actual execution of the iteration, but rather whether the result of the iteration execution is stored into memory. Thus, in embodiments, an instruction that is not executed due to the prediction flag can refer to an instruction or iteration that is being executed, but the result of the execution is prevented or excluded from changing the state of the memory 206. Instruction level prediction and lane level prediction can be included. Instruction level prediction can be used to indicate whether the entire instruction should be executed, while lane level prediction can be used to indicate which data lanes should be executed or not executed.

[0102] In some embodiments, after the loop variable exhausts the iteration count, any subsequent execution of the store instruction is automatically predicted to inhibit further writes to the memory 206. In this way, the automatic store prediction feature can allow for clean write of code by rounding iteration counts that are not multiples of 6 (or another unroll factor) to the next multiple of 6 and by not changing iteration counts that are not multiples of 6 (or another unroll factor). Although a factor of 6 is used, this is not intended to be limiting and any unroll factor can be used without departing from the scope of the present disclosure. Code 4 below includes a vector addition example with automatic store prediction.

[0103] Code 4:

[0104]

[0105] Code 4 with an original iteration count (niter) of 64 can run the unrolled loop 11 times with 1.03125 cycles per iteration (e.g., 11x6 / 64 = 1.03125). Another approach to address the limitation of iteration counts that are multiples of the unroll factor is to calculate the necessary prediction flags in the loop and provide the prediction flags in the store instruction. For example, code 5 described below shows an example implementation of prediction flag calculation.

[0106] Code 5:

[0107]

[0108]

[0109] Code 5 can compile to 1.5 loops per iteration in a VPU of the present disclosure, so auto-prediction can include a performance advantage over a prediction that is computed in a loop. In an embodiment, the VPU can include a 7-way very long instruction word (VLIW) instruction scheme, and each cycle can include 2 scalar slots for scalar operations needed for prediction computation. If there are more vector operations per iteration of the loop, there can be enough scalar slots so that prediction computation can fit in the available slots and not cause a performance penalty. Even in a compute loop where real-time computation of a prediction has no impact on performance, auto-prediction can still have an advantage in code size and energy consumption.

[0110] Thus, software can be used to configure the number of iterations (e.g., N1-N6), and the software can cause execution of address generator-based loads / stores - typically in a loop. The address generator hardware can maintain the loop variable (e.g., variables I1-I6), and can advance the address pointer as appropriate. When the address generator-based loads / stores have been executed for more than a pre-configured number of iterations, the address pointer can stall at the last valid address, and auto-prediction can be turned off (e.g., by setting a prediction flag) to block subsequent stores into memory. In this way, an "auto-prediction off' internal Boolean state can be included in the address generator 204, and the loop variable iteration logic can be configured to support turning off auto-prediction. For example, and with respect to FIG. 2B When the address generator is initialized, a value for the parameter auto-prediction off ("auto_pred_off") (e.g., prediction bit 208) can be initialized or reset to "0" in addition to the loop variables I1-I6. After the loop variables exhaust the programmed iteration count, auto_pred_off can be updated to "1". As a result of the prediction bit being "1", any subsequent execution of the store instruction can then be auto-predicted, and further writes to memory can be prevented.

[0111] In the example of FIG. 2B , the number of iterations for the address generators of registers N1-N6 can be programmed as N1=4, N2=2, N3=N4=N5=N6=1. The total programmed iteration count can thus be 4*2*1*1*1*1=8, and the results can be executed in the sequence shown. As shown, the initial state and the subsequent 7 executions (e.g., the first 8 iterations) can correspond to the auto_pred_off bit being 0, and the 8th and 9th executions (e.g., the last 2 iterations) can correspond to the auto_pred_off bit being 1, thereby preventing the results of the 9th and 10th executions from being stored into memory. FIG. 2C

[0112] ​In practice, a VPU can be configured to process a number of vector units working simultaneously - for example, 8, 16, etc. - and therefore the VPU may require the array to be a multiple of the number of vector units. This setup works well if the array is a multiple of the number of vector units. However, typically, an array may not be a multiple of vector units (for example, because there is no guarantee that the data will be calculated based on arrays of the same size), so the array is padded so that processing is always performed on batches of the same size. For example, the remaining iterations can be padded with "0" values, but this still requires an additional loop in the software to process the padded values. Therefore, padding can be inefficient because the added data leads to wasted computation, and it also complicates the software - a common problem in single instruction multiple data (SIMD) software. Therefore, automatic storage prediction can be used to address this issue.

[0113] As a non-limiting example, using batches of 16, as many batches of 16 as possible can be generated from an array, and the remaining values ​​can be included in the final batch, with the remaining or remaining space within the 16 batches being predicted off using the lookahead flag. For a specific example, if an array is 82 in size, five complete sets of 16 might be generated, and in the final iteration, the remaining two elements might be included, while the other 14 elements might be automatically predicted off—thus minimizing the computational waste of padding the batch with 14 values ​​and performing unnecessary computations on the padding data. As another example, if the vector processing granularity includes a width of 32 and the array has 100 elements, three complete 32-element vectors can be processed, and the remaining four elements can be processed using four of the 32 channels (e.g., the lookahead flag might be on), while the other 28 channels might be predicted off. In this way, programmers might be able to vectorize arrays whose number of samples is not a multiple of the number of cells. For example, for each store, the hardware might actually calculate the number of elements to be written to the memory and communicate this information to the store unit. Therefore, even if the mathematical operations for padding or appending elements could be performed and stored, such additional computation and storage would be inefficient. Therefore, a prediction flag can be set so that no additional reads are required and the write of the calculated values ​​from the padding values ​​to memory does not occur (e.g., is blocked or excluded). This automatic prediction can occur at the instruction level, and software can be added to additionally perform lane-level prediction.

[0114] In addition, for auto-prediction, no additional information can be needed, as the address generator can be programmed for multiple iterations - thus the address generator has memory to support auto-prediction - and software instructions can be added to automatically move between prediction store and prediction off store. In this way, in the last iteration, the hardware can determine when to store the full result or when to store less than the full result - e.g., due to prediction being off or otherwise signaled - which can be performed at zero cost while maintaining performance. In the case of using software alone, this process would require additional cycles, thus slowing the process.

[0115] In some embodiments, prediction can be used at the per-lane level, such that these implementations can not only handle iteration counts that are not multiples of the loop unrolling factor, but can also efficiently handle any problem size that is not a multiple of the vector width. In such embodiments, the vector registers can be used to drive the per-lane prediction, which can provide the advantage of computing the information in real-time, and by using the vector registers can implement a shortcut to eliminate the requirement to copy from the vector registers to scalar prediction registers, and then use the scalar prediction registers to apply the prediction flags on each lane. For example, the per-lane prediction can be performed from the vector registers, which can be beneficial when computing the per-lane prediction information in the loop, and can vectorize the computation.

[0116] For example, to perform a certain value replacement in an array - e.g., replace any value over 100 with 999 - the code can be written as follows:

[0117]

[0118] While this code can be functionally correct, it can result in poor performance. Thus, the code can be vectorized by incorporating the use of per-lane prediction, such as the following code:

[0119]

[0120] When the prediction computation is vectorized in this way, and the per-lane prediction can only be communicated through scalar prediction registers, the prediction information needs to be copied from the vector registers to the scalar prediction registers, thus increasing the execution time.

[0121] However, rather than performing bit packing and moving the prediction mask from vector lane 0 to a scalar register, the per-lane prediction can be driven directly from the vector register features described herein in this example, as shown in the following code:

[0122]

[0123]

[0124] Referring now to the drawings FIG. 2A Each block of the method 220 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 220 can also be embodied as computer-usable instructions stored on computer storage media. The method 220 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. Although described with respect to the system 200, FIG. 2C The method 220 can be performed by any one system or any combination of systems, including but not limited to those described herein, although described with respect to the system 200.

[0125] FIGS. 13A-13D is a flowchart illustrating a method 220 for automatic store prediction according to some embodiments of the present disclosure. At block B202, the method 220 includes determining a total number of iterations. For example, the address generator 204 can determine a total number of iterations to perform for a given instruction.

[0126] At block B204, the method 220 includes separating the total number of iterations into groups of iterations. For example, the address generator 204 can separate the iterations by a unroll factor to generate a loop body that includes multiple iterations of a loop.

[0127] At block B206, the method 220 includes determining that a group of iterations of the groups of iterations includes a first number of iterations that is less than a second number of iterations that corresponds to other groups of iterations of the groups of iterations. For example, the address generator 204 can determine that after separating the iterations by a unroll factor, a group of iterations includes fewer iterations than other groups. For example, where the unroll factor is 6 and the number of iterations is 62, there can be 11 groups of iterations—10 groups include 6 iterations and 1 group includes 2 iterations. As such, the address generator 204 can determine that 2 iterations in that group of iterations including the remaining 2 iterations should be executed and that the other four iterations should be predicted off.

[0128] At block B208, the method 220 includes generating a prediction flag corresponding to at least one iteration of the group of iterations during execution of the group of iterations. For example, upon determining that the group of iterations does not include a complete group of the same number of iterations as other groups of iterations, the address generator 204 can enable the prediction flag (change the value of the prediction off bit 208) to indicate that the results of the extra iterations should be stored or written to memory.

[0129] At block B210, the method 220 includes preventing a value corresponding to at least one iteration of the group of iterations from being written to memory based at least in part on the prediction flag. For example, based on the prediction flag being set, the calculated value can be prevented or computed from being written to memory.

[0130] Enhanced SIMD datapath organization for vector processors

[0131] In a traditional single-instruction, multiple-data (SIMD) architecture, each SIMD processing unit operates in parallel and independently of each other on its own data path. Some machines allow each SIMD processing unit to communicate directly with its immediate neighbors (e.g., left and right neighbors as a linear array of processing units, or north, south, east, and west neighbors in a two-dimensional (2D) array or processing). However, communicating only between adjacent data paths is limiting and makes operations requiring multiple input operands expensive to implement. For example, convolution is a common operation in fields such as image processing, computer vision, and machine learning. During convolution, various filters can be applied to adjacent pixels, such as, for non-limiting example, a three-tap one-dimensional (1D) filter involving three data operands and three coefficient operands. If these operands cannot be shared across the data paths of a SIMD architecture, six operands must be brought into each data path to produce a result for that specific path. With this in mind, some common approaches implement multiple read ports on a register file, but this requires additional surface area of ​​the SIMD architecture and additional operating power.

[0132] In order to address the defects of traditional SIMD architecture, the SIMD architecture of the present disclosure can allow communication between channels by defining slices (such as vector processing units (VPUs)) in a processor, and the processor includes multiple channels as a group. For non-limiting examples, in a processor, the SIMD channel organization can include a hierarchical organization, which includes a 384-bit data path that can be divided into, for example, 8 48-bit (extended word) channels, 16 24-bit (extended half-word) channels, or 32 12-bit channels (extended bytes). In such an example, each byte can be extended by 4 bits. The first layer of communication above the individual channels can be called a SIMD slice, and can be (for example, but not limited to) 96-bit width, consisting of two extended word channels (for example, two 48-bit channels), four extended half-word channels (for example, 4 24-bit channels) or 8 extended byte channels (for example, 8 12-bit channels). In a non-limiting embodiment, in all four (or other number of) SIMD slices and all channels, the entire processor data path can include four SIMD slices, and the second layer of communication can be global. In this way, operand sharing between lanes of each slice can be achieved, which may be useful in instructions such as filtering, dot products, payload sorting, etc. The SIMD architecture can be included in VPUs or other processor types, such as FIG. 14 An example of an autonomous vehicle 1300 processor, FIG. 15 The example computing device 1400 and / or FIG. 3A Example data center 1500.

[0133] Due to the physical routing of the SIMD architecture, the instruction set architecture (ISA) of the SIMD can allow sharing between a certain number (e.g., 8) of lanes within a slice. For example, as shown. As shown, within each slice, communication between 32-bit word data types, 16-bit half-word data types, and 8-bit byte data types is possible. As a result, in the example, a filtering operation such as shown can be performed in half-words with 8-bit by 8-bit multiplication and accumulation with four input operands and four coefficients, where the coefficients can be shared with data from different lanes. In a traditional SIMD architecture, each lane would need to load all 8 operands to perform the same computation that can be performed in the SIMD architecture of the present disclosure using only three input operands. Thus, since each read port is associated with increased surface area and energy consumption, only three read ports are needed which can save space and power to perform such instructions. In operation, due to the sharing between lanes within a slice, four accumulators (e.g., 0, 1, 2, and 3) can be filled with the results of the following computations. FIG. 3B FIG. 3C ACC[0] += D[0]*C[0] + D[1]*C[1] + D[2]*C[2] + D[3]*C[3]

[0134] ACC[1] += D[1]*C[0] + D[2]*C[1] + D[3]*C[2] + D[4]*C[3]

[0135] ACC[2] += D[2]*C[0] + D[3]*C[1] + D[4]*C[2] + D[5]*C[3]

[0136] ACC[3] += D[3]*C[0] + D[4]*C[1] + D[5]*C[2] + D[6]*C[3]

[0137]

[0138] ​​As shown, for example, ACC[0] can access other lanes of src la, including D[l], D[2], and D[3], and can also access other lanes of src2, including C[l], C[2], and C[3]. Similarly, other accumulators (ACCs) can access various lanes of src l and src2. In a conventional vector processor that shares limited or minimal between-lane, this type of operation is not possible. For example, these computations can include a sliding window approach, where each accumulator includes the result of moving a sliding window relative to the previous accumulator. For example, the first accumulator operates on D[0], D[l], D[2], and D[3], the second accumulator on D[l], D[2], D[3], and D[4], and so on. Each accumulator uses the same coefficients C[0], C[l], C[2], and C[3]. This is possible because of the shared physical routing between lanes of the SIMD architecture slice.

[0139] As another example implementation of the SIMD architecture of the present disclosure, and with respect to FIG. 8, FIG. 3D As shown, a dot product in a vector multiplication operation can be performed using lane sharing. In such an example, two indices (e.g., D[0][0]) indicate which lane the data belongs to and which output set the data belongs to. For the dot product computation, each lane uses only data operands from its own lane, but the coefficients are shared between lanes. Thus, the output from each lane can use all four coefficients at some time during the dot product operation. In operation, four accumulators (e.g., 0, 1, 2, and 3) can fill with the results of the following computations due to the sharing between lanes within the slice.

[0140] ACC[0] += D[0][0] * C[0] + D[l][0] * C[l] + D[2][0] * C[2] + D[3][0] * C[3] ACC[l] += D[0][l] * C[0] + D[l][l] * C[l] + D[2][l] * C[2] + D[3][l] * C[3] ACC[2] += D[0][2] * C[0] + D[l][2] * C[l] + D[2][2] * C[2] + D[3][2] * C[3] ACC[3] += D[0][3] * C[0] + D[l][3] * C[l] + D[2][3] * C[2] + D[3][3] * C[3]

[0141] As another example operation that can benefit from the SIMD architecture of the present disclosure, a FIG. 3Etwo-point sort operation. For two-point sort, two values are used to sort the payload. This two-point sort leverages communication between pairs of lanes within a slice and is useful in, for example, various computer vision applications. For example, there is an entry 0 key in lane 0 and the corresponding payload is in lane 1, and so on, and the payloads can be sorted according to the comparison of the keys—e.g., each key / payload pair as in the following code:

[0142]

[0143] Referring now to FIG. 3E The various functions can be performed, for example, by a processor executing instructions stored in memory. The method 300 can also be embodied as computer-usable instructions stored on computer storage media. The method 300 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. Although described with respect to the SIMD architecture of the present disclosure, the method 300 can be performed by any one system or any combination of systems, including but not limited to those described herein.

[0144] FIG. 3B A flowchart of a method 300 that includes computing an output using shared operands across lanes of a SIMD architecture in accordance with some embodiments of the present disclosure. At block B302, the method 300 includes dividing a bit width of a processor into a plurality of data slices, each data slice including a second bit width that is less than the first bit width, each data slice of the plurality of data slices including a plurality of lanes, each lane including a third bit width that is less than the second bit width. For example, a vector processor can be divided into some number (e.g., 4) of slices, and each slice can include some number of lanes.

[0145] At block B304, the method 300 includes loading a first vector into a first vector register such that a first lane of the plurality of lanes includes a first operand of the first vector and a second lane of the plurality of lanes includes a second operand of the first vector. For example, with respect to FIG. 3B A first data operand D[0] of the first vector can be loaded into the first lane, and a second data operand D[l] corresponding to the first vector can be loaded into the second lane.

[0146] At block B306, the method 300 includes loading a second vector into a second vector register such that the first lane of the plurality of lanes includes a third operand of the second vector and the second lane of the plurality of lanes includes a fourth operand of the second vector. For example, with respect to FIG. 3BThe first coefficient operand C[0] of the third can be loaded into the first lane, and the second coefficient operand C[l] corresponding to the third vector can be loaded into the second lane.

[0147] At block B308, the method 300 includes computing an output using the instruction based at least in part on the first operand, the second operand, the third operand, and the fourth operand. For example, with respect to FIG. 3B , a first accumulator (ACC[0]) can receive a result of the computation ACC[0] += D[0] * C[0] + D[l] * C[l] + D[2] * C[2] + D[3] * C[3], including the values of D[0], D[l], C[0], C[l], etc. This computation can occur due to the internal sharing and routing between lanes of each slice.

[0148] At block B310, the method 300 includes storing the output to a register. For example, with respect to FIGS. 13A-13D , the output of the computation can be stored to the accumulator register ACC[0], which can then be stored to memory.

[0149] Transposed load and store operations with stride parameters

[0150] In conventional vector single instruction multiple data (SIMD) processors, the size of the local data memory can be adjusted to match the vector processing width. For example, for a 256-bit vector SIMD processor capable of processing 32 8-bit lanes, 16 16-bit lanes, or 8 32-bit lanes, the local data memory can include a 256-bit wide memory or a 512-bit wide memory (e.g., twice the processing bit width), for example. In such examples, the local data memory is organized as a single memory bank with full-width memory words. However, a wide vector SIMD processor with a single full-width memory word bank can be inefficient—especially for unaligned memory accesses. For example, to load an array of a 16 element 32-bit array at byte addresses 4 to 67, the processor can need two memory reads—e.g., one to read addresses 0 to 63 (including addresses 0 to 3, which are not needed for the current operation) and a second to read addresses 64 to 127 (including addresses 68 to 127, which include data not needed for the current operation). Thus, without the grouping memory architecture of the present disclosure, access patterns can be implemented with multiple loads or stores, which can result in slower compute kernels, reduced performance, and increased power consumption.

[0151] With this in mind, a single wide memory group may instead be organized as multiple memory groups - for example, 16-bit memory groups (e.g., 32 16-bit memory groups providing 512 bits of memory bandwidth per clock cycle). In this way, read and / or write operations may be performed within any 16-bit aligned range - thereby reducing the number of redundant read / write operations, such as those described in the examples above. Using such a memory organization, reading addresses 4 to 67 may only require a single memory read. In addition to the memory group organization including smaller individual memory groups, transposed load and / or store functions may also be implemented. For example, a channel offset parameter K may be used to define a row address offset applied to each subsequent channel in the memory. The channel size may correspond to the data element size - for example, 8 bits, 16 bits, 32 bits, etc. When a 2D array is stored in memory with a row spacing of W*K+1 elements, the interleaved access mode may be converted to a vertical mode, where K is the offset parameter and W is 64 / channel size (or the size of the data element). For example, for 32-bit data elements, the row spacing may be 16*K+1. In some embodiments, a SIMD processor may be included as a component and / or may include a processor that is compatible with FIG. 14 Example of autonomous vehicle 1300, FIG. 15 The example computing device 1400 and / or FIG. 4A Components, features and / or functions similar to those of the example data center 1500 may be used.

[0152] As an example, and with respect to FIG. FIG. 4A As shown, table 400 may include an illustration of a logical view of a transposed load and a memory bank view with 17 transposed loads with a row spacing exceeding 256 bits. The memory banks are shown as 18 separate 16-bit banks in the memory bank view for illustration purposes only. For example, the memory banks may have a total of 256 bits, a total of 512 bits, or some other total number of bits—for example, each memory bank may be 16 bits wide. In the memory bank view using transposed loads, with a row spacing of 17, a single load operation can be performed to retrieve each highlighted value of the array.

[0153] While using this technique for transposed loads is beneficial for many operations, some algorithms—for example, some computer vision algorithms—may need to access even when using the FIG. 4BThe transpose load techniques described also cannot accomplish data patterns that are completed with a single read and / or write. For example, it can be desirable to load a sub-matrix of 8 high by 2 elements wide, a matrix of 4 high by 4 elements wide, or other matrix or sub-matrix sizes, rather than loading a vertical vector of 16 high. For example, in a dot product operation, the accumulation can be for two rows of 16 elements, 16 bits at a time, so when storing the output, it can be desirable to have a T16 transpose store option with the appropriate row spacing so that the two rows can be written out as one memory write transaction. To address this, a stride parameter can be used with the transpose load and / or store. In some embodiments, the stride parameter can include a power of 2 stride (although this is not limiting), such as a stride of 2, 4, 8, 32, etc., which can be referred to as T2, T4, T8, T32, etc. Examples of different transpose loads with stride parameters are shown in Table 410 of FIG. 41, which includes a logical view and a memory bank view of the transpose loads. FIG. 4A FIG. 4B Examples of the T2, T4, T8, and T32 transpose loads are shown in Table 410 of FIG. 41, which includes a logical view and a memory bank view of the transpose loads. FIG. 4A Examples of the T2, T4, T8, and T32 transpose loads are shown in Table 410 of FIG. 41, which includes a logical view and a memory bank view of the transpose loads.

[0154] In such examples, the row spacing constraint can be adjusted according to the stride. For a type T-transpose access, the row spacing can be 16K+1, for a type T2-transpose access (e.g., for a stride of 2), the row spacing can be 16K+2, for a type T4-transpose access (e.g., for a stride of 4), the row spacing can be 16K+4, and so on. Thus, the row spacing can be equal to 16K+stride value, or 16K+1+(T-1), where T is the stride parameter.

[0155] ​In operation, the architecture of the VMEM of the VPU and the instruction set architecture (ISA) of the VPU can be configured to perform transposed load and / or store operations, with or without stride parameters, to allow reading or writing data organized by column in a logical view in a single read operation. For example, the ISA can be configured to receive an indication of a starting address from which to read data or to which to write data (e.g., for reading or writing data from a register file), an indication of a write type (e.g., a transposed write operation, with or without a stride parameter), a row stride value (e.g., a K value in 16*K+1), and / or a stride parameter value. It is noted that the value of 16 corresponds to a number of data elements for a particular implementation, but the value of 16 (or W) can be different in different embodiments. Thus, when writing data to memory according to a transposed write operation, the ISA can receive a starting address to write into the VMEM, a row stride, and / or a stride parameter. As a result, when writing values, rather than writing them out to a single memory bank in a single column of data, the data can be written out according to a transpose or offset as shown in, for example FIGS. 4C-4D and 4B During a read operation, according to a transposed load instruction (and / or stride parameter), the ISA can receive a starting address, a load type (e.g., a transposed load, with or without a stride parameter), a row stride value (e.g., a K value), and a stride parameter value (e.g., a data type indicator, such as byte, halfword, etc.). The ISA can then access data from individual memory banks according to the transposed load instruction (and / or stride parameter) to retrieve a column (or columns) of data in a single read cycle. In this way, by retrieving one element from each memory bank, a single vector can be returned from a single read operation.

[0156] Referring now to FIG. 4C , each block of the methods 420 and 430 described herein includes a computational procedure that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The methods 420 and 430 can also be embodied as computer-usable instructions stored on a computer storage medium. The methods 420 and 430 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although described with respect to the SIMD architecture of the present disclosure, the methods 420 and 430 can be performed by any one system or any combination of systems, including but not limited to those described herein.

[0157] FIG. 4DA flow diagram of a method 420 to configure a transpose store operation using a stride parameter in accordance with some embodiments of the present disclosure. At block B402, the method 420 includes determining dimensions of a matrix. For example, a width of the matrix can be determined.

[0158] At block B404, the method 420 includes determining a stride parameter and a row stride for storing the matrix based on the dimensions. For example, the row stride can be determined using a 16K+ stride value, and the stride value can be determined based on the width of the matrix.

[0159] At block B406, the method 420 includes using the stride parameter and the row stride such that values of the matrix are stored into memory. For example, once the row stride and stride are determined, the values of the matrix can be stored in memory such that the row stride and stride parameter values do not cause memory bank conflicts when reading the matrix values from memory.

[0160] Reference is now made to FIG. 4D , FIGS. 13A-13D A flow diagram of a method 430 to configure a transpose store operation using a stride parameter in accordance with some embodiments of the present disclosure. At block B408, the method 430 includes receiving data representing a row stride and a starting memory address in a memory bank of a plurality of memory banks, the starting memory address corresponding to an element of a plurality of elements, the plurality of elements corresponding to a column in an array.

[0161] At block B410, the method 430 includes reading the plurality of elements from the plurality of memory banks in a single read operation, each element of the plurality of elements being read from a respective memory bank of the plurality of memory banks based at least in part on the row stride.

[0162] Permutation and zero insertion in a single instruction load

[0163] In a conventional processor instruction set, a load instruction can form a memory address through some index calculation, read the requested memory data from local memory, and deposit the memory data into a register. If additional data manipulation is required by the application, additional instructions can be used to manipulate the memory data in the register. In some cases, the data manipulation can include simple data reorganization. In a conventional processor, even this simple data manipulation in the register file requires additional instructions, and thus additional delay. For example, a conventional system can load data, perform a permutation on the loaded data, and then use the reorganized data to perform one or more operations. If such a data reorganization capability is enhanced to the load instruction, some processing time can be saved, and the compute kernel can be executed with higher performance and lower power consumption.

[0164] To address these shortcomings, the systems and methods of the present disclosure add a load with permute instruction that sends a permute pattern along with a memory address to a local memory. As a result, existing data routing and multiplexing for handling unaligned loads can be used to perform the permute without requiring substantial additional logic. In addition to saving instructions that would otherwise be spent (e.g., 5 instructions to perform a permute using a double vector input and double vector output), the overall latency of the permute operation can be reduced. For example, instead of a load-to-use latency and a compute latency (e.g., for performing the permute), the only latency is the load-to-use latency. In some embodiments, the loads with permute and / or zero insertion described herein can be included in the components, features, and / or functionality of the example autonomous vehicle 1300 of FIG. 14 the example computing device 1400 of FIG. 15 and / or the example data center 1500 of FIG. 3C or can be similar to these components, features, and / or functionality.

[0165] Accordingly, loads with permute features can be used to manipulate load data from memory into a desired format for operations. As an example, the coefficient data required for various filter and dot product instructions can include a specific repeating pattern that can be implemented through a load and permute. With respect to filter operations, for example as described in FIG. 5A the coefficients of 0, 1, 2, and 3 can repeat over the width of the vector (e.g., 16 bits) - e.g., as shown in FIG. 5B In such an example, the writeout to the first register can start with D[0]-D

[15] , and then a sliding window of 4 can be used to start the next register with D[0]-D

[19] , and so on. In this filter example, the coefficients C[0]-C[3] can repeat over the width of the vector, so using a permute load can help to write the coefficients in this order directly from the load, rather than loading all of the data, then performing a permute, and then writing the vector to a register. Accordingly, in this example, the permute pattern for the coefficient data can be {0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3}. In the same example, the permute pattern for the data operand can be {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19}. In this way, the data operand and the coefficient operand can be read out according to the permute order, rather than being sequentially read out and then permuted before being written to a register for computation. As another example, for example as shown in FIG. 5CAs shown, the permute instruction can include a two-way vector coefficient operand, and thus can include a permute pattern such as {0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7}. The permute pattern can be static or fixed, or can be dynamically computed through an algorithm, which allows the permute pattern to be flexible and dynamic. In embodiments where the pattern is a repeating pattern, a first instance of the repeating element can be loaded, then copied, and then written out to the SIMD lanes of the SIMD unit.

[0166] In some cases, it can be preferable to mask certain portions of memory data to zero values. For example, to visualize more easily in software development or to consume less energy (e.g., compared to preserving random data values), zeros can be inserted for unused entries. In other examples, zeros can be inserted to delineate data blocks in a data structure, for example, where the length of each data block is not fixed. In such examples, the zero values can indicate a gap between two data blocks. When processing one constant-sized image block, for example, where some variable-length information is extracted from each image block (e.g., the location of feature points), zeros can be used to fill the remaining data that does not correspond to the extracted information.

[0167] In practice, the permute index can typically include 32 or 16 elements in the read - for example, in the range 0-31 or 0-15, respectively. To include zero values in the readout, a negative index value can be included in the load using a permute operation so as to write a zero in the corresponding lane of the target register. Thus, during the write operation, for example, a negative value can be written in the corresponding lane of the SIMD architecture instead of a zero.

[0168] As an example, a 30-wide by 30-high image patch can be processed using 16 consecutive entry vector operations at a time. Since the width of 30 is not divisible by 16, each row can be processed by two vector operations, the first processing a full vector width of 16 entries, and the second processing a partial vector width of 14 entries. In such an example, it can be beneficial to use zero padding of the load of the second 14-entry vector to fill the last two vector lanes, instead of random data values that can currently exist in memory.

[0169] In one or more embodiments, a fill of zeros can be inserted into desired lane positions of a SIMD architecture, for example, to save processing time needed to write zeros to these lane positions. In the case where there are 16 lanes, a normal permutation pattern can consist of 16 lane indices - for example, 0-15. In such an example, if there are values of {100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115} and the indices are {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, -1, -1} as the permutation pattern, the values that end up in the target register should be {100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 0, 0}. Thus, the two values of -1 are automatically converted to 0 in the target register based on the permutation pattern including negative values. In the previous approach, -1, -1 would include 14, 15 respectively, and the values at 14, 15 in memory would be written to the register. However, these can include random values that can require additional processing time compared to including 0 values.

[0170] To implement a load with a permutation feature, routing and multiplexing in the memory logic can be used - for example, similar routing and logic used to perform unaligned memory loads. For example, to support a full memory width load (e.g., 32x16 bits) from any 16-bit address (or 16x32-bit lanes from any 32-bit address), the memory logic can include multiplexing logic to select any one of the 32 lanes of memory data to route to any target register lane. For example, for an unaligned memory load, the following logic can be driven:

[0171] output lane[0] = select(start lane, memory lane[0..31]);

[0172] output lane[1] = select((start lane + 1) % 32, memory lane[0..31]);

[0173] output lane[2] = select((start lane + 2) % 32, memory lane[0..31]);

[0174] …

[0175] output lane

[31] = select ((start lane + 31) % 32, memory lane [0..31]).

[0176] In embodiments, a modulo operator (%) can be used to wrap around the total number of lanes. Thus, for example, where the start lane is lane 3, lanes 3, 4, 5,..., 31, 0, 1, 2 will be used as output to the register lanes.

[0177] For a load with a permute feature, this same logic can be substantially reused, but can include modified logic to perform the permute operation. An example of the modified logic is as follows:

[0178] output lane [0] = select ((start lane + permute [0]) % 32, memory lane [0..31]);

[0179] output lane [1] = select ((start lane + permute [1]) % 32, memory lane [0..31]);

[0180] output lane [2] = select ((start lane + permute [2]) % 32, memory lane [0..31]);

[0181] …

[0182] output lane

[31] = select ((start lane + permute

[31] ) % 32, memory lane [0..31])

[0183] As an example, and with respect to the figures as FIG. 5CAs shown, hardware fabric 500 (which can be included as part of a VPU, SIMD unit, SoC, or other device type) with crossbar 510 (e.g., a crossbar) can be used to fetch data from any location of memory 512. And drive the data to any lane in the SIMD through respective multiplexers (muxes) 514A-514N. In this way, any of the 16 inputs (or other width memory or register) can be able to be written to any of the 16 output locations or lanes. This can facilitate unaligned access, such that a load operation can start from any address and then align down. For example, if reading data from memory from locations 2-18, the data can be read from 2-18 but aligned to lanes 0-16 (e.g., 2 into lane 0, 3 into lane 1, etc.). This is not possible in traditional systems where vector loads need to start from a multiple of 16 locations, e.g., 0, 16, 32, etc. As FIG. 5C As shown, because data from any memory index can be output to any lane in the SIMD unit of, for example, a VPU, permutation can also be accomplished. Multiplexers 518 can be used to inject or insert a permutation control for each lane to inform the multiplexers 514 of crossbar 510 from which memory location to read data based on a starting location (which can be aligned or unaligned) and a permutation mode. Thus, instead of simply extracting data from aligned locations, a permutation mode can be used to update the location of the memory read such that each multiplexer 514 sends the correct data to each lane of the SIMD unit. Further, multiplexers 516 can be used to insert zeros for permutation modes that include negative values or other values that indicate zero insertion (e.g., in cases where values other than negative values are used to cause zero insertion). In this way, once the memory access locations are sent from multiplexers 518 to crossbar 510, and the values from the memory access are sent to multiplexers 516 for zero insertion, values corresponding to negative values in the permutation mode can be converted to zero values to fill the values of the respective SIMD lanes. Although only four sets of lanes, multiplexers, and memory indices are shown in FIG. 5D This is not limiting and any number of sets can be included without departing from the scope of the present disclosure.

[0184] FIG. 5D An example use of hardware fabric 500 is illustrated. For example, FIG. 5C from FIG. 5D. 5D can be based on the following information:

[0185] crossbar_mode = 1;

[0186] start_lane = 2;

[0187] permute pattern={3,1,-1,…,2}={011b,001b,111b,…,010b};

[0188] mem read bus={100,101,102,…,103}

[0189] permute_low = {3, 1, 3, ..., 2}; / / lower 2-position permute

[0190] permute_sign = {0,0,1,…,0}; / / permuted bit 3

[0191] read data output={103,101,0,…,102}

[0192] Alternatively, the following C code can describe FIG. 5D and 5D The logic circuit of the hardware architecture:

[0193]

[0194] In this way, FIG. 5E In the example of FIG, a bit value of 1 in multiplexer 518 can indicate that the load permutation value should be selected, and these values ​​{3, 1, 3, ..., 2} can be transmitted to the corresponding multiplexer 514 of the crossbar switch 510. As such, the values ​​of {103, 101, 103, ..., 102} can be read from the memory and sent to the multiplexer 516, where the permutation pattern can include a third value of -1, so that the value of 103 can be converted to 0 by zero insertion. Thus, the final value of {103, 101, 0, ..., 102} can be read back into the vector register.

[0195] Now refer to the figure FIG. 5C As shown, each block of the method 550 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 550 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 550 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, by way of example, FIG. 5E The method 550 is described with reference to a hardware structure of FIG. However, the method 550 may additionally or alternatively be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0196] FIG. 6Ais a flowchart illustrating a method 550 for performing a load using a permute operation, in accordance with some embodiments of the present disclosure. At block B502, the method 550 includes determining a permute pattern for loading data from memory. For example, the permute pattern can be static or dynamically computed. The permute pattern can be aligned (e.g., 0 to 16, or 0 to 32), unaligned (e.g., 2 to 18), repeating (e.g., 0, 1, 2, 3, 0, 1, 2, 3,..., etc.), and / or other pattern types.

[0197] At block B504, the method 550 includes determining a memory address location for each lane of the plurality of lanes based at least in part on the permute pattern. For example, the permute pattern can indicate which memory address location should have data for a particular lane or register loaded. The permute pattern can be implemented using the multiplexer 518 such that the correct memory address according to the permute pattern is sent to the crossbar 512.

[0198] At block B506, the method 550 includes loading a value to each lane of the plurality of lanes based at least in part on the memory address location. For example, based on the memory address location, the multiplexer 514 from the crossbar 512 can retrieve a corresponding value from memory to write to one or more lanes within one or more vector registers. In some embodiments, the multiplexer 516 can also be used to convert values associated with negative values in the permute pattern (or other values indicating to pad with zeros) to zeros. Thus, in the event one or more negative values are included in the permute pattern, the values loaded from memory can be converted to zeros before being written to the vector registers.

[0199] At block B508, the method 550 includes performing one or more operations within each lane of the plurality of lanes using the values and at least one instruction. For example, once the processing lanes of a vector register or SIMD unit are populated, one or more operations— such as arithmetic instructions, logical instructions, shift / circular instructions, bit manipulation instructions, comparison instructions, conversion instructions, constant generation instructions, and / or the like— can be performed using one or more processing units corresponding to the one or more processing lanes.

[0200] Mixed multi-point lookup for performing table lookups

[0201] In conventional processors with vector SIMD computation, local memory can include a bit width that matches the bit width of the vector SIMD. As a result, these processors can typically only support read and / or write alignment and granularity that correspond to the bit width. However, table lookups are a common technique in embedded environments such as digital signal processing (DSP) and computer vision to implement various nonlinear functions. For example, square root, logarithm, sine, and cosine functions can require performing a table lookup. To perform these functions, an input space can be uniformly sampled in a one-dimensional (ID) grid, and outputs can be recorded at these input points in a one-dimensional table. However, when implementing nonlinear functions using table lookups, there is typically a tradeoff between table size (e.g., number of entries in the table) and accuracy. To improve accuracy without requiring a large table size, interpolation lookups can be performed, where two points are looked up around a fractional index to perform linear interpolation, or three points are looked up around a fractional index to perform quadratic interpolation.

[0202] As an example, where a sine function is implemented using a lookup table, and the sine values are tabulated in integer degrees, then table[0] = sin(0 degrees), table[1] = sin(l degree), table[2] = sin(2 degrees), etc. In such an example, if an evaluation is sin(l.7 degrees), then a linear interpolation between two integer degree entries can be performed using table[l] * 0.3 + table[2] * 0.7 of the fraction. In this example, the second entry of table[2] gets the fraction as a weight, and the first entry gets 1 minus the fraction, so the second entry is weighted more highly the closer the fraction is to 1.0 or the position corresponding to the second entry.

[0203] As another example, an image or patch of an image can be resampled, which can involve looking up available pixels around some fractional pixel coordinate, and then performing an interpolation lookup. In such an example, the table can include an image patch, and can be two-dimensional. In this case, bilinear interpolation can be performed to interpolate in both dimensions, each dimension being linear. For example, a patch at position Y = 5.1, X = 7.6 can be interpolated according to the following calculations:

[0204] (patch[5][7]*0.4 + patch[5][8]*0.6)*0.9 + (patch[6][7]*0.4 + patch[6][8]*0.6)*0.1 However, performing this type of interpolation lookup in a traditional processor is expensive because a separate lookup needs to be performed for each value in each table. To speed up this process, the table can be replicated to allow any number of lookups to be performed simultaneously using different instances of the table. For example, in the example above, when looking up the patches at 5, 6, 7, and 8, the table can be replicated at least 4 times to allow the lookups to be performed in four tables in parallel. For example, where the processor (e.g., VPU) supports 32-way parallelism, the table can be replicated 32 times. However, while replicating the table can increase the throughput per cycle, the replication also requires additional memory capacity and usage, which can not be available or optimal in some implementations.

[0205] With this in mind, the systems and methods described herein use two-point and / or 2-by-2 (2x2) point lookup operations to increase throughput (or match the throughput of, for example, 32-way parallelism) while saving memory space. For example, using a per-memory bank address bus and associated logic and routing, a two-point or 2x2 point (e.g., 4-point) parallel lookup can be performed with less memory usage. Thus, a single lookup into a table can yield two points in a two-point lookup or four points in a 2x2 point lookup. This can be done based on hardware settings—e.g., bank address, logic, routing, etc.—and storage patterns in memory that allow multiple data to be read without bank conflicts. As noted above, without these capabilities, to implement, for example, a 32-way parallel lookup, the table would need to be replicated 32 times. For example, this 32-way parallel lookup can be performed using the following C code:

[0206]

[0207] In this example, the lookup portion of the loop can perform 32 lookups per cycle for two cycles (lookup and blend are performed in memory and vector math slots, respectively, and each iteration is pipelined to two cycles), and interpolate to produce 32 outputs. Thus, the entire lookup / interpolation is 16 outputs per cycle, and requires 32 copies of the table.

[0208] As a further example, and with reference to FIG. FIG. 6B, showing a 16-way parallel table organization for performing a single point lookup with index vector {0, 1, 2, 3, 4, 5, 4, 3,...}. In such an example, using traditional architecture and memory layout techniques, a first lookup and a second lookup would need to be performed sequentially to read two entries from each memory bank. For example, the first memory bank T0 includes values at T0[0] and T0[1] to be read out in the lookup operation, but because these values are both in the same memory bank, T0 (which can include only a single read port), the first value T0[0] is read in the first pass and the second value T0[1] is read in the second sequential pass. Using such a memory layout, a bank conflict occurs if two reads occur in the same memory bank, which can cause processing delays and / or cause the algorithm or other computation to not execute correctly.

[0209] However, using the architecture of the present disclosure, the same 32 lookups can only require 16 table copies for a two point lookup or only 8 for a 2x2 point lookup. For example, for a two point lookup, the same performance of 16 outputs per clock cycle can be achieved with 16 copies of the table, thereby reducing memory footprint by a factor of two. A 16-way parallel variant of the instruction can return a two vector, the first entry in the lower single vector and the second entry in the upper single vector. In C code, such a 16-way parallel lookup and interpolation can be represented as follows:

[0210]

[0211] In such an example, the lookup and interpolation portion of the loop can only require a single clock cycle (lookup and blend are performed in memory and vector math slots, respectively, and pipeline to one cycle per iteration), and interpolation to produce 16 outputs. Thus, lookup / interpolation is 16 outputs per cycle. As an example, and with respect to FIG. 6A , showing an 8-way parallel table organization for performing a two point lookup with index vector {0, 1, 2, 3, 4, 5, 4, 3,...}. In such an example, because each memory bank T0, T1, T2, etc. contains only a single value to be read during the lookup operation, all 16 values can be read out in a single pass, rather than FIG. 6A , an example, in FIGS. 6C-6D , because of the potential for memory bank conflicts, only 8 values can be read in each of the two passes. To this end, in embodiments, the instruction for lookup can include a single index and a pattern that includes not only the retrieval index, but also the retrieval index plus one position. Thus, the instruction can cause two values to be read for a two point lookup, and can write the values to the lookup table in such a format to allow the single read to be performed without a memory bank conflict.

[0212] As an example, when performing vector operations, each lane of a VPU can process a set of pixel values retrieved from memory. In some cases, one lane can process multiple values from the same memory bank, which can result in a memory bank conflict as that memory bank can only include a single read port. Accordingly, the methods and systems of the present disclosure distribute values among memory banks such that no memory bank conflicts occur, and each value used for a single processing lane of a VPU, for example, can access each corresponding value read cycle in the single processing lane.

[0213] In a conventional system performing 2D bilinear interpolation lookups, four lookups are required for each output (e.g., 2x2), allowing for an optimal throughput of 8 outputs and 32 table copies per clock cycle. Using 2x2 point lookups, 8 outputs per cycle can be achieved, 8 copies of the table (compared to 32), reducing the memory footprint required for parallel sub-tables by a factor of four. For example, for 2x2 point lookups, two entries can be read from one row of a 2D table, and then 2 entries from the next row. To avoid memory bank conflicts in any memory bank, the row spacing in the 2D table can be limited to m*k+2, where m is the number of entries stored horizontally in each sub-table, and k is any integer table row sufficient to store. For an 8-way parallel 16-bit table, m = 32 (16-bit memory words) / 8 (parallelism) = 4. For a 2-way parallel 32-bit table, m = 16 (32-bit memory words) / 2 (parallelism) = 8.

[0214] As an example, and with respect to FIGS. 6C-6D , the row spacing constraint can be used to avoid memory contention. In such an example, a 2-way parallel word-type table for 2x2 point lookups is illustrated, with a row spacing of 10. The number of consecutive elements in a sub-table (m) is 8, with A[0][0...7] placed consecutively in the sub-table, conforming to the formula of 8k+2, where k can be any integer. Accordingly, regardless of which index value is used to start, the 2x2 points to be retrieved can be placed in different banks, as guaranteed by the math. For example, the group numbers for 2x2 points relative to a sub-table are outlined as follows:

[0215] index % 8,

[0216] (index + 1) % 8,

[0217] (index + line_pitch) % 8 = (index + 8k + 2) % 8 = (index + 2) % 8,

[0218] (index + line_pitch + 1) % 8 = (index + 8k + 2 + 1) % 8 = (index + 3) % 8 there are typically 4 entries to be retrieved by a 2x2 in group number lookup, relative to the sub-table is index % m, (index + 1) % m, (index + 2) % m, (index + 3) % m). As long as m >= 4, there should be no bank conflicts. In FIG. 6C the example, the lookup can include 2D indices of (0, 1) and (1, 3), using Y then X as the convention for storing pixels in row-major order. In FIG. 6D , a logical view of two two-dimensional tables is shown, and in FIG. 6C , a memory layout view of the values from the tables is shown. In the logical view, the lookup is 2x2, as shown, and the memory layout view shows four points, each in a different memory bank (or different column in the figure), so each of these values can be read in a single memory cycle or pass. Based on the instruction and read pattern using indices (e.g., (0, 1) and (1, 3)), the values in the tables can be stored in memory in this way to read each value from memory in a single pass. Thus, using this memory layout and read instruction, the four entries of each sub-table can be returned in each cycle in the following format:

[0219] Lower destination single vector: A[0][1], A[0][2], B[1][3], B[1][4], (rest padded with zeros) Upper destination single vector: A[1][1], A[1][2], B[2][3], B[2][4], (rest padded with zeros) Although shown in FIG. 6D as two 10 element wide by 3 high 2D tables, e.g., A and B tables, this is not limiting, the tables can be any width and / or height, depending on the embodiment. Similarly, FIG. 6E the memory layout in, includes a 16 element wide x 3 high layout, but this is not limiting, and the memory width and / or height can be any configuration, depending on the embodiment.

[0220] In some implementations, e.g., when sampling an image patch, interpolation between a portion of pixels can be performed. In some embodiments, to interpolate the values of a lookup without additional instructions to manipulate the data, a vector horizontal interleave blend (VHBlend_I) instruction can be performed, which can include a horizontal blend of interleaved data to blend between pairs of channels. For example, using this instruction, a post-lookup bilinear interpolation can be performed in the same cycle. The instruction can handle each pair of channels according to the layout of the table in FIG. 6F . In this way, the calculation of Y0 and Y1 can be calculated as follows:

[0221] Yo = x * (1 - alpha0) + y * alpha0

[0222] Y1 = z * (1 - alpha1) + w * alpha1

[0223] Thus, this instruction can cause horizontal mixing between the pairs of x and y, z and w, and can cause the outputs to be interleaved in the destination register. For example, the following C code segment can be used to implement optimal performance on an 8-way parallel table using a 2x2 point lookup.

[0224]

[0225] In this 8-way parallel table organization, the sub-tables are designated A, B,..., H, and the loop can perform the lookup and interpolation, resulting in 16 outputs per iteration. In such an example, the inputs can be organized in the following manner:

[0226] idx.lo = {idx0, idx1, idx2, idx3, idx4, idx5, idx6, idx7, (ignore remaining)}

[0227] idx.hi = {idx8, idx9, idx10, idx11, idx12, idx13, idx14, idx15, (ignore remaining)}

[0228] x_frac.lo = {xf0, xf0, xf1, xf1,..., xf7, xf7} / / note repeating pattern

[0229] x_frac.hi = {xf8, xf8, xf9, xf9,..., xf15, xf15} / / note repeating pattern

[0230] y_frac = {yf0, xf8, yf1, yf9,..., yf15} / / note interleaving pattern FIG. 6G An example illustration of the intermediate and final results of this instruction are illustrated in

[0231] Referring now to FIG. 1 FIG. 6GAs illustrated, each block of the method 600 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 600 can also be embodied as computer-usable instructions stored on computer storage media. The method 600 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, just to name a few. In addition, the method 600 can be performed by any of a system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0232] FIGS. 9A-9C is a flow diagram illustrating a method 600 for performing a multi-point lookup (e.g., in a single clock cycle in a decoupled lookup table (DLUT) accelerator, such as described with respect to FIGS. 7A-7C The method 600 includes copying a table to memory to include a first value at a first physical address in a first memory bank and a second value at a second physical address in a second memory bank, the first and second values included in a same column in a logical memory view of the table, at block B602. For example, one table can be copied to memory any number of times to take advantage of the memory access parallelism of a system. The table can include a first value at a first logical address and a second value at a second logical address in the same column as the first value, which can cause a memory bank conflict if stored to memory in this configuration, as both values can be stored to the same memory bank. Thus, when copying the table to memory, a write instruction can write the first value to an adjacent first physical address - e.g., in another memory bank - as the second value, such that both values can be retrieved in the same cycle.

[0233] The method 600 includes determining a first index corresponding to the first physical address in the memory, at block B604. For example, a read operation can use an index indicating a first location in the memory from which to read a value.

[0234] At block B606, the method 600 includes reading, during a single cycle, a first value located at a first physical address and a second value located at a second physical address based at least in part on a read instruction corresponding to a multi-point lookup. For example, when a table is copied to memory, the table can be copied such that pairs of points in the same column or table (e.g., corresponding to pixels in the same column of pixels) are stored in separate memory banks. Thus, using a read instruction for a two-point lookup that uses an index of a first point in a pair of points to read a first point and an adjacent second point stored in different memory banks, the first value and the second value can be read from a first memory bank storing the first value and a second memory bank storing the second value in a single cycle. This operation can be performed for each pair of values in each replicated table to produce a high vector including first values from each table and a low vector including second values from each table, and these vectors can be used as vector registers in a VPU, as well as instructions to generate an output (e.g., interpolation, etc.).

[0235] At block B608, the method 600 includes performing one or more operations using the first value and the second value. For example, the first value and the second value can be loaded into one or more lanes of a VPU, and square root, logarithm, sine, and cosine functions can be performed, linear or bilinear interpolation can be performed, and / or other operation types can be performed. In the case of performing interpolation, one table is replicated 16 times, e.g., 16 two-point lookup operations can occur to produce 32 values - 2 values per vector lane of a VPU - and interpolation can be performed on each lane to output 16 results. Thus, 16 interpolations outputs can be produced using only 16 replicates of a table per cycle. This can be a result of using two-point lookups, as a table containing values can only need to be replicated half as much as a traditional single-point lookup operation (e.g., 16 times instead of 32 times) to allow the same memory footprint of half the 32 values for the same throughput.

[0236] Loading cache per memory bank in vector memory

[0237] In conventional processors, a data cache can have a width of, e.g., 32 bytes per cache line. A cache line is a unit of data tracked by hardware. For example, hardware can track cache line usage information in memory, including a full system address, whether the cache line has been written, a time of last read cache line relative to other cache lines, to determine when to evict a cache line. In some implementations, a data cache is a local memory, or a portion of a local memory, used to temporarily map larger data structures stored in an external memory to the local memory so that the data can be processed without incurring the long latency of directly processing the external memory. This type of data cache is often used in conventional desktop or laptop computers.

[0238] Programmable vision accelerators and / or VPUs, as non-limiting examples, include embedded processors that are designed to run smaller, highly optimized sets of code. In this class of processor types, data caching can not be implemented, as the programmer can manage the contents of the local data memory. The systems and methods of the present disclosure can include local memory that is managed by the programmer rather than being cached, but can also include additional data cache capability in one or more (e.g., each) memory bank. The data cache can be narrow, such as but not limited to 16 bits wide, as compared to more traditional data caches that include, for example, 32 bytes. In contrast to the primary goal of traditional data caches to reduce latency, the data cache can be primarily used to reduce power consumption.

[0239] For example, in computer vision processing, data access patterns often have some degree of locality (e.g., staying in a neighborhood for a while before moving to the next neighborhood). For example, when performing 7x7 2D filtering using the VFilt4HHW instruction described herein (computing 4 taps at a time), the data read stream can read 3 memory reads from a neighborhood, then move to another neighborhood and read 3 more, and so on. In the coefficient reads of the operation, the same array of zero-padded values can be used (e.g., 7*2*4 = 56 half-words), advancing four half-words at a time until the last set of 4 half-words is read, then starting again from the beginning of the 56 half-word array until the filtering kernel is complete.

[0240] Accordingly, to take advantage of these local access patterns and reduce power consumption due to memory accesses, a load data cache can be implemented in each memory bank with bidirectional bank associativity (holding, for example, a total of 64 half-words). When the load cache is enabled, the most recently read set of data (e.g., the most recent, the most recent two, the most recent three, etc.) can be recorded in the cache, as well as the line address and the entry of the most recent read in a tag memory. As a result, when the same memory address is read again, a cache hit can occur, and the cache can provide the data instead of requiring the data to be read again from local memory. In embodiments, the load cache can be located between the memory logging logic and the memory itself, such that whenever there is a cache hit, the memory read for that particular address or value will be stopped or not occur, to save power.

[0241] Using this cache structure, and for the 7x7 2D filtering example above, the load cache can allow the system to skip almost two-thirds of the data reads and almost all of the coefficient reads in steady state. The use of the data cache in each memory bank is illustrated in FIG. 7AThe VFilt4HHW instruction can perform a 4-tap filter task that can be larger, and can consume two single halfword data vectors - e.g., data[0-15] and data[4-19] - and a single halfword vector coefficient - e.g., coef[0-3] - repeated four times to fill a 16 element single vector. In a 7x7 2D filter implementation using the VFilt4HHW instruction in two vector math slots, the data elements and coefficient array of FIGS. 7B-7C FIG. 7D

[0242] FIG. 7E

[0243] ​​​Accordingly, in certain algorithms - e.g., computer vision algorithms with sliding windows - load caches can be used to save power. For example, if there is no load cache, then every memory bank needs to be read at every cycle, even if most of the data is the same. In the example of reading out 512 bits at every iteration, the first 512 bits can be read out, then another 512 can be read out, and so on. For example, if the sliding window is only 8 bytes, then only 64 bits are new at every iteration, and the remaining 448 bits are the same. If there is no data cache, then these 448 bits need to be read again from the data bank. However, with a data cache for every memory bank, these 448 bits can be fetched from the load cache, and only the 64 new bits need to be read out from the other memory bank. Accordingly, the power needed to read out the 448 bits from the memory bank is saved. Examples of algorithms that can benefit from using load caches are spatial filtering operations, deep learning inference operations (e.g., convolution operations), and the like.

[0244] With regard to FIG. 7E , a hardware architecture or logic for memory banks with load caches is shown. For example, for non-aligned access support in memory (e.g., vector memory (VMEM)), sliding window data access can be accelerated. This is a key memory access pattern for many computer vision algorithms, including filtering and convolution. For sliding window vector loads, most of the data from a random access memory (RAM) bank 702 remains the same. In such examples, when sliding 4B, only 4B of data changes in a 64B vector load, so only 4B of new data is read from the RAM bank 702. To optimize power for VMEM RAM, a micro-cache called a "load cache" can be appended to each bank for each super bank - so there are 3 super banks x 32 banks = 96 load caches in total for each VMEM. In non-limiting embodiments, the configuration of each load cache can include a two-way (2x2B = 4B) capacity, full associativity, and a pseudo least recently used (pLRU) replacement policy.

[0245] The data cache stores where the most recently accessed places are split into two parts - tag storage 706 and data storage 704. In the tag storage 706, control information corresponding to previously accessed cache addresses can be stored, and in the data storage 704, data from the previously accessed can be stored. The control information in the tag storage 706 can include valid flags (e.g., whether an entry is valid), dirty flags (e.g., whether an entry has been modified and needs to be written back to memory), and / or last use flags (e.g., if an entry is to be replaced, a least recently used policy is used to indicate which entry to replace). Because the cache is a load cache, write data can not update the cache, but valid and last use flags can be included in the tag storage 706. The valid flag or bit can be used to qualify address matches, and any write should invalidate the entry. The last use flag can be updated each time an access occurs.

[0246] As described herein, for the cache scheme to be effective, the storage capacity of the load cache is much smaller than the storage capacity of the memory or RAM bank 702 to reduce access time and save power. In one embodiment, each load cache can correspond to a single RAM bank 702, which can each be a 2048 x 16-bit memory, and the load cache can each be a 2 x 16-bit data storage 704 with a 23-bit tag storage 706 (e.g., 2 entries x (11-bit address + 1-bit valid) + 1-bit last use).

[0247] In operation, the offset 722, the row address 724, and the increment 726 can be used to generate a memory address for a memory access. The memory address can be tapped to be compared to the tag store 706 - e.g., to some number of previous access addresses (e.g., 2 previous accesses). The arrow into the top of the tag store 706 can represent the memory address. In some embodiments, the tag store 706 can use the entire memory address to compare to stored memory addresses from previous accesses. In other embodiments, a subset of address bits from the memory address can be used to address a subset of tags, so only a subset of tags is compared to the memory address. For example, where a large number of previous access tags are stored in the tag store 706, only a subset of tags can be compared using a subset of the memory address bits to reduce area and save power. In a load cache design with fewer tags - e.g., corresponding to two previous accesses - the entire tag of a previous entry can be compared to the entire memory address. The "===" decision block 720 compares the current memory address of the RAM bank 702 to the addresses stored in the tag store 706. When there is a miss (e.g., the tag and memory address do not match), the read of the RAM bank 702 can be enabled using the read enable 708 and the read data multiplexer (rd data mux) 712, which can select and read out the RAM bank 702 to send to the staging flip-flop 716. When there is a hit (e.g., the tag and memory address match), the data store 704 can be addressed with a 0 or 1 (in embodiments with two entries) to indicate which previous access the hit corresponds to. The corresponding entry in the data store can be sent through the read data multiplexer 712 to the staging flip-flop 716. The staging flip-flop 716 can return the read back data to the processor pipeline, ultimately routing to the destination scalar or vector register of the load instruction.

[0248] The staging flip-flop 714 can correspond to a parity check. For example, a large enough memory can be required to have a parity bit (e.g., in the parity terminal 710) to allow for error detection and / or error correction. In the memory (e.g., VMEM), error detection can be used, and / or error correction logic can be implemented on the read back data.

[0249] Thus, the load cache can include tag bits in tag storage 706 for way 0 and way 1, each of which can include 11 bits of address and 1 bit of valid. The load cache can also include 1 bit of pLRU, and data bits in data storage 704 for way 0 and way 1, each of which includes 16 bits of data and 2 bits of parity. The load cache is enabled for lookup at Dl stage. To minimize power consumption, only the load cache of the bank 702 involved in the load can be enabled. For example, for a single vector load, only 16 of the 32 load caches can be looked up. Upon a load hit (e.g., where the load cache includes the data to be accessed), the read enable of the given bank 702 can be suppressed, thereby preventing the bank 702 from lighting up. The pLRU 720 can also be updated at Dl stage. At D2 stage, the data and parity bits can be read from the load cache hit way and multiplexed with the RAM result.

[0250] Upon a load cache miss, at Dl stage, in a victim fashion, the existing entry to be evicted to make room for the new entry can be determined based on the valid bit and pLRU. The tag of the victim way can then be updated with the miss address, and the read enable 708 of the bank 702 can not be suppressed. At D2 stage, the data / parity from the bank 702 is not only sent to the read data crossbar, but also fills the evicted cache line with data. The store can also be looked up in the load cache when enabled and involved. A store hit can cause a hit way invalidation, and a store miss can be ignored.

[0251] Upon a hit in the load cache, power is saved in reading the bank 702. On the other hand, a miss in the load cache not only causes power to read the bank 702, but also consumes power to look up the load cache to fill the victim way. Since not all types of memory access patterns get a high hit rate in the load cache - especially when accessing a supergroup in indexed addressing mode - only vector linear loads can be looked up in the load cache.

[0252] When enabled, all stores can be looked up in the load cache to ensure that the load cache is never out of sync with the data in, for example, the VMEM bank 702. For a given supergroup, software can be used to disable the load cache of the bank 702 of that supergroup to minimize store lookup capability, as described in more detail below.

[0253] For example, in some embodiments, the use of the data cache can not provide a benefit. For example, in operations where the access pattern does not repeat, the data cache can not be useful, so the additional task of checking the cache before reading can waste time and / or effort because it can be necessary to read the database to access the correct data. Thus, the load cache can be enabled or disabled, thereby reducing the power loss due to access patterns with high load cache miss rates, and also allowing the load cache to be used for access patterns where the data cache can save power. In some embodiments, the enabling or disabling can be programmed using application code, so the programmer can program the code to enable the data cache when needed and to disable the data cache when not needed. In other embodiments, the enabling or disabling can be performed by hardware analyzing the read patterns and detecting overlapping patterns. For example, for a threshold amount of overlap between consecutive read operations, the hardware can enable the load cache. However, in cases where the overlap is less than the threshold, the load cache can be disabled. As non-limiting examples, the threshold can be 25%, 40%, 50%, 75%, or a different threshold amount of overlap between reads.

[0254] When the load cache is disabled, and with reference to the diagram as FIG. 7F shown, the tag memory 706 can not be accessed, and the read enable 708 can be set so that a read to the RAM bank 702 is enabled for each read. Similarly, the data memory 704 can not be accessed, and the read data multiplexer 712 can always pass the data of the RAM bank 702 to the stage flip-flop 716.

[0255] Further, in some embodiments, the memory bank structure can include multiple super-banks - for example, three super-banks - and each super-bank can enable or disable the load cache according to the particular access pattern within each super-bank. For example, in cases where three super-banks are used, each super-bank can include 32 RAM memory banks, and the data cache for each memory bank can include two entries, where each entry is one word, so 16 bits. In cases where two or more super-banks are used, the super-banks can be any size, different sizes, the same size, or a combination thereof. For example, the first super-bank can be 128 KB, the second super-bank can be 256 KB, and the third super-bank can be 512 KB.

[0256] Reference is now made to the diagram as FIG. 7FAs illustrated, each block of the method 750 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 750 can also be embodied as computer-usable instructions stored on computer storage media. The method 750 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. In addition, the method 750 can be performed by any of the systems, structures, or components described herein or any combination of systems, structures, or components, including but not limited to those described herein.

[0257] FIG. 8A is a flow diagram illustrating a method 750 for using a data cache for read operations, in accordance with some embodiments of the present disclosure. At block B702, the method 750 includes receiving data representing a memory read address. For example, after a first read operation using some memory banks, a second read operation can be performed that includes one or more memory banks in addition to one or more additional or other memory banks. Because the first read operation can have included storing the output of the read in a data cache corresponding to each respective memory bank, these values can be reused rather than requiring another read of the memory banks. As such, a memory read address corresponding to a next read operation can be received, and a load cache— when enabled— can be accessed to determine whether any data is stored in the load cache.

[0258] At block B704, the method 750 includes comparing the memory read address to load cache memory addresses corresponding to previous memory reads stored in the load cache. For example, after a previous memory read, data from the memory read can be stored in a load cache corresponding to a particular RAM bank 702. To remember this information, the tag memory 706 can include one or more previous memory addresses corresponding to reads from the RAM bank 702.

[0259] At block B706, the method 750 includes determining that the memory read address at least partially overlaps with the load cache memory addresses. For example, the memory read address can be compared to previous memory read addresses of previous reads stored in the tag memory 706. If there is a hit, then the load cache can be used to read out at least some of the data corresponding to the memory read address of the current memory read.

[0260] At block B708, the method 750 includes reading at least a portion of the data corresponding to the memory read address from the load cache. For example, due to the hit in the load cache determined from the tag store 706, the portion of data from the overlapping memory address can be read out of the load cache and the remaining portion of the data, if any, can be read out of the RAM bank 702.

[0261] Decoupled configurable accelerator

[0262] To optimize the performance of a processor for a particular application, such as a real-time application, the instruction set architecture (ISA) can be enhanced to create custom instructions to speed up common operations. This allows the processor to reduce the number of cycles required to perform a particular task. The process of implementing a custom ISA is performed until the performance goals of the system are met. However, these new instructions are added to operate on data in the processor's register file, or directly as memory as an operand, and these instructions are executed using the existing processor controller as well as the existing memory addressing and access hardware. In such examples, it is desirable for the new instructions to fit the processor's register file read / write operand count (e.g., reuse existing ports), fit the register file width (e.g., fit the processor's data types), and fit the processor's pipeline stages. Due to these requirements for successfully adding instructions to the ISA, the flexibility of adding new instructions is limited. Furthermore, as the ISA is created for a processor with a multi-stage (e.g., 30, 40, 50, etc.) pipeline, the configuration of the ISA becomes complex.

[0263] Further, processors provide a high degree of flexibility at the expense of power consumption - as each added instruction requires fetch, decode / dispatch, read / write register file and / or memory, etc. Thus, adding additional functional units to implement these custom instructions increases the stress on the register file read / write ports, resulting in required area (e.g., additional read / write ports can be required) and power (e.g., additional load register files can be implemented). Further, processing pipelines for embedded applications often have multiple stages - with the output of one stage feeding the input to the next stage. Techniques such as executing multiple threads (e.g., for different processing stages) in the processor can reduce the expansion time, providing reduced latency. However, multi-threading is at the expense of hardware - instructions must be fetched / decoded / scheduled from multiple threads, state information must be maintained for each state of each thread (e.g., in a register file), and control logic is included to handle multiple threads in the processor. This results in increased area and power requirements, while making verification and programming of the processor more complex. Thus, while various methods exist for reducing latency in processing pipelines, existing methods require additional surface area of the processor hardware, require additional power consumption due to the additional hardware, and increase the complexity of programming the processor to perform various tasks.

[0264] To address the limitations of a host processor configuration and the deficiencies of a multi-threaded processor, the system and method of the present disclosure uses a host processor or one or more units of the host processor - e.g., a single-threaded processor such as a VPU - in conjunction with a domain-specific accelerator or co-processor - e.g., a vector memory (VMEM) - that is decoupled from the host processor and communicates with the host processor through shared memory. In this way, the accelerator can operate as a subunit of the host processor, but once configured, the accelerator can execute independently of the host processor's instructions, rather than requiring processor instructions to execute. For example, accelerator access instructions can be used to allow the host processor to configure and order the accelerator, and shared memory can allow for sharing of inter-stage data structures between the host processor and the accelerator. Once the host processor initiates or turns on the accelerator (e.g., through a common accelerator interface, and using one or more load / store instructions), the host processor is free to process different stages (thereby providing the ability to work on multiple stages of a processing pipeline simultaneously and reduce run-time) or transition to a low-power or minimal-power state while waiting for the accelerator to complete processing (e.g., to minimize power consumption when not actively processing). In this way, each of the one or more accelerators can operate independently and concurrently with the host processor once configured by the host processor. The host processor and the accelerator can synchronize during processing through a handshaking interface so that the host processor knows when the accelerator has completed processing and / or is ready to perform a new task, or vice versa. Shared memory can store configuration messages (e.g., for configuring the accelerator when configuration instructions cannot be effectively sent through the accelerator interface due to size limitations), input buffers (e.g., to store data for the accelerator to process), and / or output the results of the accelerator (e.g., after processing is complete, data from the accelerator - e.g., a register file - can be stored back to a location in shared memory indicated from the host processor's configuration instructions). Thus, once triggered, the accelerator can read configuration parameters and / or input data structures from shared memory and can write output result data structures to shared memory.

[0265] As a result, this combined system of a host processor, shared memory, and decoupled accelerator allows for the flexibility of a programmable host processor while achieving the power consumption levels of fixed-function hardware (e.g., because high computational processing stages of a processing pipeline can be implemented as accelerators) without requiring a significant increase in complexity of the host processor (e.g., because the host processor can only need additional accelerator configuration or access instructions to program the accelerator). For example, the pipeline and data types (e.g., data width) of the accelerator can be independent of those of the host processor, thereby allowing for further customization and optimization that can only be achievable by a host processor that requires instructions to adapt to the register file read / write operand count, register file width, and pipeline stages of the processor.

[0266] In some embodiments, the accelerator and the main processor can be coupled at instruction execution time in order to enable some power savings of the accelerator when coupling execution to the main processor pipeline. However, in such embodiments, the ability to concurrently process different stages of the pipeline will be reduced as instructions will be interleaved between the accelerator and the main processor. In one or more embodiments, the accelerator and the main processor can be coupled through a higher level second level (L2) memory rather than through a shared memory connection. However, in such embodiments, the higher level decoupling (e.g., removing coupling through shared memory to the higher level) can increase communication overhead with the main processor.

[0267] The decoupled accelerator can be used for any task within any domain, such as, for non-limiting examples, performing ID, 2D, etc. lookups as a decoupled lookup table accelerator to detect and resolve memory bank conflicts, performing ID / 2D interpolation, etc., for computer vision algorithms such as feature tracking, object tracking, image warping, pyramid creation, etc., for sensor processing such as matrix multiplication or other operations on LiDAR data, RADAR data, and / or the like, for machine learning or deep learning applications. Thus, the topology described herein can be applied to any processing pipeline where a portion of the processing can be offloaded to an accelerator.

[0268] Depending on the implementation, there can be any number of decoupled accelerators on one or more chips that communicate with one or more main processors through shared memory. For example, a system on a chip (SoC) or other integrated circuit (IC) can include a main processor and one or more accelerators that a programmer can be aware of and write instructions or code that use the accelerators, which can improve performance of any of a variety of tasks of the system. Although the main processor is primarily described as a VPU, this is not intended to be limiting and the main processor can include any processor type such as a CPU, GPU, DPU, or other processor without departing from the scope of the present disclosure.

[0269] Reference is now made to FIG. 8A , FIGS. 13A-13DA system 800 including one or more decoupled accelerators is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. In addition, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For instance, various functions can be performed through execution of instructions stored in memory by a processor. In some embodiments, system 800 can be included in and / or can include components, features, and / or functionality similar to those of FIG. 14 example autonomous vehicle 1300 of FIG. 1, FIG. 15 example computing device 1400 of FIG. 2, and / or FIG. 8B example data center 1500 of FIG. 3.

[0270] System 800 can include a processor 802 (e.g., a main processor), such as a VPU, CPU, GPU, DPU, etc., a decoupled accelerator 804, and / or a shared memory 806 (e.g., a vector memory or VMEM). Processor 802 can be coupled to an instruction cache (I-cache) 810, which can cache instructions for execution by processor 802. Processor 802 can include general purpose input / output (GPIO) 808 (e.g., a digital signal pin on an IC that can be used as an input, output, or both, and can be controllable at runtime), and an IC configurator 812. In some embodiments, as shown, processor 802 can use an advanced extensible interface (AXI) for on-chip communication, such as but not limited to a 256-bit AXI interface. IC configurator 812 can be used to configure system 800.

[0271] The processor 802 can communicate directly with the decoupled accelerators 804— e.g., via a coprocessor or accelerator interface, such as an Advanced Peripheral Bus (APB) interface, and / or a handshake, program, or event interface. For example, the processor 802 can configure the accelerators 804 using an accelerator interface (or configuration bus), initiate or trigger processing of the accelerators 804 using an event interface, and synchronize with the accelerators 804 using a handshake or event interface. As such, each accelerator 804 can include a mechanism configured to communicate with the processor 802 over a respective accelerator interface or configuration bus. For example, when processing is complete, the accelerator 804 can indicate as much to the processor 802 through a handshake mechanism, or when the processor 802 is waiting for the accelerator 804 to complete processing, the processor 802 can periodically poll the accelerator 804 to request status or an end time. In some embodiments, the accelerator interface can include a 32-bit interface (or other smaller size interface) such that configuration instructions can be transferred to the accelerator 804. However, in some embodiments, configuration messages can be large (e.g., greater than 32 bits, or some multiple thereof), and the configuration messages can instead be stored in the shared memory 806, and the accelerator 804 can be sent the location of the configuration information in the memory 806 over the accelerator interface to indicate where to retrieve the configuration information.

[0272] The configuration bus can thus configure the accelerator 804, and the event (or programming interface) can be used to allow the processor 802 to trigger or initiate processing by the accelerator 804. Once triggered or initiated, the accelerator 804 can operate on its own, with the processor 802 waiting for the processing to complete and / or performing different processing tasks or phases. For example, an application programmer can program the processor 802 and the accelerator 804 knowing what each is capable of, and thus can split the application into multiple parts - some parts for the processor 802 and some parts for the accelerator 804. Thus, in embodiments, processing can be performed in parallel between the processor 802 and the accelerator 804 to reduce run time and increase efficiency. Configuration messages - shared through the accelerator interface and / or through the shared memory 806 - can be generated by the processor 802 and used to indicate to the accelerator 804 where to start in the shared memory 806 for data to be processed, how much data to process, and where to write the results back into the shared memory 806. The processor 802 can generate an input buffer at a specified location in the shared memory 806 that includes data operations for the accelerator 804. Once the configuration message is sent and the input buffer is stored in the shared memory 806, the accelerator 804 can receive a trigger signal from the processor 802 through the event interface (e.g., programming interface), and the accelerator 804 can be processing data. Once the accelerator 804 is triggered, the processor 802 can then perform other work or enter a low power state, and once the accelerator 804 completes processing, the accelerator 804 can indicate the same to the processor 802 and can wait for additional work.

[0273] The processor 802 can set up an input buffer or input data structure for the accelerator 804 to process and store to the memory 806. The accelerator 804 can be configured by the processor 802 using load / store operations, with the processor 802 dedicated to configuring the accelerator 804 and communicating with the accelerator 804. The configuration message can configure various registers of the accelerator 804 (e.g., 256 32-bit registers in one embodiment). For example, for a decoupled lookup table accelerator (as described in more detail herein), the configuration information can indicate that the lookup is a ID lookup with interpolation, a 2D lookup with bilinear interpolation, and / or another type of lookup. The accelerator 804, once it knows the particular mode or function, can configure the registers to correctly read data from the memory 806, process the data, and write the data back to the memory 806.

[0274] In some embodiments, the processor 802 can configure the accelerator 804 to perform multiple tasks at once to improve efficiency. For example, where the accelerator 804 is to perform various smaller tasks, individually configuring the accelerator 804 can increase run time, as each task can be completed quickly, requiring the processor 802 to stop processing, configure the accelerator 804 for another task, and so on. To this end, a first task message can include an address for a second task message, allowing for self-linking of multiple tasks. In this way, the processor 802 can generate configuration messages for multiple tasks at once, and generate configuration information and input buffers for each task, such that the accelerator 804 can perform the various tasks consecutively before indicating to the processor 802 that processing has completed and the accelerator 804 is ready to receive more work. Furthermore, to improve efficiency, the accelerator 804 can be configured to overlap tasks, such that when a first task is nearing completion, the accelerator 804 can begin decoding and configuring registers for a next task. Ultimately, through the inclusion of separate instructions for the processor 802 and the accelerator 804, the accelerator 804 can be able to operate on data formats or types that differ from what the processor 802 will support. This can be a result of the architecture and layout of the accelerator 804’s registers differing and being specialized for particular processing tasks.

[0275] In embodiments, the processor 802 can communicate with the shared memory 806 through any number of memory interfaces (e.g., a 512-bit static random access memory (SRAM) interface). Similarly, as shown, the accelerator 804 can communicate with the shared memory 806 through any number of memory interfaces (e.g., a 512-bit SRAM interface). An arbiter 814 can decide which of the processor 802 and / or the accelerator 804 is allowed to access the shared memory 806 for each cycle.

[0276] Referring now to FIG. 8A The blocks of the method 850 described herein each include a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 850 can also be embodied as computer-usable instructions stored on computer storage media (e.g., memory). The method 850 can be provided by a standalone application, a service or hosted service, an add-on to another product, or a plug-in to another product, to name a few. FIG. 8B The method 850 is described with respect to the system 800, but the method 850 can be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0277] FIG. 9Ais a flowchart illustrating a method 850 for using a decoupled accelerator, in accordance with some embodiments of the present disclosure. At block B802, the method 850 includes receiving configuration information for one or more first processing tasks of a processing pipeline. For example, the accelerator 804 can receive configuration information from the processor 802 (e.g., via a configuration message of an accelerator interface).

[0278] At block B804, the method 850 includes configuring one or more registers of the accelerator based at least in part on the configuration information. For example, the accelerator 804 can configure the one or more registers based on the configuration information.

[0279] At block B806, the method 850 includes reading data from an input buffer in memory based at least in part on an indication of a starting location of the input buffer included in the configuration information. For example, the configuration information can include an indication of where the input buffer is stored in the memory 806, and the accelerator 804 can read data from the input buffer into the registers.

[0280] At block B808, the method 850 includes processing the data from the input buffer to compute output data. For example, the accelerator 804 can process the data from the input buffer to generate or compute output.

[0281] At block B810, the method 850 includes writing the output data to memory at a location determined based at least in part on the configuration information. For example, the accelerator 804 can write the computed results out to the memory 806, and can indicate to the processor 802 that the processing is complete. The processor 802 can then use the output data to perform one or more second processing tasks of the processing pipeline.

[0282] Decoupled lookup table accelerator

[0283] Parallel processing is used to accelerate many computing tasks, including but not limited to: computer vision applications, deep learning applications, sensor processing applications, and / or other applications that benefit from parallelism (e.g., where processing tasks are independent of other processing tasks). For example, vector processors can operate on multiple elements in the same operation to achieve the efficiency required to execute these types of parallel processing algorithms in real-time while consuming low power. For example, a common operation for computer vision or deep learning tasks is to perform a lookup from a lookup table, image patch, or surface according to an index or coordinate location. To do this, a single vector load or store operation can be used to access data from multiple elements. Unless the index being looked up is regular (e.g., a continuous or fixed integer stride in a horizontal or vertical or depth direction), it results in random index access in memory.

[0284] To support regular but unaligned vector accesses from memory, a processor can use smaller banks of RAM to construct a vector memory. In this way, the hardware is able to create interesting addressing patterns for the vector memory by independently generating unique addresses for each bank of RAM. For irregular indexed vector load operations in memory, since the indices for different vector elements can be independent of each other, this can result in bank conflicts in one or more banks of memory. Bank conflicts can not be statically determined since they are data dependent, thus not allowing the compiler to schedule around bank conflicts.

[0285] In some conventional systems, to support irregular indexed vector load operations, various architectural designs can be implemented. For example, multiple read ports can be added to the bank of RAM. In such an example, if the hardware can handle 32 vectors, then each bank would need 32 read ports, which would increase cost, area, and power, and increase layout and routing congestion around the bank of RAM. Another example includes reducing the throughput of the index lookup to perform a single scalar lookup for each load. However, this creates a bottleneck for vector execution and becomes a limiting factor for execution time. Another example includes making multiple copies of the data structure in memory so that each vector lane can access data from a single bank. While this example can solve some of the throughput problems of the other methods, the memory capacity is limited by the data structure taking N times (where N is the number of entries to access) the space, which can result in a decrease in overall performance of the associated algorithm in addition to the overhead of making the copies. However, this method is more suitable in cases where the data structure is small. In some examples, conflicts can be dynamically detected and resolved by serializing the conflicting lookups. However, this can result in increased hardware complexity since the bank conflicts must be dynamically detected and resolved. Furthermore, these additional stages increase the load-use latency of these operations, which affects the compiler’s ability to effectively schedule the code. Additionally, data dependent execution latency can be introduced, which is a problem for the compiler to effectively schedule. In some examples, a combination of these methods can be performed.

[0286] To address these shortcomings of other architectures, the systems and methods of the present disclosure include a decoupled lookup table accelerator configured to support non-regular indexed vector load operations. The decoupled lookup table accelerator can be included as accelerator 804 of system 800 and can communicate with processor 802, e.g., a VPU, through shared memory 806. The decoupled lookup table (DLUT) can support a variety of modes for performing table lookups, such as a ID lookup mode, a 2D lookup mode, a 2D conflict-free lookup mode, a ID lookup with interpolation mode, a 2D lookup with interpolation mode, a table reformatting mode, and / or other modes. In any of the lookup modes, the DLUT can accept an index array in VMEM, which can be in ID (x) format or 2D (x, y) format. For example, each element can include 16 bits or 32 bits, which can be unsigned. The DLUT can then perform prescribed index computations, which can include 2D to ID mapping, truncation / rounding, integer / fractional split, and / or valid range detection, as non-limiting examples. For example, the DLUT can detect or merge duplicate reads, detect bank conflicts within an index, and issue read requests to VMEM to look up requested table entries. Each element can include 8 bits, 16 bits, or 32 bits, which can be signed or unsigned. The DLUT can then perform post-interpolation processing as configured and write the output back to VMEM. Each of these processing operations can be performed in a pipeline to increase throughput, reduce latency, and reduce power consumption.

[0287] As a result, the DLUT accelerator overcomes the deficiencies of implementing dynamic conflict detection and resolution in a processor pipeline, allowing the compiler to effectively schedule the deterministic execution latency of all memory operations while avoiding the complexity of performing inline conflict detection. Since the accelerator operates as a tightly coupled accelerator—e.g., through a shared VMEM with the VPU— the processor can configure and launch the accelerator while continuing to process other independent portions or phases of the processing pipeline or algorithm. In some embodiments, the accelerator can include additional features to further reduce the load on the host processor, such as offloading index generation for patches with specific lookup modes, performing optional ID mixing and 2D interpolation on data for lookups, and / or providing table reformatting support without lookups or interpolation. In practice, the entire system (including the processor 802 and accelerator 804 for performing lookups) has been shown to accelerate the processing of various computer vision algorithms (e.g., feature tracking, object tracking, image warping, pyramid creation, etc.) by a factor of two while reducing energy consumption by more than 50% compared to executing the entire algorithm on the host processor alone.

[0288] Reference is now made to FIG. 9A , FIG. 8AA system 900 including a decoupled lookup table (DLUT) accelerator is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the system 900 may be included in and / or may include a processor that is configured to perform a plurality of operations. FIGS. 13A-13D System 800, FIG. 14 Example of autonomous vehicle 1300, FIG. 15 The example computing device 1400 and / or FIG. 8A Components, features and / or functions similar to those of the example data center 1500 may be used.

[0289] System 900 may include one or more processors 902 (which may correspond to FIG. 8A processor 802), memory 904 (which may correspond to FIG. 8A shared memory 806) and a decoupled lookup table (DLUT) accelerator 906 (which can be used as FIG. 9BThe accelerator 804 of the accelerator 804 is included). In embodiments, the processor 902 can include a VPU and the memory 904 can include a VMEM. The DLUT accelerator 906 (or“DLUT 906”) can include a processing unit (PU) interface (I / F) 908 for communication with the processor 902, a controller 912 for communication with the processor 902, and a configurator 910 for configuring the DLUT 906 based on information shared across the PU interface 908 from the processor 902 and / or information from the memory 904 based on an indication from the processor 902 as to where in the memory 904 the configuration message or information is. For example, the PU interface 908 and the controller 912 can correspond to an advanced peripheral bus (APB) and an event or programming interface of the system 800, respectively. The controller 912 can receive a start or trigger command or signal from the processor 902 (e.g., by the arrow labeled“start”), indicating that the DLUT 906 can start processing and / or can receive a polling signal from the processor 902 to help synchronize the processor 902 with the DLUT 906. Further, when the DLUT 906 completes processing one or more assigned tasks, the DLUT 906 can generate a signal to the processor 902 (e.g., by the arrow labeled“done”), so that the processor 902 can start configuring the DLUT 906 for the next task.

[0290] During configuration, the processor 902 may configure the DLUT 906 directly via the PU interface 908 and / or indirectly by indicating the location of configuration information in the memory 904 via the PU interface 908. In the latter example, the DLUT 906 may retrieve the configuration information from the memory via, for example, the shared read port strm1_dm_rd, and may use the stored configuration information to configure the DLUT 906 (e.g., configuration subunits (e.g., IAUs, CDRUs, PPUs, etc.) and / or other components of the DLUT 906) for performing one or more tasks. For example, the processor 902 may set up data structures in the memory 904 that the DLUT 906 needs to perform the one or more tasks. For example, for a 1000-coordinate lookup, the processor 902 may set up a data structure in the memory 904 that has each of the 1000 coordinates and may further allocate a buffer in the memory 904 to which the DLUT 906 writes the output. Processor 902 can also instruct DLUT 906 which operations to perform—e.g., 1D or 2D lookups, with or without interpolation, table reformatting, etc.—and DLUT 906 can use this information to configure the subunits. Configuration information set by processor 902 can also include an indication of the bit width of coordinate indices, as well as an indication of the bit width of table entries, etc. Thus, once the input and output buffers are set up in memory 904 and configuration information such as bit width, operation type, etc. is sent to DLUT 906, processor 902 can initiate or trigger DLUT 906 to begin processing. Consequently, compared to a system relying solely on processor 902, processor 902 can perform other tasks while DLUT 906 performs lookups, interpolation, table reformatting, etc., thereby reducing runtime and improving efficiency.

[0291] In operation, the DLUT 906 may receive a list of indices corresponding to coordinates from the memory 904, and the DLUT 906 may extract a value from a table corresponding to the index (e.g., where the value is an integer value) and / or may pull surrounding values ​​of a fractional value (e.g., the left and right values ​​for a one-dimensional lookup or the upper left, lower left, upper right, and lower right values ​​for a two-dimensional lookup) and perform interpolation or other operations on the surrounding values. Once the final values ​​are determined (e.g., directly through a lookup without performing post-processing, or after processing by a post-processing unit (PPU) 930), these values ​​may be written to an output buffer in the memory 904, corresponding one-to-one to the index in the input buffer. To efficiently perform these tasks, in embodiments, an index address unit (IAU) 922, a conflict detection and resolution unit (CDRU) 924, a control (CTL) first-in, first-out (FIFO) 928, a fractional (FRAC) FIFO 926, for example, may be used by a post-processing unit (PPU) 930, a data merge unit (DCU) 932, and / or other components.

[0292] For example, an index (IDX) stream 916 can include an index stream read out of memory 904 (e.g., via read port strmldm_rd) that is to be looked up in one or more lookup tables, and the value corresponding to the index can be read out of memory 904 (e.g., via read port strmO_dm_rd) via a lookup table (LUT) stream 918. An output (OUT) stream 920 can be a value written back to memory 904 (e.g., by write port strmO_dm_wr) after processing using DLUT 906.

[0293] Processor 902 can indicate to IDX stream 916 how to access the data structure for the index during configuration. For example, for a one-dimensional lookup, where the interface to memory 904 is 64 bytes wide, 64 bytes can be read out in each cycle. In the case of performing a ID lookup, a single coordinate can be read for each index value (e.g., an (x) value), while for a 2D lookup, two coordinate indices can be read for each index (e.g., (x,y) values). In non-limiting embodiments, each index can be 16 bits or 32 bits, so in each 64 byte read, there can be 8, 16, or 32 coordinates from IDX stream 916.

[0294] The IDX stream 916 data can be sent to the IAU 922 as raw indices in raw format, and each coordinate can be an integer value or a fractional value. The IAU 922 (where the indices are fractional values) can split the fractional values to provide fractional bits to the FRACFIFO 926 to help blend the surrounding values found using the PPU 930 in the lookup table. The IAU 922 can then determine a set of indices to send to the CDRU 924, where the number of indices sent can correspond to the number of lookups the LUT stream 918 can perform in a single cycle. For example, if the LUT stream 918 can perform 32 lookups in one cycle (based on the bit width of each value in the lookup table), the IAU 922 can send 32 indices to the CDRU 924 at each iteration. In some examples, such as where the values from the IDX stream 916 are integer values, the IAU 922 can send each set of indices without any processing. However, where the values from the IDX stream 916 are fractional values, the IAU 922 can determine which indices need to be looked up to obtain each surrounding value needed (e.g., 2 indices for ID interpolation or 4 indices for 2D interpolation) to perform interpolation or other operations to obtain a blended value corresponding to the fractional value. For example, where the fractional value is (5.3, 6.2) corresponding to (x, y) coordinates for 2D lookup and interpolation, the IAU 922 can determine that lookups will occur at (5, 6), (5, 7), (6, 6), and (6, 7), and then the PPU 930 can blend these values to generate a final value corresponding to the index (5.3, 6.2). For example, the values can be blended equally weighted, or can be blended using bilinear interpolation such that the value closer to (5, 6) than (6, 7) is more heavily weighted to compute the final value for (5.3, 6.2).

[0295] The lookup set (e.g., 32 lookup indices that LUT stream 918 can read 32 values in each read cycle) can be sent to CDRU 924 in an appropriate order corresponding to the order of indices in the input buffer in memory 904 read using IDX stream 916. CDRU 924 then performs conflict detection and resolution by identifying the bank conflicts that would result if the lookup table reads in LUT stream 918 were to occur in the order received from IAU 922, and resolves the bank conflicts by changing the order of the indices to avoid the bank conflicts. For example, where the lookups of a set of indices would result in bank conflicts, and another set (e.g., later or earlier) of indices sets are available for another lookup cycle, CDRU 924 can find non-conflicting lookups from other lookup cycles and swap the non-conflicting lookups with the conflicting lookups of that cycle. As a result, one or more bank conflicts can be avoided, increasing throughput. For example, where IAU sends 32 indices per cycle, and 6 of the indices of a given cycle have bank conflicts, CDRU 924 can determine up to 6 indices from another lookup that will not result in a current lookup conflict, and can perform the 32 lookups - e.g., 26 lookups from the original 32 and 6 lookups from another set sent by IAU 922. Once the lookups are determined (e.g., with or without replacement to resolve conflicts), the lookup set can be read out from memory 904 using LUT stream 918.

[0296] To account for out-of-order lookups that occur substitutions, the CDRU 924 can use the CTL FIFO 928 to indicate to the data merge unit the order of lookups for each set of lookups from the IAU 922. For example, for an initial set of 32 lookups, the DCU can determine that 8 were performed in the first cycle, then 8 in another cycle, then 16 in another cycle, then can determine that the entire set of 32 has been processed, then can push the 32 lookups to the PPU 930 for post-processing if applicable, or can push them directly to the OUT stream 920 to be written to an output buffer in the memory 904. This additional information indicating the actual order of lookups determined by the CDRU 924 and read out to the indices of the LUT stream 918 can be communicated to the DCU 932 by the CTL FIFO 928. Thus, whatever changes the CDRU 924 makes to the order of the indices received from the IAU 922, the DCU 932 can account for. The CTL FIFO 928 can be useful because the number of cycles through the IAU 922, CDRU 924, etc. is indeterminate and data dependent. For example, because collisions are not known ahead of time (e.g., because the data can be non-deterministic), and there is no perfect solution to avoid collisions because they are a result of the programming, the CTL FIFO 928 helps indicate to the DCU 932 to organize the lookups as a result of the collision resolution.

[0297] The PPU 930 can compute a final value for each index that can be read out to the memory 904 when needed, e.g., in cases where additional operations are needed to be performed on the lookup table values. In cases where no post-processing is needed, the PPU 930 can not need to collect results. For example, in cases where a normal ID or 2D lookup is performed on indices that directly map to integer values in the lookup table, the PPU 930 and FRAC FIFO 926 can not be used to perform additional processing. In performing interpolation (e.g., linear on ID lookups or bilinear on 2D lookups) and / or other operations, the PPU 930 and FRAC FIFO 926 can be used to convert the collected results into updated results or values to be written out to the memory 904.

[0298] In some embodiments, DLUT 906 can be used for table reformatting mode. For example, IDX stream 916 and OUT stream 920 can be used to update addresses for access and / or transposition. In such examples, where there is a buffer in memory 904 and the indices in the buffer are to be transposed, this operation can be offloaded to DLUT 906 (rather than having the address generation unit of processor 902 perform the transposition). Configuration information from processor 902, e.g., from the address generation unit, can indicate a read pattern for reading from the buffer in memory 904 and a write pattern for writing addresses back to memory 904 in a different pattern. For example, where the programmer knows that many conflicts will result from a particular access pattern, the programmer can program processor 902 to configure DLUT 906 to perform table reformatting to scramble the data so that fewer or no conflicts can occur.

[0299] As another example, DLUT 906 can be used to return out-of-range sentinel values, or out-of-range prediction to shut off output writes. Thus, for example, where coordinates in IDX stream 916 are outside of a given image block and should not write a corresponding value, DLUT 906 can write out a sentinel value, which can indicate to processor 902 when information in the output buffer is not to be relied on or used in processing the sentinel value. In some embodiments, this sentinel value can indicate to processor 902 that these values are not to be written to memory, so values that are identified as error values can not be stored.

[0300] Thus, DLUT 906 can be implemented as a pipeline of sub-units that work together to perform a particular task or operation. Each sub-unit can run independently and communicate with other sub-units through a shared interface. With respect to FIG. 9C In particular, table 940 illustrates the tasks of various sub-units of DLUT 906 during processing of a particular operation.

[0301] Due to the DLUT accelerator described herein, the processor pipeline can maintain determinism by offloading dynamic collision detection and resolution to a decoupled accelerator. Further, the accelerator can be able to run independently and concurrently with the host processor (e.g., VPU), thereby reducing runtime. The DLUT accelerator can also allow for 1D and / or 2D lookups from one common table with collision detection / resolution. The accelerator can perform various post-processing operations, such as 1D lookups with linear interpolation, 2D lookups with bilinear interpolation, out-of-range detection sentinel return (1D and 2D), and / or out-of-range prediction turn off output write (1D and 2D). The DLUT accelerator can be configured to perform interpolation using a configurable number of fractional bits, and can support various index and data formats, such as 8, 16, and 32-bit signed and unsigned data formats and 16 and 32-bit 1D and 2D coordinate index formats. The DLUT accelerator can also use configurable X / Y offsets to convert between global and local coordinates. The DLUT accelerator can also support dataflow units to read index buffers from VMEM, perform lookups from VMEM, and write results (or lookups or interpolations) to VMEM. The dataflow units can support up to 2D addressing for linear and transposed accesses. To optimize the number of cycles required for lookups / interpolations, the lookup indices can be reordered to minimize memory bank conflicts—e.g., if the VMEM supports N lookups, the accelerator can use MxN indices to maximize what can survive in collision detection—and can perform duplicate detection to filter out duplicate indices that are guaranteed to cause conflicts. Further, the 2D lookup and interpolation modes of the DLUT accelerator can include indices that are automatically generated within the accelerator according to several parameters (referred to as automatic index mode), as opposed to the programmer providing the index data block. This offloads the preparation of the indices from the host processor to the accelerator.

[0302] Referring now to FIG. 9A , each block of the method 950 described herein includes a computational procedure that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be carried out by a processor executing instructions stored in memory. The method 950 can also be embodied as computer-usable instructions stored on computer storage media. The method 950 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. Further, the system 900 described with respect to FIG. 9C , the method 950 can be executed by any one of the systems, structures, or components or any combination of systems, structures, or components, including but not limited to those described herein.

[0303] FIG. 10Cis a flowchart illustrating a method 950 for using a decoupled lookup table accelerator, according to some embodiments of the present disclosure. At block B902, the method 950 includes configuring one or more subunits of a DLUT accelerator based at least in part on configuration information generated using a processor. For example, the DLUT 906 can use information received from the processor 902 and / or retrieved from the memory 904 to configure subunits of the DLUT 906.

[0304] At block B904, the method 950 includes reading a first set of indices from a stream of memory, determining a first subset of indices that have no bank conflicts. For example, the IAU 922 can generate a set of indices for the CDRU 924 to process conflicts, and the CDRU 924 can determine a subset of the set of indices that have no memory bank conflicts.

[0305] At block B906, the method 950 includes determining a second subset of indices from the second set of indices read from the stream of memory that do not have a bank conflict with the first subset of indices. For example, the IAU 922 can generate another set of indices for the CDRU 924 to process conflicts, and the CDRU 924 can determine to replace one or more indices from the first set that have conflicts with one or more indices from the second set that do not cause conflicts with the first set of indices.

[0306] At block B908, the method 950 includes performing lookups of one or more lookup tables using the first subset of indices and the second subset of indices in a single read cycle from memory to retrieve a plurality of values. For example, the DLUT 906 can read values from the memory 904 into the LUT stream 918 using a subset of values from the first set of indices and a subset of values from the second set of indices that were determined to have no conflicts with the subset of values from the first set of indices.

[0307] At block B910, the method 950 includes writing the plurality of values to memory. For example, the values from the LUT stream 918 can be written to the memory 904 in the output stream 920. Before being written out, the DCU 932 can reorganize the data so that the data is in one-to-one order with the read out indices of the input buffer of the IDX stream 916. In some embodiments, the PPU 930 can perform one or more operations on the retrieved values, such as interpolation, before writing the final values to the memory 904 in the OUT stream 920.

[0308] Hardware sequencer for a direct memory access system

[0309] Direct memory access (DMA) systems can be used to move data from different memory locations without the need for a central processing unit (CPU). For example, a DMA can operate as a data movement engine to move data from a source to a destination - e.g., from a source such as external memory (e.g., DRAM) or internal memory such as an L2 buffer or a vector memory (VMEM) of a vector processing unit (VPU) to a destination such as a VPU. DMA systems can perform additional operations in practice, such as but not limited to padding frame data, manipulating addresses, managing overlapping data, managing different traversal orders, and considering different frame sizes.

[0310] In digital signal processing, structured tile data movement between external memory and a processor (e.g., VPU) can be described using multiple DMA resources. For example, these DMA resources can include descriptors, channels, triggers, and / or registers. For example, a descriptor can describe tile movement such as source location, destination location, inter-row spacing, tile width, tile height, circular buffer arrangement, etc. However, tile data movement with spatial and temporal dependencies of an image surface presents additional programming model challenges to a user and requires many different DMA configuration resources. These tile data dependencies can also complicate control code and control sequences in processor (e.g., VPU) code. For example, a typical processing operation can include a filtering, such as a 3x3 filtering. This type of operation introduces spatial dependencies, as each output pixel will depend on the respective values of the 3x3 pixels around the output pixel. In such an operation, a 3x3 matrix of values can be used to perform the filtering, and the operation can be referred to as a spatially dependent operation. In practice, each tile of a frame can have the same size - e.g., 64x64 - to reduce programming challenges. However, if a 3x3 filter is used on a 64x64 tile, then adjacent tiles will require additional pixels up and down - e.g., as shown by the shaded areas of FIGS. 10A-10G Thus, this information needs to be encoded in the DMA resources to allow for proper data fetch across tiles - which results in an additional programming burden to complete.

[0311] Reference is made to FIGS. 10A-10G , FIG. 10A Various challenges of data movement when using a DMA system are illustrated. For example, FIG. 10AThe visualization 1000 can correspond to a padded frame data. In the visualization 1000, there can be nine sections, a top-left section, a top section, a top-right section, a left section, a center section, a right section, a bottom-left section, a bottom section, and a bottom-right section. In such an example, each section can include one or more tiles - for example, the top-left section can include one tile, while the top section can include, for example, four tiles. Thus, to accurately define the segmentation, in existing approaches, nine descriptors (e.g., one for each section), three channels (e.g., one for the left column, one for the center column, and one for the right column), and three triggers (e.g., one for each channel) can be used.

[0312] With respect to padding, for example, due to spatial dependencies, when performing operations on data near the boundaries of tiles or sections of a frame, the DMA system can pad values or manufacture values for pixels outside the boundaries of the image. This can be because, in certain implementations, requesting data outside of the memory region for an image can trigger a fault. Thus, the DMA can be used to pad or manufacture values after acquiring image data from the corresponding memory region to avoid triggering a fault. If there is no padding, the structure of the data can not match the kernel size, for example, if a filtering operation is performed. The acquired data with the additional padded values can then be sent to a destination (e.g., a VPU) so that the VPU can process the data according to its configuration, and the data can be processed in the same way across the (padded) frame. When padding, zero padding can be used (e.g., where each new data point includes a zero value), a repeating value can be used (e.g., copying the pixel value of an adjacent pixel from the acquired data), and / or another padding mechanism can be used. Further, padding can be added to any side of the frame, and different padding can be added for different sides. For example, in FIG. 10B Padding increases the complexity of DMA programming when moving data from a source to a destination (e.g., from memory to VMEM), and also increases the complexity of VPU programming when processing larger padded frames.

[0313] Referring now to FIG. 10B , FIG. 10BVisualization 1010 corresponds to address manipulation of the DMA system. For example, different descriptor addresses can be manipulated and programmed to fetch consecutive frame data. In order for the DMA to effectively execute, the address descriptions of the data movement can be consecutive. Thus, the address of each descriptor can be manipulated and this manipulation must be passed from one descriptor to another. For example, when padding values are filled as shown, the start address of each descriptor can be manipulated to cause the fetched data to include padding values. To do so, the programmer uses the start address and tile width and the number of tiles in each section and uses this information to generate the next descriptor address. For example, the first descriptor can cause the fetching of data starting from the top left corner, followed by the top, followed by the top right corner, followed by the left, followed by the center, and so on, as indicated by the arrows in FIG. 10C However, the start descriptor address adds complexity to the DMA programming when moving data to the destination (e.g., VMEM).

[0314] As another example, and with respect to FIG. 10C In order to ensure consecutive data processing, the DMA system can need to read vertically and horizontally overlapping data from adjacent tiles. For example, as indicated by the shaded areas in FIGS. 10D-10F , the DMA system can need to read overlapping data from the top left tile and the adjacent tile in the top in the same operation. Similarly, the DMA system can need to read overlapping data from the top left tile and the adjacent tile in the left in the same operation. To do so, the descriptors need to be updated or moved to include the overlapping sections. For example, the base descriptor can include an address at the start of the top, but to capture data from the adjacent tile of the top left tile, the descriptor for the top needs to be updated (e.g., moved to the left) to capture data from the top left tile. This updating requires additional programming complexity, especially as the number of descriptors increases.

[0315] Further, with respect to FIG. 10D The DMA system can need to support different traversal orders in order to read data from memory in a consecutive manner. For example, whether performing filtering, convolution, matrix multiplication, and / or other operations, the associated traversal order can be different. With this in mind, various traversal orders can be supported, such as those shown in FIG. 10E , including a raster traversal order starting from the top left (visualization 1030), a raster traversal order starting from the top right (visualization 1032), a raster traversal order starting from the bottom left (visualization 1034), and / or a raster traversal order starting from the bottom right (visualization 1036). Similarly, with respect to FIG. 10F visualization 1038, for cube images, the DMA system can support various cube traversal orders. FIG. 10GVarious vertical digging traversal orders that can be supported by the DMA system are shown, such as a vertical digging traversal order starting from the top left (visualization 1040), a vertical digging traversal order starting from the top right (visualization 1042), a vertical digging traversal order starting from the bottom left (visualization 1046), and / or a vertical digging traversal order starting from the bottom right (visualization 1048). To support each of these different traversal orders for moving data to memory (e.g., VMEM), the complexity of DMA programming is increased.

[0316] about FIG. 10A The DMA system may also need to support different frame sizes, such as moving multiple frames of different sizes (e.g., Luma / Chroma composites or different pyramid levels). For example, a processor (e.g., a VPU) may process frames of different sizes to generate the final desired output. FIG. 10I An example visualization 1048 of pyramid processing corresponding to a frame for an optical flow estimation operation is illustrated. In such an example, the movement of pixels can be first calculated for a smaller frame size, then calculated for a larger frame size using hints from the output of the smaller frame size, then calculated for the larger frame size using hints from the larger frame size, and so on. Thus, the DMA system can support the acquisition of frame data for various frame sizes, but this capability requires additional programming complexity for the DMA system. For example, descriptors must be programmed or updated for each different frame size.

[0317] To simplify the programming of these various operations supported by the DMA system, the DMA system and method of the present disclosure can use a hardware sequencer in conjunction with the DMA engine to solve the data movement problem. For example, the data movement of a complete image can be explicitly and completely described in a hardware sequencing mode, which has a simplified programming model that handles tile ordering (triggering), padding, overlap (offset), traversal order, and different frame sizes (e.g., the image structure of the frame, e.g., FIG. 10I The hardware sequencer can reduce DMA resource usage (e.g., reduce the number of descriptors, triggers, channels, etc. required), offload control of VPU control processing from the VPU, and reduce the complexity of DMA programming. This can be achieved by loading image or frame descriptor views from local programmable memory in the form of command sequences (e.g., as shown in the figure). FIG. 10H These hardware sequence commands can incorporate each of the operations that increase programming complexity, as described in this article—including image padding, tile overlap or offset, frame offset, image traversal order, and image size at tile granularity. In addition to descriptor information (e.g., from image commands or from a separate descriptor memory or SRAM), the hardware sequencer can also read image commands from memory and sequence tile movements to traverse and draw the entire frame.

[0318] Reference is now made to FIG. 10H , FIGS. 13A-13D A DMA system 1050 including a hardware sequencer is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted and / or subdivided. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by an entity can be performed by hardware, firmware and / or software. For instance, various functions can be performed by execution of instructions stored in memory by a processor. In some embodiments, system 1050 can be included in and / or can include components, features and / or functionality similar to those of FIG. 14 example autonomous vehicle 1300 of FIG. 15 example computing device 1400 of and / or FIG. 10H example data center 1500 of

[0319] System 1050 can include a DMA engine 1056, register control 1058, hardware (HW) sequencer controller 1060, descriptor SRAM 1052, and / or hardware (HW) sequencer command SRAM 1054. Existing systems can include only DMA engine 1056 and descriptor SRAM 1052 storing frame descriptors. Thus, as described herein, when sending data from a source to a destination, DMA engine 1056 previously needs to perform all padding, address manipulation, etc. operations and needs a VPU or other source to perform sequencing by handshaking with the DMA system (e.g., with the VPU as the primary node and the DMA as the secondary node). In such examples, DMA engine 1056 would process at the tile level, using descriptors for individual portions of a frame, each portion including one or more tiles, so as to retrieve one tile at a time for sending to the destination, and subsequent tiles would be retrieved based on descriptors based on an indication from the VPU to retrieve the next tile.

[0320] However, using system 1050 of FIG. 10H can process a frame at the frame level - e.g., a single descriptor can be used for FIG. 10I, which previously required nine descriptors. Therefore, in practice, when the DMA engine 1056 attempts to load a descriptor from the descriptor SRAM 1052 (or more generally, the descriptor memory 1052), the HW sequencer control 1060 can intercept the descriptor load and use the command sequence processing structure to handle multiple frames, tile rows / columns, and multiple descriptors. To do this, the frame format 1070 ( FIG. 10I ), which describes a higher-level frame by processing tile rows / columns (depending on the traversal order) in hardware rather than at the tile level. For example, instead of filling a tile, the frame format 1070 can be used to fill an entire frame, thereby filling many frames with a single fill command. Therefore, the entire frame can be understood, such as where to fill, where to overlap, how to automatically manipulate addresses, etc. In addition, because the DMA engine 1056 can directly extract descriptors from the descriptor SRAM 1052 without the intervention of the hardware sequencer control 1060, the legacy format can still be supported for operations that may not benefit from the HW sequencer control 1060.

[0321] The HW sequencer control 1060 can operate, for example, as a state machine that reads the HW sequencer command SRAM 1054 (or more generally, the HW sequencer command memory 1054), where a frame format 1070 including sequenced commands is stored. A processing controller—e.g., an R5 processor, a CPU, an ARM processor, etc.—can program or configure the hardware sequencer command SRAM 1054 and the descriptor SRAM 1052 using programming code and / or settings from a higher-level engine.

[0322] The descriptor SRAM 1054 may include one or more descriptors that may define tile dimensions (e.g., tile width dx and tile height dy), the starting point of the image or frame (e.g., top left, bottom right, etc.), trigger type, and / or other micro-information about the scan type of the descriptor.

[0323] The HW sequencer command SRAM 1054 can store frame format 1070 defining a frame as a whole, size of the frame, frame padding, etc. For example, the frame format 1070 can include frame header for header control, offset control, and padding control, and can include column header and / or row header for columns or rows of the frame (e.g., column header for vertical scan mode and row header for raster scan mode). The frame header control can include frame repetition factor to identify how many times a particular frame is to be repeated, and number of descriptor rows and / or descriptor columns. The frame header offset control can include frame tile offset (e.g., offset from tile to tile) and frame offset (e.g., offset between two or more frames that can be read using a single channel, e.g., YUV frames can be processed to include three separate planes). The frame padding header can indicate how many rows or pixels of padding are to be added at the frame level (as opposed to each tile level of existing approaches), e.g., padding the left side of the frame, top of the frame, right side of the frame, and / or bottom of the frame, filling the entire frame, rather than padding each tile within each section of the frame at the tile level.

[0324] Column headers can be used for vertical traversal order, while row headers can be used for raster or horizontal traversal order. The column and / or row headers can include column or row offset (e.g., offset between each column or row), column or row repetition factor (e.g., how many times the same column or row is repeated across the frame, e.g., N-1 times, where N is the number of times the column or row is processed), and number of descriptors for each column or row (e.g., a single descriptor can be used for the same tile repeated across rows or columns, or a first descriptor can be used for traversing one portion of a row, while a second descriptor can be used for traversing another portion of the row, and so on). The descriptor ID can be described such that the descriptor - e.g., stored in the descriptor SRAM 1052 - can be pulled out and used to describe the row or column. For example, the descriptor ID can indicate which descriptor to use for a particular column and / or row, and how many times the descriptor is repeated (e.g., N-1 times, where N is the number of times the descriptor is used in total). In embodiments, there can be a set of descriptors (e.g., 64), and the descriptor ID can be used to determine which descriptor should be used for a particular column and / or row. In this way, the hardware sequencer controller 1060 sees the upper level structure of the frame on top of the base descriptors from the descriptor SRAM 1052, which allows for simplification of the resources needed by the DMA engine 1056 to implement the same data transfer. Further, the hardware sequencer controller 1060 can prefetch tiles ahead of time (e.g., using the register control 1058) to reduce latency, and the tile data can be immediately available when requested by the DMA engine 1056.

[0325] In operation, the HW sequencer control 1060 can read the image structure (e.g., frame format 1070) from the HW sequencer command SRAM 1054 as well as the descriptor information from the descriptor SRAM 1052 and can combine this information to order the frames for the DMA engine 1056. Thus, the HW sequencer control 1060 can read the image structure, pull in the descriptors, and order the frames for the DMA engine 1056 with the correct descriptor format instead of requiring the DMA engine 1056 to be separately coded for each descriptor, trigger, channel, etc. In embodiments, the register control 1058 can help control the traversal order, prefetching, and / or other frame addressing controls. The HW sequencer control 1060 further simplifies the code for the VPU so that the VPU does not have to account for multiple channels. Instead, the VPU can request one tile, then the next tile, then the next tile, etc. The HW sequencer control 1060 knows the current position in the frame, so it knows the next tile to fetch for the DMA engine 1056, and the DMA engine 1056 does not have to track this information internally.

[0326] The system 1050 can thus be backwards compatible with previous approaches in that the system can still support the use of various descriptors, triggers, channels, etc., but can also understand at the frame level to reduce complexity. The system 1050 can support image padding at all corners of a frame with different pixel padding sizes, vertical and / or horizontal overlapping tiles to allow the VPU to access adjacent tiles for processing along tile boundaries, and traversal of the frame in different traversal orders. Further, the system 1050 can support automatic tile offset adjustment by the hardware sequencer control 1060 at the VMEM destination. Because the descriptors in the frame are linked by hardware, the user does not need to link or stitch the descriptors together. The hardware sequencer control 1060 can manage the address ordering of the descriptors / tiles across the frame without additional programming complexity, and the hardware sequencer control 1060 can prefetch tiles to improve performance.

[0327] In some embodiments, the descriptors can be included in the image or frame structure instead of being stored separately in the descriptor SRAM 1052. For example, without implementing legacy compatibility, the entire ordering structure and tile structure can be described in the framework structure. In such examples, FIG. 10J The frame format of FIG. 7 can be used to include additional information for the descriptors, such as tile width, trigger type, etc., to result in the same information being available to the HW sequencer control 1060 as when the descriptors are stored separately in the descriptor SRAM 1052.

[0328] Reference is made to FIG. 10J , FIG. 10I is a method of implementing a raster scan sequence according to some embodiments of the present disclosure FIG. 10K1070A is an example of a frame format in raster mode with frame address processing, using a single channel, a single trigger, and a single descriptor. In this example, the tile structure may be 16x8. FIG. 10J is an example of such a tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, where frame address processing is performed using example frame format 1070A. For example, for each tile row, the same descriptor (e.g., tile dimension) may be used (as indicated by "D1" in visualization 1072) so that the same tile is applied 16 times along each row (from C1 to C16) and then repeated for 8 rows from top to bottom (from R1 to R8). The sequence may include 20 bytes, as shown in frame format 1070A, and each row may have N*2+ bytes, where N represents the number of entries per row (as indicated by "D1" in visualization 1072). FIG. 10L ). Thus, to order the frames as shown in visualization 1072, frame format 1070A may include no frame repetition, the number of descriptor rows may be zero, no tile offset, no frame offset, a frame padding of 3 rows of pixels on the left (PL), right (PR), top (PT), and bottom (PB), the row may be repeated 7 times (for a total of 8 rows), each row may be offset by the tile height (Ty) (such that each row is offset by the tile height), one descriptor may be used with descriptor ID D1, and the descriptor may be repeated 15 times in each row (for a total of 16 times). Thus, in practice, the HW sequencer control may use the descriptor corresponding to D1 from descriptor SRAM 1052 (which includes the tile height and tile width), and may use the image structure from frame format 1072 stored in HW sequencer control SRAM 1054 to order the image tiles tile by tile (16 tiles per row), row by row (from R1 to R8) for the target processor (e.g., VPU). In this way, a single descriptor, a single trigger, and a single channel may be used, thereby reducing programming complexity while also allowing the DMA system 1050 to be the primary or controlling component in the interaction between the DMA system 1050 and the VPU.

[0329] In some embodiments, as an extension of the HW sequencer control 1060, DMA trigger mode can be used to reduce software intervention for VPU programming by having the DMA system 1050 command a sequence of descriptors. For example, the DMA system 1050 can read an image from external memory, tile the image, and sequence the tiles for the VPU. To facilitate this, the VPU can expose a start and done signal. The VPU start can be driven by the DMA system 1050, and the VPU can send a done signal to the DMA system 1050 when the VPU completes processing of an instruction block. Thus, the DMA system 1050 (e.g., hardware sequencer control 1060) and the VPU can participate in a handshake regime where the DMA system 1050 is the primary node and the VPU is the secondary node. This DMA trigger mode can minimize VPU tile control overhead and simplify the programming model for the DMA engine 1056. For example, specific code for double buffered DMA data movement can not be needed, and the DMA kernel code can be independent of the VPU kernel code. Thus, the DMA trigger mode simplifies the VPU code because the DMA system handles tile sequencing using the HW sequencer control 1060. The following example code illustrates the VPU code before and after DMA trigger addition.

[0330] Before:

[0331]

[0332] After:

[0333]

[0334]

[0335] As a result, in cases where the VPU has been requesting tile movement to VMEM all along, now, because the HW sequencer control 1060 controls sequencing, the DMA system 1050 can trigger tile movement to VMEM and VPU as targets. In this way, the DMA system 1050 can fetch data to be processed by the VPU ahead of time, and when the VPU indicates that processing is complete, the DMA system 1050 can have the next data to be processed immediately available (e.g., in VMEM) and can indicate the same to the VPU.

[0336] When processing one or more frames is performed, the HW sequencer control 1060 can retrieve one or more descriptors (which can indicate tile size, trigger type, etc.) from the descriptor SRAM 1052 and can retrieve the image structure from the hardware sequencer command SRAM 1054. The HW sequencer command 1060— in conjunction with the register control 1058— can then begin traversing the first row or column according to the traversal order and using the first (and only in embodiments) descriptor, and can then move to a second descriptor based on the number of repetitions encountered (e.g., 1-N) using two or more descriptors, and so on. When each tile is determined, the DMA engine 1056 can retrieve the tile data from the source data and write the tile data to the destination data (e.g., in VMEM). Once the data is written to the data destination, the processor (e.g., VPU) can be notified by the hardware sequencer control 1060 that the data is available for the processor to begin processing. Then, during processing, the DMA system 1050 can fetch the next data tile based on the sequence from the hardware sequencer control 1060 and write the data to the data destination such that when the processor indicates that processing is complete, the hardware sequencer control 1060 can indicate (by handshaking) to the VPU that the next data to be processed is available, and so on until processing is complete.

[0337] Referring now to the drawings FIG. 10H , each block of the method 1080 described herein comprises a computational procedure that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 1080 can also be embodied as computer-usable instructions stored on computer storage media. The method 1080 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service) or a plug-in to another product, just to name a few. Further, the method 1080 is described with respect to the system of FIG. 10L , the method 1080 can be executed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0338] FIG. 11A is a flow diagram of a method 1080 of a DMA system including a hardware sequencer in accordance with some embodiments of the present disclosure. At block B1002, the method 1080 includes retrieving a tile structure from a descriptor memory and retrieving a frame structure corresponding to a frame from a hardware sequencer command memory. For example, the hardware sequencer control 1060 can retrieve a descriptor from the descriptor SRAM 1052.

[0339] At block B1004, the method 1080 includes ordering the retrieval of the tiles from the frame in the source memory. For example, the hardware sequencer control 1060— in embodiments combined with the register control 1058— can order the tile retrieval from the source memory by the DMA engine 1056 according to the frame (or image) structure and the tile descriptions from the descriptor.

[0340] At block B1006, the method 1080 includes writing the retrieved data corresponding to the tiles to the destination memory. For example, the DMA engine 1056 can write the retrieved data corresponding to the tiles to the target memory (e.g., VMEM) for processing by the target processor (e.g., VPU).

[0341] At block B1008, the method 1080 includes providing an indication to the processor associated with the destination memory that the retrieved data is stored in the destination memory. For example, the HW sequencer control 1060 can indicate to the processor that the data for the next tile is ready for processing.

[0342] At block B1010, the method 1080 includes receiving an indication that the processing of the retrieved data is complete. For example, upon completion of processing, the processor can indicate to the DMA system 1050 that the processing is complete, at which point the next data tile can be loaded (or possibly already pre-loaded) to the destination memory, and the DMA system 1050 can indicate the same to the processor.

[0343] Configuring a DMA system for region dependent data movement using a VPU

[0344] When acquiring known data patterns, a processing controller can configure a direct memory access (DMA) system and a processor (e.g., a vector processing unit (VPU)) can trigger and order the DMA. However, when processing different data points or features of irregular or unknown data patterns, challenges can be introduced in reconfiguring the data movement as the feature or object locations are dynamically computed. For example, object tracking algorithms, feature tracking algorithms, object detection algorithms, deep learning algorithms using variable size regions of interest (ROIs), and / or other region dependent data movement algorithms require dynamic adjustment of the address and data pairs so that the DMA system can retrieve the appropriate information for the processor (e.g., VPU) to process. In traditional systems, when acquiring unknown data patterns— such as in object tracking— a processing controller (e.g., an R5 processor core used to control a programmable vision accelerator (PVA)) can need to interrupt to intervene in the processing cycle to determine the updated information computed by the processor (e.g., VPU) and reconfigure the DMA for the next iteration. Thus, the processing controller introduces additional latency to, for example, a tracking algorithm, which requires a shorter response time.

[0345] To address shortcomings of conventional systems that require handling controller intervention, the system and method of the present disclosure can use a DMA and a processor (e.g., VPU) to configure a tightly coupled processing loop that allows the DMA to reconfigure its descriptors based on the output of the processor. Thus, the DMA can be dynamically reprogrammed at runtime to handle certain algorithms that require region-dependent data movement. This VPU configuration mode can be used to update the DMA’s descriptors to track feature data (including locations) based on runtime VPU computations. Thus, the VPU can specify a list of address and data pairs in memory (e.g., VMEM) and then trigger the DMA to update its own descriptors to collect data from regions with new computed addresses. By relying on the interface between the VPU and DMA, no intervention by a processing controller (e.g., R5 or ARM processing core) can be required once the processing controller initially configures the VPU and DMA to start processing. This batched, fast, and synchronous MMIO access for updating the function descriptors thus reduces latency for object tracking, feature tracking, object detection, deep learning, and / or other algorithms with region-dependent data movement.

[0346] Reference will now be made to FIG. 11A , FIGS. 13A-13D A dataflow diagram 1100 of a process for configuring a direct memory access (DMA) system using a vector processing unit (VPU) is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted altogether. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be carried out by hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. In some embodiments, the process 1100 can be performed by a system including components, features, and / or functionality similar to the example autonomous vehicle 1300 of FIG. 14 the example computing device 1400 of FIG. 15 and / or the example data center 1500 of FIG. 11B .

[0347] The system executing process 1100 can include a processing controller 1102 (e.g., an R5 processor, an ARM processing core, an instruction set architecture (ISA), an X86 architecture, etc.), a direct memory access (DMA) system 1104, a vector processing unit (VPU) 1108 (or another processor type), a vector memory (VMEM) 1110 (or another memory type), and a descriptor RAM 1106. In fact, the VPU configuration mode can configure the DMA to fetch descriptors by writing a series of non-contiguous address / data pairs to the DMA descriptor SRAM. Process 1100 can be described with respect to an example feature or object tracking algorithm. However, this is not intended to be limiting, and process 1100 and the underlying system can be used to execute any type of algorithm, such as those with region dependent data movement.

[0348] For example, a first operation can include the processing controller 1102 configuring the DMA 1104 and the VPU 1108 to perform processing on some data, and then triggering both the DMA 1104 and the VPU 1108 to perform the processing. For example, the processing controller 1102 can load the descriptor RAM 1106 into memory for the starting point of the processing, and can configure the registers of the VPU 1108 for the specific type of operation to be performed on the data for the VPU 1108.

[0349] For a second operation, the VPU 1108 can trigger the DMA 1104 to read the initial feature data point in the VMEM 1110. For example, to start working, the VPU 1108 needs data from the DMA 1104, so the VPU 1108 configures the DMA 1104 to load the data point to the VMEM 1110 at a location that the VPU 1108 knows to retrieve data for processing.

[0350] In a third operation, the VPU 1108 can process the current feature data set, and calculate the next tracked object or feature location. As a result, the VPU 1108 can now have calculated a new or updated location of the tracked feature or object.

[0351] In a fourth operation, the VPU 1108 can update the VMEM 1110 with the updated location using the VPU configuration format (referenced in FIG. 11B Description) and can then trigger the DMA 1104 to update its descriptor in the descriptor RAM 1106. For example, FIG. 11B is a table 1120 illustrating a VPU configuration format written by a VPU into a vector memory (VMEM) and read by a DMA system in accordance with some embodiments of the present disclosure. For example, for each address / data pair, the format can include four bytes of address and four bytes of data.

[0352] In a fifth operation, DMA 1104 can update the descriptor in descriptor RAM 1106 in order to retrieve the appropriate data for the next processing iteration of VPU 1108. For example, DMA 1104 can read the address / data pair forming the VPU configuration format to patch the descriptor with the updated location. In embodiments, there can be a one-to-one correspondence between a feature point and a descriptor, such that each tracked feature, object, or point can include an associated descriptor. As such, the address / data pair for each tracked feature, object, or point can be updated over time using a separate descriptor.

[0353] In a sixth operation, DMA 1104 can use the newly updated descriptor in descriptor RAM 1106 to fetch the new feature data for the location. For example, DMA 1104 can indicate to VPU 1108 that the descriptor has been updated, and VPU 1108 can trigger DMA 1104 to read the new data to VMEM 1110, etc.

[0354] As a result, operations two through six can be repeated after the first configuration operation of the processing controller to form a tightly synchronized VPU configuration loop that requires processing controller intervention—thereby reducing latency for short response times required for tracking or detection algorithms. Moreover, because DMA 1104 is overwriting the address in memory with a new updated address, DMA 1104 is updating the code that DMA 1104 needs to look at to determine what to fetch next. By doing so, there is an increase in throughput compared to traditional systems that rely on a control bus to update a register with an address and data. Thus, the benefits of defining an address / data protocol are realized, where variable address locations with variable amounts of data can be updated as well as how the address / data pair is updated. This allows DMA 1104— which can have a width greater than the width of a control bus (e.g., 512 bits and 32 bits, respectively)— to update up to (e.g., but not limited to) 8 address / data pairs at a time (where each address / data pair is defined using 8 bytes, as shown). FIG. 10H

[0355] Moreover, although DMA is shown as being updated using the VPU configuration mode of process 1100, additional or alternative elements or components of the system can be updated. For example, the instruction cache of VPU 1108 can be updated using a VPU using a similar approach. As another example, an updated hardware sequencer program can be written to update a hardware sequencer memory by giving it address data. This would essentially include writing a hardware sequencer RAM with a new program— e.g., hardware sequencer RAM 1054 of hardware sequence controller 1060 for FIG. 11C

[0356] Reference is now made to​​FIG. 11A Each block of the method 1150 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 1150 can also be embodied as computer-usable instructions stored on computer storage media. The method 1150 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, just to name a few. In addition, the method 1150 described with respect to the system FIG. 11C The method 1150 described with respect to the system

[0357] FIG. 11B is a flowchart of a method 1150 of configuring a DMA system using a VPU in accordance with some embodiments of the present disclosure. At block B1102, the method 1150 includes using a processor and based at least in part on first data written to a memory using a DMA system, computing first output corresponding to one or more first updated locations of a tracked feature. For example, the VPU 1108 can access data from the VMEM 1110 written to the VMEM 1110 using the DMA 1104 and can process the data to compute one or more object locations corresponding to a tracked feature, object, point, etc.

[0358] At block B1104, the method 1150 includes using a processor to update the memory to include second data representing one or more address / data pairs corresponding to the one or more first updated locations. For example, after computing the one or more locations, the VPU 1108 can update the VMEM 1110 with address / data pairs in a format, such as the format described with respect to FIG. 12A the system

[0359] At block B1106, the method 1150 includes using a DMA system and based at least in part on the one or more address / data pairs, updating one or more descriptors corresponding to the tracked feature. For example, the DMA 1104 can access the address / data pairs from the VMEM 1110 and use the address / data pairs to update descriptors in the descriptor RAM 1106 for a next read operation.

[0360] At block B1108, the method 1150 includes using a DMA system and based at least in part on the one or more descriptors, writing third data to the memory. For example, the DMA 1104 can write updated data from the address / data pairs corresponding to addresses / data pairs identified using the descriptors to the VMEM 1110.

[0361] At block Bll 110, the method 1150 includes calculating, using the processor and based at least in part on the third data, a second output corresponding to one or more second updated locations of the tracked feature. For example, once the updated data is in the VMEM 1110, the VPU 1108 can calculate a next set of updated address / data pairs corresponding to the tracked feature, object, point, etc., and this process can repeat until processing is complete.

[0362] Permanent fault detection in programmable vision accelerators (PVAs)

[0363] In safety-critical applications such as autonomous and semi-autonomous machine applications, there are strict requirements for permanent fault detection and isolation. For example, when deep learning, computer vision, sensor processing, and / or other applications are performed in a machine, permanent fault detection must be performed periodically within an allotted time budget in order to perform accurate testing while also allowing the application to perform correctly - e.g., with low latency. With respect to Automotive Safety Integrity Level (ASIL) D, an application performed in an autonomous or semi-autonomous machine can require 90% or more permanent fault coverage. To this end, end-to-end coverage can be required with low latency while also meeting a runtime budget for each particular application. Conventional methods use built-in self-tests (BISTs) to identify faults, but these BIST techniques either do not include sufficient coverage, introduce too much latency in the system, and / or do not meet a runtime budget for certain applications.

[0364] To address these deficiencies of conventional methods, the present systems and methods can perform multiple-input signature register (MISR) BIST - e.g., to perform fault detection of a programmable vision accelerator (PVA) of a system on a chip (SoC). For example, in various embodiments of the present disclosure, the PVA can include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) - e.g., R5 processors and ARM processors, CPUs, and / or the like. As such, each component of the PVA can need to be tested, and the present systems and methods perform MISR BIST to detect permanent faults in an end-to-end manner. In this way, permanent fault detection can be performed to cover end-to-end blocks of control and data logic, report errors directly to a safety processor to reduce latency, and be customized for a particular application to meet an associated runtime budget.

[0365] In various embodiments, MISRs can be used in a PVA to implement software logic BIST for permanent fault detection. MISR hardware (herein with respect to FIG. 12Aand / or 12B) can include cyclic redundancy check (CRC) hardware that is initialized (e.g., using a known seed value) using the processing controller. While executing the PVA application, the processing controller can allocate a portion of the timing budget (e.g., about 10% or less of the timing budget) to run a known software MISR test with known inputs that have deterministic pre-computed outputs with correct signatures or golden values. For example, in the case where the timing budget corresponds to 30 frames per second, the processing controller can allocate timing budget corresponding to 3 or fewer frames to the MISR test. At the allocated time, the processing controller can start the MISR test and wait for the test to complete to terminate the MISR CRC computation. Once the test is complete, the MISR hardware can read back the final CRC value and check the final CRC value against the pre-computed golden value. In the case of a mismatch, the MISR hardware can report the error directly to the security processor of the SoC to take further action to handle the security error - e.g., cause the output of the application to be ignored, resolved, or resolved permanently, etc.

[0366] Thus, the MISR hardware in the DMA block can monitor one or more (e.g., all in embodiments) transactions on the advanced extensible interface (AXI) master port of the PVA. By checking all output stages from the PVA, in embodiments, the security integrity of the PVA can be checked against permanent defects (e.g., output information) that can corrupt the output stages that can be consumed by the PVA and / or another engine while executing the application. The MISR hardware can thus detect errors across different blocks of the PVA (e.g., the processing controller, VPU, and DMA system) as these components all cooperate and interact in producing the output stages. The signature computed in the MISR hardware can represent the state of these different PVA blocks during the MISR test.

[0367] In embodiments, the MISR scheme can include CRC checking of both the write address (e.g., 40-bit control) and write data (e.g., 512-bit data) leaving the AXI master port. This feature can allow control path faults (e.g., addressing errors) to be isolated from data path faults (e.g., calculation errors). Due to the configuration of the MISR hardware (as described herein), each DMA AXI port can be able to be checked. In embodiments, control bits can be used to disable address and data output for all channels (chans) participating in the MISR calculation in order to save bandwidth consumption in the memory subsystem and during memory allocation. Further, the MISR scheme can include control register bits for each channel to exclude or mask specific channels from the MISR calculation - e.g., isolate non-secure channels. In embodiments, the DMA can use the IEEE 802 and MPEG CRC-32 primitive polynomials to calculate the MISR CRC: 32 + X 26 + X 23 + X 22 + X 16 + X 12 + X 11 + X 10 + X 8 + X 7 + X 5 + X 4 + X 2 + X + 1. The MISR SET register can be used to set the initial CRC value (e.g., seed value) for the address and data CRC calculations. The MISR REF register can be used to compare the CRC values of the address and data CRC calculations.

[0368] To support 512-bit data for the MISR, 8:1 bit data compression can be applied - for example, each data byte can be compressed to 1 data bit by an 8>1 exclusive OR (XOR) operation to form 2x32-bit message data. To support 40-bit addresses for the MISR, 9 most significant bits can be compressed - for example, 9 most significant bits can be compressed by a 9>1 XOR operation to form 32-bit message addresses. Changes to the test mode and instructions can be used to cover aliasing related to compression. The likelihood of aliasing occurring can be low because error failures do not produce address CRC errors when there are an even number of errors in one byte on the output image. Additionally, aliasing can be unlikely because a reference CRC can be computed on the output image with the same pattern on the same even error bit position throughout the MISR test. During experimentation, aliasing was shown to cause an average 0.25% loss of coverage. In embodiments, due to the width of the bus (e.g., 512 bits), data compression with this low aliasing is valuable, and without compression, the MISR test can not meet the system’s delay or runtime budget.

[0369] A MISR timer register can be used to time out the MISR computation, and the MISR timer register can be decremented on each AXI clock. The timeout function can be helpful in case of a fault that causes the MISR test to hang, which can prevent the MISR hardware from reporting errors. When the MISR test is finished, the processing controller can use a software event to stop the MISR computation. The DMA system can compare the MISR REF values to the MISR VAL values for the data and address outputs of the MISR test, and the DMA hardware can update the MISR status register based on the comparison. For example, the MISR status register can include one of the following values: 0: idle; 1: done: fail data; 3: busy; 4: done: both address and data fail; 5: done: fail timeout; 6: RSVD; and 7: done: pass. In the case of a MISR timeout error, the DMA can generate a timeout signal to the safety processor, and in the case of a CRC check error in the data and / or address, the DMA can assert a safety error to the safety processor.

[0370] Reference FIG. 12A , FIG. 10His a diagram of a built-in self-test (BIST) system for performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) in accordance with some embodiments of the present disclosure. It will be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the MISR hardware 1250 may include FIGS. 13A-13D DMA system 1050, FIG. 14 Example of autonomous vehicle 1300, FIG. 15 The example computing device 1400 of , and / or FIG. 10H 1500. For example, the MISR hardware 1200 may be included in a DMA block of a PVA, such as FIG. 12A As shown, the MISR hardware 1200 can operate at the output level of data movement, and addressing (or control) can access the output of the DMA engine 1056. FIG. 12A As shown, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used according to an embodiment.

[0371] In operation, the processing controller can control the DMA system 1050 and the MISR hardware 1200 - as well as one or more processing components of the system, such as a VPU. In performing MISR testing on the DMA system 1050, in embodiments, the test code can include all 0s, all Is, alternating 0s and Is, and / or random code sequences. In this way, high coverage of the DMA system 1050 can be achieved. For example, when testing a VPU, the test code can include application-specific or custom code. For example, during testing of coverage of a particular application, the components or portions of the VPU used (e.g., registers, logic, etc.) can be determined, and test code can be generated so that those particular components or portions of the VPU are included in the execution of the test code. For example, random data with different instructions can be included in the test code so that the test is ordered through different instructions to use different areas of the VPU logic. In this way, the coverage of the VPU is increased overall, particularly the coverage of the particular application executing on the VPU. By performing DMA and VPU testing in this way, and because the processing controller is involved in the control and interaction between the various components (e.g., DMA system 1050 and VPU), the processing controller can have high coverage because the output and addressing of the data movement is affected by the processing controller interaction.

[0372] During testing, where different code patterns are used, the code patterns can be used in an alternating pattern, or one code can be used for a first time frame (e.g., equivalent to 30 fps of time), another code for a second time frame (e.g., equivalent to 30 fps of time), another code for a third time frame (e.g., equivalent to 30 fps of time), and so on. For example, in the DMA code example, the code of 0s can be used for a first time frame, then the code of Is for a second time frame, then the code of alternating 0s and Is (e.g., 0101010101...) for a third time frame, then a random code (e.g., 011100100010...) for a fourth time frame, and then these four codes can repeat, and so on.

[0373] In practice, when testing the DMA, for example, the processing controller can interact with the MISR controller 1206 to write set reference values to the MISR data set register 1210 and the MISR address set register 1216. These values can be different for data and address and can be referred to as seed values for the CRC calculation. The processing controller can then initialize a channel in the DMA engine 1056 that performs data movement, and because the location of the test code in memory is known to the processing controller, the descriptors (e.g., configured by the processing controller in the descriptor SRAM) 1052) can be used to order the DMA engine 1056 by the data under MISR test. The processing controller can set a timer 1226 on the MISR hardware 1200 to enable the MISR test, and then can trigger the channel of the DMA engine 1056 to begin reading the test data from the source destination and outputting the data to the MISR hardware 1200 for the MISR test. Thus, when testing the DMA, the data movement being tested (e.g., correct addressing and correct data in the addressed locations), and thus the MISR hardware 1200 can tap into the output of the DMA engine 1056 as the DMA engine performs the data movement of the test code. This tapped into output stage can be used to FIG. 12AAs an external memory, the sequence funneling (one data channel at a time, one address channel at a time) by the processing controller can be instructed. For example, for the data channels, the processing controller can order through each of the 16 data channels, for example, and the respective AXI write data (wdata) for each channel can be fed through the CH0-CH16 data CRC computation 1202— e.g., in series. For example, the processing controller can configure the channel output register 1220 to pass through the channels one at a time according to a configuration order from the processing controller. In embodiments, the channel mask register 1208 (e.g., programmed by the MISR controller 1206 based on interaction with the processing controller) can be configured by the processing controller to mask or remove various channels (e.g., channels not under test) from the CRC computation. In embodiments, this masking can be performed using AND gates. With one or more channels masked, the golden value in the MISR data reference register 1222 (which can be provided to the MISR controller 1206 by the processing controller) can only correspond to the CRC computation for the unmasked channels. For each unmasked channel, the data on the channel (generated using test code read from memory) can be applied to (e.g., compressed or uncompressed) the polynomial of the CRC data computation 1202 to generate a MISR data value 1214 for that channel. Once the channel completes the computation, the processing controller can receive an indication and can cause the next data channel to be sent to the CRC computation 1202 to compute the next MISR data value 1214, and so on, until each unmasked channel has a respective MISR data value 1214. Once each MISR data value 1214 for a particular iteration has been computed, these values 1214 can be combined to generate a final MISR data value, which can be compared to the golden value in the MISR data reference register 1222 to generate a MISR data status determination (e.g., which can include a status corresponding to values 0-7 described above).

[0374] As another example, for an address channel, the processing controller can order each of, for example, 16 address or control channels, and can feed the corresponding AXI write address (waddress) address for each channel - e.g., concatenated - through the CH0-CH16 CRC calculation 1204. In embodiments, a channel mask register 1208 can be configured by the processing controller to mask or remove various channels from the CRC calculation - e.g., channels not under test. In embodiments, this masking can be performed using an AND gate. With one or more channels masked out, the golden value in the MISR data reference register 1224 can only correspond to the CRC calculation for the unmasked channels. For each unmasked channel, the address on the channel (generated using test code read from memory) can be applied to (e.g., compressed or uncompressed) the polynomial of the CRC address calculation 1204 to generate a MISR address value 1218 for that channel. Once the channel completes the calculation, the processing controller can receive an indication and can cause the next channel of address data to be sent to the CRC calculation 1204 to calculate the next MISR address value 1218, and so on, until each unmasked channel has a corresponding MISR address value 1218. Once each MISR address value 1218 for a particular iteration is calculated, these values 1218 can be combined to generate a final address MISR value, which can be compared to the golden value in the MISR reference register 1224 address to generate a MISR address status determination (e.g., which can include the states corresponding to the values 0-7 described above).

[0375] In some embodiments, the MISR test can be iterative, such that a first code can be processed, an output can be tested, and then the output can be used for a next iteration that can be tested, and so on. In such embodiments, the MISR test can include multiple phases, and a completed MISR test can include executing each phase.

[0376] In cases where the MISR hardware 1200 is specifically for testing a VPU, for example, the DMA system 1050 can move test code into VMEM, the VPU can process the test code and write results back to VMEM, and the DMA engine 1056 can read the results from VMEM back to a destination location. When writing the results back to the destination location, the MISR hardware 1200 can tap into the DMA output and perform a MISR on the data (e.g., including data and addresses), and perform a MISR similar to that discussed herein. In this way, the VPU can be tested for interaction with the test code using the MISR hardware 1200.

[0377] After completing the MISR test, the processing controller may receive an interrupt. For example, the processing controller may receive a completion interrupt and, in the absence of an error, may wait for the next MISR test cycle. In the event that the interrupt is an error interrupt, the type of error may be determined—e.g., failed data, failed address, failed both, etc.—and a security error may be asserted to the security processor. For example, in some embodiments where the MISR hardware 1200 is hung or idle (e.g., with a timeout error), the DMA may generate a timeout signal to the security processor of the SoC.

[0378] In some embodiments, to speed up MISR calculations to calculate CRC on one or more (e.g., in an embodiment, such as 16) channels without serializing or staging the channel-MISR calculations, the channels may be demultiplexed based on the channel IDs present in the AXIID field to parallelize the channel calculations. For example, since CRC calculations are done at different rates, FIG. 12B The method involves serial processing of channels one by one. However, using FIG. 12B In a system with a MISR, as described below, these calculations can be performed in parallel. For example, when the processing controller terminates the MISR calculation, the MISR controller can sequence all channel outputs to calculate a final signature that can be compared to a reference or golden value for the address and data outputs. This feature can speed up permanent fault detection without requiring an additional programmer register interface—for example, because the same control registers can be used for all channels.

[0379] Likewise, and with reference to FIG. 12B , FIG. 10H is a diagram of a built-in self-test (BIST) system for parallel channel cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the MISR hardware 1250 may include FIGS. 13A-13D DMA system 1050, FIG. 14 Example of autonomous vehicle 1300, FIG. 15 The example computing device 1400 and / or FIG. 10Hsimilar components, features, and / or functions of the example data center 1500. For example, the MISR hardware 1250 can be included in a DMA block of the PVA, as shown in FIG. 12A Thus, the MISR hardware 1250 can operate on the output stage of data movement, and the addressing (or control) can access the output of the DMA engine 1056. As shown in FIG. 12A there can be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels can be used, according to embodiments.

[0380] The MISR hardware 1250 can operate similarly to the MISR hardware 1200 of FIG. 12A The MISR hardware 1250 can be configured for parallel data channel and parallel address channel CRC calculations. For example, the processing controller can configure the MISR data set register 1256 with a seed or reference value for each data CRC calculation 1260A-1260N (corresponding to AXI data channels 0-15, respectively), and can configure the MISR address set register 1258 with a seed or reference value for each address CRC calculation 1262A-1262N (corresponding to AXI address channels 0-15, respectively). The processing controller, similarly to that described with respect to FIG. 12A The processing controller can then trigger data movement of the DMA system 1050 (e.g., for DMA testing) and / or VPU processing (e.g., for VPU specific testing) to move data, and the MISR hardware 1250 can tap into the output stage to be tested.

[0381] Thus, the processing controller can cause 16 channels of data to be sent to a multiplexer (mux) 1252 and 16 channels of address data to be sent to a multiplexer (mux) 1254. The mux 1252 can then provide the data of the respective channel to the respective CRC calculation 1260A-1260N (e.g., channel 0 AXI data to channel 0 CRC calculation 1260, channel 1 data to channel 1 CRC calculation 1260B, etc.), and each CRC calculation 1260 can use that data along with a CRC polynomial having a reference value to calculate a MISR data value 1284A-1284N (e.g., channel 0 CRC calculation 1260A can calculate a MISR data 0 value 1284A, channel 1 CRC calculation 1260B can calculate a MISR data 1 value 1284B, etc.). The MISR data values 1284A-1284N can then be ordered out of a multiplexer (mux) 1264 according to a MISR sequence from the MISR control 1270, as configured by the processing controller. In embodiments, for example with respect to FIG. 12AAs described, one or more channels can not be included in a particular MISR test, and thus the channel mask register 1268 can be configured by the processing controller to update the MISR sequence such that the MISR data values 1284 corresponding to the one or more masked channels are not provided to the channel 0-16 data CRC calculation 1274 for use in calculating the final CRC value. For unmasked channels, the multiplexer 1264 can output the MISR data values 1284 according to the MISR sequence. In this way, the different channel and CRC calculation 1260 timing is accounted for in that the MISR data values 1284 are forced to be output according to the MISR sequence, rather than sent to the CRC calculation 1274 according to the timing at which the CRC calculation is being completed. Once the MISR sequence of MISR data values 1284 is output by the multiplexer 1264 to the CRC calculation 1274, the CRC calculation 1274 can calculate the final CRC value and store the final CRC value to the VAL register 1276. The final CRC value in the VAL register 1276 can then be compared to the golden value in the MISR data reference register 1272 (configured by the MISR control 1270 from the processing controller) to determine the MISR data status.

[0382] Similarly, the processing controller can cause the 16 address channels to be sent to a multiplexer (mux) 1254, which can then provide the corresponding address channels to corresponding CRC calculations 1262A-1262N (e.g., channel 0 AXI address to channel 0 CRC calculation 1262, channel 1 address to channel 1 CRC calculation 1262B, etc.), and each CRC calculation 1262 can use the address and a CRC polynomial with a reference value to calculate MISR address values 1286A-1286N (e.g., channel 0 CRC calculation 1262A can calculate MISR address 0 value 1286A, channel 1 CRC calculation 1262B can calculate MISR address 1 value 1286B, etc.). The MISR address values 1286A-1286N can then be ordered from a multiplexer (mux) 1266 according to the MISR sequence from the MISR control 1270, as configured by the processing controller. In embodiments, for example, with respect to the 16 address channels, the processing controller can configure the mux 1266 to output the MISR address values 1286A-1286N in the order of the MISR sequence from the MISR control 1270, and the processing controller can configure the mux 1254 to output the 16 address channels in the order of the MISR sequence from the MISR control 1270. FIG. 12AAs described, one or more channels can not be included in a particular MISR test, and thus the channel mask register 1268 can be configured by the processing controller to update the MISR sequence such that the MISR address values 1286 corresponding to the one or more masked channels are not provided to the channel 0-16 address CRC calculation 1280 for use in calculating the final CRC value. For unmasked channels, the MISR address values 1286 can be output by the multiplexer 1266 according to the MISR sequence. In this way, the different computation times of different channels and CRC calculations 1262 are accounted for because the MISR address values 1286 are forced to be output according to the MISR sequence, rather than sent to the CRC calculation 1280 according to the timing at which the CRC calculation is being completed. Once the multiplexer 1266 outputs the MISR sequence of MISR address values 1286 to the CRC calculation 1280, the CRC calculation 1280 can calculate the final CRC value and store the final CRC value to the VAL register 1282. The final CRC value in the VAL register 1282 can then be compared to the golden value in the MISR address reference register 1278 (configured by the MISR control 1270 from the processing controller) to determine the MISR address status.

[0383] The MISR data status and MISR address status can be checked and used similarly to the descriptions above regarding FIG. 12C .

[0384] Referring now to the drawings, and more particularly to FIGS. FIG. 12A As such, each block of the method 1290 described herein can comprise a FIG. 12C computer usable or executable instructions implemented as "software" stored in a memory of the computing device. The software can include a plurality of programs and

[0385] FIG. 13A is a flow diagram of an execution (BIST) method 1290 for permanent fault detection in a PVA in accordance with some embodiments of the present disclosure. At block B1202, the method 1290 includes receiving a plurality of data channels from a DMA system one at a time and based on an ordering by a processing controller. For example, the MISR hardware 1200 can receive one data channel (or one address data channel) at a time according to an order determined by the processing controller.

[0386] At block B1204, the method 1290 includes calculating a plurality of MISR values by performing a CRC calculation using a polynomial of the CRC calculation and respective data corresponding to the channels to calculate the MISR values. For example, for each channel, the CRC calculation 1202 (or address of 1204) can use the data (or address) from the channel and the polynomial of the CRC calculation 1202 to calculate a MISR data value 1214 (or MISR address value 1216 for the address) (starting from the seed value of the CRC MISR data set register 1210 or the MISR address set register 1216).

[0387] At block B1206, the method 1290 includes calculating a final MISR value using the plurality of MISR values. For example, the MISR data values 1214 from each channel (or MISR address values from each channel) can be combined to generate a final MISR value.

[0388] At block B1208, the method 1290 includes comparing the final MISR value to a signature value. For example, the final MISR value generated from the individual MISR values 1214 (or address values 1216) can be compared to a signature or golden value of the MISR data reference register 1222 (or MISR address reference register 1224 for the address).

[0389] At block B1210, the method 1290 includes outputting a MISR status based at least in part on the comparison. For example, based on the comparison of block B1208, a status can be determined - e.g., failed data, failed address, both failed, complete, etc. - and the status can be used to notify a security processor of the SoC where the error status originated.

[0390] Example autonomous vehicle

[0391] FIG. 13Cis an illustration of an example autonomous vehicle 1300 according to some embodiments of the present disclosure. Autonomous vehicle 1300 (alternatively referred to herein as “vehicle 1300”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, drone, vehicle coupled to a trailer, and / or another type of vehicle (e.g., a vehicle that is driverless and / or that accommodates one or more passengers). Autonomous vehicles are generally described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and prior and future versions of this standard). Vehicle 1300 can be capable of implementing functionality that complies with one or more of Levels 3-5 of autonomous driving. For example, depending on the embodiment, vehicle 1300 can be capable of implementing conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0392] Vehicle 1300 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. Vehicle 1300 can include a propulsion system 1350 such as an internal combustion engine, a hybrid power plant, a fully electric motor, and / or another type of propulsion system. Propulsion system 1350 can be connected to a drivetrain of vehicle 1300 that can include a transmission in order to effect propulsion of vehicle 1300. Propulsion system 1350 can be controlled in response to receiving a signal from a throttle / accelerator 1352.

[0393] A steering system 1354, which can include a steering wheel, can be used to steer vehicle 1300 (e.g., along a desired path or route) while propulsion system 1350 is operating (e.g., while the vehicle is in motion). Steering system 1354 can receive a signal from a steering actuator 1356. For full automation (Level 5) functionality, a steering wheel can be optional.

[0394] A braking sensor system 1346 can be used to operate vehicle brakes in response to receiving a signal from a brake actuator 1348 and / or a brake sensor.

[0395] One or more system on chips (SoCs) 1304 FIG. 13C One or more controllers 1336 of one or more GPUs can provide signals (e.g., signals representative of commands) to one or more components and / or systems of vehicle 1300. For example, the one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 1348, to operate a steering system 1354 via one or more steering actuators 1356, to operate a propulsion system 1350 via one or more throttle / accelerator 1352. The one or more controllers 1336 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representative of commands) to enable autonomous driving and / or to assist a human driver in driving vehicle 1300. The one or more controllers 1336 can include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 can handle two or more of the above functions, two or more controllers 1336 can handle a single function, and / or any combination thereof.

[0396] The one or more controllers 1336 can provide signals to control one or more components and / or systems of vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, global navigation satellite system sensors 1358 (e.g., global positioning system sensors), RADAR sensors 1360, ultrasonic sensors 1362, LIDAR sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-angle cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360 degree cameras), long and / or medium range cameras 1398, speed sensors 1344 (e.g., to measure a speed of vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of brake sensor system 1346), and / or other sensor types.

[0397] One or more of the controllers 1336 can receive input from the instrument cluster 1332 of the vehicle 1300 (e.g., represented by input data) and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1334, audible annunciators, speakers, and / or via other components of the vehicle 1300. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., HD map 1322) of the vehicle 1300, e.g., position on the map, direction, position of other vehicles (e.g., occupancy grid), information about objects and object states as perceived by the controllers 1336, etc. For example, the HMI display 1334 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, exit 34B in two miles, etc.). FIG. 13B

[0398] The vehicle 1300 also includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or modems. For example, the network interface 1324 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 1326 can also enable communication between objects (e.g., vehicles, mobile devices, etc.) in the environment using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0399] FIG. 13A An example camera position and field of view for an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure. FIG. 13B An example camera position and field of view for an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure.

[0400] ​The camera type used for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120fps, 240fps, and the like, depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a clear pixel camera such as a camera with an RCCC, RCCB, and / or RBGC color filter array may be used in an effort to improve light sensitivity.

[0401] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0402] One or more of the cameras can be mounted in a mounting assembly, such as a custom-designed (3-D printed) assembly, to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that could interfere with the camera's ability to capture image data. With respect to the wing mirror mounting assembly, the wing mirror assembly can be custom 3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0403] A camera (e.g., a front-facing camera) having a field of view that includes a portion of the environment in front of the vehicle 1300 can be used for surround vision to help identify the forward path and obstacles, as well as assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1336 and / or control SoCs. The front-facing camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0404] A variety of cameras can be used in the front-facing configuration, including, for example, monocular camera platforms including CMOS (complementary metal-oxide semiconductor) color imagers. Another example can be a wide-view camera 1370, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although FIG. 13B Although only one wide-view camera is illustrated in FIG. 13, any number of wide-view cameras 1370 can be present on the vehicle 1300. In addition, long-range cameras 1398 (e.g., long-view stereo camera pairs) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. The long-range cameras 1398 can also be used for object detection and classification and basic object tracking.

[0405] One or more stereo cameras 1368 can also be included in the front-facing configuration. The stereo cameras 1368 can include an integrated control unit that includes a scalable processing unit, which can provide a multi-core microprocessor with integrated CAN or Ethernet interfaces and programmable logic (FPGA) on a single chip. Such a unit can be used to generate a 3-D map of the vehicle's environment, including distance estimates for all points in the image. Alternative stereo cameras 1368 can include compact stereo vision sensors, which can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 can be used in addition to or instead of those described herein.

[0406] Cameras with fields of view that include portions of the environment to the sides of the vehicle 1300 (e.g., side-view cameras) can be used for surround view, providing information used to create and update an occupancy grid and to generate side-crash collision warnings. For example, surround cameras 1374 (e.g., four surround cameras 1374 as shown in FIG. 13C The surround cameras 1374 can include wide-view cameras 1370, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1374 (e.g., left, right, and back), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0407] Cameras with fields of view that include portions of the environment behind the vehicle 1300, such as rearview cameras, can be used to assist with parking, surround view, rear collision warning, and creating and updating the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long and / or mid-range cameras 1398, stereo cameras 1368, infrared cameras 1372, etc.) as described herein.

[0408] FIG. 13A FIG. 1 illustrates an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure. FIG. 13C FIG. 1 illustrates an example autonomous vehicle 1300 for use in accordance with some embodiments of the present disclosure.

[0409] FIG. 13A Each of the components, features, and systems of the vehicle 1300 are illustrated as being connected via a bus 1302. The bus 1302 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1300 that is used to assist in controlling various features and functions of the vehicle 1300, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0410] Although bus 1302 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 1302 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 1302, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 1302 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 1302 can be used for collision avoidance functions, and a second bus 1302 can be used for drive control. In any example, each bus 1302 can communicate with any component of vehicle 1300, and two or more buses 1302 can communicate with the same components. In some examples, each SoC 1304, each controller 1336, and / or each computer within the vehicle can have access to the same input data (e.g., input from sensors of vehicle 1300), and can be connected to a common bus, such as a CAN bus.

[0411] Vehicle 1300 can include one or more controllers 1336, such as those described herein with respect to ​ controllers. Controllers 1336 can be used for a wide variety of functions. Controllers 1336 can be coupled to any other different components and systems of vehicle 1300, and can be used for control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment for vehicle 1300, and / or the like.

[0412] Vehicle 1300 can include one or more system on chips (SoCs) 1304. SoCs 1304 can include CPUs 1306, GPUs 1308, processors 1310, caches 1312, accelerators 1314, data stores 1316, and / or other components and features not illustrated. In a wide variety of platforms and systems, SoCs 1304 can be used to control vehicle 1300. For example, one or more SoCs 1304 can be in a system (e.g., a system of vehicle 1300) in conjunction with HD map 1322, which can obtain map refreshes and / or updates from one or more servers (e.g., one or more servers 1378) via network interface 1324. Figure 13D

[0413] ​CPU 1306 can include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). CPU 1306 can include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1306 can include eight cores in a coherent multi-processor configuration. In some embodiments, CPU 1306 can include four dual-core clusters, with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). CPU 1306 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1306 can be active at any given time.

[0414] CPU 1306 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. CPU 1306 can further implement enhanced algorithms for managing power states, with specified allowed power states and desired wake-up times, and the hardware / microcode determines the optimal power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work offloaded to microcode.

[0415] GPU 1308 can include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU 1308 can be programmable and efficient for parallel workloads. In some examples, GPU 1308 can use an enhanced tensor instruction set. GPU 1308 can include one or more streaming microprocessors, where each streaming microprocessor can include an Ll cache (e.g., an Ll cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 1308 can include at least eight streaming microprocessors. GPU 1308 can use a compute application programming interface (API). In addition, GPU 1308 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0416] In the case of automotive and embedded uses, the GPU 1308 can be power-optimized for best performance. For example, the GPU 1308 can be fabricated on a fin field-effect transistor (FinFET) for lower power consumption. However, this is not intended to be limiting, and the GPU 1308 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. Further, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and address computation to provide efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0417] The GPU 1308 can include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, a synchronous graphics random access memory (SGRAM) such as a fifth generation graphics double data rate synchronous random-access memory (GDDR5) can be used.

[0418] The GPU 1308 can include a unified memory technology that includes an access counter to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow the GPU 1308 to access CPU 1306 page tables directly. In such examples, when the GPU 1308 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 1306. In response, the CPU 1306 can look up the virtual-to-physical mapping for the address in its page tables and transmit the translation back to the GPU 1308. In this way, the unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1306 and the GPU 1308, simplifying GPU 1308 programming and porting applications to the GPU 1308.

[0419] In addition, GPU 1308 can include an access counter that can track how often GPU 1308 accesses memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0420] SoC 1304 can include any number of caches 1312, including those described herein. For example, caches 1312 can include an L3 cache that is available to both CPU 1306 and GPU 1308 (e.g., connected to both CPU 1306 and GPU 1308). Caches 1312 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4MB or more, although smaller cache sizes can also be used.

[0421] SoC 1304 can include an arithmetic logic unit (ALU) that can be utilized in processing any of a variety of tasks or operations related to vehicle 1300, such as processing a DNN. In addition, SoC 1304 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 1306 and / or GPU 1308.

[0422] SoC 1304 can include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1304 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement GPU 1308 and offload some of the tasks of GPU 1308 (e.g., freeing up more cycles of GPU 1308 for performing other tasks). As one example, accelerators 1314 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0423] The accelerators 1314 (e.g., a cluster of hardware accelerators) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPU can be an accelerator that is configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and is optimized for performing image processing functions. The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPU can perform several functions including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0424] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not by way of limitation: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or safety related events.

[0425] The DLA can perform any of the functions of the GPU 1308, and by using an inferencing accelerator, the designer can target the DLA or the GPU 1308 for any function, for example. For example, the designer can focus the processing and floating point operations of the CNNs on the DLA and leave other functions to the GPU 1308 and / or other accelerators 1314.

[0426] The accelerators 1314 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example and not by way of limitation.

[0427] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0428] The DMA can enable components of the PVA to access system memory independently of the CPU 1306. The DMA can support any number of features to provide optimization to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0429] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0430] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of those PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0431] The accelerator 1314 (e.g., hardware accelerator cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1314. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0432] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0433] In some examples, the SoC 1304 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison to LIDAR data for purposes of localization and / or other functionality, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0434] The accelerator 1314 (e.g., a hardware accelerator cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0435] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0436] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0437] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections above the threshold to be true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1366 output related to vehicle 1300 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 1364 or RADAR sensor 1360), etc.

[0438] SoC 1304 can include one or more data stores 1316 (e.g., memory). Data stores 1316 can be on-chip memory of SoC 1304, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 1316 can be large enough in capacity to store multiple instances of a neural network. Data stores 1312 can include L2 or L3 cache 1312. References to data stores 1316 can include references to memory associated with PVA, DLA, and / or other accelerators 1314 as described herein.

[0439] SoC 1304 can include one or more processors 1310 (e.g., embedded processors). The processors 1310 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 1304 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 1304 thermal and temperature sensor management, and / or SoC 1304 power state management. Each temperature sensor can be implemented as a ring oscillator, whose output frequency is proportional to temperature, and the SoC 1304 can use the ring oscillator to detect the temperature of the CPU 1306, GPU 1308, and / or accelerators 1314. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 1304 in a lower power state and / or place the vehicle 1300 in a driver safe park mode (e.g., safely park the vehicle 1300).

[0440] The processors 1310 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0441] The processors 1310 can also include an always-on processor engine, which can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0442] The processors 1310 can also include a security cluster engine, which includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0443] The processors 1310 can also include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0444] The processor 1310 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0445] The processor 1310 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 1370, the surround camera 1374, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0446] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0447] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1308 does not need to continuously render new surfaces. Even when the GPU 1308 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 1308 to improve performance and responsiveness.

[0448] The SoC 1304 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 1304 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0449] The SoC 1304 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1304 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor 1364, RADAR sensor 1360, etc. that can be connected over Ethernet), data from the bus 1302 (e.g., speed of the vehicle 1300, steering wheel position, etc.), data from GNSS sensor 1358 (connected over Ethernet or CAN bus). The SoC 1304 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 1306 from routine data management tasks.

[0450] The SoC 1304 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1304 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 1314, when combined with the CPU 1306, GPU 1308, and data storage 1316, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0451] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0452] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1320) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0453] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 1308.

[0454] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 1300. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1404 provides security against theft and / or carjacking.

[0455] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphone 1396 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 1304 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine that slows the vehicle, pulls over to the side of the road, stops the vehicle, and / or idles the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensor 1362.

[0456] The vehicle can include a CPU 1318 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 can include, for example, an X86 processor. The CPU 1318 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1304, and / or monitoring the status and health of the controller 1336 and / or infotainment SoC 1330.

[0457] The vehicle 1300 can include a GPU 1320 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 1304 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 1320 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 1300.

[0458] The vehicle 1300 can also include a network interface 1324 that can include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1324 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 1378 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a client device of a passenger). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 1300 with information about vehicles that are approaching the vehicle 1300 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1300). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1300.

[0459] The network interface 1324 can include a SoC that provides modulation and demodulation functionality and enables the controller 1336 to communicate over a wireless network. The network interface 1324 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0460] The vehicle 1300 may also include data storage 1328, which may include off-chip storage (e.g., outside the SoC 1304). The data storage 1328 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, a hard disk, and / or other components and / or devices that can store at least one bit of data.

[0461] The vehicle 1300 may also include a GNSS sensor 1358. The GNSS sensor 1358 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist with mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1358 may be u...

Claims

1. A system comprising: Memory; a direct memory access (DMA) system coupled to the memory; as well as a processor coupled to the memory, The system performs the following operations, including: calculating, using the processor and based at least in part on first data written to a memory using the DMA system, a first output corresponding to one or more first updated positions of a tracked feature; updating, using the processor, the memory to include second data representing one or more address / data pairs corresponding to the one or more first updated locations; updating one or more descriptors corresponding to the tracked feature using the DMA system and based at least in part on the one or more address / data pairs; writing third data to the memory using the DMA system and based at least in part on the one or more descriptors; and A second output corresponding to one or more second updated positions of the tracked feature is calculated using the processor and based at least in part on the third data.

2. The system of claim 1, further comprising a process controller, wherein the system further performs operations including configuring the DMA system and the processor using the process controller.

3. The system of claim 1 , further comprising a process controller, wherein the system further performs operations including triggering the DMA system and the processor using the process controller. 4 . The system of claim 1 , further comprising a descriptor memory, wherein the one or more descriptors are updated in the descriptor memory. The system of claim 1 , wherein the operations are performed in a loop in each of a plurality of iterations.

6. The system of claim 1 , wherein the second data corresponds to a data format comprising one or more bytes representing addresses of the one or more address / data pairs and one or more bytes representing data of the one or more address / data pairs.

7. The system of claim 1, wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing deep learning operations; System on chip (SoC); A system including a programmable vision accelerator (PVA); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system for merging one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

8. A system comprising: Vector memory VMEM; a direct memory access (DMA) system coupled to the VMEM; as well as a vector processing unit VPU, which is coupled to the VMEM, The system performs the following operations, including: calculating, using the VPU and based at least in part on first data written to the VMEM using the DMA system, a first output corresponding to one or more first updated positions of a tracked feature; updating, using the VPU, the VMEM to include second data representing one or more address / data pairs corresponding to the one or more first updated locations; updating one or more descriptors corresponding to the tracked feature using the DMA system and based at least in part on the one or more address / data pairs; writing third data to the VMEM using the DMA system and based at least in part on the one or more descriptors; and A second output corresponding to one or more second updated positions of the tracked feature is calculated using the VPU and based at least in part on the third data.

9. The system of claim 8, further comprising a processing controller, wherein the system further performs operations including configuring the DMA system and the VPU using the processing controller.

10. The system of claim 8, further comprising a process controller, wherein the system further performs operations including triggering the DMA system and the VPU using the process controller.

11. The system of claim 8, further comprising a descriptor memory, wherein the one or more descriptors are updated in the descriptor memory.

12. The system of claim 8, wherein the operations are performed in a loop in each of a plurality of iterations.

13. The system of claim 8, wherein the second data corresponds to a data format comprising one or more bytes representing addresses of the one or more address / data pairs and one or more bytes representing data of the one or more address / data pairs.

14. The system of claim 8, wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing deep learning operations; System on chip (SoC); A system including a programmable vision accelerator (PVA); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system for merging one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

15. A method comprising: calculating, using a vector processing unit (VPU) and based at least in part on first data written to a vector memory (VMEM) using a direct memory access (DMA) system, a first output corresponding to one or more first updated positions of the tracked feature; updating, using the VPU, the VMEM to include second data representing one or more address / data pairs corresponding to the one or more first updated locations; updating one or more descriptors corresponding to the tracked feature using the DMA system and based at least in part on the one or more address / data pairs; writing third data to the VMEM using the DMA system and based at least in part on the one or more descriptors; as well as A second output corresponding to one or more second updated positions of the tracked feature is calculated using the VPU and based at least in part on the third data.

16. The method of claim 15, further comprising configuring the DMA system and the VPU using a processing controller.

17. The method of claim 15, further comprising triggering the DMA system and the VPU using a process controller.

18. The method of claim 15, further comprising updating the one or more descriptors in a descriptor memory.

19. The method of claim 15, wherein the method is performed in a loop for each of a plurality of iterations.

20. The method of claim 15, wherein the second data corresponds to a data format including one or more bytes representing addresses of the one or more address / data pairs and one or more bytes representing data of the one or more address / data pairs.

Images