Using a hardware sequencer in a direct memory access system of a system on a chip

Abstract

The present disclosure relates to the use of a hardware sequencer in a direct memory access system of a system on a chip. In various examples, a VPU and associated components can be optimized to improve VPU performance and throughput. For example, the VPU can include a min / max collector, an auto store prediction function, a SIMD data path organization that allows inter-lane sharing, a transpose load / store with a stride parameter function, a load with permute and zero insertion function, a hardware, logic, and memory layout function to allow two-point and two-point-by-two-point lookups, and a per-memory bank load cache function. Further, a decoupled accelerator can be used to offload VPU processing tasks to improve throughput and performance, and a hardware sequencer can be included in the DMA system to reduce programming complexity of the VPU and DMA system. The DMA and VPU can perform a VPU configuration mode that allows the VPU and DMA to operate without a processing controller for performing dynamic region based data movement operations.

Description

Background Art

[0001] Vector Processing Units (VPUs) are used to execute Single Instruction Multiple Data (SIMD) operations in parallel. Popular uses of VPUs include operations such as image processing, computer vision, signal processing, and deep learning (e.g., for convolution operations).

[0002] In some computer vision applications, for example, the dynamic range of intermediate values ​​is well understood. Therefore, to detect anomalies, calculated values ​​can be compared against this dynamic range. However, conventional solutions for detecting and analyzing these minimum and maximum values ​​involve writing all values ​​to memory and then analyzing the values ​​in memory, which requires additional processing cycles. Furthermore, to achieve high throughput despite load latency, high-clock-rate processors can implement software pipelining and / or loop unrolling. However, if the original iteration count does not divide evenly by the unroll factor, some iterations may remain after the unrolled loop completes, requiring an additional remainder loop to calculate the value for one or more final iterations. This remainder loop increases system code size and latency—for example, because it cannot be unrolled for optimal performance. In traditional single-instruction, multiple-data (SIMD) operations, each SIMD unit can operate in parallel and independently of each other in its own data lane. Some architectures may allow sharing between adjacent neighbors, but this limited sharing is restrictive and often requires that the implementation of an operation copy the same operand to each data lane for processing. Furthermore, vector SIMD processors may require that each memory read operation use a standard or consistent unit, for example, equal to the vector processing width. This can be inefficient when the memory banks are wide. For example, reading from elements 4 to 67 in a 64-byte memory bank may require two memory reads—for example, one from 0 to 63 and one from 64 to 67. However, this results in reading many additional values—for example, values ​​0-3 and values ​​68-127—even if these values ​​are not needed for the current operation. In traditional instruction sets that require additional data operations, after the data is read out and stored in a register, additional instructions can be used to operate on the memory data in the register. For example, this may require loading the data, performing a permutation on the data, and then performing the operation using the reorganized data. Consequently, data operations require additional cycles and increase latency. When performing table lookups using existing VPUs, the table can be replicated so that each individual value can be retrieved from the replicated table, or additional read ports can be added to each memory bank to allow multiple values ​​to be read from the same table in the same bank. However, replicating the table for each value requires additional memory and processing, and adding additional read ports requires additional space on the chip. In traditional VPUs, since the VPU is programmed to execute on a small set of highly optimized code, data caching may not be possible because the programmer may manage the contents of the local data memory. However, by doing so, each access requires reading a value from each memory bank, even if the data for the next iteration includes an overlap with one or more previous read operations.

[0003] To optimize the performance of a processor (such as a VPU), the instruction set architecture (ISA) can be enhanced to create custom instructions to speed up common operations - such as table lookups, convolution operations, etc. However, using the ISA in this way requires the processor itself to also perform these operations, which means that the processor is busy during the execution of these enhanced instructions.

[0004] Additionally, the VPU can use a direct memory access (DMA) system to retrieve data for VPU processing. This allows the DMA system to operate as a data movement engine, but can also perform additional operations such as image padding, address manipulation, overlapping data management, traversal order management, and frame size management. However, as the number of DMA resources (e.g., descriptors, channels, triggers, etc.) increases, so does the programming complexity of programming the DMA system and the VPU. In situations where the tiles of a frame contain spatial or temporal dependencies, dynamic updates of DMA resources become a processing burden on the system. When acquiring unknown or data-dependent data, traditional DMA systems require a processing controller (e.g., an R5 or ARM processing core) to intervene in the processing cycle to determine the updated information used to guide the next processing iteration. For example, in object or feature tracking, the VPU can calculate the next position of the object or feature, and the processing controller will then intervene to update the memory addressing information and trigger the DMA system to use the updated information. However, this processing controller intervention increases latency and requires more complex programming to operate with region-dependent data movement algorithms.

[0005] Furthermore, in safety-critical applications such as autonomous and semi-autonomous machine applications, there are stringent requirements for permanent fault detection and isolation. For example, when deep learning, computer vision, sensor processing, and / or other applications are executed in a machine, permanent fault detection must be performed regularly within the allocated time budget to allow accurate testing while also allowing the application to execute correctly—e.g., with low latency. To this end, end-to-end coverage may be required, with low latency, while meeting the runtime budget of each specific application. Traditional approaches use built-in self-test (BIST) to identify faults, but these BIST techniques either do not include sufficient coverage, introduce excessive latency in the system, and / or do not meet the runtime budget of some applications. Summary of the Invention

[0006] Embodiments of the present disclosure relate to improvements to a vector processing unit (VPU), a decoupled accelerator that can be used to handle offload processing from the VPU, and a direct memory access (DMA) system that supports data movement between memory and the VPU. To address various shortcomings of traditional or existing solutions, the VPU of the present disclosure may include a minimum / maximum hardware collector that is included in the data path from the VPU to the memory, so that the minimum / maximum values ​​can be stored before being stored in the memory. In this way, the minimum / maximum values ​​can be available immediately after the memory write operation is completed, thereby reducing the delay in determining the minimum / maximum values ​​after the values ​​are stored in the memory. In addition, the VPU may include an automatic prediction function that can apply a prediction flag by setting a prediction bit for each value calculated in the iteration after the final iteration. As a result, each group of iterations may include the same number of executed iterations, but one or more values ​​from the final iteration group may not be written out to the memory due to the prediction flag. To address the limitations of sharing between data channels of existing solutions, the SIMD architecture of the present disclosure can define slices in the processor, each slice including multiple channels, and each channel can be configured to communicate between each other. This allows operands from one channel to be used by other channels, eliminating the requirement to copy each operand to each channel for processing. To address the inefficiencies of loading from a single wide memory bank, the VPU can include multiple smaller memory banks to allow for smaller bit alignments—for example, 16-bit alignment, where the memory banks are 16 bits each. Thus, for example, reading the values ​​4 to 67 can occur in a single memory read, rather than two memory reads for 0-63 and 64-127. In addition to this memory bank organization, the VPU can include transpose load and / or store functionality to allow stored values ​​to be offset within the memory banks, eliminating memory bank conflicts and allowing more data to be read or written per cycle. To address data manipulation limitations of traditional instruction sets, a load with permute instruction can be used to send a permute pattern along with the memory address to local memory, allowing data to be retrieved from memory according to the permute or data manipulation pattern. This allows data manipulation and data loads to be performed in the same cycle, reducing latency. To address the drawbacks of table duplication for each value or additional read ports for table lookups, a two-point or two-by-two-point lookup can be performed so that each table can look up two or four points per cycle, respectively. To achieve this, an offset storage pattern of table and per-memory bank address buses and associated logic and routing can be used to allow for parallel lookups of two or four points. In an embodiment, each memory bank can include an associated data cache that can be enabled or disabled based on a given operation.For example, for filtering operations where there is a large amount of data overlap between iterations, a data cache can be used to store values ​​from one or more previous lookups so that only minimal reads are required from each memory bank, thereby conserving energy and power for the system.

[0007] To address the shortcomings of traditional ISAs for VPUs or other processor types, the systems and methods of the present disclosure can use decoupled accelerators that can be configured by the VPU and communicate with the VPU through shared memory, but can perform specific tasks independently of the VPU to allow the VPU to continue other processing tasks in parallel with the accelerator. For example, a decoupled lookup table (DLUT) accelerator can be used to improve the performance of the system when executing lookup tables. In this way, the DLUT accelerator can identify conflicts, resolve conflicts, and increase the throughput of the system, rather than the VPU performing memory bank conflict detection and resolution online.

[0008] To address the shortcomings of conventional DMA systems, the systems and methods of the present disclosure may include a hardware sequencer that operates on frame data that includes a command sequence for the hardware sequencer. For example, the hardware sequencer can operate at the frame level rather than the tile level and can perform sequencing for the DMA engine, eliminating the programming complexity of programming the DMA engine to perform the same operations (e.g., fill, address manipulation, etc.). In some embodiments, the DMA system can include a DMA trigger mode, where the DMA engine controls the movement of tiles to vector memory (VMEM), rather than requiring the VPU to trigger the DMA to load the next tile. Thus, the command sequence is reversed, with the DMA becoming the trigger for the VPU. To address the shortcomings of region-dependent data movement operations in DMA systems, the DMA system can use the DMA and VPU to operate in a tightly coupled loop without the need for process controller intervention. For example, the VPU can update the location information in VMEM for various features and / or objects being tracked, and the DMA can use this updated information to update descriptors in the descriptor memory so that the next data provided to the VPU for processing corresponds to the next location of the feature or object. This process can be repeated until processing is complete, eliminating the need for process controller intervention and reducing system latency.

[0009] Furthermore, to address the deficiencies of conventional approaches to BIST, the present systems and methods may perform a Multiple Input Signature Register (MISR) BIST—for example, to perform fault detection on a programmable vision accelerator (PVA) of a system-on-chip (SoC). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) (e.g., an R5 processor and an ARM processor, CPU, and / or the like). Thus, each component of the PVA may require testing, and the present systems and methods perform a MISR BIST to detect permanent faults in an end-to-end manner. In this manner, permanent fault detection may be performed to cover end-to-end blocks of control and data logic, report errors directly to a safety processor to reduce latency, and be customized for a specific application to meet an associated runtime budget. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present system and method for improving a vector processing unit (VPU) are described in detail below with reference to the accompanying drawings, wherein:

[0011] Figure 1A is an example min / max collection system according to some embodiments of the present disclosure;

[0012] Figure 1B is a flow chart illustrating a method for min / max collection according to some embodiments of the present disclosure;

[0013] Figure 2A is an example system including a processor having an address generation unit with automatic prediction capabilities according to some embodiments of the present disclosure;

[0014] Figure 2B is a table showing a sequence of state changes over time according to some embodiments of the present disclosure;

[0015] Figure 2C is a flow chart illustrating a method for automatic storage prediction according to some embodiments of the present disclosure;

[0016] Figure 3A is a diagram of an example single instruction multiple data (SIMD) datapath organization according to some embodiments of the present disclosure;

[0017] Figures 3B-3D shows operand sharing between slices of a SIMD architecture for filter operations, dot product operations, and sort operations with payloads, respectively, according to some embodiments of the present disclosure;

[0018] Figure 3E A flowchart of a method for computing output using shared operands across lanes of a SIMD architecture according to some embodiments of the present disclosure is included.

[0019] Figure 4A is a logical view of a transposed load for reading and writing memory and a memory bank view corresponding to the transposed load of the logical view according to some embodiments of the present disclosure;

[0020] Figure 4B is a logical view of a transposed load with various row spacing and stride parameters for reading and writing memory and a memory bank view corresponding to the transposed load of the logical view according to some embodiments of the present disclosure;

[0021] Figure 4C is a flow chart illustrating a method for configuring a write operation of a transposed load with a stride parameter according to some embodiments of the present disclosure;

[0022] Figure 4D is a flow chart illustrating a method of performing a write operation of a transposed load using a stride parameter according to some embodiments of the present disclosure;

[0023] Figures 5A-5B shows a data and coefficient layout table in a SIMD architecture for different functions according to some embodiments of the present disclosure;

[0024] Figure 5C shows a hardware architecture for performing loads with permutations and zero insertions according to some embodiments of the present disclosure;

[0025] Figure 5D Some embodiments of the present disclosure are shown Figure 5C Example usage of the hardware architecture;

[0026] Figure 5E is a flowchart illustrating a method of utilizing permutation loading according to some embodiments of the present disclosure;

[0027] Figure 6A shows a 16-way parallel table organization for single-point lookup according to some embodiments of the present disclosure;

[0028] Figure 6B shows an 8-way parallel table organization for two-point lookup according to some embodiments of the present disclosure;

[0029] Figure 6C shows a logical view of a 2-way parallel word type table for 2x2 point lookup according to some embodiments of the present disclosure;

[0030] Figure 6D Some embodiments of the present disclosure are shown Figure 6C Memory view of a 2-way parallel word type table for 2x2 point lookup;

[0031] Figure 6E shows a layout for processing channel pairs using horizontal mixing with interleaved data operations according to some embodiments of the present disclosure;

[0032] Figure 6F shows intermediate and final results of horizontal mixing with interleaved data operations according to some embodiments of the present disclosure;

[0033] Figure 6G is a flowchart of a method for performing a multi-point lookup according to some embodiments of the present disclosure;

[0034] Figure 7A shows elements of data and coefficient arrays according to some embodiments of the present disclosure;

[0035] Figures 7B-7C shows the read operations required for data operands and coefficient operands respectively using a data cache for a memory bank according to some embodiments of the present disclosure;

[0036] Figure 7D shows a memory bank organization for use with a load cache according to some embodiments of the present disclosure;

[0037] Figure 7E shows a hardware architecture for using a data cache in a memory bank according to some embodiments of the present disclosure;

[0038] Figure 7F is a flowchart of a method of using a data cache for a memory bank according to some embodiments of the present disclosure;

[0039] Figure 8A A system including one or more decoupled accelerators according to some embodiments of the present disclosure is shown;

[0040] Figure 8B is a flowchart of a method of performing one or more operations using a decoupled accelerator according to some embodiments of the present disclosure;

[0041] Figure 9A A system including a decoupled lookup table accelerator according to some embodiments of the present disclosure is shown;

[0042] Figure 9B is a table illustrating actions of different components of a decoupled lookup table accelerator when performing various operations according to some embodiments of the present disclosure;

[0043] Figure 9C is a flowchart of a method of performing one or more operations using a decoupled lookup table accelerator according to some embodiments of the present disclosure;

[0044] Figure 10Ais a visualization illustrating filling a frame with a fill value according to some embodiments of the present disclosure;

[0045] Figure 10B is a visualization illustrating address manipulation of a frame's descriptor according to some embodiments of the present disclosure;

[0046] Figure 10C is a visualization showing overlapping data between tiles of a frame according to some embodiments of the present disclosure;

[0047] Figure 10D including visualizations illustrating various raster traversal orders according to some embodiments of the present disclosure;

[0048] Figure 10E is a visualization illustrating a three-way traversal order according to some embodiments of the present disclosure;

[0049] Figure 10F including visualizations illustrating various vertical drill-down traversal orders according to some embodiments of the present disclosure;

[0050] Figure 10G is a visualization illustrating various image sizes in a pyramid configuration according to some embodiments of the present disclosure;

[0051] Figure 10H is a direct memory access (DMA) system including a hardware sequencer according to some embodiments of the present disclosure;

[0052] Figure 10I According to some embodiments of the present disclosure, Figure 10H The frame format of the sequencing command controlled by the hardware sequencer of the DMA system;

[0053] Figure 10J is a method for raster scanning sequences according to some embodiments of the present disclosure Figure 10I An example of a frame format;

[0054] Figure 10K is an example tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, using Figure 10J Perform frame address processing on the example frame format;

[0055] Figure 10L is a flowchart of a method of using a hardware sequencer in a DMA system according to some embodiments of the present disclosure;

[0056] Figure 11A A data flow diagram illustrating a process for configuring a direct memory access (DMA) system using a vector processing unit (VPU) according to some embodiments of the present disclosure;

[0057] Figure 11B is a table illustrating a VPU configuration format written into vector memory (VMEM) by the VPU and read by the DMA system according to some embodiments of the present disclosure;

[0058] Figure 11C is a flowchart of a method for configuring a DMA system using a VPU according to some embodiments of the present disclosure;

[0059] Figure 12A is a diagram of a built-in self-test (BIST) system for performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) according to some embodiments of the present disclosure;

[0060] Figure 12B is a diagram of a BIST system for parallel channel CRC calculation for PVA according to some embodiments of the present disclosure;

[0061] Figure 12C is a flow chart of a method for performing BIST in a PVA according to some embodiments of the present disclosure;

[0062] Figure 13A is an illustration of an example autonomous vehicle according to some embodiments of the present disclosure;

[0063] Figure 13B According to some embodiments of the present disclosure Figure 13A Examples of camera positions and fields of view for autonomous vehicles;

[0064] Figure 13C According to some embodiments of the present disclosure Figure 13A a block diagram of an example system architecture for an example autonomous vehicle;

[0065] Figure 13D is a method for one or more cloud-based servers and Figure 13A System diagram of an example of communication between autonomous vehicles;

[0066] Figure 14 is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0067] Figure 15 is a block diagram of an example data center suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0068] Systems and methods are disclosed relating to various components of a system on a chip (SoC)—e.g., vector processing units (VPUs), direct memory access (DMA) controllers, and hardware accelerators (e.g., programmable vision accelerators (PVAs), such as a PVA including one or more pairs of VPUs and DMAs). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) (e.g., an R5 processor and an ARM processor, CPU, and / or the like). Although the present disclosure (including the various components of the SoC) may be described with respect to an example autonomous vehicle 1300 (also referred to herein as "vehicle 1300" or "ego vehicle 1300"), examples thereof are described with respect to FIG. Figures 13A-13D ), which is not limiting. For example, the systems and methods described herein may be used by, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), driving and non-driving robots or robotic use platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, while the present disclosure may be described with respect to computer vision, machine learning, artificial intelligence, image processing, etc., this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology space in which a vector processing unit (VPU), direct memory access (DMA) system, instruction set architecture (ISA), programmable vision accelerator (PVA), decoupled accelerators, decoupled lookup tables, hardware sequencers, single-input multiple data (SIMD) architecture, and / or one or more other components of a SoC may be used. Furthermore, although the components and related processes described herein may be described with respect to a SoC, this is not meant to be limiting, and these components may be implemented as standalone components, discrete components of a system, and / or integrated into a SoC. In some embodiments, the systems, components, features, functions, and / or methods of the present disclosure may be integrated into a Figures 13A-13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 In the example data center 1500.

[0069] Min / Max Hardware Collector for Anomaly Detection

[0070] For example, in computer vision applications, especially safety-critical vision applications, calculating the dynamic range of intermediate results is an important task. For example, in order to detect noise or errors in intermediate calculations, a known or expected dynamic value range can be used to identify values ​​that fall outside these ranges. In such an example, when values ​​fall outside the known or expected dynamic range, these values ​​can be marked as corresponding to noise, error and / or another problem. Therefore, it may be necessary to collect minimum (min) and maximum (max) values ​​of intermediate results to detect data anomalies. In practice, these anomalies may be caused by, but are not limited to, noise in the image sensor, algorithmic extremes, or data corruption in memory or interconnects. To address these issues, collecting minimum / maximum values ​​is an effective way to detect outliers in this data. Minimum / maximum values ​​are also used in certain algorithms.

[0071] For a specific example, in an autonomous vehicle application, runtime exceptions—such as infinity or non-number—may be invalid values ​​or produce errors, leading to failures or other undesirable results. With this in mind, algorithms executed as part of the autonomous vehicle platform can be evaluated to determine the range of values ​​(intermediate or otherwise) that may be produced during processing. Once the range of values ​​is known, the actual calculated values ​​can be compared to the known range, and values ​​that exceed the minimum or maximum threshold can be flagged as errors. In the event of a flagged error, changes in the processing can be implemented—such as ignoring data for a given iteration, identifying and fixing problems, etc. In this way, because potential runtime exceptions are taken into account, runtime exceptions are not allowed and are not relied upon by the autonomous vehicle.

[0072] As another example, min / max collection can be used in certain algorithms to normalize intermediate results to a certain numerical range, thereby achieving higher accuracy in processing—for example, block floating point. This normalization process can include a dynamic range collection step to collect the minimum and / or maximum values ​​of the array, and an adjustment step to apply a scaling factor to the array. However, to collect the min / max values, the traditional process requires writing all values ​​to memory, then analyzing the min / max values ​​and adjusting the scaling.

[0073] Therefore, these traditional methods for min / max evaluation are performed in software and require additional processing cycles. For example, the algorithm itself may be run to calculate the value, and then software may be run to determine the min / max value and compare it to a known range of values ​​to identify anomalies. The software must execute additional instructions to read the elements in the intermediate result array and then perform the min / max operation. As a result, the runtime of the system for detecting anomalies increases because the algorithm is executed to completion and then an additional process is performed to calculate the min / max value of the algorithm's output. This may cause downstream processing to be delayed until the min / max value is calculated and compared to the threshold, or it may cause downstream tasks to begin performing calculations on data containing errors while the min / max evaluation is being performed. This not only increases runtime but also increases the system's processing requirements and energy consumption due to the execution of these additional cycles to identify anomalous data.

[0074] refer to Figure 1A , Figure 1A is an example processor architecture 100 for min / max collection according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the architecture 100 may include Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features and / or functions similar to those of the example data center 1500 may be used.

[0075] In order to address the deficiencies of traditional minimum / maximum evaluation processes such as those described herein, the present disclosure includes systems and methods for using hardware to perform minimum / maximum collection. For example, during a calculation, the calculated values ​​can be written to memory 106 (e.g., local memory) and used for downstream calculations within the same algorithm or another algorithm. In order to reduce runtime and processing, minimum / maximum collection hardware (e.g., minimum / maximum collector 104) can be used to capture minimum / maximum values ​​before or when they are written to memory 106 - for example, rather than waiting for values ​​to be read out of memory 106 and then analyzing the value of the minimum / maximum value. For example, an enable bit can be used to enable the minimum / maximum collection function of minimum / maximum collector 104, and once enabled, as the values ​​are calculated using processor 102 and written to memory 106 (e.g., before or concurrently with being stored to memory 106), minimum / maximum collector 104 can update the minimum / maximum values. In an embodiment, an enable bit may indicate the type of array being calculated—e.g., signed or unsigned—so that the min / max collector 104 is configured to collect the min / max values ​​for the specific type of array. For example, an enable bit or another type of control feature may be used to disable the min / max collector 104 and / or configure the min / max collector 104 to collect unsigned min / max values ​​or to collect signed min / max values. In the data storage data path, min / max collection logic of the min / max collector 104 may be included to read the values ​​as they are calculated by the processor 102 and stored in the register file to update or maintain the min / max values.

[0076] For example, during operation, the current minimum and / or maximum values ​​can be maintained in the minimum / maximum collector 104, and the current minimum and / or current maximum values ​​can be updated with a new, lower minimum and / or new, higher maximum value, which is written to the memory 106. In the event that the newly calculated value is greater than the minimum and / or less than the maximum, the current minimum and / or maximum values ​​can be maintained by the minimum / maximum collector 104. In this way, the minimum / maximum collector 104 can maintain the current minimum and / or maximum values ​​as each value is calculated throughout the calculation. Once the calculation for a given iteration is complete, the minimum / maximum values ​​are immediately available in the minimum / maximum collector 104, and software and / or hardware can be used to compare these stored values ​​to minimum and / or maximum thresholds associated with the specific algorithm or calculation being executed to determine whether an anomaly exists. For example, a mechanism can be included that allows the collected minimum / maximum values ​​to be read for evaluation. Thus, in contrast to previous approaches, another loop is not required to calculate the minimum / maximum values ​​after the algorithm is fully executed, as the minimum / maximum values ​​are immediately available. Furthermore, in embodiments, the min / max collector 104 (e.g., comprising hardware and / or logic) may be aware of store predictions such that min / max collection may exclude a particular data item if, for example, it is prohibited from being stored to the memory 106 via per-channel store predictions. For example, where the address from the address generator includes a store prediction flag, the calculated value may be ignored for both storing to the memory 106 and updating the min / max collector 104.

[0077] In some embodiments, the min / max collector 104 can be implemented as a feature of a system that includes an address generator—such as one or more of the address generators described in U.S. Non-Provisional Application No. 15 / 141,703, filed April 28, 2016, which is incorporated herein by reference in its entirety. The address generator can be included in any type of processor or other processing unit—such as a vector processing unit (VPU), a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a data processing unit (DPU), and / or another type of processing unit (e.g., a processor with a processor that is ... Figures 13A-13D , 14 and / or 15). In some embodiments, one or more VPUs may be included in a programmable vision accelerator (PVA) and / or as part of a system on a chip (SoC).

[0078] As a non-limiting example, the input for a particular sensor type or algorithm may be limited to 16-bit units. In order to determine the dynamic range of that particular sensor and / or algorithm, the operations associated with the algorithm that processes the sensor input can be evaluated. In such an example, assuming the first operation is the addition of two 16-bit numbers, the first intermediate result is a 17-bit number. The 17-bit number can then be multiplied by a 5-bit number to produce a 22-bit number. If this is the end of the algorithm, it can be determined that the output is likely to be no more than 22 bits. Similarly, the minimum value can be evaluated. Therefore, during deployment, if the minimum / maximum value exceeds this known range (e.g., 22 bits), the output may be flagged.

[0079] In some embodiments, the storage data path (e.g., between the processor 102 and the memory 106) may include saturation and / or rounding logic 108 to constrain values ​​stored to the memory 106 to be between certain upper and lower limits or thresholds, or to be rounded according to certain conventions. Thus, in conventional approaches, the evaluation of minimum / maximum values ​​may be performed after saturation and / or rounding. In the presence of anomalies, these conventional approaches may fail to detect the anomaly because the saturation and / or rounding may hide the anomaly—for example, a low value and / or a high value may saturate between the upper and lower limits configured for the saturation logic.

[0080] However, for a particular implementation, it may be valuable or desirable to have unsaturated, unrounded, or absolute minimum / maximum values—for example, in addition to or instead of saturated minimum / maximum values. Therefore, the minimum / maximum collector 104 of the present disclosure can collect minimum / maximum values ​​from raw or unsaturated data (for example, before the values ​​are manipulated using the saturation / rounding logic 108) for anomaly detection. In an embodiment, the collection of the average value of the data or the average absolute value of the data can be performed. The average value can be calculated, for example, by summing the elements, reading back the sum from the address generator configuration register, and dividing by multiple stored data items (which the application may know). In this way, the minimum / maximum value of the absolute value, the sum of the values, and / or the sum of the absolute values ​​can be added to the processor storage data path, and the configuration and collection of the result statistics can be performed—for example, it can be added to the address generator configuration function set, or it can be managed separately. In some embodiments, the minimum / maximum value collector 104 can collect values ​​before and / or after saturation, rounding, or other calculations using the saturation / rounding logic 108.

[0081] Now refer to Figure 1B, each block of the method 110 described herein comprises a computing process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The method 110 may also be embodied as computer usable instructions stored on a computer storage medium. The method 110 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although the description of Figure 1A While described with respect to architecture 100 , method 110 may be performed by any one system or any combination of systems, including but not limited to those described herein.

[0082] Figure 1B 1 is a flow chart illustrating a method 110 for minimum / maximum collection according to some embodiments of the present disclosure. At block B102, the method 110 includes calculating one or more values. For example, when executing one or more algorithms—e.g., neural networks, computer vision algorithms, filtering algorithms, etc.—the one or more values ​​may be calculated using the processor 102.

[0083] At block B104, method 110 includes comparing the value of one or more values ​​to a currently stored minimum value and a currently stored maximum value. For example, the minimum / maximum collector 104 can compare each value of any number of values ​​to be stored in memory 106 (e.g., a value in a register file) to a currently stored minimum value and a currently stored maximum value (e.g., currently stored by the hardware minimum / maximum collector 104). In such an example, the minimum / maximum collector can compare the value to the currently stored minimum and / or maximum value as the value is calculated and before or while the value is stored in memory. In one or more embodiments, the minimum / maximum collector can be included in a data path between a hardware unit that calculates one or more values ​​and a memory unit that stores one or more values.

[0084] At block B106, the method 110 includes determining whether the value is greater than one of the currently stored maximum values ​​or less than one of the currently stored minimum values. For example, based on the comparison at block B104, the system (e.g., the hardware minimum / maximum collector 104) can determine whether each value to be stored in memory is greater than one of the currently stored maximum values ​​or less than one of the currently stored minimum values.

[0085] At block B108, method 110 includes updating the currently stored minimum value to the value based on the value being less than the currently stored minimum value. For example, if the calculated value to be stored to the memory is less than the minimum value currently stored by the hardware minimum / maximum collector, the hardware minimum / maximum collector may update the currently stored minimum value to the calculated value.

[0086] At block B110, method 110 includes updating the currently stored maximum value to the value based on the value being greater than the currently stored maximum value. For example, if the calculated value to be stored in the memory is greater than the currently stored maximum value of the hardware minimum / maximum collector, the hardware minimum / maximum collector may update the currently stored maximum value to the calculated value.

[0087] In this way, the min / max values ​​can be dynamically updated during storage of values ​​so that once some (e.g., all) values ​​are stored, the min / max values ​​are immediately available by reading them out from the values ​​currently stored in the min / max collector.

[0088] Automatic storage prediction

[0089] In high-clock-rate processors, a popular implementation approach is to configure the processor into multiple pipeline stages. Consequently, there may be a delay between issuing an instruction to load a register from local memory and the time the register becomes available for another instruction operation—e.g., a load-to-use delay. To achieve high throughput despite load-to-use delays, processor compilers and application developers can employ software pipelining and / or loop unrolling. For example, software pipelining can be used to overlap the execution of multiple iterations of a loop, while loop unrolling can be used to extend the loop body by repeating the loop body's contents multiple times. Together, these techniques can allow multiple iterations of the loop's contents to execute concurrently, thereby reducing (ideally, eliminating) idle cycles in the schedule. When performing loop unrolling, the compiler may divide the loop iteration count by the unrolling factor. For example, the compiler may assume that the original iteration count is a multiple of the unrolling factor, so that the unrolled loop can execute with equivalent functional behavior. In such an example, if the original iteration count is 60 and the loop is unrolled by a factor of 6, the unrolled loop can run for 10 iterations. However, if the original loop iteration count is 64, 64 / 6 also results in 10 through normal integer division, so the loop will not execute enough times (for example, the additional 4 iterations may not be executed), resulting in different code behavior after unrolling, which may cause the application to fail. In some techniques, assert statements are added to ensure that the iteration count is indeed a multiple of the unroll factor.

[0090] The collection of steps or operations in a loop body can have a narrow range of optimal or desired expansion factors. For example, the lower limit of the expansion factor may be the minimum number of copies of the loop code to be scheduled to fill gaps due to various delays and achieve optimal performance, and its upper limit may also be the maximum number of copies that can be scheduled with the limited capacity in the register file - for example, this may result in excessive register spills (saving to and restoring from the stack) and lead to suboptimal scheduling. As another example, due to the feasibility of selecting tile width and tile height combinations, allowing iteration counts to be powers of 2 (e.g., 2, 4, 8, etc.), expansion in powers of 2 is acceptable for many applications. However, in embodiments, the loop body may also be optimally expanded 6 or 7 times, while expansion 4 or 8 times may be inefficient. In any case, loop expansion to achieve optimal scheduling may impose inconvenient limits on the number of iterations. Therefore, traditional techniques for addressing this problem may result in performance degradation and increased code size.

[0091] For example, a restriction on the number of iterations is inconvenient, so the programmer may write two loops—for example, a "many" loop and a "remainder" loop—when such a restriction on the number of iterations should not be placed. As an example, the following illustrative code snippets show: Code 1 – a vector addition loop without loop unrolling; Code 2 – the same loop with loop unrolling of 6, which only works when the iteration count is a multiple of 6; and Code 3 – a two-loop solution that works for any iteration count, but the remainder loop is not unrolled and is therefore less efficient, also resulting in larger code size due to the additional loop and iteration count calculation.

[0092] Code 1:

[0093]

[0094] Code 2:

[0095]

[0096] Code 3:

[0097]

[0098] Using the vector processing unit (VPU) of the present disclosure, code 1 can achieve 6 cycles per iteration, code 2 can achieve 1 cycle per iteration, and the performance of code 3 can depend on the iteration count. For the number of iterations (niter), niter = 60 (a multiple of 6, so the remainder does not run), code 3 may achieve 1.0 cycle per iteration, and for niter = 64 (the remainder loop runs 4 times), code 3 may achieve an average of 1.3125 cycles per iteration (e.g., (60*1+4*6) / 64=84 / 64=1.3125).

[0099] refer to Figure 2A , Figure 2A is an example system 200 including a processor 202 (e.g., a VPU) including an address generation unit with automatic prediction capabilities in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the processor 202 may be included in and / or may include a processor that is configured to perform a plurality of operations. Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features, and / or functions similar to those of the example data center 1500.

[0100] In an embodiment of the present disclosure, loads and stores in a code segment may use an address generator 204 in a processor 202 (e.g., a VPU). For example, in each load and store, an address generator (agen) parameter (agen_a, agen_b, agen_c) may be provided to the load / store function. The argument may identify an address generator register that contains parameters that may be used for address calculation for a particular load and / or store operation—e.g., an address pointer, an iteration count, a current loop variable value, etc. In some embodiments, the VPU may be designed to support six (or other values) addressing dimensions per address generator register, thus including six (or other values) iteration counts and six (or other values) loop variables.

[0101] To address the limitation of loop unrolling on the number of iterations, the systems and methods of the present disclosure may include an address generator 204 having logic (e.g., a prediction flag or bit 208) for automatically predicting storage from the address generator 204. For example, the prediction can be used to provide an indication of conditional execution, such as whether (or not) something should be executed. The value of the prediction bit 208 (e.g., 0 indicates storage or 1 indicates prevention of storage, or vice versa) can be used to indicate whether the instruction will be executed. Execution may not refer to the actual execution of the iteration, but rather to whether the result value of the iteration execution is stored in the memory. Therefore, in an embodiment, an instruction that is not executed due to a prediction flag may refer to an instruction or iteration that is being executed, but the result of the execution is prevented or excluded from changing the state of the memory 206. Instruction-level prediction and channel-level prediction may be included. Instruction-level prediction can be used to indicate whether the entire instruction should be executed, while channel-level prediction can be used to indicate which data channels should be executed or not executed.

[0102] In some embodiments, after the loop variable exhausts the iteration count, any subsequent execution of the store instruction is automatically predicted to prohibit further writes to memory 206. In this way, the automatic store prediction feature can allow code to be written cleanly by rounding iteration counts that are not multiples of 6 (or another unroll factor) to the next multiple of 6 and by not changing iteration counts that are not multiples of 6 (or another unroll factor). Although a factor of 6 is used, this is not intended to be limiting, and any unroll factor can be used without departing from the scope of this disclosure. Code 4 below includes an example of vector addition with automatic store prediction.

[0103] Code 4:

[0104]

[0105] Code 4, with an original iteration count of 64, can run the unrolled loop 11 times with 1.03125 cycles per iteration (e.g., 11 x 6 / 64 = 1.03125). Another way to account for the restriction on iteration counts that are multiples of the unroll factor is to calculate the necessary prediction flags in the loop and provide the prediction flags in the store instruction. For example, Code 5, described below, shows an example implementation of the prediction flag calculation.

[0106] Code 5:

[0107]

[0108] Code 5 can be compiled into 1.5 cycles per iteration in the VPU of the present disclosure, so automatic prediction can include performance advantages over predictions calculated in the loop. In an embodiment, the VPU may include a 7-way very long instruction word (VLIW) instruction scheme and may include 2 scalar slots per cycle for predicting the scalar operations required for the calculation. If the loop has more vector operations per iteration, there may be enough scalar slots so that the prediction calculation can fit into the available slots and will not cause performance degradation. Even in computation loops where real-time calculation of the prediction has no impact on performance, automatic prediction may still have advantages in code size and energy consumption.

[0109] Thus, software can be used to configure a number of iterations (e.g., N1-N6), and the software can cause address generator based loads / stores to be performed—typically in a loop. The address generator hardware can maintain loop variables (e.g., variables I1-I6) and can advance the address pointer appropriately. When address generator based loads / stores have executed for more than a preconfigured number of iterations, the address pointer may be left at the last valid address and auto-prediction may be turned off (e.g., by setting a prediction flag) to block subsequent stores to memory. Thus, an “auto-prediction off” internal Boolean state may be included in the address generator 204, and the loop variable iteration logic may be configured to support turning off auto-prediction. For example, and with respect to Figure 2B When initializing the address generator, in addition to loop variables I1-I6, the value of the parameter auto-prediction off ("auto_pred_off") (e.g., prediction bit 208) may be initialized or reset to "0". After the loop variable exhausts the programmed iteration count, auto_pred_off may be updated to "1". As a result of the prediction bit being "1", any subsequent execution of the store instruction may then be auto-predicted and further writes to memory may be prevented.

[0110] exist Figure 2B In the example of , the number of iterations of the address generator of registers N1-N6 can be programmed as N1=4, N2=2, N3=N4=N5=N6=1. The total programming iteration count can therefore be 4*2*1*1*1*1=8, and the result can be performed Figure 2B As shown, the initial state and the following 7 executions (e.g., the first 8 iterations) may correspond to the auto_pred_off bit having a value of 0, and the 8th and 9th executions (e.g., the last 2 iterations) may correspond to the auto_pred_off bit having a value of 1, thereby preventing the results of the 9th and 10th executions from being stored in memory.

[0111] In practice, a VPU can be configured to process a number of vector units working simultaneously - for example, 8, 16, etc. - and therefore the VPU may require the array to be a multiple of the number of vector units. This setup works well if the array is a multiple of the number of vector units. However, typically, an array may not be a multiple of vector units (for example, because there is no guarantee that the data will be calculated based on arrays of the same size), so the array is padded so that processing is always performed on batches of the same size. For example, the remaining iterations can be padded with "0" values, but this still requires an additional loop in the software to process the padded values. Therefore, padding can be inefficient because the added data leads to wasted computation, and it also complicates the software - a common problem in single instruction multiple data (SIMD) software. Therefore, automatic storage prediction can be used to address this issue.

[0112] As a non-limiting example, using batches of 16, as many batches of 16 as possible can be generated from an array, and the remaining values ​​can be included in the final batch, with the remaining or remaining space within the 16 batches being predicted off using the lookahead flag. For a specific example, if an array is 82 in size, five complete sets of 16 might be generated, and in the final iteration, the remaining two elements might be included, while the other 14 elements might be automatically predicted off—thus minimizing the computational waste of padding the batch with 14 values ​​and performing unnecessary computations on the padding data. As another example, if the vector processing granularity includes a width of 32 and the array has 100 elements, three complete 32-element vectors can be processed, and the remaining four elements can be processed using four of the 32 channels (e.g., the lookahead flag might be on), while the other 28 channels might be predicted off. In this way, programmers might be able to vectorize arrays whose number of samples is not a multiple of the number of cells. For example, for each store, the hardware might actually calculate the number of elements to be written to the memory and communicate this information to the store unit. Therefore, even if the mathematical operations for padding or appending elements could be performed and stored, such additional computation and storage would be inefficient. Therefore, a prediction flag can be set so that no additional reads are required and the write of the calculated values ​​from the padding values ​​to memory does not occur (e.g., is blocked or excluded). This automatic prediction can occur at the instruction level, and software can be added to additionally perform lane-level prediction.

[0113] Furthermore, with automatic prediction, additional information may not be needed because the address generator can be programmed for multiple iterations—thus having memory to support automatic prediction—and software instructions can be added to automatically move between predicted stores and prediction-off stores. That way, in the final iteration, the hardware can determine when to store a complete result or when to store a less-than-complete result—for example, because prediction was turned off or otherwise signaled—and this can be performed at zero cost, maintaining performance. Using software alone, this process would require additional cycles, slowing the process.

[0114] In some embodiments, prediction can be used at a per-lane level, allowing these implementations to not only handle iteration counts that are not multiples of the loop unrolling factor, but also to efficiently handle any problem size that is not a multiple of the vector width. In such embodiments, vector registers can be used to drive per-lane prediction, which can provide the advantage of computing information in real time, and by using vector registers, a shortcut can be implemented that eliminates the requirement to copy from the vector registers to the scalar prediction registers, which are then used to apply prediction flags on each lane. For example, per-lane prediction can be performed from vector registers, which can be beneficial when computing per-lane prediction information in a loop and can vectorize the computation.

[0115] For example, to perform certain value replacements in an array—such as replacing any value over 100 with 999—the code could be written as follows:

[0116]

[0117] While this code may be functionally correct, it may result in poor performance. Therefore, the code can be vectorized by incorporating per-lane predictions, such as the following:

[0118]

[0119] When the prediction computation is vectorized in this way, and the prediction per lane can only be transferred via scalar prediction registers, the prediction information needs to be copied from the vector registers to the scalar prediction registers, increasing execution time.

[0120] However, instead of performing bit packing and moving the prediction mask from vector lane 0 to a scalar register, it is possible to use per-lane prediction driven directly from the vector register feature described in this paper in this example, as shown in the following code:

[0121]

[0122]

[0123] Now refer to Figure 2C , each block of the method 220 described herein comprises a computing process that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. The method 220 may also be embodied as computer usable instructions stored on a computer storage medium. The method 220 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although regarding Figure 2A While described with respect to system 200 , method 220 may be performed by any one system or any combination of systems, including but not limited to those described herein.

[0124] Figure 2C 2 is a flow chart illustrating a method 220 for automatic storage prediction according to some embodiments of the present disclosure. At block B202, the method 220 includes determining a total number of iterations. For example, the address generator 204 may determine a total number of iterations to be executed for a given instruction.

[0125] At block B204, the method 220 includes dividing the total number of iterations into multiple groups of iterations. For example, the address generator 204 may divide the iterations by an unrolling factor to generate a loop body including multiple iterations of the loop.

[0126] At block B206, the method 220 includes determining that one of the plurality of groups of iterations includes a first number of iterations that is less than a second number of iterations, the second number of iterations corresponding to the other groups of iterations in the plurality of groups of iterations. For example, the address generator 204 may determine that after separating the iterations by the expansion factor, one group of iterations includes fewer iterations than the other groups. For example, with an expansion factor of 6 and a number of iterations of 62, there may be 11 groups of iterations—10 groups including 6 iterations and one group including 2 iterations. Thus, the address generator 204 may determine that two of the group of iterations, including the remaining two iterations, should be executed and that the other four iterations should be predicted closed.

[0127] At block B208, the method 220 includes generating a prediction flag corresponding to at least one iteration of the set of iterations during execution of the set of iterations. For example, upon determining that the set of iterations does not include a complete set of the same number of iterations as the other sets of iterations, the address generator 204 may enable the prediction flag (change the value of the prediction off bit 208) to indicate that results of the excess iterations should be stored or written to memory.

[0128] At block B210, method 220 includes preventing a value corresponding to at least one iteration of the set of iterations from being written to memory based at least in part on a prediction flag. For example, a calculated value may be prevented from being written to memory based on a set prediction flag.

[0129] Enhanced SIMD datapath organization for vector processors

[0130] In a traditional single-instruction, multiple-data (SIMD) architecture, each SIMD processing unit operates in parallel and independently of each other on its own data path. Some machines allow each SIMD processing unit to communicate directly with its immediate neighbors (e.g., left and right neighbors as a linear array of processing units, or north, south, east, and west neighbors in a two-dimensional (2D) array or processing). However, communicating only between adjacent data paths is limiting and makes operations requiring multiple input operands expensive to implement. For example, convolution is a common operation in fields such as image processing, computer vision, and machine learning. During convolution, various filters can be applied to adjacent pixels, such as, for non-limiting example, a three-tap one-dimensional (1D) filter involving three data operands and three coefficient operands. If these operands cannot be shared across the data paths of a SIMD architecture, six operands must be brought into each data path to produce a result for that specific path. With this in mind, some common approaches implement multiple read ports on a register file, but this requires additional surface area of ​​the SIMD architecture and additional operating power.

[0131] In order to address the defects of traditional SIMD architecture, the SIMD architecture of the present disclosure can allow communication between channels by defining slices (such as vector processing units (VPUs)) in a processor, and the processor includes multiple channels as a group. For non-limiting examples, in a processor, the SIMD channel organization can include a hierarchical organization, which includes a 384-bit data path that can be divided into, for example, 8 48-bit (extended word) channels, 16 24-bit (extended half-word) channels, or 32 12-bit channels (extended bytes). In such an example, each byte can be extended by 4 bits. The first layer of communication above the individual channels can be called a SIMD slice, and can be (for example, but not limited to) 96-bit width, consisting of two extended word channels (for example, two 48-bit channels), four extended half-word channels (for example, 4 24-bit channels) or 8 extended byte channels (for example, 8 12-bit channels). In a non-limiting embodiment, in all four (or other number of) SIMD slices and all channels, the entire processor data path can include four SIMD slices, and the second layer of communication can be global. In this way, operand sharing between lanes of each slice can be achieved, which may be useful in instructions such as filtering, dot products, payload sorting, etc. The SIMD architecture can be included in VPUs or other processor types, such as Figures 13A-13D An example of a processor of an autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Example data center 1500.

[0132] Due to the physical routing of the SIMD architecture, the SIMD instruction set architecture (ISA) can allow sharing between a certain number (e.g., 8) of channels within a slice. For example, as shown in FIG. Figure 3A As shown, within each slice, communication between 32-bit word data types, 16-bit half-word data types, and 8-bit byte data types is possible. As a result, in the example, for example Figure 3B The filter operation shown, with four input operands and four coefficients, can perform 8-bit by 8-bit multiplication and accumulation in a half-word, where the coefficients can be shared with data from different channels. In a traditional SIMD architecture, each channel would need to load all 8 operands to perform the same calculation that can be performed using only three input operands in the SIMD architecture of the present disclosure. Therefore, since each read port is associated with increased surface area and energy consumption, only three read ports are required to save space and power for executing such instructions. In operation, due to sharing between channels within a slice, four accumulators (e.g., 0, 1, 2, and 3) may be filled with the results of the following calculations.

[0133] ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3]

[0134] ACC[1]+=D[1]*C[0]+D[2]*C[1]+D[3]*C[2]+D[4]*C[3]

[0135] ACC[2]+=D[2]*C[0]+D[3]*C[1]+D[4]*C[2]+D[5]*C[3]

[0136] ACC[3]+=D[3]*C[0]+D[4]*C[1]+D[5]*C[2]+D[6]*C[3]

[0137] As shown in the figure, for example, ACC[0] can access other channels of src1a, including D[1], D[2], and D[3], and can also access other channels of src2, including C[1], C[2], and C[3]. Similarly, other accumulators (ACC) can access individual channels of src1 and src2. In traditional vector processors with limited or minimal sharing between channels, this type of operation is not possible. For example, these calculations can include a sliding window approach, where each accumulator includes the result of moving a sliding window relative to the previous accumulator. For example, the first accumulator operates on D[0], D[1], D[2], and D[3], the second accumulator operates on D[1], D[2], D[3], and D[4], and so on. Each accumulator uses the same coefficients C[0], C[1], C[2], and C[3]. This is possible because of the shared physical routing between the channels of the SIMD architecture slice.

[0138] As another example implementation of the SIMD architecture of the present disclosure, and with respect to FIG. Figure 3C As shown, the dot product in the vector multiplication operation can be performed using channel sharing. In such an example, two indices (e.g., D[0][0]) indicate which channel the data belongs to and which output set the data belongs to. For the dot product calculation, each channel uses only data operands from its own channel, but the coefficients are shared between channels. Therefore, the output from each channel may use all four coefficients at some time during the dot product operation. In operation, due to sharing between channels within a slice, four accumulators (e.g., 0, 1, 2, and 3) may be filled with the results of the following calculations.

[0139] ACC[0]+=D[0][0]*C[0]+D[1][0]*C[1]+D[2][0]*C[2]+D[3][0]*C[3]

[0140] ACC[1]+=D[0][1]*C[0]+D[1][1]*C[1]+D[2][1]*C[2]+D[3][1]*C[3]

[0141] ACC[2]+=D[0][2]*C[0]+D[1][2]*C[1]+D[2][2]*C[2]+D[3][2]*C[3]

[0142] ACC[3]+=D[0][3]*C[0]+D[1][3]*C[1]+D[2][3]*C[2]+D[3][3]*C[3]

[0143] As another example operation that may benefit from the SIMD architecture of the present disclosure, one may perform Figure 3DA two-point sort operation. For a two-point sort, the payloads are sorted using two values. This two-point sort exploits the correspondence between channel pairs within a slice and is useful in various computer vision applications, for example. For example, channel 0 has the key for entry 0, channel 1 has the corresponding payload, and so on. The payloads can be sorted based on the comparison of the keys - for example, for each key / payload pair in the following code:

[0144]

[0145] Now refer to Figure 3E , each block of the method 300 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 300 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 300 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although described with respect to the SIMD architecture of the present disclosure, the method 300 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0146] Figure 3E A flow chart of a method 300 for computing outputs using shared operands across lanes of a SIMD architecture according to some embodiments of the present disclosure is provided. At block B302, the method 300 includes dividing the bit width of a processor into a plurality of data slices, each data slice including a second bit width less than the first bit width, each data slice of the plurality of data slices including a plurality of lanes, each lane including a third bit width less than the second bit width. For example, a vector processor may be divided into a certain number (e.g., 4) of slices, and each slice may include a certain number of lanes.

[0147] At block B304, method 300 includes loading the first vector into a first vector register such that a first lane of the plurality of lanes includes a first operand of the first vector and a second lane of the plurality of lanes includes a second operand of the first vector. Figure 3B , a first data operand D[0] of the first vector may be loaded into the first lane, and a second data operand D[1] corresponding to the first vector may be loaded into the second lane.

[0148] At block B306, method 300 includes loading the second vector into the second vector register such that a first lane of the plurality of lanes includes the third operand of the second vector and a second lane of the plurality of lanes includes the fourth operand of the second vector. Figure 3B, the first coefficient operand C[0] of the third vector can be loaded into the first channel, and the second coefficient operand C[1] corresponding to the third vector can be loaded into the second channel.

[0149] At block B308, method 300 includes using the instruction to calculate an output based at least in part on the first operand, the second operand, the third operand, and the fourth operand. Figure 3B , the first accumulator (ACC[0]) can receive the result of the calculation ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3], including the values ​​of D[0], D[1], C[0], C[1], etc. This calculation is possible due to internal sharing and routing between channels of each slice.

[0150] At block B310, method 300 includes storing the output to a register. Figure 3B , the output of the calculation can be stored in the accumulator register ACC[0] and then can be stored in the memory.

[0151] Transposed load and store operations with stride parameters

[0152] In a conventional vector single instruction multiple data (SIMD) processor, the size of the local data memory can be adjusted to match the vector processing width. For example, for a 256-bit vector SIMD processor capable of processing 32 8-bit lanes, 16 16-bit lanes, or 8 32-bit lanes, the local data memory can include, for example, 256-bit wide memory or 512-bit wide memory (e.g., twice the processing bit width). In such an example, the local data memory is organized as a single memory bank with full-width memory words. However, a wide vector SIMD processor with a single full-width memory word bank can be inefficient—especially for unaligned memory accesses. For example, to load an array of 16-element 32-bit arrays at byte addresses 4 to 67, the processor may require two memory reads—e.g., one read from addresses 0 to 63 (including addresses 0 to 3, whose data is not needed for the current operation) and a second read from addresses 64 to 127 (including addresses 68 to 127, which includes data not needed for the current operation). Thus, without the banked memory architecture of the present disclosure, an access pattern may be implemented with multiple loads or stores, which may result in a slower computing core, reduced performance, and increased power consumption.

[0153] With this in mind, a single wide memory group may instead be organized as multiple memory groups - for example, 16-bit memory groups (e.g., 32 16-bit memory groups providing 512 bits of memory bandwidth per clock cycle). In this way, read and / or write operations may be performed within any 16-bit aligned range - thereby reducing the number of redundant read / write operations, such as those described in the examples above. Using such a memory organization, reading addresses 4 to 67 may only require a single memory read. In addition to the memory group organization including smaller individual memory groups, transposed load and / or store functions may also be implemented. For example, a channel offset parameter K may be used to define a row address offset applied to each subsequent channel in the memory. The channel size may correspond to the data element size - for example, 8 bits, 16 bits, 32 bits, etc. When a 2D array is stored in memory with a row spacing of W*K+1 elements, the interleaved access mode may be converted to a vertical mode, where K is the offset parameter and W is 64 / channel size (or the size of the data element). For example, for 32-bit data elements, the row spacing may be 16*K+1. In some embodiments, a SIMD processor may be included as a component and / or may include a processor that is compatible with Figures 13A-13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features and / or functions similar to those of the example data center 1500 may be used.

[0154] As an example, and with respect to FIG. Figure 4A As shown, table 400 may include an illustration of a logical view of a transposed load and a memory bank view with 17 transposed loads with a row spacing exceeding 256 bits. The memory banks are shown as 18 separate 16-bit banks in the memory bank view for illustration purposes only. For example, the memory banks may have a total of 256 bits, a total of 512 bits, or some other total number of bits—for example, each memory bank may be 16 bits wide. In the memory bank view using transposed loads, with a row spacing of 17, a single load operation can be performed to retrieve each highlighted value of the array.

[0155] While using this technique for transposed loads is beneficial for many operations, some algorithms—for example, some computer vision algorithms—may need to access even when using the Figure 4AThe transposed load technique described also operates on data patterns that cannot be completed with a single read and / or write. For example, instead of loading a 16-high vertical vector, it may be necessary to load a sub-matrix that is 8 high by 2 elements wide, a matrix that is 4 high by 4 elements wide, or other matrix or sub-matrix sizes. For example, in a dot product operation, the accumulation may be over two rows of 16 elements, 16 bits at a time, so when storing the output, a T16 transposed store option may be needed with the appropriate row spacing so that the two rows can be written out as one memory write transaction. To address this issue, a stride parameter may be used with the transposed loads and / or stores. In some embodiments, the stride parameter may include strides that are powers of 2 (although this is not limiting), such as strides of 2, 4, 8, 32, etc., which may be referred to as T2, T4, T8, T32, etc. Examples of different transposed loads with stride parameters are shown in FIG. Figure 4B , which includes a logical view and a memory bank view of the transposed load. Figure 4A An example of this, mirrored at Figure 4B , includes a stride parameter of 1, however, other stride parameters are multiples of 2. For example, T2 has a row spacing of 18, allowing a matrix that is 2 elements wide by 8 high to be stored as a transposed load, such that each value can be retrieved using a single load transaction. Similarly, for T4, with a row spacing of 20 and a stride of 4, a matrix that is 4 elements wide by 4 high can be stored, allowing each value to be retrieved using a single load transaction, and so on. Although described as load transactions, this type of format can also be used for store transactions, storing data in memory according to the transpose plus stride parameter.

[0156] In such an example, the line spacing constraint can be adjusted based on the stride. For font T-transposed access, the line spacing can be 16K+1, for font T2-transposed access (e.g., for a stride of 2), the line spacing can be 16K+2, for font T4-transposed access (e.g., for a stride of 4), the line spacing can be 16K+4, and so on. Therefore, the line spacing can be equal to 16K+stride value, or 16K+1+(T-1), where T is the stride parameter.

[0157] In operation, the architecture of the VMEM of the VPU and the instruction set architecture (ISA) of the VPU can be configured to perform transposed load and / or store operations, with or without a stride parameter, to allow data organized in columns in a logical view to be read or written in a single read operation. For example, the ISA can be configured to receive an indication of a starting address for reading data from or writing data to (e.g., for reading or writing data from a register file), an indication of the type of write (e.g., a transposed write operation, with or without a stride parameter), a row spacing value (e.g., the value of K in 16*K+1), and / or a stride parameter value. It should be noted that the value of 16 corresponds to the number of data elements of a particular implementation, but the value of 16 (or W) may be different in different embodiments. Therefore, when writing data to memory according to a transposed write operation, the ISA can receive a starting address, a row spacing, and / or a stride parameter to be written to the VMEM. As a result, when writing values, instead of writing them out in a single data column to a single memory bank, they can be written according to, for example Figure 4A and 4B , or offset as shown in . In the case of using a stride parameter, the first value of the stride can be written to the memory, followed by the next number of elements corresponding to the stride, and then the row spacing can be applied to write the next set of values ​​to the memory bank so that each value can be written to the memory in one cycle. Similarly, during a read operation, according to the transposed memory organization or writing data, the ISA can receive the starting address, the load type (e.g., transposed load, with or without a stride parameter), the row spacing value (e.g., the value of K), and the stride parameter value (e.g., a data type indicator, such as byte, halfword, etc.). The ISA can then access data from various memory banks according to the transposed load instruction (and / or stride parameter) to retrieve one (or more) columns of data in a single read cycle. In this way, a single vector can be returned from a single read operation by retrieving one element from each memory bank.

[0158] Now refer to Figures 4C-4D , each block of methods 420 and 430 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. Methods 420 and 430 can also be embodied as computer-usable instructions stored on a computer storage medium. Methods 420 and 430 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Although described with respect to the SIMD architecture of the present disclosure, methods 420 and 430 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0159] Figure 4CFlowchart of method 420 for configuring a transpose store operation using a stride parameter according to some embodiments of the present disclosure. At block B402, method 420 includes determining the dimensions of a matrix. For example, the width of the matrix may be determined.

[0160] At block B404, method 420 includes determining a stride parameter and a row spacing for storing the matrix based on the dimensions. For example, the row spacing may be determined using 16K+stride value, and the stride value may be determined based on the width of the matrix.

[0161] At block B406, method 420 includes using the stride parameter and the row spacing so that the matrix values ​​are stored in the memory. For example, once the row spacing and stride are determined, the matrix values ​​can be stored in the memory so that the row spacing and stride parameter values ​​do not cause a memory bank conflict when the matrix values ​​are read from the memory.

[0162] Now refer to Figure 4D , Figure 4D A flowchart of a method 430 for configuring a transpose store operation using a stride parameter according to some embodiments of the present disclosure is included. At block B408, the method 430 includes receiving data representing a row pitch in a memory bank in a plurality of memory banks and a starting memory address corresponding to an element in a plurality of elements corresponding to a column in an array.

[0163] At block B410 , the method 430 includes reading a plurality of elements from a plurality of memory banks in a single read operation, reading each element of the plurality of elements from a corresponding memory bank in the plurality of memory banks based at least in part on a row pitch.

[0164] Load with permutation and zero insertion in a single instruction

[0165] In a traditional processor instruction set, a load instruction can form a memory address through some index calculations, read the requested memory data from local memory, and store the memory data in a register. If the application requires additional data manipulation, additional instructions can be used to operate on the memory data in the register. In some cases, the data manipulation may include simple data reorganization. In a traditional processor, even such simple data manipulation in the register file requires additional instructions and therefore additional latency. For example, a traditional system may load data, perform a permutation on the loaded data, and then use the reorganized data to perform one or more operations. If the load instruction is enhanced with this data reorganization capability, some processing time can be saved, and the computing kernel can be executed with higher performance and lower power consumption.

[0166] To address these shortcomings, the systems and methods of the present disclosure add a load with permute instruction that sends the permute pattern along with the memory address to the local memory. As a result, the existing data routing and multiplexing used to handle unaligned loads can be used to perform the permute without requiring a large amount of additional logic. In addition to saving instructions that would otherwise be expended (e.g., performing a permute with dual vector inputs and dual vector outputs requires executing 5 instructions), the overall latency of the permute operation may be reduced. For example, there is no load-to-use delay and no compute delay (e.g., to perform the permute), the only delay is the load-to-use delay. In some embodiments, the loads with permute and / or zero insertion described herein may be included in Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 The components, features, and / or functions of the example data center 1500 may be included in, or may be similar to, these components, features, and / or functions.

[0167] Thus, loads with permutation characteristics can be used to manipulate loaded data from memory into a desired format for the operation. As an example, the coefficient data required by various filtering and dot product instructions may include a specific repeating pattern that can be implemented using loads and permutations. Figure 3C As described, coefficients 0, 1, 2, and 3 may be repeated across the vector width (e.g., 16 bits)—e.g., as Figure 5A As shown. In such an example, the write out to the first register could start with D[0]-D

[15] , then a sliding window of 4 could be used to start the next register with D[0]-D

[19] , and so on. In this filtering example, the coefficients C[0]-C[3] may repeat across the width of the vector, so using permuted loads may help to write the coefficients in this order directly from the load, rather than loading all the data, then performing the permutation, and then writing the vector to the register. Thus, in this example, the permutation pattern for the coefficient data could include {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3). In the same example, the permutation pattern for the data operands could be {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}. In this way, the data operands and coefficient operands can be read out according to the permuted order, rather than being read out sequentially and then permuted before being written into registers for calculation. As another example, for example, Figure 5BAs shown, the filter instruction may include a dual vector coefficient operand and may therefore include a permutation pattern such as {0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 4, 5, 6, 7, 4, 5, 6, 7}. The permutation pattern may be static or fixed, or may be dynamically calculated by the algorithm, which allows the permutation pattern to be flexible and dynamic. In the case where the pattern is a repeating pattern, in an embodiment, the first instance of the repeating element may be loaded, then copied, and then written out to the SIMD lane of the SIMD unit.

[0168] In some cases, it may be preferable to mask certain portions of memory data to zero values. For example, to facilitate visualization in software development or to consume less energy (e.g., compared to retaining random data values), zeros may be inserted for unused entries. In other examples, zeros may be inserted to delineate blocks of data in a data structure, such as where the length of each block of data is not fixed. In such an example, a zero value may indicate a gap between two blocks of data. When processing a constant-sized image block, for example, when extracting some variable-length information (e.g., the location of a feature point) from each image block, zeros may be used to fill in the remaining data that does not correspond to the extracted information.

[0169] In practice, permutation indices can typically include 32 or 16 elements in the readout—for example, in the range 0-31 or 0-15, respectively. To include zero values ​​in the readout, a permutation operation can be used to include negative index values ​​in the load so that zero is written in the corresponding lane of the target register. Thus, during a write operation, for example, a negative value can be written to the corresponding lane of a SIMD architecture instead of zero.

[0170] As an example, an image patch that is 30 wide by 30 high can be processed with a vector operation that uses 16 consecutive entries at a time. Since a width of 30 is not divisible by 16, each row can be processed with two vector operations, the first processing the full vector width of 16 entries, and the second processing the partial vector width of 14 entries. In such an example, it may be beneficial to use zero padding to fill the last two vector lanes instead of random data values ​​that may currently be in memory.

[0171] In one or more embodiments, padding zeros may be inserted into desired lane positions in a SIMD architecture, for example, to save processing time required to write zeros to these lane positions. In the case where there are 16 lanes, a normal permutation pattern might consist of 16 lane indices—for example, 0-15. In this example, if there are values ​​of {100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115} and the index is a permutation pattern of {0,1,2,3,4,5,6,7,8,9,10,11,12,13,-1,-1}, the value finally loaded into the destination register should be {100,101,102,103,104,105,106,107,108,109,110,111,112,113,0,0}. Therefore, the two values ​​of -1 are automatically converted to 0 in the destination register based on the permutation pattern that includes negative values. In the previous method, -1 and -1 would include 14 and 15 respectively, and the values ​​at 14 and 15 in memory would be written to the register. However, these may include random values ​​which may require additional processing time compared to including zero values.

[0172] To implement loads with permutation characteristics, routing and multiplexing in the memory logic can be used—for example, similar routing and logic used to perform unaligned memory loads. For example, to support loading the full memory width (e.g., 32x16 bits) from any 16-bit address (or loading 16x32-bit channels from any 32-bit address), the memory logic can include multiplexing logic to select any of the 32 channels in the memory data to be routed to any target register channel. For example, for unaligned memory loads, this can be driven according to the following logic:

[0173] output_lane[0]=select(start_lane,memory_lane[0..31]);

[0174] output_lane[1]=select((start_lane+1)%32,memory_lane[0..31]);

[0175] output_lane[2]=select((start_lane+2)%32,memory_lane[0..31]);

[0176] …

[0177] output_lane

[31] =select((start_lane+31)%32,memory_lane[0..31]).

[0178] In an embodiment, a modulo operator (%) may be used to wrap the total number of lanes. Thus, for example, where the starting lane is lane 3, lanes 3, 4, 5, ..., 31, 0, 1, 2 would be used as outputs to the register lanes.

[0179] For loads with a permutation feature, this same logic can essentially be reused, but with modified logic included to perform the permutation operation. An example of the modified logic is as follows:

[0180] output_lane[0]=select((start_lane+permute[0])%32,memory_lane[0..31]);

[0181] output_lane[1]=select((start_lane+permute[1])%32,memory_lane[0..31]);

[0182] output_lane[2]=select((start_lane+permute[2])%32,memory_lane[0..31]);

[0183] …

[0184] output_lane

[31] =select((start_lane+permute

[31] )%32,memory_lane[0..31])

[0185] As an example, and with respect to FIG. Figure 5CAs shown, a hardware structure 500 (which may be included as part of a VPU, SIMD unit, SoC, or other device type) having a crossbar switch 510 (e.g., a crossbar switch) can be used to obtain data from any location in memory 512. And drive the data to any channel in the SIMD through corresponding multiplexers (muxes) 514A-514N. In this way, any of the 16 inputs (or other width memories or registers) may be able to be written to any of the 16 output locations or channels. This may facilitate unaligned accesses, so that load operations can start from any address and then align downward. For example, if data in memory is read from locations 2-18, the data can be read from 2-18 but aligned with channels 0-16 (e.g., 2 goes into channel 0, 3 goes into channel 1, and so on). This is not possible in traditional systems, where vector loads need to start at locations that are multiples of 16, such as 0, 16, 32, etc. As Figure 5C As shown, since data from any memory index can be output to any lane in, for example, a SIMD unit of a VPU, permutations can also be accomplished. Multiplexer 518 can be used to inject or insert permutation controls for each lane to inform the multiplexers 514 of the crossbar 510 which memory location to read data from based on a starting location (which can be aligned or unaligned) and a permutation pattern. Thus, instead of simply extracting data from an aligned location, a permutation pattern can be used to update the location of the memory read so that each multiplexer 514 sends the correct data to each lane of the SIMD unit. Additionally, multiplexer 516 can be used to insert zeros for permutation patterns that include negative values ​​or other values ​​indicating zero insertion (e.g., where values ​​other than negative values ​​are used to cause zero insertion). Thus, once the memory access location is sent from multiplexer 518 to the crossbar 510, and the values ​​from the memory access are sent to multiplexer 516 for zero insertion, the values ​​corresponding to the negative values ​​in the permutation pattern can be converted to zero values ​​to fill the values ​​of the corresponding SIMD lane. Although in Figure 5C Only four groups of channels, multiplexers, and memory indexes are shown in FIG, which is not limiting and any number of groups may be included without departing from the scope of the present disclosure.

[0186] Figure 5D An example use of the hardware architecture 500 is illustrated. For example, Figure 5D The illustration in .5D may be based on the following information:

[0187] crossbar_mode=1;

[0188] start_lane = 2;

[0189] permute pattern={3,1,-1,…,2}={011b,001b,111b,…,010b};

[0190] mem read bus={100,101,102,…,103}

[0191] permute_low = {3, 1, 3, ..., 2}; / / lower 2-position permute

[0192] permute_sign = {0,0,1,…,0}; / / permuted bit 3

[0193] read data output={103,101,0,…,102}

[0194] Alternatively, the following C code can describe Figure 5C and 5D The logic circuit of the hardware architecture:

[0195]

[0196] In this way, Figure 5D In the example of FIG, a bit value of 1 in multiplexer 518 can indicate that the load permutation value should be selected, and these values ​​{3, 1, 3, ..., 2} can be transmitted to the corresponding multiplexer 514 of the crossbar switch 510. As such, the values ​​of {103, 101, 103, ..., 102} can be read from the memory and sent to the multiplexer 516, where the permutation pattern can include a third value of -1, so that the value of 103 can be converted to 0 by zero insertion. Thus, the final value of {103, 101, 0, ..., 102} can be read back into the vector register.

[0197] Now refer to the figure Figure 5E As shown, each block of the method 550 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 550 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 550 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, by way of example, Figure 5C The method 550 is described with reference to a hardware structure of FIG. However, the method 550 may additionally or alternatively be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0198] Figure 5E5 is a flow chart illustrating a method 550 for performing a load using a permutation operation according to some embodiments of the present disclosure. At block B502, the method 550 includes determining a permutation pattern for loading data from memory. For example, the permutation pattern can be static or dynamically calculated. The permutation pattern can be aligned (e.g., 0 to 16, or 0 to 32), unaligned (e.g., 2 to 18), repeating (e.g., 0, 1, 2, 3, 0, 1, 2, 3, ..., etc.), and / or other pattern types.

[0199] At block B504, method 550 includes determining a memory address location for each of the plurality of channels based at least in part on a permutation pattern. For example, the permutation pattern may indicate from which memory address location data for a particular channel or register should be loaded. The permutation pattern may be implemented using multiplexer 518 so that the correct memory address according to the permutation pattern is sent to crossbar switch 512.

[0200] At block B506, method 550 includes loading a value into each of the plurality of channels based at least in part on the memory address location. For example, based on the memory address location, multiplexer 514 from crossbar 512 may retrieve a corresponding value from memory to write to one or more channels within one or more vector registers. In some embodiments, multiplexer 516 may also be configured to convert a value associated with a negative value in the permutation pattern (or other value indicating zero padding) to zero. Thus, in the event that one or more negative values ​​are included in the permutation pattern, the value loaded from memory may be converted to zero before being written to the vector register.

[0201] At block B508, method 550 includes executing one or more operations within each of the plurality of lanes using the values ​​and the at least one instruction. For example, once the vector registers or processing lanes of the SIMD unit are filled, one or more operations—e.g., arithmetic instructions, logical instructions, shift / rotate instructions, bit manipulation instructions, comparison instructions, conversion instructions, constant generation instructions, and / or the like—can be executed using one or more processing units corresponding to the one or more processing lanes.

[0202] Multi-point lookup with hybrid for performing table lookups

[0203] In traditional processors with vector SIMD computations, local memory may include a bit width that matches the bit width of the vector SIMD. As a result, these processors may typically only support read and / or write alignment and granularity corresponding to the bit width. However, table lookups are a common technique in embedded environments such as digital signal processing (DSP) and computer vision to implement various nonlinear functions. For example, square root, logarithm, sine, and cosine functions may require table lookups. To perform these functions, the input space can be uniformly sampled in a one-dimensional (1D) grid, and the outputs can be recorded at these input points in a one-dimensional table. However, when using table lookups to implement nonlinear functions, a trade-off is typically made between table size (e.g., the number of entries in the table) and accuracy. In order to improve accuracy without requiring a large table size, an interpolation lookup can be performed, where two points are looked up around the fractional index for linear interpolation, or three points are looked up around the fractional index for quadratic interpolation.

[0204] As an example, where a sine function is implemented using a lookup table, and the sine values ​​are tabulated in integer degrees, then table[0] = sin(0 degrees), table[1] = sin(1 degree), table[2] = sin(2 degrees), and so on. In such an example, if the evaluation is sin(1.7 degrees), then the score of table[1] * 0.3 + table[2] * 0.7 can be used to linearly interpolate between the two integer degree entries. In this example, the second entry of table[2] gets the score as the weight, and the first entry gets 1 minus the score, so the closer the score is to 1.0 or the position corresponding to the second entry, the higher the second entry is weighted.

[0205] As another example, an image or patch of an image can be resampled, which can involve looking up available pixels around a certain fractional pixel coordinate and then performing an interpolation lookup. In such an example, the table can include image patches and can be two-dimensional. In this case, bilinear interpolation can be performed to interpolate in two dimensions, each dimension being linear. For example, a patch at position Y=5.1, X=7.6 can be interpolated according to the following calculation:

[0206] (patch[5][7]*0.4+patch[5][8]*0.6)*0.9+(patch[6][7]*0.4+patch[6][8]*0.6)*0.1

[0207] However, performing this type of interpolation lookup in a conventional processor is expensive because a separate lookup needs to be performed for each value in each table. To speed up this process, the table can be replicated to allow any number of lookups to be performed simultaneously using different instances of the table. For example, in the example above, when looking up patches at 5, 6, 7, and 8, the table might be replicated at least 4 times to allow parallel lookups in the four tables. For example, where a processor (such as a VPU) supports 32-way parallelism, the table might be replicated 32 times. However, while replicating the table may increase throughput per cycle, the replication also requires additional memory capacity and usage, which may not be available or optimal in some implementations.

[0208] With this in mind, the systems and methods described herein use two-point and / or two-by-two (2x2) point lookup operations to increase throughput (or match the throughput of, for example, 32-way parallelism) while saving memory space. For example, using a per-memory bank address bus and associated logic and routing, a two-point or 2x2-point (e.g., 4-point) parallel lookup can be performed with less memory usage. Therefore, a single lookup of the table may produce two points in a two-point lookup or four points in a 2x2-point lookup. This can be done based on the hardware settings - for example, bank addresses, logic, routing, etc. - and the storage mode in the memory, allowing multiple data to be read without bank conflicts. As mentioned above, without these features, to implement, for example, a 32-way parallel lookup, the table would need to be copied 32 times. For example, this 32-way parallel lookup can be performed using the following C code:

[0209]

[0210] In this example, the lookup portion of the loop can perform 32 lookups per cycle, lasting two cycles (the lookup and blend are performed in memory and the vector math slot, respectively, and are pipelined to two cycles per iteration), and interpolate to produce 32 outputs. Therefore, the entire lookup / interpolation is 16 outputs per cycle, and requires 32 copies of the table.

[0211] As a further example, and with respect to FIG. Figure 6A, shows a 16-way parallel table organization for performing a single-point lookup with an index vector {0,1,2,3,4,5,4,3,...}. In such an example, using conventional architectures and memory layout techniques, a first lookup and a second lookup would need to be performed sequentially to read two entries from each memory bank. For example, a first memory bank T0 includes values ​​at T0[0] and T0[1] to be read in a lookup operation, but because these values ​​are all in the same memory bank, T0 (which may only include a single read port), the first value T0[0] is read in the first pass and the second value T0[1] is read in the second sequential pass. Using such a memory layout, if two reads occur in the same memory bank, a bank conflict will occur, which may cause processing delays and / or cause algorithms or other computations to not execute correctly.

[0212] However, using the architecture of the present disclosure, the same 32 lookups may require only 16 table copies for a two-point lookup or only 8 for a 2x2 point lookup. For example, for a two-point lookup, the same performance of 16 outputs per clock cycle can be achieved with 16 copies of the table, reducing the memory footprint by a factor of two. A 16-way parallel variant of the instruction can return a double vector with the first entry in the lower single vector and the second entry in the upper single vector. In C code, this 16-way parallel lookup and interpolation can be expressed as follows:

[0213]

[0214]

[0215] In such an example, the lookup and interpolation portion of the loop might only require a single clock cycle (the lookup and blend are performed in memory and vector math slots, respectively, and are pipelined to one cycle per iteration), and interpolated to produce 16 outputs. Thus, the lookup / interpolation is 16 outputs per cycle. As an example, and with respect to Figure 6B , illustrates an 8-way parallel table organization for performing a two-point lookup with an index vector {0,1,2,3,4,5,4,3,...}. In such an example, since each memory bank T0, T1, T2, etc. contains only a single value to be read during the lookup operation, all 16 values ​​can be read out in a single pass, rather than Figure 6A For example, Figure 6AIn the example, only eight values ​​can be read in each of the two passes due to possible bank conflicts. To this end, in an embodiment, the instruction for the lookup can include a single index and a pattern that includes not only the search index but also the search index plus a position. Thus, the instruction can result in two values ​​being read for a two-point lookup, and the values ​​can be written to the lookup table in this format to allow this single read to be performed without bank conflicts.

[0216] As an example, when performing vector operations, each channel of the VPU can process a set of pixel values ​​retrieved from memory. In some cases, a channel can process multiple values ​​from the same memory bank, which can lead to memory bank conflicts because the memory bank may only include a single read port. Therefore, the method and system of the present disclosure distributes the values ​​between the memory banks so that memory bank conflicts do not occur, and for example, each value for a single processing channel of the VPU can access each corresponding value in the single processing channel during a read cycle.

[0217] In a conventional system performing 2D bilinear interpolation lookups, four lookups (e.g., 2x2) are required per output, allowing for an optimal throughput of 8 outputs and 32 table copies per clock cycle. Using a 2x2 point lookup, 8 outputs can be achieved per cycle, with 8 copies of the table (compared to 32), thereby reducing the memory footprint required for the parallel sub-tables by a factor of four. For example, for a 2x2 point lookup, two entries can be read from one row of the 2D table, followed by 2 entries from the next row. To avoid memory bank conflicts in any memory bank, the row spacing in the 2D table can be limited to m*k+2, where m is the number of entries stored horizontally in each sub-table and k is sufficient to store any integer in one row of the table. For an 8-way parallel 16-bit table, m=32(16-bit memory words) / 8(parallelism)=4. For a 2-way parallel 32-bit table, m=16(32-bit memory words) / 2(parallelism)=8.

[0218] As an example, and with regard to Figures 6C-6D , row spacing constraints can be used to avoid memory contention. In such an example, a 2-way parallel font table for 2x2 point lookup is illustrated with a row spacing of 10. The number of consecutive elements in the subtable (m) is 8, where A[0][0…7] is placed consecutively in the subtable, conforming to the formula 8k+2, where k can be any integer. Therefore, no matter which index value is used to start, the 2x2 points to be retrieved can be placed in different groups, which is mathematically guaranteed. For example, the group numbers of the 2x2 points relative to the subtable are summarized as follows:

[0219] index%8,

[0220] (index+1)%8,

[0221] (index+line_pitch)%8=(index+8k+2)%8=(index+2)%8,

[0222] (index+line_pitch+1)%8=(index+8k+2+1)%8=(index+3)%8

[0223] There are typically 4 entries to retrieve via a 2x2 lookup in the group number, with index %m, (index+1)%m, (index+2)%m, (index+3)%m relative to the subtable. As long as m>=4, there should be no bank conflicts. Figures 6C-6D In the example, the lookup could include 2D indices of (0,1) and (1,3), using Y then X as a convention for storing pixels in row-major order. Figure 6C In , a logical view of two two-dimensional tables is shown, and in Figure 6D The memory layout view of the values ​​from the table is shown in Figure 1. In the logical view, the lookup is 2x2, as shown in the figure, and the memory layout view shows four points, each in a different memory bank (or a different column in the figure), so each of these values ​​can be read in a single memory cycle or pass. Based on instructions and read patterns that use indexes (for example, (0,1) and (1,3)), the values ​​in the table can be stored in memory in such a way that each value can be read from memory in a single pass. Therefore, using this memory layout and read instructions, the four entries of each sub-table can be returned in the following format each cycle:

[0224] Single vector with lower destination: A[0][1], A[0][2], B[1][3], B[1][4], (the rest are filled with zeros)

[0225] Single vector with higher destination: A[1][1], A[1][2], B[2][3], B[2][4], (the rest are filled with zeros)

[0226] Despite Figure 6C Although shown as two 2D tables of 10 elements wide by 3 elements high, such as Table A and Table B, this is not limiting and the tables can be of any width and / or height, depending on the embodiment. Similarly, Figure 6D The memory layout in , includes a 16 element wide x 3 high layout, but this is not limiting, and the memory width and / or height may be any configuration depending on the embodiment.

[0227] In some implementations, for example, when sampling an image patch, interpolation between a portion of pixels may be performed. In some embodiments, to interpolate the looked-up values ​​without additional instructions to manipulate the data, a Vector Horizontal Interleave Blend (VHBlend_I) instruction may be executed that may include a horizontal blend with interleaved data to blend between channel pairs. For example, using this instruction, post-lookup bilinear interpolation may be performed in the same loop. This instruction may be used to interpolate the looked-up values ​​without additional instructions to manipulate the data. Figure 6E The table layout is used to handle each channel pair. In this way, the calculation of Y0 and Y1 can be calculated as follows:

[0228] Y0=x*(1–alpha0)+y*alpha0

[0229] Y1=z*(1–alpha1)+w*alpha1

[0230] Thus, the instruction may cause horizontal mixing between lane pairs x and y, z and w, and may cause the outputs to be interleaved in the destination register. For example, the following C code snippet may be used to achieve optimal performance on an 8-way parallel table using a 2x2 point lookup.

[0231]

[0232] In this 8-way parallel table organization, the subtables are designated A, B, ..., H, and the loop can perform lookups and interpolations, resulting in 16 outputs per iteration. In such an example, the inputs can be organized as follows:

[0233]

[0234] Examples of intermediate and final results of this instruction are shown in Figure 6F , which includes arrows indicating the mixing and interleaving patterns of the data.

[0235] Now refer to the figure Figure 6G As shown, each block of the method 600 described herein includes a computing process that can be performed using any combination of hardware, firmware and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 600 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 600 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. In addition, the method 600 can be performed by any one system, structure or component or any combination of systems, structures or components, including but not limited to those described herein.

[0236] Figure 6Gis a diagram illustrating a method for performing a multi-point lookup (eg, in a single clock cycle in a decoupled lookup table (DLUT) accelerator) according to some embodiments of the present disclosure, such as with respect to FIG. Figures 9A-9C Flowchart of method 600 of the embodiment of the present invention (described in detail in block B602). At block B602, method 600 includes copying a table to memory to include a first value at a first physical address in a first memory bank and a second value at a second physical address in a second memory bank, the first value and the second value being included in the same column in the logical memory view of the table. For example, a table can be copied to memory any number of times to exploit the system's memory access parallelism. The table can include a first value at a first logical address and a second value at a second logical address in the same column as the first value. If stored to memory in this configuration, a memory bank conflict may result because the two values ​​may be stored to the same memory bank. Therefore, when the table is copied to memory, a write instruction can write the first value to an adjacent first physical address—e.g., in another memory bank—as the second value, so that both values ​​can be retrieved in the same cycle.

[0237] At block B604, method 600 includes determining a first index corresponding to a first physical address in the memory.For example, a read operation may use an index indicating a first location in the memory from which to begin reading a value.

[0238] At block B606, method 600 includes reading a first value at a first physical address and a second value at a second physical address during a single cycle based at least in part on a read instruction corresponding to a multi-point lookup. For example, when a table is copied to memory, the table can be copied so that pairs of points in the same column or table (e.g., corresponding to pixels in the same column of pixels) are stored in separate memory banks. Thus, using a read instruction for a two-point lookup that uses the index of the first point in the point pair to read a first point and an adjacent second point stored in different memory banks, the first value and the second value can be read from a first memory bank storing the first value and a second memory bank storing the second value in a single cycle. This operation can be performed for each pair of values ​​in each copied table to generate a high vector including the first value from each table and a low vector including the second value from each table, and these vectors can be used as vector registers in the VPU and instructions that generate output (e.g., interpolation, etc.).

[0239] At block B608, method 600 includes performing one or more operations using the first value and the second value. For example, the first value and the second value may be loaded into one or more channels of the VPU, and square root, logarithm, sine, and cosine functions may be performed, linear or bilinear interpolation may be performed, and / or other types of operations may be performed. In the case of performing interpolation, a table is replicated 16 times, for example, 16 two-point lookup operations may occur to produce 32 values—two values ​​for each vector channel of the VPU—and interpolation may be performed on each channel to output 16 results. Thus, 16 interpolated outputs may be produced using only 16 copies of the table per cycle. This may be a result of using a two-point lookup, because the table containing the values ​​may only need to replicate half of the traditional single-point lookup operation (e.g., 16 times instead of 32 times), allowing for the same 32 values ​​with half the memory usage.

[0240] Each memory bank in the vector memory loads the cache

[0241] In conventional processors, a data cache may have a width of, for example, 32 bytes per cache line. A cache line is a unit of data that is tracked by the hardware. For example, the hardware may track cache line usage information in tag memory, including the full system address, whether the cache line has been written to, and the time the cache line was last read relative to other cache lines, to determine when to evict a cache line. In some implementations, a data cache is local memory, or a portion of local memory, that is used to temporarily map larger data structures stored in external memory to local memory so that the data can be processed without being subject to the long latency of processing the external memory directly. This type of data cache is often used in conventional desktop or laptop computers.

[0242] As a non-limiting example, a programmable vision accelerator and / or VPU comprises an embedded processor designed to run a smaller set of highly optimized code. In this type of processor, data caching may not be implemented because the programmer can manage the contents of the local data memory. The systems and methods of the present disclosure may include local memory that is managed by the programmer rather than being cached, but may also include additional data caching capabilities in one or more (e.g., each) memory bank. The data cache can be narrow, such as but not limited to 16 bits wide, compared to a more traditional data cache that includes, for example, 32 bytes. In contrast to traditional data caches where the primary goal is to reduce latency, the data cache can be used primarily to reduce power consumption.

[0243] For example, in computer vision processing, data access patterns often have a certain degree of locality (e.g., staying in a certain neighborhood for a while before moving on to the next neighborhood). For example, when performing a 7x72D filter using the VFilt4HHW instruction described here (computing 4 taps at a time), the data read flow can read from the neighborhood 3 times from memory, then move to another neighborhood and read 3 more times, etc. In the coefficient read operation, the same array of zero-filled values ​​can be used (e.g., 7*2*4=56 halfwords), advancing four halfwords at a time until the last group of 4 halfwords is read, and then starting to work back from the beginning of the 56 halfword array again until the filter kernel is completed.

[0244] Therefore, in order to take advantage of these local access patterns and reduce power consumption due to memory accesses, a load cache of data can be implemented in each memory bank with two-way set associativity (holding, for example, a total of 64 halfwords). When the load cache is enabled, the set of most recent reads of the read data (e.g., most recent, two most recent, three most recent, etc.) can be recorded in the cache, along with the row address and the most recently read entry in the tag memory. As a result, when the same memory address is read again, a cache hit may occur, and the cache may provide the data, rather than requiring the data to be read again from local memory. In an embodiment, the load cache can be located between the memory logging logic and the memory itself, so that whenever there is a cache hit, the memory read for that particular address or value will be stopped or not occur, saving power.

[0245] Using this cache structure, and for the 7x7 2D filtering example above, the load cache can allow the system to skip almost two-thirds of the data reads and almost all of the coefficient reads in the steady state. The use of the data cache in each memory bank is described in Figures 7A-7C For example, the VFilt4HHW instruction can perform a 4-tap filter of a potentially larger task and may consume two single halfword data vectors—for example, data[0-15] and data[4-19]—and a single halfword vector of coefficients—for example, coef[0-3]—repeated four times to fill a single vector of 16 elements. In a 7x7 2D filter implementation using the VFilt4HHW instruction in two vector math slots, you can use Figure 7A. Since the VPU of the present disclosure can be configured to read double vectors, data[y][0-15] and data[y][16-31] can be read as double vectors. Similarly, data[y][4-19] and data[y][20-35], and data[y][8-23] and data[y][24-39] can be read as double vectors. Likewise, the data and coefficient reading mode can correspond to Figures 7B-7C Assume that the row spacing of the data is 100 and the row spacing of the coefficients is 8.

[0246] Figure 7D Illustrate the memory bank organization. For example, a 2-entry fully associative cache holds two locations' worth of data in any superbank, and data and coefficients can be placed into different superbanks to allow the cache to work efficiently. In a coefficient read, memory banks 0-3 might first hold coefficient elements 0-3, add elements 32-35, and then reading elements 64-67 will evict elements 0-3, which will be repeated as a pattern in the next coefficient read. In the steady state with the load cache enabled, only four memory banks can be read per scan of the coefficient read pattern. Therefore, the memory bank read savings by using the load cache for data might be (3*32-(32+4+4)) / (3*32)=58.3%, and for coefficients it might be (14*16-4) / (14*16)=98.2%.

[0247] Therefore, in certain algorithms—such as computer vision algorithms with sliding windows—a load cache can be used to save power. For example, without a load cache, each memory bank would need to be read every cycle, even if most of the data is the same. In the example of reading out 512 bits on each iteration, the first 512 bits can be read, then another 512 bits can be read, and so on. For example, if the sliding window is only 8 bytes, then only 64 bits are new on each iteration, and the remaining 448 bits are the same. Without a data cache, these 448 bits would need to be read again from the data bank. However, by using a data cache for each memory bank, these 448 bits can be fetched from the load cache, and only the 64 new bits need to be read from the other memory banks. Thus, the power required to read 448 bits from the memory banks is saved. Examples of algorithms that can benefit from using a load cache are spatial filtering operations, deep learning inference operations (such as convolution operations), etc.

[0248] about Figure 7E, shows the hardware architecture or logic for a memory bank with a load cache. For example, support for unaligned access in a memory, such as a vector memory (VMEM), can accelerate sliding window data access. This is a key memory access pattern for many computer vision algorithms, including filtering and convolution. For sliding window vector loads, most of the data from the random access memory (RAM) bank 702 remains unchanged. In such an example, when sliding 4B, only 4B of data changes in a 64B vector load, so only 4B of new data is read from the RAM bank 702. To optimize the power of the VMEM RAM, a micro-cache called a "load cache" can be attached to each bank for each super bank - so there are a total of 3 super banks x 32 banks = 96 load caches per VMEM. In a non-limiting embodiment, the configuration of each load cache can include a two-line (2x2B=4B) capacity, full associativity, and a pseudo least recently used (pLRU) replacement policy.

[0249] The data cache, where recently accessed data is stored, is divided into two parts: a tag store 706 and a data store 704. In the tag store 706, cache addresses and control information corresponding to previous accesses can be stored, and in the data store 704, data from previous accesses can be stored. The control information in the tag store 706 can include a valid flag (e.g., whether the entry is valid), a dirty flag (e.g., whether the entry has been modified and needs to be written back to memory), and / or a last-used flag (e.g., if an entry is to be replaced, a least recently used policy is used to indicate the entry to be replaced). Because the cache is a load cache, writing data may not update the cache, but the valid and last-used flags may be included in the tag store 706. The valid flag or bit can be used to qualify an address match, and any write should invalidate the entry. The last-used flag may be updated with each access.

[0250] As described herein, in order for the caching scheme to be effective, the storage capacity of the load cache is much smaller than the storage capacity of the memory or RAM bank 702 to reduce access time and save power. In one embodiment, each load cache may correspond to a single RAM bank 702, which may each be a 2048x16-bit memory, and the load caches may each be a 2x16-bit data store 704 with a 23-bit tag store 706 (e.g., 2 entries x (11-bit address + 1-bit valid) + 1-bit last used).

[0251] In operation, the offset 722, row address 724, and increment 726 may be used to generate a memory address for a memory access. This memory address may be tapped for comparison with the tag store 706—for example, with some number of previously accessed addresses (e.g., two previously accessed addresses). The arrow entering the top of the tag memory 706 may represent a memory address. In some embodiments, the tag memory 706 may use the entire memory address to compare with the stored memory address from the previously accessed address. In other embodiments, a subset of the address bits from the memory address may be used to address a subset of the tags, thus only comparing the subset of tags with the memory address. For example, in the case of a large number of previously accessed tags stored in the tag memory 706, only a subset of the tags may be compared with the subset of memory address bits used to reduce area and save power. In load cache designs with fewer tags—for example, corresponding to two previously accessed tags—the entire tag of the previous entry may be compared with the entire memory address. The "==?" decision block 720 compares the current memory address of the RAM bank 702 with the address stored in the tag memory 706. When there is a miss (e.g., the tag and memory address do not match), the read of the RAM bank 702 can be enabled using the read enable 708 and the read data multiplexer (rd data mux) 712, and the RAM bank 702 can be selected and read out to send to the staging flip-flop 716. When there is a hit (e.g., the tag and memory address match), the data store 704 can be addressed with a 0 or 1 (in an embodiment with two entries) to indicate which previous access the hit corresponds to. The corresponding entry in the data memory can be sent to the staging flip-flop 716 via the read data multiplexer 712. The staging flip-flop 716 can return the read-back data to the processor pipeline, ultimately routing it to the destination scalar or vector register of the load instruction.

[0252] The staging flip-flop 714 may correspond to a parity check. For example, a sufficiently large memory may be required to have a parity bit (e.g., in the parity terminal 710) to allow error detection and / or error correction. Error detection may be used in the memory (e.g., VMEM), and / or error correction logic may be implemented on the readback data.

[0253] Thus, the load cache may include tag bits in the tag storage 706 for way 0 and way 1, each tag bit including an 11-bit address and a 1-bit valid bit. The load cache may also include a 1-bit pLRU, as well as data bits in the data storage 704 for way 0 and way 1, each data bit including 16 bits of data and 2 bits of parity. Once the load cache is enabled, it may be searched in the D1 phase. To minimize power consumption, only the load caches of the RAM banks 702 participating in the load may be enabled. For example, for a single vector load, only 16 of the 32 load caches may be searched. Upon a load hit (e.g., when the load cache contains the data to be accessed), the read enable for a given RAM bank 702 may be suppressed, thereby preventing the RAM bank 702 from being illuminated. The pLRU 720 may also be updated in the D1 phase. In the D2 phase, the data and parity bits may be read from the load cache hit way and multiplexed with the RAM result.

[0254] On a load cache miss, in the D1 phase, in victim mode, an existing entry to be evicted to make room for the new entry can be determined based on the valid bit and pLRU. The tag of the victim path can then be updated with the miss address, and the read enable 708 of the RAM bank 702 can be unchecked. In the D2 phase, the data / parity from the RAM bank 702 is not only sent to the read data crossbar, but the data is also filled into the evicted cache line. Stores can also look up the load cache when enabled and engaged. A store hit may result in a hit-mode invalidation, and a store miss may be ignored.

[0255] On a hit in the load cache, power is saved on reading RAM bank 702. On the other hand, a miss in the load cache not only results in power on reading RAM bank 702, but also consumes power to look up the load cache to populate the victim way. Since not all types of memory access patterns have a high hit rate in the load cache—particularly when accessing supergroups in indexed addressing modes—only vector linear loads can be looked up in the load cache.

[0256] When enabled, all stores may be looked up in the load cache to ensure that the load cache is never out of sync with data in, for example, the VMEM RAM bank 702. For a given supergroup, software may be used to disable the load cache for the RAM bank 702 of that supergroup to minimize store lookup capabilities, as described in more detail below.

[0257] For example, in some embodiments, the use of a data cache may not provide a benefit. For example, in operations where access patterns are non-repetitive, a data cache may not be useful, so performing the additional task of checking the cache before a read may waste time and / or energy, as a read to the database may be required to access the correct data. Therefore, a load cache can be enabled or disabled, reducing the power penalty caused by access patterns with high load cache miss rates while also allowing the load cache to be used for access patterns where a data cache would save power. In some embodiments, enabling or disabling can be programmatically performed using application code, allowing programmers to program the code to enable the data cache when needed and disable it when not. In other embodiments, enabling or disabling can be performed by hardware analyzing read patterns and detecting overlapping patterns. For example, if a threshold amount of overlap occurs between consecutive read operations, the hardware may enable the load cache. However, if the overlap is less than the threshold, the load cache may be disabled. As non-limiting examples, the threshold may be 25%, 40%, 50%, 75%, or a different threshold amount of overlap between reads.

[0258] When the load cache is disabled, and the Figure 7E As shown, tag memory 706 may not be accessed, and read enable 708 may be set so that for every read, reading of RAM bank 702 is enabled. Similarly, data memory 704 may not be accessed, and read data multiplexer 712 may always pass data from RAM bank 702 to staging flip-flop 716.

[0259] In addition, in some embodiments, the memory bank structure can include multiple super banks - for example, three super banks - and each super bank can enable or disable the load cache based on the specific access pattern within each super bank. For example, where three super banks are used, each super bank can include 32 RAM memory banks, and the data cache of each memory bank can include two entries, where each entry is a word and therefore 16 bits. Where two or more super banks are used, the super banks can be of any size, different sizes, the same size, or a combination thereof. For example, the first super bank can be 128KB, the second super bank can be 256KB, and the third super bank can be 512KB.

[0260] Now refer to the figure Figure 7FAs shown, each block of the method 750 described herein includes a computing process that can be performed using any combination of hardware, firmware and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 750 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 750 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product, to name a few. In addition, the method 750 can be performed by any one system, structure or component or any combination of systems, structures or components, including but not limited to those described herein.

[0261] Figure 7F 7 is a flow chart illustrating a method 750 for utilizing a data cache for read operations according to some embodiments of the present disclosure. At block B702, the method 750 includes receiving data representing a memory read address. For example, after a first read operation utilizing some memory banks, a second read operation may be performed that includes one or more memory banks in addition to one or more additional or other memory banks. Because the first read operation may have already included storing the output of the read in the data cache corresponding to each respective memory bank, these values ​​can be reused rather than requiring another read of the memory banks. In this way, a memory read address corresponding to the next read operation may be received, and the load cache, if enabled, may be accessed to determine whether any data is stored in the load cache.

[0262] At block B704, the method 750 includes comparing the memory read address with the load cache memory addresses corresponding to previous memory reads stored in the load cache. For example, after a previous memory read, data from a memory read may be stored in the load cache corresponding to a particular RAM bank 702. To remember this information, the tag memory 706 may include one or more previous memory addresses corresponding to reads from the RAM bank 702.

[0263] At block B706, method 750 includes determining that the memory read address at least partially overlaps with a load cache memory address. For example, the memory read address may be compared with a previous memory read address of a previous read stored in tag memory 706. If there is a hit, the load cache may be used to read out at least some of the data corresponding to the memory read address of the current memory read.

[0264] At block B 708, method 750 includes reading at least a portion of the data corresponding to the memory read address from the load cache. For example, due to a hit in the load cache determined from tag storage 706, a portion of the data from the overlapping memory address may be read from the load cache, and the remainder of the data, if any, may be read from RAM bank 702.

[0265] Decoupled configurable accelerator

[0266] To optimize the performance of a processor for a specific application (e.g., a real-time application), the instruction set architecture (ISA) can be enhanced to create custom instructions to accelerate common operations. This allows the processor to reduce the number of cycles required to perform a specific task. The process of customizing the ISA is continued until the system's performance goals are met. However, these new instructions are added to operate on data in the processor's register file or directly on memory as operands, and these instructions are executed using the existing processor controller and existing memory addressing and access hardware. In such examples, it is desirable for the new instructions to fit the processor's register file read / write operand count (e.g., reusing existing ports), the register file width (e.g., fitting the processor data type), and the processor pipeline stages. Due to these requirements for successfully adding instructions to the ISA, the flexibility of adding new instructions is limited. Furthermore, when creating an ISA for a pipeline with multiple processing stages (e.g., 30, 40, 50, etc. stages), the configuration of the ISA becomes complex.

[0267] Furthermore, processors offer a high degree of flexibility at the expense of power consumption—because each added instruction requires fetching, decoding / dispatching, reading / writing to register files and / or memory, etc. Therefore, adding additional functional units to implement these custom instructions increases pressure on register file read / write ports, leading to increases in required area (e.g., additional read / write ports may be required) and power (e.g., additional load register files may be implemented). Furthermore, processing pipelines for embedded applications often have multiple stages—the output of one stage feeds the input to the next stage. Techniques such as executing multiple threads within a processor (e.g., for different processing stages) can reduce scaling time, thereby providing reduced latency. However, multithreading comes at the expense of hardware—instructions must be fetched / decoded / dispatched from multiple threads, state information must be maintained for each state of each thread (e.g., in register files), and control logic must be included to handle multiple threads within the processor. This increases area and power requirements while making processor verification and programming more complex. Thus, while various methods exist for reducing delays in processing pipelines, existing methods require additional surface area of ​​the processor hardware, require additional power consumption due to the additional hardware, and increase the complexity of programming the processor to perform various tasks.

[0268] To address the limitations of the host processor configuration and the shortcomings of multi-threaded processors, the systems and methods of the present disclosure use a host processor or one or more units of the host processor - for example, a single-threaded processor such as a VPU - in addition to a domain-specific accelerator or coprocessor - for example, a vector memory (VMEM) - that is decoupled from the host processor and communicates with the host processor via shared memory. In this way, the accelerator can operate as a sub-unit of the host processor, but once configured, the accelerator can execute independently of the host processor's instructions, rather than requiring processor instructions to execute. For example, accelerator access instructions can be used to allow the host processor to configure and sort the accelerator, and the shared memory can allow inter-stage data structures to be shared between the host processor and the accelerator. Once the host processor starts or turns on the accelerator (for example, through a common accelerator interface and using one or more load / store instructions), the host processor can freely process different stages (thereby providing the ability to work simultaneously on multiple stages of the processing pipeline and reduce runtime) or transition to a low-power or minimum-power state while waiting for the accelerator to complete processing (for example, to minimize power consumption when not actively processing). In this way, once configured by the host processor, each of the one or more accelerators can operate independently and concurrently with the host processor. The host processor and the accelerator can synchronize via a handshake interface during processing so that the host processor knows when the accelerator has completed processing and / or is ready to execute a new task, or vice versa. The shared memory can store configuration messages (e.g., for configuring the accelerator when configuration instructions cannot be effectively sent through the accelerator interface due to size limitations), input buffers (e.g., to store data for the accelerator to process), and / or output results of the accelerator (e.g., after processing is completed, data from the accelerator, such as a register file, can be stored back to the location in the shared memory indicated in the configuration instructions from the host processor). Thus, once triggered, the accelerator can read configuration parameters and / or input data structures from the shared memory and can write output result data structures to the shared memory.

[0269] As a result, this combined system of a host processor, shared memory, and decoupled accelerator allows for the flexibility of a programmable host processor while achieving the power consumption levels of fixed-function hardware (e.g., because high computational processing stages of a processing pipeline can be implemented as accelerators) without significantly increasing the complexity of the host processor (e.g., because the host processor may only require additional accelerator configuration or access instructions to program the accelerator). For example, the pipeline and data types (e.g., data width) of the accelerator can be independent of those of the host processor, allowing for further customization and optimization that may not be possible with a host processor alone, which requires instructions to adapt to the processor's register file read / write operand count, register file width, and pipeline stages.

[0270] In some embodiments, the accelerator and the main processor can be coupled during instruction execution to achieve some power savings of the accelerator when the execution is coupled to the main processor pipeline. However, in such an embodiment, the ability to process different stages of the pipeline concurrently will be reduced because instructions will be interleaved between the accelerator and the main processor. In one or more embodiments, the accelerator and the main processor can be coupled via a higher level second level (L2) memory, rather than being connected via shared memory. However, in such an embodiment, higher level decoupling (e.g., removing the coupling to the higher level via shared memory) may increase communication overhead with the main processor.

[0271] The decoupled accelerator can be used for any task in any domain, such as, for non-limiting example, performing 1D, 2D, etc. lookups as a decoupled lookup table accelerator to detect and resolve memory bank conflicts, performing 1D / 2D interpolation, etc., for computer vision algorithms such as feature tracking, object tracking, image warping, pyramid creation, etc., for sensor processing such as matrix multiplication or other operations on LiDAR data, RADAR data, and / or the like, for machine learning or deep learning applications. Therefore, the topology described herein can be applied to any processing pipeline where a portion of the processing can be offloaded to the accelerator.

[0272] Depending on the implementation, there may be any number of decoupled accelerators on one or more chips that communicate with one or more main processors through shared memory. For example, a system on a chip (SoC) or other integrated circuit (IC) may include a main processor and one or more accelerators, and a programmer may be aware of various accelerators and write instructions or code that use the accelerators to improve the performance of any of the various tasks of the system. Although the main processor is primarily described as a VPU, this is not intended to be limiting, and the main processor may include any processor type, such as a CPU, GPU, DPU, or other processor, without departing from the scope of the present disclosure.

[0273] Now refer to Figure 8A , Figure 8AA system 800 including one or more decoupled accelerators according to some embodiments of the present disclosure is shown. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the system 800 may be included in and / or may include a processor that is configured to perform the functions described herein. Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features, and / or functions similar to those of the example data center 1500.

[0274] The system 800 may include a processor 802 (e.g., a main processor), such as a VPU, CPU, GPU, DPU, etc., a decoupled accelerator 804, and / or a shared memory 806 (e.g., a vector memory or VMEM). The processor 802 may be coupled to an instruction cache (I-cache) 810, which may cache instructions for execution by the processor 802. The processor 802 may include a general-purpose input / output (GPIO) 808 (e.g., a digital signal pin on an IC that can be used as input, output, or both and can be controlled at runtime), and an IC configurator 812. In some embodiments, as shown, the processor 802 may use an Advanced eXtensible Interface (AXI) for on-chip communication, such as, but not limited to, a 256-bit AXI interface. The IC configurator 812 may be used to configure the system 800.

[0275] The processor 802 can communicate directly with the decoupled accelerator 804—for example, via a coprocessor or accelerator interface, such as an advanced peripheral bus (APB) interface, and / or a handshake, programming, or event interface. For example, the processor 802 can configure the accelerator 804 using an accelerator interface (or configuration bus), start or trigger processing of the accelerator 804 using an event interface, and synchronize with the accelerator 804 using a handshake or event interface. Thus, each accelerator 804 can include a mechanism configured to communicate with the processor 802 via a corresponding accelerator interface or configuration bus. For example, when processing is completed, the accelerator 804 can indicate the same to the processor 802 via a handshake mechanism, or when the processor 802 is waiting for the accelerator 804 to complete processing, the processor 802 can periodically poll the accelerator 804 to request status or end time. In some embodiments, the accelerator interface can include a 32-bit interface (or other smaller size interface) so that configuration instructions can be transmitted to the accelerator 804. However, in some embodiments, the configuration message may be large (e.g., greater than 32 bits, or some multiple thereof), and the configuration message may instead be stored in shared memory 806, and the location of the configuration information in memory 806 may be sent to the accelerator 804 via the accelerator interface to indicate where to retrieve the configuration information.

[0276] The configuration bus can thus configure the accelerator 804, and events (or programming interfaces) can be used to allow the processor 802 to trigger or initiate processing by the accelerator 804. Once triggered or initiated, the accelerator 804 can operate on its own while the processor 802 waits for processing to complete and / or performs different processing tasks or phases. For example, an application programmer can program the processor 802 and accelerator 804 to know what each is capable of, thereby splitting the application into multiple parts—some for the processor 802 and some for the accelerator 804. Thus, in embodiments, processing can be performed in parallel between the processor 802 and the accelerator 804 to reduce runtime and improve efficiency. Configuration messages—through the accelerator interface and / or shared through the shared memory 806—can be generated by the processor 802 and used to indicate to the accelerator 804 where the data to be processed begins in the shared memory 806, how much data to process, and where to write the results back to the shared memory 806. The processor 802 can generate an input buffer at a specified location in the shared memory 806, which contains data operations for the accelerator 804. Once the configuration message is sent and the input buffer is stored in the shared memory 806, the accelerator 804 may receive a trigger signal from the processor 802 via an event interface (e.g., a programming interface), and the accelerator 804 may be processing the data. Once the accelerator 804 is triggered, the processor 802 may then perform other work or enter a low-power state, and once the accelerator 804 completes processing, the accelerator 804 may indicate the same to the processor 802 and may wait for additional work.

[0277] The processor 802 may set up an input buffer or input data structure for processing by the accelerator 804 and store it to the memory 806. The accelerator 804 may be configured by the processor 802 using load / store operations, and the processor 802 is specifically configured to configure and communicate with the accelerator 804. Configuration messages may configure various registers of the accelerator 804 (e.g., in one embodiment, 256 32-bit registers). For example, for a decoupled lookup table accelerator (as described in more detail herein), the configuration information may indicate whether the lookup is for a 1D lookup with interpolation, a 2D lookup with bilinear interpolation, and / or another type of lookup. Once the accelerator 804 knows a particular mode or function, it may configure the registers to correctly read data from the memory 806, process the data, and write the data back to the memory 806.

[0278] In some embodiments, the processor 802 can configure the accelerator 804 to execute multiple tasks simultaneously to improve efficiency. For example, if the accelerator 804 will be performing a variety of smaller tasks, configuring the accelerator 804 individually may increase runtime because each task may complete quickly, requiring the processor 802 to stop processing and configure the accelerator 804 for another task, and so on. To this end, a first task message can include the address of a second task message, allowing for self-chaining of multiple tasks. In this way, the processor 802 can generate configuration messages for multiple tasks at once, generating configuration information and input buffers for each task, allowing the accelerator 804 to execute various tasks consecutively before indicating to the processor 802 that processing is complete and the accelerator 804 is ready to receive more work. Furthermore, to improve efficiency, the accelerator 804 can be configured to overlap tasks, so that as one task nears completion, the accelerator 804 can begin decoding the next task and configuring registers for the next task. Finally, by including separate instructions for the processor 802 and the accelerator 804, the accelerator 804 may be able to operate on data formats or types different from those supported by the processor 802. This may be a result of the architecture and layout of the registers of accelerator 804 being different and specialized for specific processing tasks.

[0279] In an embodiment, the processor 802 can communicate with the shared memory 806 via any number of memory interfaces (e.g., a 512-bit static random access memory (SRAM) interface). Similarly, as shown, the accelerator 804 can communicate with the shared memory 806 via any number of memory interfaces (e.g., a 512-bit SRAM interface). The arbiter 814 can decide for each cycle which of the processor 802 and / or the accelerator 804 is allowed to access the shared memory 806.

[0280] Now refer to Figure 8B , each block of the method 850 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 850 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 850 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, with respect to Figure 8A The method 850 is described with reference to the system 800 , and can be performed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0281] Figure 8B804. FIGURE 805 is a flow chart illustrating a method 850 for using a decoupled accelerator according to some embodiments of the present disclosure. At block B802, the method 850 includes receiving configuration information for one or more first processing tasks of a processing pipeline. For example, the accelerator 804 may receive the configuration information from the processor 802 (e.g., via a configuration message of the accelerator interface).

[0282] At block B 804 , the method 850 includes configuring one or more registers of the accelerator based at least in part on the configuration information. For example, the accelerator 804 may configure the one or more registers based on the configuration information.

[0283] At block B806, method 850 includes reading data from an input buffer in memory based at least in part on an indication of a starting location of the input buffer included in the configuration information. For example, the configuration information may include an indication of where the input buffer is stored in memory 806, and the accelerator 804 may read the data from the input buffer into a register.

[0284] At block B808 , the method 850 includes processing the data from the input buffer to compute output data. For example, the accelerator 804 may process the data from the input buffer to generate or compute output.

[0285] At block B810, the method 850 includes writing the output data to the memory at a location determined at least in part based on the configuration information. For example, the accelerator 804 may write the computation result to the memory 806 and may indicate to the processor 802 that the processing is complete. The processor 802 may then use the output data to perform one or more second processing tasks of the processing pipeline.

[0286] Decoupled Lookup Table Accelerator

[0287] Parallel processing is used to accelerate many computing tasks, including but not limited to: computer vision applications, deep learning applications, sensor processing applications, and / or other applications that benefit from parallelism (e.g., where processing tasks are independent of other processing tasks). For example, vector processors can operate on multiple elements in the same operation to achieve the efficiency required to execute these types of parallel processing algorithms in real time while consuming low power. For example, a common operation for computer vision or deep learning tasks is to perform a lookup from a lookup table, image block, or surface based on an index or coordinate position. To do this, a single vector load or store operation can be used to access data from multiple elements. Unless the index being looked up is regular (e.g., continuous or fixed integer strides in the horizontal or vertical or depth direction), it will result in random index access in memory.

[0288] To support regular but unaligned vector accesses from memory, the processor can construct vector memory using smaller RAM banks. This allows the hardware to create interesting addressing modes for the vector memory by independently generating unique addresses for each RAM bank. For non-regularly indexed vector load operations from memory, this can lead to bank conflicts in one or more banks of RAM, as the indices for different vector elements can be independent of each other. Bank conflicts may not be statically determined because they are data-dependent, thus preventing the compiler from scheduling around bank conflicts.

[0289] In some conventional systems, various architectural designs can be implemented to support non-normally indexed vector load operations. For example, multiple read ports can be added to the RAM bank. In such an example, if the hardware can handle 32 vectors, each bank would require 32 read ports, which increases cost, area, and power, and increases placement and routing congestion around the RAM bank. Another example involves reducing the throughput of indexed lookups to perform a single scalar lookup for each load. However, this creates a bottleneck for vector execution and becomes the limiting factor in execution time. Another example involves making multiple copies of data structures in memory, allowing each vector lane to access data from a single bank. While this example can address some of the throughput issues of other approaches, memory capacity is limited by the data structure occupying N times the space (where N is the number of entries to be accessed), which can lead to overall performance degradation for the associated algorithm, in addition to the overhead of making copies. However, this approach is more suitable for smaller data structures. In some examples, conflicts can be dynamically detected and resolved by serializing conflicting lookups. However, this can increase hardware complexity due to the need to dynamically detect and resolve bank conflicts. Furthermore, these additional stages increase the load latency of these operations, impacting the compiler's ability to schedule the code efficiently. Furthermore, data-dependent execution latency may be introduced, which is problematic for compilers to schedule efficiently. In some examples, a combination of these approaches may be performed.

[0290] To address these shortcomings of other architectures, the systems and methods of the present disclosure include a decoupled lookup table accelerator that is configured to support unconventional index vector load operations. The decoupled lookup table accelerator can be included as an accelerator 804 of the system 800 and can communicate with the processor 802, such as a VPU, via a shared memory 806. The decoupled lookup table (DLUT) can support multiple modes for performing table lookups, such as a 1D lookup mode, a 2D lookup mode, a 2D conflict-free lookup mode, a 1D lookup with an interpolation mode, a 2D lookup with an interpolation mode, a table reformatting mode, and / or other modes. In any lookup mode, the DLUT can accept an index array in the VMEM, which can be in a 1D (x) format or a 2D (x, y) format. For example, each element can include 16 bits or 32 bits, which can be unsigned. The DLUT can then perform the specified index calculations, which can include 2D to 1D mapping, truncation / rounding, integer / fractional segmentation, and / or valid range detection, as non-limiting examples. For example, the DLUT can detect or merge duplicate reads, detect bank conflicts within an index, and issue a read request to the VMEM to find the requested table entry. Each element can consist of 8, 16, or 32 bits, and they can be signed or unsigned. The DLUT can then perform post-interpolation processing as configured and write the output back to the VMEM. Each of these processing operations can be performed in a pipeline to increase throughput, reduce latency, and lower power consumption.

[0291] As a result, the DLUT accelerator overcomes the drawbacks of implementing dynamic conflict detection and resolution in the processor pipeline, allowing the compiler to efficiently schedule deterministic execution latencies for all memory operations while avoiding the complexity of performing inline conflict detection. Because the accelerator operates as a tightly coupled accelerator—for example, via a shared VMEM with the VPU—the processor can configure and start the accelerator while continuing to process other independent parts or stages of the processing pipeline or algorithm. In some embodiments, the accelerator can include additional features to further reduce the load on the host processor, such as offloading index generation for patches with specific lookup patterns, performing optional 1D blending and 2D interpolation on the looked-up data, and / or providing table reformatting support without the need for lookups or interpolation. In practice, the entire system (including the processor 802 for performing lookups and the accelerator 804) has been demonstrated to accelerate the processing of various computer vision algorithms (e.g., feature tracking, object tracking, image warping, pyramid creation, etc.) by a factor of two, while reducing energy consumption by more than 50% compared to executing the entire algorithm solely on the host processor.

[0292] Now refer to Figure 9A , Figure 9AA system 900 including a decoupled lookup table (DLUT) accelerator is shown in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the system 900 may be included in and / or may include a processor that is configured to perform a plurality of operations. Figure 8A System 800, Figures 13A-13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features and / or functions similar to those of the example data center 1500 may be used.

[0293] System 900 may include one or more processors 902 (which may correspond to Figure 8A processor 802), memory 904 (which may correspond to Figure 8A shared memory 806) and a decoupled lookup table (DLUT) accelerator 906 (which can be used as Figure 8A804 is included). In embodiments, the processor 902 may include a VPU and the memory 904 may include a VMEM. The DLUT accelerator 906 (or "DLUT 906") may include a processing unit (PU) interface (I / F) 908 for communicating with the processor 902, a controller 912 for communicating with the processor 902, and a configurator 910 for configuring the DLUT 906 based on information shared across the PU interface 908 from the processor 902 and / or information from the memory 904 based on an indication from the processor 902 of where configuration messages or information are located in the memory 904. For example, the PU interface 908 and the controller 912 may correspond to an advanced peripheral bus (APB) and an event or programming interface of the system 800, respectively. The controller 912 may receive a start or trigger command or signal from the processor 902 (e.g., via an arrow labeled “Start”) indicating that the DLUT 906 may begin processing and / or may receive a polling signal from the processor 902 to help synchronize the processor 902 with the DLUT 906. Additionally, when the DLUT 906 completes processing one or more assigned tasks, the DLUT 906 may generate a signal to the processor 902 (e.g., via an arrow labeled “Done”) so that the processor 902 may begin configuring the DLUT 906 for the next task.

[0294] During configuration, the processor 902 may configure the DLUT 906 directly via the PU interface 908 and / or indirectly by indicating the location of configuration information in the memory 904 via the PU interface 908. In the latter example, the DLUT 906 may retrieve the configuration information from the memory via, for example, the shared read port strm1_dm_rd, and may use the stored configuration information to configure the DLUT 906 (e.g., configuration subunits (e.g., IAUs, CDRUs, PPUs, etc.) and / or other components of the DLUT 906) for performing one or more tasks. For example, the processor 902 may set up data structures in the memory 904 that the DLUT 906 needs to perform the one or more tasks. For example, for a 1000-coordinate lookup, the processor 902 may set up a data structure in the memory 904 that has each of the 1000 coordinates and may further allocate a buffer in the memory 904 to which the DLUT 906 writes the output. Processor 902 can also instruct DLUT 906 which operations to perform—e.g., 1D or 2D lookups, with or without interpolation, table reformatting, etc.—and DLUT 906 can use this information to configure the subunits. Configuration information set by processor 902 can also include an indication of the bit width of coordinate indices, as well as an indication of the bit width of table entries, etc. Thus, once the input and output buffers are set up in memory 904 and configuration information such as bit width, operation type, etc. is sent to DLUT 906, processor 902 can initiate or trigger DLUT 906 to begin processing. Consequently, compared to a system relying solely on processor 902, processor 902 can perform other tasks while DLUT 906 performs lookups, interpolation, table reformatting, etc., thereby reducing runtime and improving efficiency.

[0295] In operation, the DLUT 906 may receive a list of indices corresponding to coordinates from the memory 904, and the DLUT 906 may extract a value from a table corresponding to the index (e.g., where the value is an integer value) and / or may pull surrounding values ​​of a fractional value (e.g., the left and right values ​​for a one-dimensional lookup or the upper left, lower left, upper right, and lower right values ​​for a two-dimensional lookup) and perform interpolation or other operations on the surrounding values. Once the final values ​​are determined (e.g., directly through a lookup without performing post-processing, or after processing by a post-processing unit (PPU) 930), these values ​​may be written to an output buffer in the memory 904, corresponding one-to-one to the index in the input buffer. To efficiently perform these tasks, in embodiments, an index address unit (IAU) 922, a conflict detection and resolution unit (CDRU) 924, a control (CTL) first-in, first-out (FIFO) 928, a fractional (FRAC) FIFO 926, for example, may be used by a post-processing unit (PPU) 930, a data merge unit (DCU) 932, and / or other components.

[0296] For example, an index (IDX) stream 916 may include a stream of indices read from memory 904 (e.g., via read port strm1_dm_rd) to be looked up in one or more lookup tables, and values ​​corresponding to the indices may be read from memory 904 (e.g., via read port strm0_dm_rd) via a lookup table (LUT) stream 918. An output (OUT) stream 920 may be values ​​written back to memory 904 (e.g., via write port strm0_dm_wr) after processing using DLUT 906.

[0297] During configuration, the processor 902 can instruct the IDX stream 916 how to access the data structure used for indexing. For example, for a one-dimensional lookup, where the interface to the memory 904 is 64 bytes wide, 64 bytes can be read in each cycle. In the case of a 1D lookup, a single coordinate can be read for each index value (e.g., an (x) value), while for a 2D lookup, two coordinate indices (e.g., (x, y) values) can be read for each index. In non-limiting embodiments, each index can be 16 bits or 32 bits, so in each 64-byte read, there may be 8, 16, or 32 coordinates from the IDX stream 916.

[0298] The IDX stream 916 data may be sent to the IAU 922 in its raw format as raw indices, and each coordinate may be an integer value or a fractional value. The IAU 922 (where the indices are fractional values) may split the fractional values ​​to provide the fractional bits to the FRACFIFO 926 to facilitate use of surrounding values ​​found in the PPU 930 blending table. The IAU 922 may then determine a set of indices to send to the CDRU 924, where the number of indices sent may correspond to the number of lookups that the LUT stream 918 may perform in a single cycle. For example, if the LUT stream 918 may perform, for example, 32 lookups in one cycle (based on the bit width of each value in the lookup table), the IAU 922 may send 32 indices to the CDRU 924 on each iteration. In some examples, such as where the values ​​from the IDX stream 916 to the IAU 922 are integer values, the IAU 922 may send each set of indices without any processing. However, in the case where the value from the IDX stream 916 is a fractional value, the IAU 922 can determine which indices need to be looked up to obtain each of the surrounding values ​​required (e.g., 2 indices for 1D interpolation or 4 indices for 2D interpolation) to perform interpolation or other operations to obtain a blended value corresponding to the fractional value. For example, in the case where the fractional value is (5.3, 6.2) corresponding to the (x, y) coordinates for 2D lookup and interpolation, the IAU 922 can determine that the lookups will occur at (5, 6), (5, 7), (6, 6), and (6, 7). The PPU 930 can then blend these values ​​to generate the final value corresponding to the index (5.3, 6.2). For example, these values ​​can be blended equally, or bilinear interpolation can be used to blend so that values ​​closer to (5, 6) than to (6, 7) are weighted more heavily to calculate the final value of (5.3, 6.2).

[0299] The lookup set (e.g., 32 lookup indices for the LUT stream 918 capable of reading 32 values ​​in each read cycle) can be sent to the CDRU 924 in the appropriate order corresponding to the order of the indices in the input buffer in the memory 904 read using the IDX stream 916. The CDRU 924 then performs conflict detection and resolution by identifying the bank conflicts that would result if the lookup table reads in the LUT stream 918 occurred in the order received from the IAU 922, and resolves the bank conflicts by changing the order of the indices to avoid the bank conflicts. For example, when a lookup of an index set would result in a bank conflict, and another set of indices (e.g., later or earlier) is available for another lookup cycle, the CDRU 924 can find a non-conflicting lookup from the other lookup cycle and swap the non-conflicting lookup with the conflicting lookup for that cycle. As a result, one or more bank conflicts can be avoided, thereby increasing throughput. For example, where the IAU sends 32 indices per cycle, and 6 indices from a given cycle have bank conflicts, the CDRU 924 may determine up to 6 indices from another lookup that does not cause a conflict with the current lookup, and may perform those 32 lookups—e.g., 26 lookups from the original 32 and 6 lookups from the other set sent by the IAU 922. Once the lookups are determined (e.g., with or without replacement to resolve the conflict), the set of lookups may be read out from the memory 904 using the LUT stream 918.

[0300] To account for out-of-order lookups where replacements occur, the CDRU 924 can use the CTL FIFO 928 to indicate to the data merge unit the order of the lookups for each set of lookups from the IAU 922. For example, for an initial set of 32 lookups, the DCU can determine that 8 were performed in the first cycle, then 8 in another cycle, then 16 in another cycle, and can then determine that the entire set of 32 has been processed. The 32 lookups can then be pushed to the PPU 930 for post-processing, if applicable, or they can be pushed directly to the OUT stream 920 to be written to the output buffer in the memory 904. This additional information indicating the actual order of the lookups determined by the CDRU 924 and read out to the LUT stream 918 index can be passed to the DCU 932 via the CTL FIFO 928. Therefore, any changes made by the CDRU 924 to the order of the indexes received from the IAU 922 can be taken into account by the DCU 932. The CTL FIFO 928 may be useful because the number of cycles through the IAU 922, CDRU 924, etc. is uncertain and data dependent. For example, because conflicts are not known in advance (e.g., because the data may be non-deterministic) and are a result of programming, there is no solution to completely avoid conflicts, so the CTL FIFO 928 helps instruct the DCU 932 to organize the lookup as a result of conflict resolution.

[0301] When needed—e.g., where additional operations need to be performed on the lookup table values—the PPU 930 may calculate a final value for each index that may be read out to memory 904. In cases where no post-processing is required, the PPU 930 may not need to collect the results. For example, in the case of performing a normal 1D or 2D lookup on an integer-valued index that maps directly to a location in a lookup table, the PPU 930 and FRAC FIFO 926 may not be used to perform the additional processing. When performing interpolation (e.g., linear on a 1D lookup or bilinear on a 2D lookup) and / or other operations, the PPU 930 and FRAC FIFO 926 may be used to convert the collected results into updated results or values ​​to be written out to memory 904.

[0302] In some embodiments, the DLUT 906 can be used in a table reformatting mode. For example, the IDX stream 916 and the OUT stream 920 can be used to update addresses for access and / or transposition. In such an example, if a buffer exists in the memory 904 and the index in the buffer is to be transposed, this operation can be offloaded to the DLUT 906 (rather than having the address generation unit of the processor 902 perform the transposition). Configuration information from the processor 902, such as from the address generation unit, can indicate a read mode for reading from the buffer in the memory 904 and a write mode for writing addresses back to the memory 904 in a different mode. For example, if the programmer knows that many conflicts will result from a particular access pattern, the programmer can program the processor 902 to configure the DLUT 906 to perform table reformatting to shuffle the data so that fewer or no conflicts are likely to occur.

[0303] As another example, the DLUT 906 can be used to return a sentinel value for out-of-range detection, or to disable output writing for out-of-range prediction. Thus, for example, if a coordinate in the IDX stream 916 is outside a given image block and the corresponding value should not be written, the DLUT 906 can write a sentinel value, which can indicate to the processor 902 when processing information in the output buffer that does not rely on or use the sentinel value in processing. In some embodiments, the sentinel value can indicate to the processor 902 that these values ​​are not to be written to memory, and thus values ​​identified as error values ​​may not be stored.

[0304] Thus, the DLUT 906 can be implemented as a pipeline of subunits that work together to perform a specific task or operation. Each subunit can operate independently and communicate with other subunits through a shared interface. Figure 9B 9, table 940 illustrates the tasks of the various sub-units of DLUT 906 during the processing of a particular operation.

[0305] Thanks to the DLUT accelerator described herein, the processor pipeline can remain deterministic by offloading dynamic conflict detection and resolution to the decoupled accelerator. In addition, the accelerator is able to run independently and concurrently with the main processor (e.g., VPU), thereby reducing runtime. The DLUT accelerator can also allow 1D and / or 2D lookups from a common table with conflict detection / resolution. The accelerator can perform various post-processing operations, such as 1D lookups with linear interpolation, 2D lookups with bilinear interpolation, out-of-range detection sentinel returns (1D and 2D), and / or out-of-range prediction shutdown output writes (1D and 2D). The DLUT accelerator can be configured to perform interpolation using a configurable number of fractional bits and can support various index and data formats, such as 8, 16, and 32-bit signed and unsigned data formats and 16 and 32-bit 1D and 2D coordinate index formats. The DLUT accelerator can also convert between global and local coordinates using configurable X / Y offsets. The DLUT accelerator can also support data flow units to read index buffers from VMEM, perform lookups from VMEM, and write results (or lookups or interpolations) to VMEM. The data flow unit can support up to 2D addressing for linear and transposed access. In order to optimize the number of cycles required for lookups / interpolations, the lookup indexes may appear out of order to minimize memory bank conflicts - for example, if VMEM supports N lookups, the accelerator can use MxN indexes to maximize the ability to survive conflict detection - and duplicate detection can be performed to filter out duplicate indexes that are guaranteed to cause conflicts. In addition, the DLUT accelerator's 2D lookup and interpolation modes can include indexes that are automatically generated within the accelerator based on several parameters (called automatic index mode), as opposed to the programmer providing index data blocks. This offloads the preparation of the indexes from the host processor to the accelerator.

[0306] Now refer to Figure 9C , each block of the method 950 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 950 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 950 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, with respect to Figure 9A The method 950 is described with reference to the system 900 and can be performed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0307] Figure 9C9 is a flow chart illustrating a method 950 for using a decoupled lookup table accelerator according to some embodiments of the present disclosure. At block B902, the method 950 includes configuring one or more subunits of the DLUT accelerator based at least in part on configuration information generated using a processor. For example, the DLUT 906 may configure the subunits of the DLUT 906 using information received from the processor 902 and / or retrieved from the memory 904.

[0308] At block B904, the method 950 includes reading a first set of indexes in the stream from memory and determining a first subset of indexes that do not have memory bank conflicts. For example, the IAU 922 may generate a set of indexes for the CDRU 924 to handle conflicts, and the CDRU 924 may determine a subset of the set of indexes that do not have memory bank conflicts.

[0309] At block B906, the method 950 includes determining a second subset of indexes from the second set of indexes in the index stream read from the memory that do not cause a bank conflict with the first subset of indexes. For example, the IAU 922 may generate another set of indexes for the CDRU 924 to handle the conflict, and the CDRU 924 may determine to replace the one or more indexes from the first set that have a conflict with one or more indexes from the second set of indexes that do not cause a conflict with the first set of indexes.

[0310] At block B908, the method 950 includes performing a lookup of one or more lookup tables using the first subset of indices and the second subset of indices in a single read cycle from the memory to retrieve a plurality of values. For example, the DLUT 906 may read values ​​from the memory 904 into the LUT stream 918 using a subset of values ​​from the set of indices and values ​​from the second set of indices that are determined to not conflict with the subset of values ​​from the first set of indices.

[0311] At block B910, method 950 includes writing a plurality of values ​​to memory. For example, values ​​from LUT stream 918 may be written to memory 904 in output stream 920. Before being written out, DCU 932 may reorganize the data so that the data is in a one-to-one order with the index read from the input buffer of IDX stream 916. In some embodiments, PPU 930 may perform one or more operations, such as interpolation, on the retrieved values ​​before writing the final values ​​out to memory 904 in OUT stream 920.

[0312] Hardware sequencer for direct memory access systems

[0313] A direct memory access (DMA) system can be used to move data from different memory locations without requiring a central processing unit (CPU). For example, a DMA can operate as a data movement engine, moving data from a source to a destination—for example, from a source such as external memory (e.g., DRAM) or internal memory such as an L2 buffer or vector memory (VMEM) of a vector processing unit (VPU) to a destination such as a VPU. In practice, a DMA system can perform additional operations such as, but not limited to, filling frame data, manipulating addresses, managing overlapping data, managing different traversal orders, and accounting for different frame sizes.

[0314] In digital signal processing, multiple DMA resources can be used to describe structured tile data movement between external memory and a processor (e.g., a VPU). For example, these DMA resources can include descriptors, channels, triggers, and / or registers. For example, a descriptor can describe tile movement such as source location, destination location, line spacing, tile width, tile height, circular buffer layout, etc. However, tile data movement of image surfaces with spatial and temporal dependencies presents additional programming model challenges to the user and requires many different DMA configuration resources. These tile data dependencies may also complicate control code and control sequences in the processor (e.g., VPU) code. For example, a typical processing operation may include filtering, such as 3x3 filtering. This type of operation introduces spatial dependencies because each output pixel will depend on the corresponding values ​​of the 3x3 pixels surrounding the output pixel. In such an operation, filtering can be performed using a 3x3 matrix of values, and the operation can be referred to as a spatially dependent operation. In practice, each tile of a frame may have the same size—e.g., 64x64—to reduce programming challenges. However, if a 3x3 filter is used on a 64x64 tile, then the neighboring blocks will need extra pixels up and down - e.g. Figure 10C Therefore, this information needs to be encoded in the DMA resources to allow data to be correctly fetched across tiles – this results in additional programming overhead.

[0315] refer to Figures 10A-10G , Figures 10A-10G shows the various challenges of data movement when using a DMA system. For example, Figure 10AThe visualization 1000 may correspond to filling the frame data. In the visualization 1000, there may be nine sections, an upper left section, an upper section, an upper right section, a left section, a center section, a right section, a lower left section, a lower section, and a lower right section. In such an example, each section may include one or more tiles—for example, the upper left section may include one tile, while the top section may include, for example, four tiles. Thus, to accurately define the segmentation, in existing methods, nine descriptors (e.g., one for each section), three channels (e.g., one for the left column, one for the center column, and one for the right column), and three triggers (e.g., one for each channel) may be used.

[0316] With respect to padding, for example, due to spatial correlation, when performing operations on data near the boundaries of a tile or portion of a frame, the DMA system may pad values ​​or fabricate values ​​for pixels outside the boundaries of the image. This may be because, in some embodiments, requesting data outside of a memory region for an image may trigger a fault. Thus, the DMA may be used to pad or fabricate values ​​after acquiring the image data from the corresponding memory region to avoid triggering a fault. Without padding, the structure of the data may not match the kernel size, for example, if a filtering operation is performed. The acquired data with the padding values ​​attached may then be sent to the destination (e.g., a VPU) so that the VPU can process the data according to its configuration and may process the data in the same manner across the entire (padded) frame. When padding, zero padding may be used (e.g., where each new data point includes a zero value), repeated values ​​may be used (e.g., pixel values ​​for adjacent pixels are copied from the acquired data), and / or another padding mechanism may be used. Furthermore, padding may be added to any side of the frame, and different padding may be added for different sides. For example, in Figure 10A In the example, the padding area 1002 can be larger on the right side than on the left, top, or bottom of the frame. Padding increases the complexity of DMA programming when moving data from a source to a destination (e.g., from memory to VMEM), and also increases the complexity of VPU programming when handling larger padded frames.

[0317] Now refer to Figure 10B , Figure 10BThe visualization 1010 corresponds to the address operations of the DMA system. For example, different descriptor addresses can be manipulated and programmed to obtain continuous frame data. In order for DMA to perform efficiently, the address descriptions of the data movement may be continuous. Therefore, the address of each descriptor can be manipulated, and such manipulation must be passed from one descriptor to another. For example, when filling values ​​as shown in the figure, the starting address of each descriptor can be manipulated so that the extracted data includes the filled values. To do this, the programmer uses the starting address and the tile width and the number of tiles in each section and uses this information to generate the next descriptor address. For example, the first descriptor can cause the acquisition of data starting from the upper left corner, then the top, then the upper right corner, then the left, then the center, and so on, as shown in FIG. Figure 10B However, the starting descriptor address adds complexity to DMA programming when moving data to a destination such as VMEM.

[0318] As another example, and with respect to Figure 10C , in order to ensure continuous data processing, the DMA system may be required to read vertically and horizontally overlapping data from adjacent tiles. For example, Figure 10C As shown in the shaded area, it may be necessary to read overlapping data from the tile in the upper left portion and the adjacent tile in the top portion in the same operation. Similarly, it may be necessary to read overlapping data from the tile in the upper left portion and the adjacent tile in the left portion in the same operation. To do this, the descriptors need to be updated or moved to include the overlapping portions. For example, the base descriptor may include an address at the beginning of the top portion, but in order to capture data from the adjacent tile in the upper left portion, the top descriptor needs to be updated (e.g., moved to the left) to capture data from the tile in the upper left corner. This update requires additional programming complexity, especially as the number of descriptors increases.

[0319] In addition, about Figures 10D-10F , the DMA system may need to support different traversal orders in order to read data from memory in a continuous manner. For example, whether filtering, convolution, matrix multiplication and / or other operations are performed, the associated traversal order may be different. With this in mind, various traversal orders can be supported, such as Figure 10D , which include a raster traversal order starting from the top left (visualization 1030), a raster traversal order starting from the top right (visualization 1032), a raster traversal order starting from the bottom left (visualization 1034), and / or a raster traversal order starting from the bottom right (visualization 1036). Figure 10E As shown in visualization 1038, for cube images, the DMA system can support various cube traversal orders. Figure 10FVarious vertical digging traversal orders that can be supported by the DMA system are shown, such as a vertical digging traversal order starting from the top left (visualization 1040), a vertical digging traversal order starting from the top right (visualization 1042), a vertical digging traversal order starting from the bottom left (visualization 1046), and / or a vertical digging traversal order starting from the bottom right (visualization 1048). To support each of these different traversal orders for moving data to memory (e.g., VMEM), the complexity of DMA programming is increased.

[0320] about Figure 10G The DMA system may also need to support different frame sizes, such as moving multiple frames of different sizes (e.g., Luma / Chroma composites or different pyramid levels). For example, a processor (e.g., a VPU) may process frames of different sizes to generate the final desired output. Figure 10A An example visualization 1048 of pyramid processing corresponding to a frame for an optical flow estimation operation is illustrated. In such an example, the movement of pixels can be first calculated for a smaller frame size, then calculated for a larger frame size using hints from the output of the smaller frame size, then calculated for the larger frame size using hints from the larger frame size, and so on. Thus, the DMA system can support the acquisition of frame data for various frame sizes, but this capability requires additional programming complexity for the DMA system. For example, descriptors must be programmed or updated for each different frame size.

[0321] To simplify the programming of these various operations supported by the DMA system, the DMA system and method of the present disclosure can use a hardware sequencer in conjunction with the DMA engine to solve the data movement problem. For example, the data movement of a complete image can be explicitly and completely described in a hardware sequencing mode, which has a simplified programming model that handles tile ordering (triggering), padding, overlap (offset), traversal order, and different frame sizes (e.g., the image structure of the frame, e.g., Figure 10I The hardware sequencer can reduce DMA resource usage (e.g., reduce the number of descriptors, triggers, channels, etc. required), offload control of VPU control processing from the VPU, and reduce the complexity of DMA programming. This can be achieved by loading image or frame descriptor views from local programmable memory in the form of command sequences (e.g., as shown in the figure). Figure 10I These hardware sequence commands can incorporate each of the operations that increase programming complexity, as described in this article—including image padding, tile overlap or offset, frame offset, image traversal order, and image size at tile granularity. In addition to descriptor information (e.g., from image commands or from a separate descriptor memory or SRAM), the hardware sequencer can also read image commands from memory and sequence tile movements to traverse and draw the entire frame.

[0322] Now refer to Figure 10H , Figure 10H A DMA system 1050 including a hardware sequencer is shown in accordance with some embodiments of the present disclosure. It will be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the system 1050 may be included in and / or may include a processor that is configured to perform a plurality of operations. Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 Components, features, and / or functions similar to those of the example data center 1500.

[0323] The system 1050 may include a DMA engine 1056, a register control 1058, a hardware (HW) sequencer controller 1060, a descriptor SRAM 1052, and / or a hardware (HW) sequencer command SRAM 1054. Existing systems may only include the DMA engine 1056 and the descriptor SRAM 1052 that stores frame descriptors. Thus, as described herein, when sending data from a source to a destination, the DMA engine 1056 is required to perform all padding, address manipulation, and the like, and the VPU or other source is required to perform the sequencing by handshaking with the DMA system (e.g., with the VPU as the master node and the DMA as the slave node). In such an example, the DMA engine 1056 would process at a tile level, using descriptors for portions of a frame, each portion comprising one or more tiles, to retrieve one tile at a time to send to the destination, and subsequent tiles would be retrieved based on the descriptors, which are based on instructions from the VPU to retrieve the next tile.

[0324] However, using Figure 10H The system 1050 can be implemented to process frames at the frame level - for example, a single descriptor can be used to Figure 10H, which previously required nine descriptors. Therefore, in practice, when the DMA engine 1056 attempts to load a descriptor from the descriptor SRAM 1052 (or more generally, the descriptor memory 1052), the HW sequencer control 1060 can intercept the descriptor load and use the command sequence processing structure to handle multiple frames, tile rows / columns, and multiple descriptors. To do this, the frame format 1070 ( Figure 10I ), which describes a higher-level frame by processing tile rows / columns (depending on the traversal order) in hardware rather than at the tile level. For example, instead of filling a tile, the frame format 1070 can be used to fill an entire frame, thereby filling many frames with a single fill command. Therefore, the entire frame can be understood, such as where to fill, where to overlap, how to automatically manipulate addresses, etc. In addition, because the DMA engine 1056 can directly extract descriptors from the descriptor SRAM 1052 without the intervention of the hardware sequencer control 1060, the legacy format can still be supported for operations that may not benefit from the HW sequencer control 1060.

[0325] The HW sequencer control 1060 can operate, for example, as a state machine that reads the HW sequencer command SRAM 1054 (or more generally, the HW sequencer command memory 1054), where a frame format 1070 including sequenced commands is stored. A processing controller—e.g., an R5 processor, a CPU, an ARM processor, etc.—can program or configure the hardware sequencer command SRAM 1054 and the descriptor SRAM 1052 using programming code and / or settings from a higher-level engine.

[0326] The descriptor SRAM 1054 may include one or more descriptors that may define tile dimensions (e.g., tile width dx and tile height dy), the starting point of the image or frame (e.g., top left, bottom right, etc.), trigger type, and / or other micro-information about the scan type of the descriptor.

[0327] The HW sequencer command SRAM 1054 can store a frame format 1070 that defines a frame as a whole, the size of the frame, frame padding, etc. For example, the frame format 1070 can include a frame header for header control, offset control, and padding control, and can include column headers and / or row headers for columns or rows of the frame (e.g., a column header for vertical scan mode and a row header for raster scan mode). The frame header control can include a frame repetition factor to identify how many times a particular frame will be repeated, as well as the number of descriptor rows and / or descriptor columns. The frame header offset control can include a frame tile offset (e.g., an offset from tile to tile) and a frame offset (e.g., an offset between two or more frames that can be read using a single channel, such as a YUV frame that can be processed to include three separate planes). The frame padding header can indicate how many rows or pixels of padding to add at the frame level (as opposed to the per-tile level of existing methods), such as padding the left side of the frame, the top of the frame, the right side of the frame, and / or the bottom of the frame, thereby filling the entire frame, rather than padding each tile within each portion of the frame at the tile level.

[0328] A column header may be used when the traversal order is vertical, while a row header may be used when the traversal order is raster or horizontal. The column header and / or row header may include a column or row offset (e.g., an offset between each column or row), a column or row repetition factor (e.g., how many times the same column or row processing type is repeated across frames, such as N-1 times, where N is the number of times the column or row is processed), and the number of descriptors used for each column or row (e.g., a single descriptor may be used to repeat the same tile across rows or columns, or the first descriptor may be used to traverse a portion of a row, while the second descriptor may be used to traverse another portion of the row, and so on). The descriptor ID may be described so that a descriptor—e.g., stored in the descriptor SRAM 1052—can be pulled out and used to describe a row or column. For example, a descriptor ID may indicate which descriptor is used for a particular column and / or row, and how many times the descriptor is repeated (e.g., N-1 times, where N is the total number of times the descriptor is used). In an embodiment, there may be a set of descriptors (e.g., 64), and the descriptor ID may be used to determine which descriptor should be used for a particular column and / or row. In this way, the hardware sequencer controller 1060 looks at the upper structure of the frame above the basic descriptors from the descriptor SRAM 1052, which allows for a reduction in the resources required by the DMA engine 1056 to achieve the same data transfer. In addition, the hardware sequencer control 1060 can pre-fetch tiles in advance (e.g., using register control 1058) to reduce latency, and the tile data can be immediately available when requested by the DMA engine 1056.

[0329] In operation, the HW sequencer control 1060 can read the image structure (e.g., frame format 1070) from the HW sequencer command SRAM 1054 and the descriptor information from the descriptor SRAM 1052 and can combine this information to sequence frames for the DMA engine 1056. Thus, rather than requiring the DMA engine 1056 to be individually coded for each descriptor, trigger, channel, etc., the HW sequencer control 1060 can read the image structure, pull in the descriptors, and sequence the frames for the DMA engine 1056 with the correct descriptor format. In an embodiment, the register control 1058 can help control the traversal order, prefetch, and / or other frame addressing controls. The HW sequencer control 1060 further simplifies the VPU code so that the VPU does not have to account for multiple channels. Instead, the VPU can request one tile, then the next tile, then the next tile, and so on. The HW sequencer control 1060 knows the current position in the frame, and therefore the next tile to be fetched for the DMA engine 1056, and the DMA engine 1056 does not have to track this information internally.

[0330] The system 1050 can therefore be backward compatible with previous approaches in that the system can still support the use of various descriptors, triggers, channels, etc., but can also be understood at the frame level to reduce complexity. The system 1050 can support image padding at all corners of a frame with different pixel padding sizes, vertical and / or horizontal overlapping tiles to allow the VPU to access adjacent tiles for processing along tile boundaries, and traversal of the frame in different traversal orders. In addition, the system 1050 can support automatic tile offset adjustment at the VMEM destination by the hardware sequencer control 1060. Because the descriptors in a frame are linked by hardware, the user does not need to link or splice the descriptors together. The hardware sequencer control 1060 can manage the address ordering of descriptors / tiles across frames without additional programming complexity, and the hardware sequencer control 1060 can prefetch tiles to improve performance.

[0331] In some embodiments, the descriptor may be included in the image or frame structure rather than being stored separately in the descriptor SRAM 1052. For example, where legacy compatibility is not implemented, the entire sorting structure and tile structure may be described in the frame structure. In such an example, Figure 10I The frame format may be used to include additional information with the descriptors, such as tile width, trigger type, etc., resulting in the same information being available to the HW sequencer control 1060 as when the descriptors are stored separately in the descriptor SRAM 1052 .

[0332] refer to Figure 10J , Figure 10J10 when implemented for a raster scan sequence according to some embodiments of the present disclosure. For example, frame format 1070A is an example of a frame format in raster mode, with frame address processing, using a single channel, a single trigger, and a single descriptor. In this example, the tile structure may be 16x8. Figure 10K is an example of such a tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, where frame address processing is performed using example frame format 1070A. For example, for each tile row, the same descriptor (e.g., tile dimension) may be used (as indicated by "D1" in visualization 1072) so that the same tile is applied 16 times along each row (from C1 to C16) and then repeated for 8 rows from top to bottom (from R1 to R8). The sequence may include 20 bytes, as shown in frame format 1070A, and each row may have N*2+ bytes, where N represents the number of entries per row (as indicated by "D1" in visualization 1072). Figure 10J ). Thus, to order the frames as shown in visualization 1072, frame format 1070A may include no frame repetition, the number of descriptor rows may be zero, no tile offset, no frame offset, a frame padding of 3 rows of pixels on the left (PL), right (PR), top (PT), and bottom (PB), the row may be repeated 7 times (for a total of 8 rows), each row may be offset by the tile height (Ty) (such that each row is offset by the tile height), one descriptor may be used with descriptor ID D1, and the descriptor may be repeated 15 times in each row (for a total of 16 times). Thus, in practice, the HW sequencer control may use the descriptor corresponding to D1 from descriptor SRAM 1052 (which includes the tile height and tile width), and may use the image structure from frame format 1072 stored in HW sequencer control SRAM 1054 to order the image tiles tile by tile (16 tiles per row), row by row (from R1 to R8) for the target processor (e.g., VPU). In this way, a single descriptor, a single trigger, and a single channel may be used, thereby reducing programming complexity while also allowing the DMA system 1050 to be the primary or controlling component in the interaction between the DMA system 1050 and the VPU.

[0333] In some embodiments, as an extension of the HW sequencer control 1060, a DMA trigger mode can be used to reduce software intervention in VPU programming by having the DMA system 1050 sequence command descriptors. For example, the DMA system 1050 can read an image from external memory, tile the image, and sequence the tiles for the VPU. To facilitate this, the VPU can expose start and done signals. The VPU startup can be driven by the DMA system 1050, and when the VPU completes processing the instruction block, the VPU can send a done signal to the DMA system 1050. Thus, the DMA system 1050 (e.g., the hardware sequencer control 1060) and the VPU can participate in a handshake mechanism in which the DMA system 1050 is the primary node and the VPU is the secondary node. This DMA trigger mode can minimize VPU tile control overhead and simplify the programming model of the DMA engine 1056. For example, specific code for double-buffered DMA data movement may not be required, and the DMA kernel code can be independent of the VPU kernel code. Therefore, DMA trigger mode simplifies the VPU code because the DMA system handles tile sequencing using the HW sequencer control 1060. The following sample code illustrates the VPU code before and after the addition of DMA trigger.

[0334] Before:

[0335]

[0336] after:

[0337]

[0338]

[0339] As a result, in cases where the VPU had previously been requesting that a tile be moved to VMEM, the DMA system 1050 can now trigger the move of the tile to VMEM and the VPU as the target because the HW sequencer control 1060 controls the ordering. In this way, the DMA system 1050 can obtain data to be processed by the VPU in advance, and when the VPU indicates that the processing is complete, the DMA system 1050 can make the next data to be processed immediately available (e.g., in VMEM) and can indicate the same to the VPU.

[0340] When processing one or more frames, the HW sequencer control 1060 can retrieve one or more descriptors (which can indicate tile size, trigger type, etc.) from the descriptor SRAM 1052 and can retrieve the image structure from the hardware sequencer command SRAM 1054. The HW sequencer command 1060 - in conjunction with the register control 1058 - can then begin traversing the first row or column according to the traversal order and using the first (and only in one embodiment) descriptor, and can then move to the second descriptor based on the number of repetitions encountered (e.g., 1-N) if two or more descriptors are used, and so on. As each tile is determined, the DMA engine 1056 can retrieve the tile data from the source data and write the tile data to the destination data (e.g., in VMEM). Once the data is written to the data destination, the processor (e.g., VPU) can be notified by the hardware sequencer control 1060 that the data is available for the processor to begin processing. Then, during processing, the DMA system 1050 can fetch the next data tile and write the data to the data destination based on a sequence from the hardware sequencer control 1060, so that when the processor indicates that processing is complete, the hardware sequencer control 1060 can indicate to the VPU (via a handshaking mechanism) that the next data to be processed is available, and so on until processing is complete.

[0341] Now refer to Figure 10L , each block of the method 1080 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 1080 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 1080 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, the method 1080 is about Figure 10H Although the system described herein, method 1080 can be performed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0342] Figure 10L 1080 is a flow chart of a method 1080 for a DMA system including a hardware sequencer according to some embodiments of the present disclosure. At block B1002, the method 1080 includes retrieving a tile structure from a descriptor memory and a frame structure corresponding to a frame from a hardware sequencer command memory. For example, the hardware sequencer control 1060 may retrieve a descriptor from the descriptor SRAM 1052.

[0343] At block B1004, method 1080 includes sequencing the retrieval of tiles of a frame from source memory. For example, hardware sequencer control 1060—in an embodiment combined with register control 1058—may sequence the retrieval of tiles from source memory by DMA engine 1056 based on the frame (or image) structure and the tile description from the descriptor.

[0344] At block B 1006 , the method 1080 includes writing the retrieved data corresponding to the tile to a destination memory. For example, the DMA engine 1056 may write the retrieved data corresponding to the tile to a target memory (eg, VMEM) for processing by a target processor (eg, VPU).

[0345] At block B 1008 , the method 1080 includes providing an indication to a processor associated with the destination memory that the retrieved data is stored in the destination memory. For example, the HW sequencer control 1060 may indicate to the processor that the next tile of data is ready for processing.

[0346] At block B1010, method 1080 includes receiving an indication of completion of processing of the retrieved data. For example, upon completion of processing, the processor may indicate to DMA system 1050 that processing is complete, at which point the next data tile may be loaded (or may have been preloaded) into the destination memory, and DMA system 1050 may indicate the same to the processor.

[0347] Configuring the DMA system for region-dependent data movement using the VPU

[0348] When acquiring known data patterns, a processing controller can configure the direct memory access (DMA) system and a processor (e.g., a vector processing unit (VPU)) can trigger and sequence the DMA. However, when processing different data points or features of irregular or unknown data patterns, challenges may arise in reconfiguring data movement because the features or object positions are dynamically calculated. For example, object tracking algorithms, feature tracking algorithms, object detection algorithms, deep learning algorithms using variable-sized regions of interest (ROIs), and / or other region-dependent data movement algorithms require dynamic adjustment of address and data pairs so that the DMA system can retrieve the appropriate information for processing by the processor (e.g., VPU). In conventional systems, when acquiring unknown data patterns—such as in object tracking—a processing controller (e.g., an R5 processor core used to control a programmable vision accelerator (PVA)) may require an interrupt to intervene in the processing cycle to determine the updated information calculated by the processor (e.g., VPU) and reconfigure the DMA for the next iteration. As a result, the processing controller introduces additional latency to, for example, tracking algorithms, which require a shorter response time.

[0349] To address the shortcomings of traditional systems that require processing controller intervention, the systems and methods of the present disclosure can use DMA and a processor (e.g., a VPU) to configure a tightly coupled processing loop that allows the DMA to reconfigure its descriptors based on the processor's output. Thus, the DMA can be dynamically reprogrammed at runtime to process certain algorithms that require area-dependent data movement. This VPU configuration mode can be used to update the DMA's descriptors to track feature data (including location) based on runtime VPU calculations. Thus, the VPU can specify a list of address and data pairs in a memory (e.g., VMEM) and then trigger the DMA to update its own descriptors to collect data from the area with the newly calculated address. By relying on the interface between the VPU and DMA, once the processing controller initially configures the VPU and DMA to start processing, the processing controller (e.g., an R5 or ARM processing core) may not need to intervene. This batch, fast, and synchronous MMIO access for updating function descriptors therefore reduces latency for object tracking, feature tracking, object detection, deep learning, and / or other algorithms with area-dependent data movement.

[0350] Now refer to Figure 11A , Figure 11A A data flow diagram 1100 is shown of a process for configuring a direct memory access (DMA) system using a vector processing unit (VPU) in accordance with some embodiments of the present disclosure. It will be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the process 1100 may be performed by a processor comprising Figures 13A-13D Example autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 15 The system performs similar components, features and / or functions of the example data center 1500.

[0351] The system performing process 1100 may include a processing controller 1102 (e.g., an R5 processor, an ARM processing core, an instruction set architecture (ISA), an X86 architecture, etc.), a direct memory access (DMA) system 1104, a vector processing unit (VPU) 1108 (or another processor type), a vector memory (VMEM) 1110 (or another memory type), and a descriptor RAM 1106. In practice, the VPU configuration mode may configure the DMA to retrieve descriptors by writing a series of non-contiguous address / data pairs to the DMA descriptor SRAM. Process 1100 may be described with respect to an example feature or object tracking algorithm. However, this is not intended to be limiting, and process 1100 and the underlying system may be used to execute any type of algorithm, such as those with region-dependent data movement.

[0352] For example, a first operation may include the processing controller 1102 configuring the DMA 1104 and the VPU 1108 to perform processing on some data, and then triggering both the DMA 1104 and the VPU 1108 to perform processing. For example, the processing controller 1102 may load the descriptor RAM 1106 into memory for the starting point of the processing and may configure the registers of the VPU 1108 for the specific type of operation that the VPU 1108 will perform on the data.

[0353] For the second operation, the VPU 1108 may trigger the DMA 1104 to read the initial feature data points in the VMEM 1110. For example, to begin work, the VPU 1108 requires data from the DMA 1104, so the VPU 1108 configures the DMA 1104 to load the data points into the VMEM 1110 at a location where the VPU 1108 knows to retrieve the data for processing.

[0354] In a third operation, the VPU 1108 may process the current feature data set and calculate the next tracked object or feature position. As a result, the VPU 1108 may now have calculated a new or updated position of the tracked feature or object.

[0355] In the fourth operation, the VPU 1108 may use the VPU configuration format (see Figure 11B Description) updates VMEM 1110 with the updated location, which can then trigger DMA 1104 to update its descriptor in Descriptor RAM 1106. For example, Figure 11B is a table 1120 illustrating a VPU configuration format written into vector memory (VMEM) by the VPU and read by the DMA system according to some embodiments of the present disclosure. For example, for each address / data pair, the format may include four bytes of address and four bytes of data.

[0356] In a fifth operation, DMA 1104 may update the descriptors in descriptor RAM 1106 to retrieve appropriate data for the next processing iteration of VPU 1108. For example, DMA 1104 may read the address / data pairs forming the VPU configuration format to patch the operation descriptor with the updated location. In an embodiment, a one-to-one correspondence may exist between feature points and descriptors, such that each tracked feature, object, or point may include an associated descriptor. In this way, the address / data pairs for each tracked feature, object, or point may be updated over time using a separate descriptor.

[0357] In a sixth operation, DMA 1104 may retrieve new feature data for the location using the newly updated descriptor in Descriptor RAM 1106. For example, DMA 1104 may indicate to VPU 1108 that the descriptor has been updated, and VPU 1108 may trigger DMA 1104 to read the new data into VMEM 1110, and so on.

[0358] As a result, after the first configuration operation by the processing controller, operations two through six can be repeated to form a tightly synchronized VPU configuration loop that requires intervention by the processing controller - thereby reducing latency to account for the short response times required by the tracking or detection algorithm. In addition, because the DMA 1104 is overwriting the address in memory with the new updated address, the DMA 1104 is updating the code that the DMA 1104 needs to look at to determine what to fetch next. By doing this, throughput is increased compared to traditional systems that rely on the control bus to update registers with addresses and data. Thus, the benefits of defining an address / data protocol are realized in which variable address locations with variable amounts of data can be updated and how address / data pairs are updated. This allows the DMA 1104 - which may be larger in width than the width of the control bus (e.g., 512 bits and 32 bits, respectively) - to update up to (for example, but not limited to) 8 address / data pairs at a time (where each address / data pair is defined using 8 bytes, as shown in Figure 1). Figure 11B shown).

[0359] Furthermore, while the DMA is shown as being updated using the VPU configuration mode of process 1100, additional or alternative elements or components of the system may be updated. For example, the instruction cache of VPU 1108 may be updated using a similar method using the VPU. As another example, an updated hardware sequencer program may be written to update the hardware sequencer memory by providing address data. This would essentially involve writing the hardware sequencer RAM with a new program—e.g., for Figure 10H The hardware sequence controller 1060 and the hardware sequencer RAM 1054.

[0360] Now refer to Figure 11C , each block of the method 1150 described herein comprises a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 1150 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 1150 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, with respect to Figure 11A The method 1150 is described for a system and can be performed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0361] Figure 11C 11 is a flow chart of a method 1150 for configuring a DMA system using a VPU according to some embodiments of the present disclosure. At block B 1102, the method 1150 includes computing, using a processor and at least in part, computing a first output corresponding to one or more first updated positions of tracked features based on first data written to a memory using the DMA system. For example, the VPU 1108 may access data from the VMEM 1110 written to the VMEM 1110 using the DMA 1104 and may process the data to compute one or more object positions corresponding to tracked features, objects, points, and the like.

[0362] At block B1104, the method 1150 includes updating the memory using the processor to include second data representing one or more address / data pairs corresponding to the one or more first updated locations. For example, after calculating the one or more locations, the VPU 1108 may update the VMEM 1110 with address / data pairs in a format such as Figure 11B The format of the description.

[0363] At block B 1106, method 1150 includes using the DMA system and updating one or more descriptors corresponding to the tracked feature based at least in part on the one or more address / data pairs. For example, DMA 1104 may access the address / data pairs from VMEM 1110 and use the address / data pairs to update the descriptors in descriptor RAM 1106 for the next read operation.

[0364] At block B 1108 , method 1150 includes writing third data to memory using the DMA system and based at least in part on the one or more descriptors. For example, DMA 1104 may write updated data to VMEM 1110 corresponding to the address / data pair identified using the descriptor.

[0365] At block B1110, method 1150 includes computing, using the processor and based at least in part on the third data, a second output corresponding to one or more second updated locations of the tracked features. For example, once the updated data is in VMEM 1110, VPU 1108 may compute a next set of updated address / data pairs corresponding to the tracked features, objects, points, etc., and this process may be repeated until processing is complete.

[0366] Permanent Fault Detection in Programmable Vision Accelerators (PVA)

[0367] In safety-critical applications such as autonomous and semi-autonomous machine applications, there are strict requirements for permanent fault detection and isolation. For example, when deep learning, computer vision, sensor processing, and / or other applications are executed in a machine, permanent fault detection must be performed regularly within the allocated time budget to allow accurate testing while also allowing the application to execute correctly—i.e., with low latency. With respect to Automotive Safety Integrity Level (ASIL) D, applications executed in autonomous or semi-autonomous machines may require 90% or more permanent fault coverage. To achieve this, end-to-end coverage may be required with low latency while meeting the runtime budget of each specific application. Traditional approaches use built-in self-test (BIST) to identify faults, but these BIST techniques either do not include sufficient coverage, introduce excessive latency in the system, and / or do not meet the runtime budget of some applications.

[0368] To address the deficiencies of these conventional approaches, the present systems and methods may perform a Multiple Input Signature Register (MISR) BIST—for example, to perform fault detection of a Programmable Vision Accelerator (PVA) of a System-on-Chip (SoC). For example, in various embodiments of the present disclosure, the PVA may include one or more DMA systems and one or more VPUs that are controlled using one or more processing controllers (or control processors) (e.g., an R5 processor and an ARM processor, CPU, and / or the like). Therefore, each component of the PVA may require testing, and the present systems and methods perform a MISR BIST to detect permanent faults in an end-to-end manner. In this manner, permanent fault detection may be performed to cover end-to-end blocks of control and data logic, report errors directly to a safety processor to reduce latency, and be customized for a specific application to meet an associated runtime budget.

[0369] In various embodiments, MISRs can be used in PVAs to implement software logic BIST for permanent fault detection. Figure 12Aand / or 12B) may include cyclic redundancy check (CRC) hardware that is initialized (e.g., using a known seed value) by the processing controller. While executing the PVA application, the processing controller may allocate a portion of the timing budget (e.g., approximately 10% or less of the timing budget) to run a known software MISR test with known inputs that has deterministic pre-computed outputs with a correct signature or golden value. For example, where the timing budget corresponds to 30 frames per second, a timing budget corresponding to 3 or fewer frames may be allocated to the MISR test. At the allocated time, the processing controller may initiate the MISR test and wait for the test to complete to terminate the MISR CRC calculation. Once the test is complete, the MISR hardware may read back the final CRC value and check it against the pre-computed golden value. In the event of a mismatch, the MISR hardware may report the error directly to the SoC's security processor for further action to address the security error—e.g., causing the application's output to be ignored, addressing or resolving a permanent fault, etc.

[0370] Thus, the MISR hardware in the DMA block can monitor one or more (e.g., in an embodiment, all) transactions on one or more (e.g., in an embodiment, all) Advanced eXtensible Interface (AXI) master ports of the PVA. By checking all output stages from the PVA, in an embodiment, the safety integrity of the PVA can be checked for permanent defects (e.g., output information) that could corrupt the output stages, which could be consumed by the PVA and / or another engine when executing an application. The MISR hardware can therefore detect errors across different blocks of the PVA (e.g., the processing controller, the VPU, and the DMA system) because these components all collaborate and interact in generating the output stages. The signatures calculated in the MISR hardware can represent the state of these different PVA blocks during MISR testing.

[0371] In an embodiment, the MISR scheme may include CRC checking of both the write address (e.g., 40-bit control) and the write data (e.g., 512-bit data) leaving the AXI master port. This feature may allow control path failures (e.g., addressing errors) to be isolated from data path failures (e.g., calculation errors). Due to the configuration of the MISR hardware (as described herein), each DMAAXI port may be able to be checked. In an embodiment, control bits may be used to disable writes to the address and data outputs of all channels participating in the MISR calculation in order to save bandwidth consumption in the memory subsystem and during memory allocation. In addition, the MISR scheme may include control register bits for each channel to exclude or mask specific channels from the MISR calculation - for example, to isolate non-secure channels. In an embodiment, the DMA may use IEEE 802 and MPEG CRC-32 primitive polynomials to calculate the MISR CRC: X 32 +X 26 +X 23 +X 22 +X 16 +X 12 +X 11 +X 10 +X 8 +X 7 +X 5 +X 4 +X 2 +X+1. The MISR SET register can be used to set the initial CRC value (e.g., seed value) for the address and data CRC calculations. The MISR REF register can be used to compare the CRC values ​​for the address and data CRC calculations.

[0372] To support MISRs with 512-bit data, 8:1 bit data compression can be applied—for example, each data byte can be compressed to 1 data bit via an 8>1 exclusive OR (XOR) operation to form 2x32-bit message data. To support MISR 40-bit addresses, the 9 most significant bits can be compressed—for example, the 9 most significant bits can be compressed via a 9>1 XOR operation to form a 32-bit message address. Variations in test patterns and instructions can be used to cover aliasing associated with compression. The likelihood of aliasing can be low because error failures do not generate address CRC errors when there are an even number of errors in a byte on the output image. Furthermore, aliasing can be unlikely because the reference CRC can be calculated on the output image with the same pattern in the same even-numbered error bit positions throughout the MISR test. During experiments, aliasing was shown to result in an average coverage loss of 0.25%. In embodiments, data compression with this low aliasing is valuable due to the bus width (e.g., 512 bits), and without compression, MISR testing may not meet the system's latency or runtime budget.

[0373] The MISR timer register can be used to time out the MISR calculation, and the MISR timer register can be decremented on every AXI clock. The timeout feature can be helpful if a failure occurs that causes the MISR test to hang, which may prevent the MISR hardware from reporting an error. When the MISR test is completed, the processing controller can use a software event to stop the MISR calculation. The DMA system can compare the MISR REF value with the MISR VAL value of the data and address outputs of the MISR test, and the DMA hardware can update the MISR status register based on the comparison result. For example, the MISR status register can include one of the following values: 0: Idle; 1: Complete: Failed Data; 3: Busy; 4: Complete: Failed Both Address and Data; 5: Complete: Failed Timeout; 6: RSVD; and 7: Complete: Passed. In the event of a MISR timeout error, the DMA can generate a timeout signal to the security processor, and in the event of a CRC check error in the data and / or address, the DMA can assert a security error to the security processor.

[0374] refer to Figure 12A , Figure 12Ais a diagram of a built-in self-test (BIST) system for performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) in accordance with some embodiments of the present disclosure. It will be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the MISR hardware 1250 may include Figure 10H DMA system 1050, Figures 13A-13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 of , and / or Figure 15 1500. For example, the MISR hardware 1200 may be included in a DMA block of a PVA, such as Figure 10H As shown, the MISR hardware 1200 can operate at the output level of data movement, and addressing (or control) can access the output of the DMA engine 1056. Figure 12A As shown, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used according to an embodiment.

[0375] In operation, the processing controller can control the DMA system 1050 and the MISR hardware 1200 - as well as one or more processing components of the system, such as the VPU. When performing MISR testing on the DMA system 1050, in an embodiment, the test code can include all 0s, all 1s, alternating 0s and 1s, and / or random code sequences. In this way, high coverage of the DMA system 1050 can be achieved. For example, when testing the VPU, the test code can include application-specific or customized code. For example, during coverage testing of a specific application, the components or parts of the VPU used (e.g., registers, logic, etc.) can be determined, and test code can be generated so that those specific components or parts of the VPU are included in the execution of the test code. For example, this random data with different instructions can be included in the test code so that the test can be sequenced by different instructions to use different areas of the VPU logic. In this way, the coverage of the VPU as a whole is increased, and in particular the coverage of the specific application executed on the VPU. By performing DMA and VPU testing in this manner, and because the processing controller is involved in the control and interaction between various components (e.g., DMA system 1050 and VPU), the processing controller can have high coverage because the output and addressing of data movement are affected by the processing controller interaction.

[0376] During testing, where different code patterns are used, the code patterns may be used in an alternating pattern, or one code may be used for a first time frame (e.g., a time equivalent to 30 fps), another code for a second time frame (e.g., a time equivalent to 30 fps), another code for a third time frame (e.g., a time equivalent to 30 fps), etc. For example, in the DMA code example, a code of 0 may be used for the first time frame, then a code of 1 for the second time frame, then a code of alternating 0 and 1 (e.g., 0101010101...) for the third time frame, then a random code for the fourth time frame (e.g., 011100100010...), and then these four codes may be repeated, and so on.

[0377] In practice, when testing DMA, for example, the process controller can interact with the MISR controller 1206 to write reference values ​​to the MISR data set register 1210 and the MISR address set register 1216. These values ​​may be different for data and addresses and can be referred to as seed values ​​for the CRC calculation. The process controller can then initialize the channels in the DMA engine 1056 to perform data movement, and because the location of the test code in memory is known to the process controller, the descriptors (e.g., configured by the process controller in the descriptor SRAM 1052) can be used to sequence the DMA engine 1056 with the data for the MISR test. The process controller can set a timer 1226 on the MISR hardware 1200 to enable MISR testing, and can then trigger the channels of the DMA engine 1056 to begin reading test data from the source destination and outputting the data to the MISR hardware 1200 for MISR testing. Therefore, when testing DMA, data movement is being tested (e.g., correct addressing and correct data in the addressed location), so the MISR hardware 1200 can access the output of the DMA engine 1056 while the DMA engine performs data movement of the test code. This tap into the output stage can be Figure 12AAs indicated in FIG, as external memory, it can be funneled in the order of the processing controller (one data channel at a time, one address channel at a time). For example, for the data channels, the processing controller can sort through each of the 16 data channels, and the corresponding AXI write data (wdata) for each channel can be fed through the CH0-CH16 data CRC calculation 1202—e.g., in series. For example, the processing controller can configure the channel output register 1220 to pass through the channels one at a time according to the configuration order from the processing controller. In an embodiment, the channel mask register 1208 (e.g., programmed by the MISR controller 1206 based on interaction with the processing controller) can be configured by the processing controller to mask or remove various channels (e.g., channels not under test) from the CRC calculation. In an embodiment, this masking can be performed using an AND gate. When one or more channels are masked, the golden value in the MISR data reference register 1222 (which can be provided by the processing controller to the MISR controller 1206) may only correspond to the CRC calculation of the unmasked channels. For each unmasked channel, the data on the channel (generated using the test code read from memory) can be applied (e.g., with or without compression) to the polynomial of the CRC data calculation 1202 to generate a MISR data value 1214 for that channel. Once a channel has completed calculation, the processing controller can receive an indication and can cause the next data channel to be sent to the CRC calculation 1202 to calculate the next MISR data value 1214, and so on, until each unmasked channel has a corresponding MISR data value 1214. Once each MISR data value 1214 for a particular iteration has been calculated, those values ​​1214 can be combined to generate a final MISR data value, which can be compared to the golden value in the MISR data reference register 1222 to generate a MISR data state determination (e.g., which can include states corresponding to the values ​​0-7 described above).

[0378] As another example, for address channels, the processing controller may sequence each of, for example, 16 address or control channels and feed the corresponding AXI write address (waddress) address of each channel through the CH0-CH16 CRC calculation 1204 - for example, in series. In an embodiment, a channel mask register 1208 may be configured by the processing controller to mask or remove various channels from the CRC calculation - for example, channels not under test. In an embodiment, this masking may be performed using an AND gate. In the event that one or more channels are masked out, the golden value in the MISR data reference register 1224 may correspond only to the CRC calculation for the unmasked channels. For each unmasked channel, the address on the channel (generated using the test code read from memory) may be applied (e.g., with or without compression) to the polynomial of the CRC address calculation 1204 to generate the MISR address value 1218 for that channel. Once a channel has completed calculation, the processing controller may receive an indication and may cause the next channel of address data to be sent to the CRC calculation 1204 to calculate the next MISR address value 1218, and so on, until each unmasked channel has a corresponding MISR address value 1218. Once each MISR address value 1218 for a particular iteration has been calculated, those values ​​1218 may be combined to generate a final address MISR value, which may be compared to the golden value in the MISR reference register 1224 address to generate a MISR address state determination (e.g., which may include states corresponding to the values ​​0-7 described above).

[0379] In some embodiments, MISR testing may be iterative, such that a first code may be processed, the output may be tested, and then the output may be used for the next iteration, which may be tested, etc. In such embodiments, MISR testing may include multiple phases, and completed MISR testing may include executing each phase.

[0380] In the case where the MISR hardware 1200 is specifically used to test a VPU, for example, the DMA system 1050 can move the test code into VMEM, the VPU can process the test code and write the results back to VMEM, and the DMA engine 1056 can read the results back from VMEM to the destination location. When writing the results back to the destination location, the MISR hardware 1200 can access the DMA output and execute a MISR on the data (e.g., including the data and the address), and perform a MISR similar to that discussed herein. In this way, the interaction of the VPU with the test code can be tested using the MISR hardware 1200.

[0381] After completing the MISR test, the processing controller may receive an interrupt. For example, the processing controller may receive a completion interrupt and, in the absence of an error, may wait for the next MISR test cycle. In the event that the interrupt is an error interrupt, the type of error may be determined—e.g., failed data, failed address, failed both, etc.—and a security error may be asserted to the security processor. For example, in some embodiments where the MISR hardware 1200 is hung or idle (e.g., with a timeout error), the DMA may generate a timeout signal to the security processor of the SoC.

[0382] In some embodiments, to speed up MISR calculations to calculate CRC on one or more (e.g., in an embodiment, such as 16) channels without serializing or staging the channel-MISR calculations, the channels may be demultiplexed based on the channel IDs present in the AXIID field to parallelize the channel calculations. For example, since CRC calculations are done at different rates, Figure 12A The method involves serial processing of channels one by one. However, using Figure 12B In a system with a MISR, as described below, these calculations can be performed in parallel. For example, when the processing controller terminates the MISR calculation, the MISR controller can sequence all channel outputs to calculate a final signature that can be compared to a reference or golden value for the address and data outputs. This feature can speed up permanent fault detection without requiring an additional programmer register interface—for example, because the same control registers can be used for all channels.

[0383] Likewise, and with reference to Figure 12B , Figure 12B is a diagram of a built-in self-test (BIST) system for parallel channel cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth by way of example only. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components and in any suitable combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory. In some embodiments, the MISR hardware 1250 may include Figure 10H DMA system 1050, Figures 13A-13D Example of autonomous vehicle 1300, Figure 14 The example computing device 1400 and / or Figure 151500. For example, the MISR hardware 1250 may be included in a DMA block of a PVA, such as Figure 10H As shown. In this way, the MISR hardware 1250 can operate at the output level of the data movement, and the addressing (or control) can access the output of the DMA engine 1056. Figure 12A As shown, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used according to an embodiment.

[0384] MISR hardware 1250 can be used with Figure 12A The MISR hardware 1200 operates similarly. Except that the MISR hardware 1250 can be configured for parallel data channel and parallel address channel CRC calculations. For example, the processing controller can configure the MISR data set register 1256 to set a seed or reference value for each data CRC calculation 1260A-1260N (corresponding to AXI data channels 0-15, respectively), and can configure the MISR address set register 1258 to set a seed or reference value for each address CRC calculation 1262A-1262N (corresponding to AXI address channels 0-15, respectively). The processing controller, similar to the Figure 12A As described, data movement of the DMA system 1050 (eg, for DMA testing) and / or VPU processing (eg, for VPU-specific testing) may then be triggered to move the data, and the MISR hardware 1250 may access the output stage for testing.

[0385] Thus, the processing controller may cause 16 channels of data to be sent to multiplexer (mux) 1252 and 16 channels of address data to be sent to multiplexer (mux) 1254. Mux 1252 may then provide the corresponding channel's data to corresponding CRC calculations 1260A-1260N (e.g., channel 0 AXI data to channel 0 CRC calculation 1260, channel 1 data to channel 1 CRC calculation 1260B, etc.), and each CRC calculation 1260 may use the data and a CRC polynomial with a reference value to calculate a MISR data value 1284A-1284N (e.g., channel 0 CRC calculation 1260A may calculate MISR data 0 value 1284A, channel 1 CRC calculation 1260B may calculate MISR data 1 value 1284B, etc.). MISR data values ​​1284A-1284N may then be sorted out of multiplexer (mux) 1264 according to the MISR sequence from MISR control 1270 configured by the process controller. Figure 12AAs described, one or more channels may not be included in a particular MISR test, so the channel mask register 1268 can be configured by the processing controller to update the MISR sequence so that the MISR data values ​​1284 corresponding to one or more masked channels are not provided to the channel 0-16 data CRC calculation 1274 for use in calculating the final CRC value. For the unmasked channels, multiplexer 1264 can output the MISR data values ​​1284 according to the MISR sequence. In this manner, the different calculation times for different channels and CRC calculation 1260 are accounted for because the MISR data values ​​1284 are forced to be output according to the MISR sequence rather than being sent to the CRC calculation 1274 according to the timing of the CRC calculation being completed. Once the MISR sequence of MISR data values ​​1284 is output by multiplexer 1264 to the CRC calculation 1274, the CRC calculation 1274 can calculate the final CRC value and store the final CRC value in the VAL register 1276. The final CRC value in the VAL register 1276 may then be compared to the golden value in the MISR data reference register 1272 (configured by the MISR control 1270 from the process controller) to determine the MISR data status.

[0386] Similarly, the processing controller can cause 16 address channels to be sent to multiplexer (mux) 1254, which can then provide the corresponding address channel to the corresponding CRC calculation 1262A-1262N (e.g., channel 0 AXI address to channel 0 CRC calculation 1262, channel 1 address to channel 1 CRC calculation 1262B, etc.), and each CRC calculation 1262 can use the address and the CRC polynomial with the reference value to calculate a MISR address value 1286A-1286N (e.g., channel 0 CRC calculation 1262A can calculate MISR address 0 value 1286A, channel 1 CRC calculation 1262B can calculate MISR address 1 value 1286B, etc.). The MISR address values ​​1286A-1286N can then be sorted out of multiplexer (mux) 1266 according to the MISR sequence from MISR control 1270, as configured by the processing controller. In an embodiment, for example, with respect to Figure 12AAs described, one or more channels may not be included in a particular MISR test. Therefore, the channel mask register 1268 may be configured by the process controller to update the MISR sequence so that the MISR address values ​​1286 corresponding to one or more masked channels are not provided to the channel 0-16 address CRC calculation 1280 for use in calculating the final CRC value. For unmasked channels, the MISR address values ​​1286 may be output by multiplexer 1266 according to the MISR sequence. In this manner, the different calculation times for different channels and CRC calculation 1262 are accounted for because the MISR address values ​​1286 are forced to be output according to the MISR sequence rather than being sent to CRC calculation 1280 according to the timing of the CRC calculation being completed. Once multiplexer 1266 outputs the MISR sequence of MISR address values ​​1286 to CRC calculation 1280, CRC calculation 1280 may calculate the final CRC value and store the final CRC value in VAL register 1282. The final CRC value in VAL register 1282 may then be compared to the golden value in MISR address reference register 1278 (configured by MISR control 1270 from the process controller) to determine the MISR address status.

[0387] MISR data status and MISR address status can be similar to the above Figure 12A Description to check and use.

[0388] Now refer to the figure Figure 12C As shown, each block of the method 1290 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 1290 can also be embodied as computer-usable instructions stored on a computer storage medium. The method 1290 can be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, the method 1290 is about Figure 12A Although the system described herein, method 1290 can be performed by any one system, structure, or component or any combination of systems, structures, or components, including but not limited to those described herein.

[0389] Figure 12C 1 is a flow chart of a method 1290 for performing a built-in test (BIST) for performing a permanent fault detection in a PVA according to some embodiments of the present disclosure. At block B 1202, the method 1290 includes receiving a plurality of data channels from a DMA system, one channel at a time and based on an ordering by a processing controller. For example, the MISR hardware 1200 may receive one data channel (or one address data channel) at a time according to an order determined by the processing controller.

[0390] At block B 1204, method 1290 includes calculating a plurality of MISR values ​​by performing a CRC calculation using the CRC calculated polynomial and the corresponding data corresponding to the channel to calculate the MISR value. For example, for each channel, CRC calculation 1202 (or address 1204) may calculate a MISR data value 1214 (or MISR address value 1216 for the address) using the data (or address) from the channel and the CRC calculated polynomial 1202 (starting from a seed value in the CRC MISR data set register 1210 or the MISR address set register 1216).

[0391] At block B 1206 , the method 1290 includes calculating a final MISR value using the plurality of MISR values. For example, the MISR data values ​​1214 from each channel (or the MISR address values ​​from each channel) may be combined to generate the final MISR value.

[0392] At block B 1208, method 1290 includes comparing the final MISR value to the signature value. For example, the final MISR value generated from each MISR value 1214 (or address value 1216) can be compared to the signature or golden value of the MISR data reference register 1222 (or the MISR address reference register 1224 for the address).

[0393] At block B1210, method 1290 includes outputting a MISR status based at least in part on the comparison. For example, based on the comparison at block B1208, a status may be determined—e.g., failed data, failed address, failed both, completed, etc.—and the status may be used to notify a security processor of the SoC that an error condition has occurred.

[0394] Example autonomous vehicle

[0395] Figure 13A1 is an illustration of an example autonomous vehicle 1300 according to some embodiments of the present disclosure. Autonomous vehicle 1300 (alternatively referred to herein as "vehicle 1300") may include, but is not limited to, a passenger vehicle such as a car, a truck, a bus, a first responder vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire truck, a police vehicle, an ambulance, a boat, a construction vehicle, an underwater vessel, a drone, a vehicle coupled to a trailer, and / or another type of vehicle (e.g., a vehicle that is unmanned and / or accommodates one or more passengers). Autonomous vehicles are generally described according to the levels of automation defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE), "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, issued on June 15, 2018, Standard No. J3016-201609, issued on September 30, 2016, and previous and future versions of such standards). The vehicle 1300 may be capable of implementing functionality consistent with one or more of autonomous driving levels 3 to 5. For example, depending on the embodiment, the vehicle 1300 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0396] Vehicle 1300 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power plant, an all-electric engine, and / or another type of propulsion system. Propulsion system 1350 may be connected to a drivetrain of vehicle 1300, which may include a transmission, to achieve propulsion of vehicle 1300. Propulsion system 1350 may be controlled in response to receiving a signal from throttle / accelerator 1352.

[0397] A steering system 1354, which may include a steering wheel, may be used to steer vehicle 1300 (e.g., along a desired path or route) when propulsion system 1350 is operating (e.g., while the vehicle is in motion). Steering system 1354 may receive signals from steering actuator 1356. For fully automated (Level 5) functionality, a steering wheel may be optional.

[0398] Brake sensor system 1346 may be used to operate vehicle brakes in response to receiving signals from brake actuator 1348 and / or brake sensors.

[0399] May include one or more system on chip (SoC) 1304 ( Figure 13C ) and / or one or more GPUs can provide signals (e.g., representing commands) to one or more components and / or systems of vehicle 1300. For example, one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 1348, to operate steering system 1354 via one or more steering actuators 1356, and to operate propulsion system 1350 via one or more throttles / accelerators 1352. One or more controllers 1336 can include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1300. One or more controllers 1336 can include a first controller 1336 for autonomous driving functionality, a second controller 1336 for functional safety functionality, a third controller 1336 for artificial intelligence functionality (e.g., computer vision), a fourth controller 1336 for infotainment functionality, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 may handle two or more of the above functions, two or more controllers 1336 may handle a single function, and / or any combination thereof.

[0400] The one or more controllers 1336 may provide signals for controlling one or more components and / or systems of the vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example and without limitation, a global navigation satellite system sensor 1358 (e.g., a global positioning system sensor), a RADAR sensor 1360, an ultrasonic sensor 1362, a LIDAR sensor 1364, an inertial measurement unit (IMU) sensor 1366 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1396, a stereo camera 1368, a wide-angle camera 1370 (e.g., a fisheye camera), an infrared camera 1372, a surround camera 1374 (e.g., a 360-degree camera), a long-range and / or mid-range camera 1398, a speed sensor 1344 (e.g., for measuring the velocity of the vehicle 1300), a vibration sensor 1342, a steering sensor 1340, a brake sensor (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0401] One or more of the controllers 1336 may receive input (e.g., represented by input data) from the instrument cluster 1332 of the vehicle 1300 and provide output (e.g., represented by output data, display data, etc.) via a human machine interface (HMI) display 1334, an audible annunciator, a speaker, and / or via other components of the vehicle 1300. These outputs may include information such as vehicle speed, velocity, time, map data (e.g., Figure 13C The HMI display 1334 may display information such as an HD map 1322 of the vehicle 1300, position data (e.g., the position of the vehicle 1300 on the map), direction, the positions of other vehicles (e.g., an occupancy grid), information about objects and object states as sensed by the controller 1336, etc. For example, the HMI display 1334 may display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0402] The vehicle 1300 also includes a network interface 1324 that can communicate over one or more networks using one or more wireless antennas 1326 and / or a modem. For example, the network interface 1324 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 1326 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0403] Figure 13B For use according to some embodiments of the present disclosure Figure 13A 13. Example camera positions and fields of view for autonomous vehicle 1300. The cameras and respective fields of view are an example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or located at different locations on vehicle 1300.

[0404] The camera type used for the camera may include, but is not limited to, a digital camera that may be suitable for use with components and / or systems of the vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120fps, 240fps, and the like, depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a clear pixel camera such as a camera with an RCCC, RCCB, and / or RBGC color filter array may be used in an effort to improve light sensitivity.

[0405] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0406] One or more of the cameras can be mounted in a mounting assembly, such as a custom-designed (3-D printed) assembly, to cut off stray light and reflections from within the car (e.g., reflections from the dashboard reflected in the windshield mirror) that could interfere with the camera's ability to capture image data. With respect to the wing mirror mounting assembly, the wing mirror assembly can be custom 3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0407] A camera (e.g., a front-facing camera) having a field of view that includes a portion of the environment in front of the vehicle 1300 can be used for surround vision to help identify the forward path and obstacles, as well as assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1336 and / or control SoCs. The front-facing camera can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used for ADAS functions and systems, including lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0408] A variety of cameras can be used in the front-facing configuration, including, for example, a monocular camera platform including a CMOS (complementary metal oxide semiconductor) color imager. Another example could be a wide-angle camera 1370, which can be used to sense objects entering the field of view from the periphery (e.g., pedestrians, intersection traffic, or bicycles). Figure 13B The figure shows only one wide-angle camera, but any number of wide-angle cameras 1370 can be present on vehicle 1300. In addition, long-range cameras 1398 (e.g., a long-view stereo camera pair) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Long-range cameras 1398 can also be used for object detection and classification, as well as basic object tracking.

[0409] One or more stereo cameras 1368 may also be included in the front configuration. The stereo camera 1368 may include an integrated control unit including a scalable processing unit that may provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit may be used to generate a 3-D map of the vehicle environment, including distance estimates for all points in the image. An alternative stereo camera 1368 may include a compact stereo vision sensor that may include two camera lenses (one on the left and one on the right) and an image processing chip that may measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 may be used in addition to or alternatively to those described herein.

[0410] Cameras with a field of view that includes portions of the environment to the sides of the vehicle 1300 (e.g., side-view cameras) can be used for surround viewing, providing information used to create and update occupancy grids and generate side impact collision warnings. For example, surround cameras 1374 (e.g., Figure 13B Four surround cameras 1374 (shown in FIG) can be placed on vehicle 1300. Surround cameras 1374 can include wide-angle camera 1370, fisheye camera, 360-degree camera, and / or the like. For example, four fisheye cameras can be placed on the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1374 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0411] A camera having a field of view that includes a portion of the environment behind the vehicle 1300 (e.g., a rearview camera) can be used to assist with parking, surround view, rear collision warning, and creating and updating an occupancy grid. A variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range cameras 1398, stereo cameras 1368, infrared cameras 1372, etc.).

[0412] Figure 13C For use according to some embodiments of the present disclosure Figure 13A 13. Block diagram of an example system architecture for an example autonomous vehicle 1300. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory.

[0413] Figure 13C Each of the components, features, and systems of vehicle 1300 is illustrated as being connected via bus 1302. Bus 1302 may include a controller area network (CAN) data interface (alternatively, referred to herein as a "CAN bus"). CAN may be a network internal to vehicle 1300 that assists in controlling various features and functions of vehicle 1300, such as actuation of brakes, acceleration, braking, steering, windshield wipers, and the like. The CAN bus may be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0414] Although bus 1302 is described here as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or in lieu of a CAN bus. Furthermore, although bus 1302 is represented by a single line, this is not intended to be limiting. For example, there may be any number of buses 1302, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1302 may be used for collision avoidance functionality, and a second bus 1302 may be used for drive control. In any example, each bus 1302 may communicate with any component of vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304 , each controller 1336 , and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors of the vehicle 1300 ) and may be connected to a common bus such as a CAN bus.

[0415] The vehicle 1300 may include one or more controllers 1336, such as those described herein. Figure 13A Controller 1336 may be used for a variety of functions. Controller 1336 may be coupled to any of the other various components and systems of vehicle 1300 and may be used for control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment for vehicle 1300, and / or the like.

[0416] The vehicle 1300 may include one or more system-on-chips (SoCs) 1304. The SoCs 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. The SoCs 1304 may be used to control the vehicle 1300 in a variety of platforms and systems. For example, the one or more SoCs 1304 may be combined with an HD map 1322 in a system (e.g., a system of the vehicle 1300), which may be downloaded from one or more servers (e.g., a vehicle) via a network interface 1324. Figure 13D one or more servers 1378) to obtain map refreshes and / or updates.

[0417] The CPU 1306 may include a CPU cluster or CPU complex (alternatively, referred to herein as a "CCPLEX"). The CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1306 may include four dual-core clusters, each with a dedicated L2 cache (e.g., a 2MB L2 cache). The CPU 1306 (e.g., CCPLEX) may be configured to support simultaneous cluster operations such that any combination of CPU 1306 clusters can be active at any given time.

[0418] CPU 1306 may implement power management capabilities including one or more of the following features: each hardware block may be automatically clock gated when idle to conserve dynamic power; each core clock may be gated when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. CPU 1306 may further implement an enhanced algorithm for managing power states, in which allowed power states and expected wakeup times are specified, and hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core may support a simplified power state entry sequence in software, with this work being offloaded to the microcode.

[0419] GPU 1308 may include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU 1308 may be programmable and efficient for parallel workloads. In some examples, GPU 1308 may use an enhanced tensor instruction set. GPU 1308 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1308 may include at least eight streaming microprocessors. GPU 1308 may use a computing application programming interface (API). In addition, GPU 1308 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0420] In the case of automotive and embedded use, GPU 1308 can be power optimized to achieve optimal performance. For example, GPU 1308 can be manufactured on fin field effect transistors (FinFETs). However, this is not intended to be limiting, and GPU 1308 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed precision processing cores divided into multiple blocks. For example and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIA tensor cores for deep learning matrix arithmetic, L0 instruction cache, warp scheduler, dispatch unit and / or 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating point data paths to provide efficient execution of workloads using a mix of computation and addressing calculations. The streaming microprocessor may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. The streaming microprocessor may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0421] GPU 1308 can include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem that provides a peak memory bandwidth of approximately 900 GB / s in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as fifth generation graphics double data rate synchronous random access memory (GDDR5), can be used in addition to or in lieu of HBM memory.

[0422] The GPU 1308 may include unified memory technology that includes access counters to allow memory pages to be more accurately migrated to the processor that accesses them most frequently, thereby improving the efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support may be used to allow the GPU 1308 to directly access the CPU 1306 page tables. In such an example, when the GPU 1308 memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU 1306. In response, the CPU 1306 may look up the virtual-to-physical mapping for the address in its page table and transmit the translation back to the GPU 1308. In this way, unified memory technology may allow a single unified virtual address space to be used for memory of both the CPU 1306 and the GPU 1308, thereby simplifying GPU 1308 programming and porting of applications to the GPU 1308.

[0423] Additionally, GPU 1308 may include access counters that can track how often GPU 1308 accesses the memory of other processors. The access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0424] SoC 1304 may include any number of caches 1312, including those described herein. For example, cache 1312 may include an L3 cache available to both CPU 1306 and GPU 1308 (e.g., connected to both CPU 1306 and GPU 1308). Cache 1312 may include a write-back cache that can track the state of lines, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4MB or more, although smaller cache sizes may also be used.

[0425] The SoC 1304 may include an arithmetic logic unit (ALU) that may be utilized in performing any of a variety of tasks or operations related to the vehicle 1300, such as processing a DNN. Furthermore, the SoC 1304 may include a floating point unit (FPU) (or other math coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, the SoC 1304 may include one or more FPUs integrated as execution units within the CPU 1306 and / or GPU 1308.

[0426] SoC 1304 may include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1304 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB SRAM) may enable the hardware accelerator cluster to accelerate neural networks and other calculations. The hardware accelerator cluster may be used to supplement GPU 1308 and offload some tasks of GPU 1308 (e.g., freeing up more cycles of GPU 1308 to perform other tasks). As an example, accelerator 1314 may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to easily control acceleration. When used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0427] Accelerator 1314 (e.g., a hardware accelerator cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and reasoning. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNN, RCNN, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and reasoning. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and far exceed the performance of the CPU. The TPU may perform several functions, including a single-instance convolution function, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0428] DLA can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of a wide variety of functions, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or security-related events.

[0429] The DLA can perform any function of the GPU 1308, and by using an inference accelerator, for example, the designer can target any function to either the DLA or the GPU 1308. For example, the designer can focus the processing of CNNs and floating-point operations on the DLA and leave other functions to the GPU 1308 and / or other accelerators 1314.

[0430] The accelerator 1314 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein alternatively as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0431] The RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, and / or the like. Each of these RISC cores can include any amount of memory. Depending on the embodiment, the RISC core can use any of a number of protocols. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core can include an instruction cache and / or tightly coupled RAM.

[0432] The DMA can enable components of the PVA to access system memory independently of the CPU 1306. The DMA can support any number of features used to provide optimizations for the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0433] A vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can operate as the main processing engine of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), a very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0434] Each of the vector processors can include an instruction cache and can be coupled to dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of images. Among other things, any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of these PVAs. In addition, the PVAs can include additional error correction code (ECC) memory to enhance overall system security.

[0435] The accelerator 1314 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1314. In some examples, the on-chip memory may include at least 4MB of SRAM consisting of, for example and without limitation, eight field-configurable memory blocks that can be accessed by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA may access the memory via a backbone that provides high-speed memory access to the PVA and DLA. The backbone may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using APB).

[0436] The on-chip computer vision network can include an interface that ensures that both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-based communication for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0437] In some examples, SoC 1304 may include a real-time ray tracing hardware accelerator, such as that described in U.S. patent application Ser. No. 16 / 101,232, filed on Aug. 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulations, for general wave propagation simulations, for comparison with LIDAR data for positioning and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0438] The accelerator 1314 (e.g., a hardware accelerator cluster) has a wide range of uses in autonomous driving. The PVA can be a programmable vision accelerator that can be used in key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithmic domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-intensive or intensive rule computations, and even on small data sets that require predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, the PVA is designed to run classic computer vision algorithms because they are efficient at object detection and integer math operations.

[0439] For example, according to one embodiment of the technology, PVA is used to perform computer stereo vision. In some examples, a semi-global matching-based algorithm can be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require on-the-fly motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0440] In some examples, PVA can be used to perform dense optical flow, by processing raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR. In other examples, PVA is used for time-of-flight depth processing, by processing raw time-of-flight data to provide processed time-of-flight data.

[0441] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence measure for each object detection. Such confidence values ​​can be interpreted as probabilities, or as providing a relative "weight" of each detection compared to other detections. This confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for the confidence level and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network for regressing confidence values. The neural network may take as its input at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1366 output related to the orientation and range of the vehicle 1300, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 1364 or RADAR sensor 1360), etc.

[0442] SoC 1304 may include one or more data stores 1316 (e.g., memory). Data store 1316 may be on-chip memory of SoC 1304 that may store neural networks to be executed on the GPU and / or DLA. In some examples, data store 1316 may be large enough to store multiple instances of the neural network for redundancy and safety. Data store 1312 may include an L2 or L3 cache 1312. References to data store 1316 may include references to memory associated with the PVA, DLA, and / or other accelerators 1314 as described herein.

[0443] SoC 1304 may include one or more processors 1310 (e.g., embedded processors). Processor 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management functions and related safety implementations. The boot and power management processor may be part of the SoC 1304 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assist system low-power state transitions, SoC 1304 thermal and temperature sensor management, and / or SoC 1304 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and SoC 1304 may use the ring oscillator to detect the temperature of CPU 1306, GPU 1308, and / or accelerator 1314. If the temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and place SoC 1304 in a lower power state and / or place vehicle 1300 in a driver safety parking mode (e.g., to safely park vehicle 1300).

[0444] The processor 1310 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio through multiple interfaces and a wide range of flexible audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0445] The processor 1310 may also include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0446] Processor 1310 may also include a safety cluster engine, which includes a dedicated processor subsystem that handles safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores can operate in lockstep mode and act as a single core with comparison logic to detect any differences between their operations.

[0447] Processor 1310 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0448] Processor 1310 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0449] The processor 1310 may include a video image compositer, which may be a processing block (e.g., implemented on a microprocessor) that implements the video post-processing functions required by the video playback application to produce the final image for the player window. The video image compositer may perform lens distortion correction for the wide-angle camera 1370, the surround camera 1374, and / or for the in-cab monitoring camera sensor. The in-cab monitoring camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC and is configured to recognize in-cab events and respond accordingly. The in-cab system may perform lip reading to activate mobile phone service and place calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other circumstances.

[0450] The video image compositer can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, in the presence of motion in the video, the noise reduction appropriately weights spatial information and downweights information provided by neighboring frames. In the case where an image or portion of an image does not include motion, the temporal noise reduction performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0451] The video image compositor can also be configured to perform stereo rectification on the input stereo footage frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1308 does not need to continuously render new surfaces. Even when the GPU 1308 is powered on and active for 3D rendering, the video image compositor can be used to offload the GPU 1308 to improve performance and responsiveness.

[0452] The SoC 1304 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface for receiving video and input from a camera, and / or a video input block that may be used for camera and related pixel input functions. The SoC 1304 may also include an input / output controller that may be controlled by software and may be used to receive I / O signals that are not assigned to a specific role.

[0453] SoC 1304 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. SoC 1304 may be used to process data from cameras (connected via Gigabit multimedia serial links and Ethernet), sensors (e.g., LIDAR sensor 1364, RADAR sensor 1360, etc., which may be connected via Ethernet), data from bus 1302 (e.g., vehicle 1300 speed, steering wheel position, etc.), and data from GNSS sensor 1358 (connected via Ethernet or CAN bus). SoC 1304 may also include dedicated high-performance mass storage controllers, which may include their own DMA engines and which may be used to free up CPU 1306 from routine data management tasks.

[0454] SoC 1304 can be an end-to-end platform with a flexible architecture that spans levels 3-5 of automation, providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to provide a platform for a flexible and reliable driving software stack. SoC 1304 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with CPU 1306, GPU 1308, and data storage 1316, accelerator 1314 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0455] This technology therefore provides capabilities and functionality that cannot be achieved with conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often fail to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, a requirement for in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0456] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined to achieve Level 3-5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1320) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that can recognize, interpret, and provide semantic understanding of the signs, and pass that semantic understanding to a path planning module running on the CPU complex.

[0457] As another example, as required for Level 3, 4, or 5 driving, multiple neural networks can run simultaneously. For example, a warning sign consisting of "Caution: Flashing lights indicate icing conditions" along with a light can be interpreted by several neural networks, either independently or collectively. The sign itself can be identified as a traffic sign by a first neural network deployed (e.g., a trained neural network), and the text "Flashing lights indicate icing conditions" can be interpreted by a second neural network deployed, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icing conditions exist when the flashing lights are detected. The flashing lights can be identified by operating a third neural network deployed over multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can run simultaneously, for example, within the DLA and / or on GPU 1308.

[0458] In some examples, a CNN for facial recognition and owner recognition can use data from a camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1300. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1404 provides security against theft and / or carjacking.

[0459] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 1396 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use general classifiers to detect sirens and manually ex...

Claims

1. A system comprising: Source memory; Destination memory; processor; as well as Direct Memory Access (DMA) system for performing operations including: Retrieving a tile structure from a descriptor memory and a frame structure corresponding to a frame from a HW sequencer command memory, the frame structure including a row description or a column description; ordering retrieval of tiles of the frame from the source memory along rows or columns of the frame based at least in part on the row description or the column description and the tile structure; For each tile in the tileset, and according to the ordering: writing the retrieved data corresponding to the tile to the destination memory; as well as Based at least in part on writing the retrieved data to the destination memory, indicating to the processor that the retrieved data is in the destination memory to cause the processor to process the retrieved data from the destination memory.

2. The system of claim 1, further comprising a processing controller to configure the HW sequencer command memory and the descriptor memory prior to processing of the frame.

3. The system of claim 1 , wherein the operations further comprise: receiving, from the processor and in response to the processor completing processing of the retrieved data, an indication that the processing is complete; as well as The retrieved data for the next tile is written to the destination memory. 4 . The system of claim 1 , wherein before the processor completes processing of the retrieved data, the operations further comprise writing the retrieved data for a next tile to the destination memory.

5. The system of claim 1, wherein the frame structure further comprises a padding control for padding one or more sides of the frame at a frame level. 6 . The system of claim 1 , wherein a frame structure includes a descriptor ID, and the tile structure is determined using a descriptor corresponding to the descriptor ID from the descriptor memory.

7. The system of claim 1, wherein the rows correspond to rows of tiles and the columns correspond to columns of tiles. 8 . The system of claim 1 , wherein the processor is a vector processing unit (VPU) and the destination memory is a vector memory (VMEM).

9. The system of claim 1 , wherein the system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing deep learning operations; System on chip (SoC); A system including a programmable vision accelerator (PVA); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system for merging one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

10. A direct memory access (DMA) system comprising a processing circuit, wherein the processing circuit is configured to: Retrieving a tile structure from a descriptor memory and a frame structure corresponding to the frame from a hardware HW sequencer command memory, the frame structure including a row description or a column description; ordering retrieval of tiles of the frame from a source memory along rows or columns of the frame based at least in part on the row description or the column description and the tile structure; For each tile in the tileset, and according to the ordering: writing the retrieved data corresponding to the tile to the destination memory; as well as Based at least in part on writing the retrieved data to the destination memory, indicating to a processor that the retrieved data is in the destination memory to cause the processor to process the retrieved data from the destination memory.

11. The DMA system of claim 10 , further comprising a processing circuit configured to: receiving, from the processor and in response to the processor completing processing of the retrieved data, an indication that the processing is complete; and The retrieved data for the next tile is written to the destination memory.

12. The DMA system of claim 10, further comprising processing circuitry to write the retrieved data for a next tile to the destination memory before the processor completes processing the retrieved data.

13. The DMA system of claim 10, wherein the frame structure further comprises a padding control for padding one or more sides of the frame at a frame level. 14 . The DMA system of claim 10 , wherein a frame structure includes a descriptor ID, and the tile structure is determined using a descriptor corresponding to the descriptor ID from the descriptor memory.

15. The DMA system of claim 10, wherein the rows correspond to rows of tiles, and the columns correspond to columns of tiles. 16 . The DMA system of claim 10 , wherein the processor is a vector processing unit (VPU), and the destination memory is a vector memory (VMEM).

17. The DMA system of claim 10, wherein the DMA system is included in at least one of the following: control systems for autonomous or semi-autonomous machines; Perception systems for autonomous or semi-autonomous machines; a system for performing simulation operations; Systems for performing deep learning operations; System on chip (SoC); A system including a programmable vision accelerator (PVA); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system for merging one or more virtual machines VM; A system implemented at least in part in a data center; or A system implemented at least in part using cloud computing resources.

18. A method comprising: Retrieving a tile structure from a descriptor memory and a frame structure corresponding to a frame from a hardware HW sequencer command memory, the frame structure including a row description or a column description; ordering retrieval of tiles of the frame from a source memory along rows or columns of the frame based at least in part on the row description or the column description and the tile structure; For each tile in the tileset, and according to the ordering: writing the retrieved data corresponding to the tile to the destination memory; as well as Based at least in part on writing the retrieved data to the destination memory, indicating to a processor that the retrieved data is in the destination memory to cause the processor to process the retrieved data from the destination memory.

19. The method of claim 18, further comprising: receiving, from the processor and in response to the processor completing processing of the retrieved data, an indication that the processing is complete; as well as The retrieved data for the next tile is written to the destination memory.

20. The method of claim 18, further comprising: The retrieved data for the next tile is written to the destination memory before the processor completes processing of the retrieved data.

Images