Performing multi-point table lookups in a single cycle in a system on a chip

Abstract

The present disclosure relates to performing multi-point table lookups in a single cycle in a system on a chip. In various examples, a VPU and associated components can be optimized to improve VPU performance and throughput. For example, the VPU can include a min / max collector, an auto store prediction function, a SIMD data path organization that allows inter-channel sharing, a transpose load / store with a stride parameter function, a load with permute and zero insertion function, a hardware, logic, and memory layout function to allow two-point and two-point by one lookups, and a per-memory bank load cache function. Further, a decoupled accelerator can be used to offload VPU processing tasks to improve throughput and performance, and a hardware sequencer can be included in a DMA system to reduce programming complexity of the VPU and DMA system. The DMA and VPU can perform a VPU configuration mode that allows the VPU and DMA to operate without a processing controller for performing dynamic region based data movement operations.

Description

Background Technology

[0001] Vector processing units (VPUs) are used to execute single-instruction multiple-data (SIMD) operations in parallel. Popular uses of VPUs include operations such as image processing, computer vision, signal processing, and deep learning (e.g., for convolution operations).

[0002] In some computer vision applications, for example, the dynamic range of intermediate values ​​is well understood. Therefore, to detect anomalies, calculated values ​​can be compared to these dynamic ranges. However, conventional solutions for detecting and analyzing these minimum and maximum values ​​involve writing all values ​​to memory and then analyzing the values ​​in memory, which requires additional processing cycles. Furthermore, to achieve high throughput, high clock rate processors can execute software pipelines and / or loop unrolling despite load usage latency. However, if the original iteration count is not uniformly divided by the unrolling factor, some iterations may remain after the unrolled loop is completed, requiring additional remainder loops to compute the value of one or more final iterations. This remainder loop increases the system's code size and latency—for example, because the remainder loop cannot be unrolled for optimal performance. In traditional Single Instruction Multiple Data (SIMD) operations, each SIMD unit can operate in parallel and independently of each other in its own data channel. Some architectures may allow sharing between neighboring units, but this limited sharing is restrictive and often requires the implementation of operations to copy the same operands to each data channel for processing. Furthermore, vector SIMD processors may require each memory read operation to use a standard or consistent unit, e.g., equal to the vector processing width, which can be inefficient when memory banks are wide. For example, reading from elements 4 to 67 in a 64-byte memory bank might require two memory reads—e.g., one from 0 to 63 and one from 64 to 67. However, this results in reading many additional values—e.g., values ​​0-3 and values ​​68-127—even if the current operation does not require these values. In traditional instruction sets that require additional data operations, additional instructions can be used to operate on the memory data in the register after the data has been read and stored in the register. For example, this might require loading the data, performing a permutation on the data, and then performing an operation using the reassembled data. Therefore, data operations require additional cycles and increase latency. When performing table lookups using an existing VPU, the table can be copied so that each individual value can be extracted from the copied table, or additional read ports can be added to each memory bank to allow multiple values ​​to be read from the same table in the same bank. However, copying the table for each value requires additional memory and processing, and adding additional read ports requires additional on-chip space. In a traditional VPU, because the VPU is programmed to execute on a small set of highly optimized code, data caching may not be feasible, as the programmer might manage the contents of local data memory. However, by doing so, each access requires reading a value from each memory set, even if the data for the next iteration includes overlap with one or more previous read operations.

[0003] To optimize processor performance (such as a VPU), the instruction set architecture (ISA) can be enhanced to create custom instructions that accelerate common operations—such as table lookups and convolution operations. However, using the ISA in this way requires the processor itself to perform these operations, meaning the processor is busy executing these enhanced instructions.

[0004] Furthermore, the VPU can use a Direct Memory Access (DMA) system to retrieve data for VPU processing. In this way, the DMA system can operate as a data movement engine, but can also perform additional operations such as image filling, address manipulation, overlapping data management, traversal order management, frame size management, etc. However, as DMA resources (e.g., descriptors, channels, triggers, etc.) increase, the programming complexity of programming the DMA system and the VPU also increases. When the frame tiles contain spatial or temporal dependencies, the dynamic updating of DMA resources becomes a processing burden on the system. When acquiring unknown or data-dependent data, traditional DMA systems require a processing controller (e.g., an R5 or ARM processing core) to intervene in the processing cycle to determine updated information to guide the next processing iteration. For example, in object or feature tracking, the VPU can calculate the next location of the object or feature, then the processing controller will intervene, update the memory addressing information, and then trigger the DMA system to use the updated information. However, the intervention of the processing controller increases latency and requires more complex programming to operate using region-dependent data movement algorithms.

[0005] Furthermore, in safety-critical applications such as autonomous and semi-autonomous machine applications, there are stringent requirements for the detection and isolation of persistent faults. For example, when performing deep learning, computer vision, sensor processing, and / or other applications in a machine, persistent fault detection must be performed periodically within the allocated time budget to ensure accurate testing while also allowing the application to function correctly—for example, with low latency. This may require end-to-end coverage with low latency while meeting the runtime budget of each specific application. Traditional methods use Built-in Self-Test (BIST) to identify faults, but these BIST techniques either lack sufficient coverage, introduce excessive latency into the system, and / or fail to meet the runtime budget of some applications. Summary of the Invention

[0006] Embodiments of this disclosure relate to improvements to a vector processing unit (VPU), a decoupled accelerator for handling offload processing from the VPU, and a direct memory access (DMA) system supporting data movement between memory and the VPU. To address various drawbacks of conventional or existing solutions, the VPU of this disclosure may include a minimum / maximum value hardware collector included in the data path from the VPU to memory, allowing the minimum / maximum value to be stored before being stored in memory. In this way, the minimum / maximum value is available immediately after a memory write operation, reducing the latency of determining the minimum / maximum value after the value is stored in memory. Furthermore, the VPU may include an automatic prediction function that applies a prediction flag by setting a prediction bit for each value computed in iterations following the final iteration. As a result, each set of iterations may include the same number of executed iterations, but one or more values ​​from the final iteration set may not be written to memory due to the prediction flag. To address the limitations of sharing data channels between existing solutions, the SIMD architecture of this disclosure may define slices within the processor, each slice including multiple channels, and each channel may be configured to communicate with each other. In this way, operands from one channel can be used by other channels, eliminating the need to copy each operand to each channel for processing. To address the inefficiency of loading from a single wide memory bank, the VPU can include multiple smaller memory banks to allow for smaller bit alignment—for example, 16-bit alignment, where each memory bank is 16 bits. Thus, examples of reading values ​​4 to 67 might occur in a single memory read, instead of two memory reads for 0-63 and 64-127. In addition to this memory bank organization, the VPU can include transpose loads and / or store functions to allow stored values ​​to be offset within the memory banks, thus preventing memory bank conflicts and allowing more data to be read or written per cycle. To address the data manipulation shortcomings of traditional instruction sets, loads with substitution instructions can be used to send the substitution pattern along with the memory address to local memory for retrieving data from memory based on the substitution or data manipulation pattern. This allows data manipulation and data loading to be performed in the same cycle, reducing latency. To address the drawbacks of table copying for each value or additional read ports for table lookups, two-point or pairwise lookups can be performed, allowing each table to be looked up at two or four points per cycle. To achieve this, offset storage modes for the table and per-memory bank address bus and associated logic and routing can be used to allow parallel lookups of two or four points. In embodiments, each memory bank may include an associated data cache, which can be enabled or disabled depending on a given operation.For example, for filtering operations with significant data overlap between iterations, a data cache can be used to store values ​​from one or more previous lookups, so that each memory group requires only minimal reads, thus conserving energy and power for the system.

[0007] To address the shortcomings of traditional ISAs used with VPUs or other processor types, the systems and methods disclosed herein may utilize decoupled accelerators. These accelerators can be configured by the VPU and communicate with it via shared memory, but can independently execute specific tasks on the VPU, allowing the VPU to continue other processing tasks in parallel with the accelerator. For example, a decoupled lookup table (DLUT) accelerator can be used to improve system performance when executing lookup tables. In this way, the DLUT accelerator can identify and resolve conflicts, increasing system throughput, instead of the VPU performing memory group conflict detection and resolution online.

[0008] To address the drawbacks of conventional DMA systems, the systems and methods disclosed herein may include a hardware sequencer that operates on frame data comprising a sequence of commands for the hardware sequencer. For example, the hardware sequencer may operate at the frame level rather than the tile level and may perform ordering for the DMA engine, eliminating the programming complexity of programming the DMA engine to perform the same operations (e.g., padding, address manipulation, etc.). In some embodiments, the DMA system may include a DMA-triggered mode where the DMA engine controls tile movement to the vector memory (VMEM), rather than requiring the VPU to trigger the DMA to load the next tile. Thus, the command sequence is reversed, and the DMA becomes the trigger for the VPU. To address the drawbacks of region-dependent data movement operations in DMA systems, the DMA system may use the DMA and VPU to operate in a tightly coupled loop without the need for processing controller intervention. For example, the VPU may update position information in the VMEM for various features and / or objects being tracked, and the DMA may use this updated information to update descriptors in the descriptor memory so that the next data provided to the VPU for processing corresponds to the next location of the feature or object. This process can be repeated until processing is complete, thereby eliminating the need for processing controller intervention and reducing system latency.

[0009] Furthermore, to address the shortcomings of conventional BIST methods, this system and method can perform Multi-Input Signature Register (MISR) BIST—for example, performing fault detection on a Programmable Vision Accelerator (PVA) of a System-on-Chip (SoC). For example, in various embodiments of this disclosure, the PVA may include one or more DMA systems and one or more VPUs, controlled by one or more processing controllers (or control processors) (e.g., R5 and ARM processors, CPUs, and / or similar devices). Therefore, each component of the PVA may require testing, and this system and method perform MISR BIST to detect persistent faults in an end-to-end manner. In this way, persistent fault detection can be performed to cover end-to-end blocks of control and data logic, errors can be reported directly to a safety processor to reduce latency, and can be tailored to specific applications to meet associated runtime budgets. Attached Figure Description

[0010] The present system and method for improving the vector processing unit (VPU) are described in detail below with reference to the accompanying drawings, wherein:

[0011] Figure 1A This is an example minimum / maximum value collection system according to some embodiments of this disclosure;

[0012] Figure 1B This is a flowchart illustrating a method for collecting minimum / maximum values ​​according to some embodiments of the present disclosure;

[0013] Figure 2A This is an example system of a processor including an address generation unit with automatic prediction capability, according to some embodiments of the present disclosure;

[0014] Figure 2B It is a table showing a sequence of states changing over time according to some embodiments of the present disclosure;

[0015] Figure 2C This is a flowchart illustrating a method for automatically storing predictions according to some embodiments of the present disclosure;

[0016] Figure 3A This is a diagram illustrating an example Single Instruction Multiple Data (SIMD) data path organization according to some embodiments of this disclosure;

[0017] Figure 3B-3D Operand sharing among slices of a SIMD architecture, respectively used for filtering operations, dot product operations, and sorting operations with payloads, according to some embodiments of the present disclosure, is illustrated.

[0018] Figure 3E This includes flowcharts of methods for calculating outputs using shared operands across channels in a SIMD architecture, according to some embodiments of this disclosure.

[0019] Figure 4A This is a logical view of a transposed load for reading and writing memory according to some embodiments of the present disclosure, and a memory group view corresponding to the transposed load of the logical view;

[0020] Figure 4B It is a logical view of a transposed load having various line spacing and stride parameters for reading and writing memory, according to some embodiments of the present disclosure, and a memory group view of the transposed load corresponding to the logical view;

[0021] Figure 4C This is a flowchart illustrating a method for configuring a transposed load for a write operation using stride parameters according to some embodiments of the present disclosure;

[0022] Figure 4D This is a flowchart illustrating a method for performing a write operation of a transposed load using stride parameters according to some embodiments of the present disclosure;

[0023] Figures 5A-5B Data and coefficient layout tables for different functions in a SIMD architecture according to some embodiments of this disclosure are shown;

[0024] Figure 5C A hardware architecture for performing loads with permutation and zero insertion, according to some embodiments of the present disclosure, is shown;

[0025] Figure 5D Some embodiments according to this disclosure are shown. Figure 5C Examples of hardware architectures used;

[0026] Figure 5E This is a flowchart illustrating a method of displacement loading according to some embodiments of the present disclosure;

[0027] Figure 6A A 16-way parallel table organization for single-point lookup is shown according to some embodiments of the present disclosure;

[0028] Figure 6B An 8-way parallel table organization for two-point lookup is shown according to some embodiments of the present disclosure;

[0029] Figure 6C A logical view of a 2-way parallel word type table for 2x2 point lookup according to some embodiments of the present disclosure is shown;

[0030] Figure 6D Some embodiments of the present disclosure are shown for use with Figure 6C A memory view of a 2x2 point lookup 2-way parallel word type table;

[0031] Figure 6E The layout for processing channel pairs using horizontal mixing with interleaved data operations, according to some embodiments of the present disclosure, is shown.

[0032] Figure 6F Intermediate and final results of horizontal mixing with interleaved data operations according to some embodiments of this disclosure are shown;

[0033] Figure 6G This is a flowchart of a method for performing a multi-point search according to some embodiments of the present disclosure;

[0034] Figure 7A Elements of a data and coefficient array according to some embodiments of this disclosure are shown;

[0035] Figures 7B-7C This illustrates read operations required for data operands and coefficient operands respectively using a data cache for a memory bank, according to some embodiments of this disclosure;

[0036] Figure 7D The present disclosure illustrates a memory group organization used with a load cache according to some embodiments thereof;

[0037] Figure 7E A hardware architecture for using a data cache in a memory bank, according to some embodiments of the present disclosure, is shown;

[0038] Figure 7F This is a flowchart of a method for using a data cache for a memory group according to some embodiments of the present disclosure;

[0039] Figure 8A A system including one or more decoupled accelerators is shown according to some embodiments of the present disclosure;

[0040] Figure 8B This is a flowchart of a method for performing one or more operations using a decoupling accelerator according to some embodiments of the present disclosure;

[0041] Figure 9A A system including a decoupled lookup table accelerator is shown according to some embodiments of the present disclosure;

[0042] Figure 9B This is a table illustrating the actions of different components of a decoupled lookup table accelerator as it performs various operations, according to some embodiments of the present disclosure;

[0043] Figure 9C This is a flowchart of a method for performing one or more operations using a decoupled lookup table accelerator according to some embodiments of the present disclosure;

[0044] Figure 10AThis is a visualization illustrating the filling of a frame with padding values ​​according to some embodiments of the present disclosure;

[0045] Figure 10B This is a visualization illustrating address manipulation of frame descriptors according to some embodiments of the present disclosure;

[0046] Figure 10C This is a visualization illustrating overlapping data between tiles in a frame according to some embodiments of the present disclosure;

[0047] Figure 10D Includes visualizations showing various raster traversal sequences according to some embodiments of the present disclosure;

[0048] Figure 10E This is a visualization illustrating the three-step traversal order according to some embodiments of the present disclosure;

[0049] Figure 10F Includes visualizations showing various vertical mining traversal sequences according to some embodiments of the present disclosure;

[0050] Figure 10G This is a visualization illustrating various image sizes in a pyramid configuration according to some embodiments of the present disclosure;

[0051] Figure 10H This is a direct memory access (DMA) system including a hardware sequencer according to some embodiments of the present disclosure;

[0052] Figure 10I It is according to some embodiments of the present disclosure for storing for Figure 10H The frame format of the sequencing commands controlled by the hardware sequencer of the DMA system;

[0053] Figure 10J This is a raster scanning sequence according to some embodiments of the present disclosure. Figure 10I Examples of frame formats;

[0054] Figure 10K This is an example tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, which uses Figure 10J The example frame format is used for frame address processing;

[0055] Figure 10L This is a flowchart of a method for using a hardware sequencer in a DMA system according to some embodiments of the present disclosure;

[0056] Figure 11A A data flow diagram illustrating the process of configuring a direct memory access (DMA) system using a vector processing unit (VPU) according to some embodiments of the present disclosure is shown.

[0057] Figure 11B This is a table illustrating the VPU configuration format written by the VPU to the vector memory (VMEM) and read by the DMA system according to some embodiments of this disclosure;

[0058] Figure 11C This is a flowchart of a method for configuring a DMA system using a VPU according to some embodiments of this disclosure;

[0059] Figure 12A This is a system diagram of a built-in self-test (BIST) for performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) according to some embodiments of the present disclosure;

[0060] Figure 12B This is a BIST system diagram for parallel channel CRC calculation for PVA according to some embodiments of this disclosure;

[0061] Figure 12C This is a flowchart of a BIST (Bio-Inspection Execution) method for permanent fault detection in a PVA according to some embodiments of the present disclosure;

[0062] Figure 13A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0063] Figure 13B According to some embodiments of this disclosure Figure 13A Examples of camera positions and fields of view for autonomous vehicles;

[0064] Figure 13C According to some embodiments of this disclosure Figure 13A A block diagram of an example system architecture for an example autonomous vehicle;

[0065] Figure 13D It is for one or more cloud-based servers and according to some embodiments of this disclosure. Figure 13A A system diagram illustrating communication between autonomous vehicles;

[0066] Figure 14 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0067] Figure 15 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0068] Systems and methods related to various components of a System-on-a-Chip (SoC) are disclosed—such as vector processing units (VPUs), direct memory access (DMA) controllers, and hardware accelerators (e.g., programmable vision accelerators (PVAs), such as PVAs comprising one or more pairs of VPUs and DMAs). For example, in various embodiments of this disclosure, a PVA may include one or more DMA systems and one or more VPUs, which are controlled using one or more processing controllers (or control processors) (e.g., R5 processors and ARM processors, CPUs, and / or similar devices). Although this disclosure (including various components of the SoC) can be described with respect to an example autonomous vehicle 1300 (also referred to herein as “vehicle 1300” or “self-vehicle 1300”, an example of which is related to see [reference 1300](link 1300)). Figures 13A-13D This is not limiting. For example, the systems and methods described herein can be comprised of, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), driving and non-driving robots or robot-using platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, aircraft, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Furthermore, while this disclosure may be described with respect to computer vision, machine learning, artificial intelligence, image processing, etc., this is not intended to be limiting, and the systems and methods described herein can be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technology space where vector processing units (VPUs), direct memory access (DMA) systems, instruction set architectures (ISAs), programmable vision accelerators (PVA), decoupled accelerators, decoupled lookup tables, hardware sequencers, single-input multiple-data (SIMD) architectures, and / or one or more other components of a SoC. Furthermore, while the components and related processes described herein are described relative to a System-on-a-Chip (SoC), this is not intended to be limiting, and these components can be implemented as standalone components, discrete components of a system, and / or integrated components within a SoC. In some embodiments, the systems, components, features, functions, and / or methods of this disclosure can be integrated into... Figures 13A-13D Example autonomous vehicles 1300 Figure 14 Example computing device 1400 and / or Figure 15 In example data center 1500.

[0069] Minimum / maximum value hardware collector for anomaly detection

[0070] For example, in computer vision applications, especially in safety-critical vision applications, calculating the dynamic range of intermediate results is an important task. For instance, to detect noise or error in intermediate calculations, known or expected dynamic ranges can be used to identify values ​​falling outside these ranges. In such examples, values ​​falling outside the known or expected dynamic range can be labeled as corresponding to noise, error, and / or another problem. Therefore, it may be necessary to collect the minimum and maximum values ​​of intermediate results to detect data anomalies. In practice, these anomalies may be caused by, but are not limited to, noise in image sensors, algorithmic extremes, or data corruption in memory or interconnects. Collecting minimum / maximum values ​​is an effective method for detecting outliers in this data to address these issues. Minimum / maximum values ​​are also used in some algorithms.

[0071] In specific examples, within autonomous vehicle applications, runtime anomalies—such as infinity or non-numeric values—may be invalid values ​​or introduce errors, leading to malfunctions or other undesirable outcomes. With this in mind, algorithms executed as part of the autonomous vehicle platform can be evaluated to determine the range (intermediate or other values) of values ​​that might be generated during processing. Once the range of values ​​is known, the actually calculated values ​​can be compared to the known range, and values ​​exceeding minimum or maximum thresholds can be marked as errors. In cases where errors are marked, changes can be made to the processing—such as ignoring data from a given iteration, identifying and fixing problems, etc. In this way, because potential runtime anomalies are taken into account, runtime anomalies are not allowed and are independent of the autonomous vehicle's reliance.

[0072] As another example, minimum / maximum value collection can be used in some algorithms to normalize intermediate results to a certain numerical range, thereby achieving higher accuracy in processing—for example, in block floating-point. This normalization process may include a dynamic range collection step that gathers the minimum and / or maximum values ​​of an array, and an adjustment step that applies a scaling factor to the array. However, to collect minimum / maximum values, the conventional process requires writing all values ​​to memory, then analyzing these minimum / maximum values ​​and adjusting the scaling.

[0073] Therefore, these traditional methods for minimum / maximum evaluation are executed in software and require additional processing cycles. For example, the algorithm itself can be run to compute values, and then software can be run to determine the minimum / maximum and compare it to values ​​within a known range to identify anomalies. This software needs to execute additional instructions to read elements from an array of intermediate results and then perform the minimum / maximum operation. As a result, the runtime of the system used for anomaly detection increases because the algorithm is executed to completion, followed by additional processes to compute the minimum / maximum value output by the algorithm. This can lead to delays in downstream processing until the minimum / maximum value is computed and compared to a threshold, or it can cause downstream tasks to begin computations on data containing errors while performing minimum / maximum evaluation. This not only increases runtime but also increases the system's processing requirements and energy consumption due to the execution of these additional loops to identify anomalous data.

[0074] refer to Figure 1A , Figure 1A This is an example processor architecture 100 for minimum / maximum value collection according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, architecture 100 may include... Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 Similar components, features, and / or functions to the example data center 1500.

[0075] To address the shortcomings of conventional minimum / maximum evaluation processes such as those described herein, this disclosure includes systems and methods for collecting minimum / maximum values ​​using hardware. For example, during computation, calculated values ​​may be written to memory 106 (e.g., local memory) and used for downstream computation within the same algorithm or another algorithm. To reduce runtime and processing, minimum / maximum value collection hardware (e.g., minimum / maximum value collector 104) can be used to capture minimum / maximum values ​​before or while they are being written to memory 106—for example, instead of waiting for values ​​to be read from memory 106 and then analyzing the values. For example, an enable bit can be used to enable the minimum / maximum value collection function of minimum / maximum value collector 104, and once enabled, minimum / maximum value collector 104 can update the minimum / maximum values ​​as these values ​​are computed using processor 102 and written to memory 106 (e.g., before or concurrently with storage to memory 106). In an embodiment, an enable bit can indicate the type of array being computed—e.g., signed or unsigned—causing the minimum / maximum collector 104 to be configured to collect minimum / maximum values ​​of a specific type of array. For example, the enable bit or another type of control feature can be used to disable the minimum / maximum collector 104 and / or configure it to collect unsigned minimum / maximum values ​​or to collect signed minimum / maximum values. In the data storage path, minimum / maximum collector 104's minimum / maximum collection logic can be included, reading values ​​to update or maintain minimum / maximum values ​​as they are computed using processor 102 and stored in a register file.

[0076] For example, during operation, the current minimum and / or current maximum value can be maintained in the minimum / maximum value collector 104, and the current minimum and / or current maximum value can be updated to a new, lower minimum and / or a new, higher maximum value, which is written to memory 106. If a newly calculated value is greater than the minimum and / or less than the maximum value, the current minimum and / or maximum value can be maintained by the minimum / maximum value collector 104. In this way, the minimum / maximum value collector 104 can maintain the current minimum and / or maximum value because each value is calculated throughout the computation. Once the computation for a given iteration is complete, the minimum / maximum value is immediately available in the minimum / maximum value collector 104, and software and / or hardware can be used to compare these stored values ​​with a minimum and / or maximum threshold associated with a specific algorithm or computation performed to determine if any anomalies exist. For example, a mechanism can be included that allows the collected minimum / maximum values ​​to be read for evaluation. Therefore, compared to previous methods, no additional loop is needed to compute the minimum / maximum value after the algorithm has fully executed, because the minimum / maximum value is immediately available. Furthermore, in the embodiment, the minimum / maximum collector 104 (e.g., including hardware and / or logic) is aware of the storage prediction, such that if a specific data item is prohibited from being stored in memory 106 via, for example, per-channel storage prediction, the minimum / maximum collector may exclude that specific data item. For example, if the address from the address generator includes a storage prediction flag, the calculated value for both storing in memory 106 and updating the minimum / maximum collector 104 can be ignored.

[0077] In some embodiments, the minimum / maximum value collector 104 may be implemented as a feature of a system including an address generator—for example, one or more address generators described in U.S. Nonprovisional Application No. 15 / 141,703, filed April 28, 2016, the entire contents of which are incorporated herein by reference. The address generator may be contained in any type of processor or other processing unit—for example, a vector processing unit (VPU), a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a data processing unit (DPU), and / or another type of processing unit (e.g., regarding…). Figures 13A-13D (as described in 14 and / or 15). In some embodiments, one or more VPUs may be included in a programmable vision accelerator (PVA) and / or as part of a system-on-chip (SoC).

[0078] As a non-limiting example, the input for a particular sensor type or algorithm can be limited to 16-bit units. To determine the dynamic range of that particular sensor and / or algorithm, the operations associated with the algorithm processing the sensor input can be evaluated. In such an example, assuming the first operation is the addition of two 16-bit numbers, the first intermediate result is a 17-bit number. This 17-bit number can then be multiplied by a 5-bit number to produce a 22-bit number. If this is the end of the algorithm, it can be determined that the output is likely to be no more than 22 bits. Similarly, a minimum value can be evaluated. Therefore, during deployment, if the minimum / maximum value exceeds this known range (e.g., 22 bits), the output may be flagged.

[0079] In some embodiments, the data storage path (e.g., between processor 102 and memory 106) may include saturation and / or rounding logic 108 to restrict values ​​stored in memory 106 to a certain upper and lower limit or threshold, or to round them according to certain specific conventions. Therefore, in conventional methods, the evaluation of minimum / maximum values ​​may be performed after saturation and / or rounding. In the presence of anomalies, these conventional methods may fail to detect them because saturation and / or rounding may mask the anomalies—for example, low and / or high values ​​may saturate between the upper and lower limits configured for them by the saturation logic.

[0080] However, for a particular implementation, what is valuable or desirable might be unsaturated, unrounded, or absolute minimum / maximum values—for example, minimum / maximum values ​​other than or alternatively derived from saturated minimum / maximum values. Therefore, the minimum / maximum value collector 104 of this disclosure can collect minimum / maximum values ​​from raw or unsaturated data (e.g., before manipulating values ​​using saturation / rounding logic 108) for anomaly detection. In embodiments, the collection of the average value of the data or the average absolute value of the data can be performed. The average value can be calculated, for example, by summing the elements, reading back the sum from the address generator configuration register, and dividing by multiple stored data items (which the application may know). In this way, minimum / maximum values ​​of absolute values, sums of values, and / or sums of absolute values ​​can be added to the processor storage data path, and the configuration and collection of resulting statistics can be performed—for example, added to the address generator configuration feature set, or managed separately. In some embodiments, the minimum / maximum value collector 104 can collect values ​​before and / or after saturation, rounding, or other calculations using saturation / rounding logic 108.

[0081] Now for reference Figure 1BEach block of method 110 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 110 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 110 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Although regarding… Figure 1A The architecture 100 is described, and the method 110 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0082] Figure 1B This is a flowchart illustrating a method 110 for minimum / maximum value collection according to some embodiments of the present disclosure. In block B102, method 110 includes calculating one or more values. For example, when executing one or more algorithms—e.g., neural networks, computer vision algorithms, filtering algorithms, etc.—one or more values ​​can be calculated using processor 102.

[0083] In block B104, method 110 includes comparing the values ​​of one or more values ​​with currently stored minimum and maximum values. For example, a minimum / maximum value collector 104 may compare each of any number of values ​​to be stored in memory 106 (e.g., values ​​in a register file) with currently stored minimum and maximum values ​​(e.g., currently stored by the hardware minimum / maximum value collector 104). In such an example, the minimum / maximum value collector may compare the values ​​with currently stored minimum and / or maximum values ​​as the values ​​are computed and before or simultaneously with the values ​​being stored in memory. In one or more embodiments, the minimum / maximum value collector may be included along a data path between the hardware unit that computes one or more values ​​and the memory unit that stores one or more values.

[0084] In box B106, method 110 includes determining whether the value is greater than or less than one of the currently stored maximum or minimum values. For example, based on the comparison in box B104, the system (e.g., hardware minimum / maximum value collector 104) can determine whether each value to be stored in memory is greater than or less than the currently stored minimum value.

[0085] In box B108, method 110 includes updating the currently stored minimum value to the value based on the fact that the value is less than the currently stored minimum value. For example, if the computed value to be stored in memory is less than the currently stored minimum value of the hardware minimum / maximum value collector, the hardware minimum / maximum value collector may update the currently stored minimum value to the computed value.

[0086] In box B110, method 110 includes updating the currently stored maximum value to the value based on the value being greater than the currently stored maximum value. For example, if the computed value to be stored in memory is greater than the currently stored maximum value of the hardware minimum / maximum value collector, the hardware minimum / maximum value collector may update the currently stored maximum value to the computed value.

[0087] In this way, the minimum / maximum value can be dynamically updated during the storage of values, so that once some (e.g., all) values ​​are stored, the minimum / maximum value is immediately available by reading it from the values ​​currently stored in the minimum / maximum value collector.

[0088] Automatic Storage Prediction

[0089] In high-clock-rate processors, a popular implementation is to configure the processor as multiple pipeline stages. Therefore, there can be a delay between issuing an instruction to load a register from local memory and the time it takes for the register to become available for another instruction to operate—e.g., load-to-use latency. To achieve high throughput with load-to-use latency, processor compilers and application developers can use software pipelines and / or loop unrolling. For example, software pipelines can be used for the execution of multiple iterations of an overlapping loop, and loop unrolling can be used to expand the loop body by repeating its contents multiple times. In short, these techniques allow multiple iterations of loop contents to be executed concurrently, thereby reducing idle cycles in scheduling (ideally none). When performing loop unrolling, the compiler can divide the loop interaction count by the unrolling factor. For example, the compiler might assume the original iteration count is a multiple of the unrolling factor, so the unrolled loop can execute with equivalent functional behavior. In such an example, if the original iteration count is 60 and the loop is to be unrolled by a factor of 6, the unrolled loop can run 10 iterations. However, if the original loop iteration count is 64, normal integer division will also result in 10 for 64 / 6, so the loop won't execute many times enough (e.g., it might not perform the additional 4 iterations), leading to different code behavior after unrolling, which could cause the application to fail. In some techniques, assertion statements are added to ensure that the iteration count is indeed a multiple of the unroll factor.

[0090] The collection of steps or operations within a loop body can have a narrow range of optimal or desired unroll factors. For example, the lower bound of the unroll factor might be the minimum number of copies of the loop code to be scheduled to fill gaps caused by various delays and achieve optimal performance, while the upper bound might be the maximum number of copies to be scheduled with the limited capacity of the register file—which, for example, could lead to excessive register overflows (saving to and restoring from the stack) and suboptimal scheduling. As another example, allowing iteration counts to be powers of 2 (e.g., 2, 4, 8, etc.) is acceptable for many applications due to the feasibility of choosing combinations of tile width and tile height. However, in an embodiment, the loop body could also be optimally unrolled 6 or 7 times, while unrolling 4 or 8 times might be inefficient. In any case, loop unrolling to achieve optimal scheduling can impose inconvenient limitations on the number of iterations. Therefore, conventional techniques for addressing this issue can lead to performance degradation and increased code size.

[0091] For example, limiting the number of iterations is inconvenient, so a programmer can write two loops—for example, a "multiple" loop and a "remainder" loop—when such a limitation on the number of iterations should not be imposed. As examples, the following illustrative code snippets show: Code 1 – a vector addition loop without loop unrolling; Code 2 – the same loop unrolled to 6, which only works if the iteration count is a multiple of 6; and Code 3 – a double-loop solution that works for any iteration count, but the remainder loop is not unrolled, making it less efficient and also resulting in a larger code size due to the additional loop and iteration count calculations.

[0092] Code 1:

[0093]

[0094] Code 2:

[0095]

[0096] Code 3:

[0097]

[0098] Using the Vector Processing Unit (VPU) of this disclosure, Code 1 can achieve 6 cycles per iteration, Code 2 can achieve 1 cycle per iteration, and the performance of Code 3 can depend on the iteration count. For the iteration count (niter), if niter = 60 (a multiple of 6, so the remainder does not run), Code 3 may achieve 1.0 cycle per iteration, and if niter = 64 (the remainder loops 4 times), Code 3 may achieve an average of 1.3125 cycles per iteration (e.g., (60*1+4*6) / 64 = 84 / 64 = 1.3125).

[0099] refer to Figure 2A , Figure 2A This is an example system 200 according to some embodiments of the present disclosure, including a processor 202 (e.g., a VPU) comprising an address generation unit with automatic prediction capabilities. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the processor 202 may be included in and / or may include components with… Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 The components, features, and / or functions of the example data center 1500 are similar to those of other components, features, and / or functions.

[0100] In embodiments of this disclosure, loads and stores in the code segment may utilize an address generator 204 in the processor 202 (e.g., a VPU). For example, in each load and store, address generator (agen) parameters (agen_a, agen_b, agen_c) may be provided to the load / store function. Arguments may identify an address generator register containing parameters that can be used for address calculations for a particular load and / or store operation—e.g., address pointer, iteration count, current loop variable value, etc. In some embodiments, the VPU may be designed to support six (or other) addressing dimensions per address generator register, thus including six (or other) iteration counts and six (or other) loop variables.

[0101] To address the limitation of loop unrolling on the number of iterations, the systems and methods of this disclosure may include an address generator 204 having logic (e.g., a prediction flag or bit 208) for automatically predicting storage from the address generator 204. For example, the prediction may be used to provide an indication of conditional execution, such as whether (or not) something is performed. The value of the prediction bit 208 (e.g., 0 indicating storage or 1 indicating prevention of storage, or vice versa) may be used to indicate whether an instruction will be executed. Execution may not refer to the actual execution of an iteration, but rather to whether the result of the iteration execution is stored in memory. Therefore, in embodiments, an instruction not executed due to a prediction flag may refer to an instruction or iteration that is being executed, but the result of the execution is prevented or excluded from changing the state of memory 206. Instruction-level prediction and channel-level prediction may be included. Instruction-level prediction may be used to indicate whether an entire instruction should be executed, while channel-level prediction may be used to indicate which data channels should or should not be executed.

[0102] In some embodiments, after the loop variable exhausts the iteration count, any subsequent execution of the store instruction is automatically predicted to prevent further writes to memory 206. In this way, the automatic store prediction feature allows for clear writes by rounding iteration counts that are not multiples of 6 (or another expansion factor) to the next multiple of 6 and by not altering iteration counts that are not multiples of 6 (or another expansion factor). Although a factor of 6 is used, this is not intended to be restrictive, and any expansion factor can be used without departing from the scope of this disclosure. Code 4 below includes an example of vector addition with automatic store prediction.

[0103] Code 4:

[0104]

[0105] Code 4, with an initial iteration count of 64, can run the unroll loop 11 times at 1.03125 cycles per iteration (e.g., 11 x 6 / 64 = 1.03125). Another approach to account for the limitation on iteration counts that are multiples of the unroll factor is to compute the necessary prediction flag within the loop and provide the prediction flag in a stored instruction. For example, Code 5, described below, shows an example implementation of the prediction flag computation.

[0106] Code 5:

[0107]

[0108] Code 5 can be compiled into a VPU of this disclosure with 1.5 loops per iteration, thus auto-prediction can include performance advantages over predictions computed in loops. In an embodiment, the VPU may include a 7-way Very Long Instruction Word (VLIW) instruction scheme, and each cycle may include 2 scalar slots for the scalar operations required for prediction computation. If the loop has more vector operations per iteration, there may be enough scalar slots so that prediction computation can fit into the available slots without causing performance degradation. Even in computational loops where real-time computation of prediction has no impact on performance, auto-prediction may still have advantages in terms of code size and energy consumption.

[0109] Therefore, the software can be used to configure multiple iterations (e.g., N1-N6), and the software can cause the execution of address generator-based load / store operations—typically within a loop. The address generator hardware can maintain loop variables (e.g., variables I1-I6) and can advance the address pointer as appropriate. When the address generator-based load / store has been executed beyond a pre-configured number of iterations, the address pointer may remain at the last valid address, and auto-prediction can be disabled (e.g., by setting a prediction flag) to block subsequent storage into memory. Thus, an internal Boolean state of "auto-prediction off" can be included in the address generator 204, and the loop variable iteration logic can be configured to support disabling auto-prediction. For example, and regarding Figure 2B When the address generator is initialized, the value of the autopred_off parameter (“auto_pred_off”) (e.g., prediction bit 208), except for the loop variables I1-I6, can be initialized or reset to “0”. After the loop variables have exhausted the programmed iteration count, auto_pred_off may be updated to “1”. As a result of the prediction bit being “1”, any subsequent execution of store instructions can then be automatically predicted, and further writes to memory can be prevented.

[0110] exist Figure 2B In the example, the iteration count of the address generator for registers N1-N6 can be programmed as N1=4, N2=2, N3=N4=N5=N6=1. The total programmed iteration count can therefore be 4*2*1*1*1*1=8, and the result can be executed. Figure 2B The sequence is shown. As shown, the initial state and the subsequent 7 executions (e.g., the first 8 iterations) can correspond to the auto_pred_off bit with a value of 0, and the 8th and 9th executions (e.g., the last 2 iterations) can correspond to the auto_pred_off bit with a value of 1, thereby preventing the results of the 9th and 10th executions from being stored in memory.

[0111] In practice, a VPU might be configured to handle a number of vector units working concurrently—e.g., 8, 16, etc.—so the VPU might need an array that is a multiple of the number of vector units. This setup works well if the array is a multiple of the number of vector units. However, often an array might not be a multiple of the number of vector units (e.g., because there's no guarantee that data will be computed based on an array of the same size), so the array is padded so that processing is always performed on batches of the same size. For example, the remaining iterations could be padded with "0" values, but this still requires additional loops in the software to handle the padded values. Therefore, padding can be inefficient because the added data leads to wasted computation and also complicates the software—a common problem in Single Instruction Multiple Data (SIMD) software. Therefore, automatic store prediction can be used to address this issue.

[0112] For a non-limiting example, using 16 batches, as many as 16 batches as possible can be generated from an array, and the remaining values ​​can be included in the final batch. The remaining or spare space within the 16 batches is predicted to be off using a prediction flag. For a concrete example, if an array size is 82, it's possible to generate 5 complete sets (16 per set), and in the last iteration, it might contain 2 remaining elements, while the other 14 elements might be automatically predicted to be off—thus minimizing computational waste from filling batches with 14 values ​​and performing unnecessary calculations on the fill data. As another example, with a vector processing granularity including a width of 32 and an array having 100 elements, 3 complete 32-element vectors can be processed, and the remaining 4 elements can be processed through 4 of the 32 channels (e.g., the prediction flag might be on), while the other 28 channels might be predicted to be off. In this way, programmers may be able to vectorize arrays where the sample size is not a multiple of the number of cells. For example, for each memory, the hardware might actually calculate the number of elements to be written to memory and communicate this information to the memory cell. Therefore, even if mathematical operations for filling or appending elements can be performed and stored, such additional computation and storage are inefficient. Therefore, prediction flags can be set so that no additional reads are needed and the writing of calculated values ​​from the fill values ​​to memory does not occur (e.g., it is blocked or excluded). This automatic prediction can occur at the instruction level, and software can be added to additionally perform lane-level prediction.

[0113] Furthermore, for automatic prediction, additional information may not be needed because the address generator can be programmed for multiple iterations—therefore, the address generator has memory that supports automatic prediction—and software instructions can be added to automatically move between prediction storage and prediction-off storage. In this way, in the final iteration, the hardware can determine when to store the complete result or when to store less than the complete result—for example, due to prediction being turned off or otherwise signaled—which can be done at zero cost while maintaining performance. In the case of using software alone, this process would require additional cycles, thus slowing it down.

[0114] In some embodiments, prediction can be used at the per-channel level, allowing these implementations to handle not only iteration counts that are not multiples of the loop unrolling factor, but also any problem size that is not a multiple of the vector width. In such embodiments, a vector register can be used to drive per-channel prediction, which offers the advantage of providing real-time computation information and provides a shortcut that eliminates the need to copy from the vector register to the scalar prediction register, which is then used to apply prediction flags to each channel. For example, per-channel prediction can be performed from the vector register, which can be beneficial when computing per-channel prediction information in a loop and allows for vectorization of the computation.

[0115] For example, to perform some value replacements in an array—such as replacing any value greater than 100 with 999—the code could be written as follows:

[0116]

[0117] While this code may be functionally correct, it could lead to poor performance. Therefore, the code can be vectorized by incorporating per-lane predictions, as shown in the following code:

[0118]

[0119] When prediction computation is vectorized in this way, and the prediction for each channel can only be transferred through the scalar prediction register, the prediction information needs to be copied from the vector register to the scalar prediction register, which increases the execution time.

[0120] However, instead of performing bit packing and moving the prediction mask from vector channel 0 to the scalar register, per-channel prediction driven directly from the vector register features described in this paper can be used in this example, as shown in the following code:

[0121]

[0122]

[0123] Now for reference Figure 2C Each block of method 220 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 220 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 220 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name just a few. Although regarding… Figure 2A System 200 is described herein, and method 220 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0124] Figure 2C This is a flowchart illustrating a method 220 for automatic memory prediction according to some embodiments of the present disclosure. In block B202, method 220 includes determining the total number of iterations. For example, address generator 204 may determine the total number of iterations to be performed for a given instruction.

[0125] In box B204, method 220 includes dividing the total number of iterations into multiple groups of iterations. For example, address generator 204 can separate the iterations by an expansion factor to generate a loop body that includes multiple iterations of loops.

[0126] In box B206, method 220 includes determining that one set of iterations in a plurality of sets includes a first set of iterations less than a second set of iterations, the second set of iterations corresponding to the other sets of iterations in the plurality of sets of iterations. For example, address generator 204 may determine that, after separating the iterations by an expansion factor, one set of iterations includes fewer iterations than the other sets. For example, with an expansion factor of 6 and an iteration count of 62, there may be 11 sets of iterations—10 sets including 6 iterations and 1 set including 2 iterations. Thus, address generator 204 may determine that 2 iterations in that set of iterations, including the remaining 2 iterations, should be executed and that the other four iterations should be predicted to be closed.

[0127] In box B208, method 220 includes generating a prediction flag corresponding to at least one iteration of the group iterations during the execution of the group iterations. For example, when it is determined that the group iterations do not include a complete group of iterations with the same number of iterations as other group iterations, address generator 204 may enable the prediction flag (by changing the value of prediction off bit 208) to indicate that the results of the extra iterations should be stored or written to memory.

[0128] In box B210, method 220 includes preventing the writing of values ​​corresponding to at least one iteration of the set of iterations to memory, based at least in part on a prediction flag. For example, based on the set prediction flag, the writing of calculated values ​​to memory can be prevented.

[0129] Enhanced SIMD Data Path Organization for Vector Processors

[0130] In traditional Single Instruction Multiple Data (SIMD) architectures, each SIMD processing unit operates in parallel and independently on its own data path. Some machines allow each SIMD processing unit to communicate directly with its nearest neighbors (e.g., left and right neighbors as a linear array of processing units, or a two-dimensional (2D) array, or north, south, east, and west neighbors in processing). However, communication only between adjacent data paths is limited and makes the implementation of operations requiring multiple input operands very costly. For example, convolution is a common operation in image processing, computer vision, machine learning, and other fields. During convolution, various filters can be applied to adjacent pixels; for example, a three-tap one-dimensional (1D) filter involving three data operands and three coefficient operands, for a non-limiting example. If these operands cannot be shared between the data paths of a SIMD architecture, then six operands need to be fed into each data path to produce the result for that particular path. With this in mind, some common approaches implement multiple read ports on a register file, but this requires additional surface area and additional operating power for the SIMD architecture.

[0131] To address the shortcomings of traditional SIMD architectures, the SIMD architecture disclosed herein allows communication between channels by defining slices (e.g., Vector Processing Units (VPUs)) within a processor, which includes multiple channels as groups. For a non-limiting example, the SIMD channel organization within the processor may include a hierarchical organization comprising a 384-bit data path that can be divided, for example, into eight 48-bit (extended word) channels, sixteen 24-bit (extended half-word) channels, or 32 12-bit (extended byte) channels. In such an example, each byte may be extended by 4 bits. The first layer of communication above individual channels may be referred to as a SIMD slice and may be (e.g., but not limited to) 96-bit wide, consisting of two extended word channels (e.g., two 48-bit channels), four extended half-word channels (e.g., four 24-bit channels), or eight extended byte channels (e.g., eight 12-bit channels). In a non-limiting embodiment, the entire processor data path may include four SIMD slices across all four (or other number) SIMD slices and all channels, and the second layer of communication may be global. In this way, operand sharing between channels of each slice can be achieved, which can be useful in instructions such as filtering, dot product, and payload sorting. SIMD architectures can be included in VPUs or other processor types, for example... Figures 13A-13D Example of an autonomous vehicle's 1300 processor, Figure 14 Example computing device 1400 and / or Figure 15 Example data center 1500.

[0132] Due to the physical routing of the SIMD architecture, the SIMD instruction set architecture (ISA) can allow sharing among a certain number (e.g., 8) channels within a slice. For example, as shown in the figure. Figure 3A As shown, communication between 32-bit word data types, 16-bit half-word data types, and 8-bit byte data types is possible within each slice. As a result, in the example, for instance... Figure 3B The filtering operation shown, with four input operands and four coefficients, can perform 8-bit multiplication and accumulation in a half-word, where the coefficients can be shared with data from different channels. In a conventional SIMD architecture, each channel would need to load all eight operands to perform the same computation that can be performed with only three input operands in the SIMD architecture of this disclosure. Therefore, since each read port is associated with increased surface area and power consumption, only three read ports are needed to save space and power for executing such instructions. In operation, due to sharing between channels within a slice, four accumulators (e.g., 0, 1, 2, and 3) may fill the results of the following computations.

[0133] ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3]

[0134] ACC[1]+=D[1]*C[0]+D[2]*C[1]+D[3]*C[2]+D[4]*C[3]

[0135] ACC[2]+=D[2]*C[0]+D[3]*C[1]+D[4]*C[2]+D[5]*C[3]

[0136] ACC[3]+=D[3]*C[0]+D[4]*C[1]+D[5]*C[2]+D[6]*C[3]

[0137] As shown in the figure, for example, ACC[0] can access other channels of src1a, including D[1], D[2], and D[3], and can also access other channels of src2, including C[1], C[2], and C[3]. Similarly, other accumulators (ACC) can access individual channels of src1 and src2. This type of operation is not possible in traditional vector processors that share a limited or minimal amount of data between channels. For example, these computations may include a sliding window approach, where each accumulator includes the result of moving a sliding window relative to the previous accumulator. For example, the first accumulator operates on D[0], D[1], D[2], and D[3], the second accumulator on D[1], D[2], D[3], and D[4], and so on. Each accumulator uses the same coefficients C[0], C[1], C[2], and C[3]. This is possible because of the shared physical routing between channels in a SIMD architecture slice.

[0138] As another example implementation of the SIMD architecture disclosed herein, and regarding the diagram as follows Figure 3C As shown, the dot product in vector multiplication can be performed using channel sharing. In such an example, two indices (e.g., D[0][0]) indicate which channel the data belongs to and which output set the data belongs to. For the dot product calculation, each channel uses only the data operands from its own channel, but the coefficients are shared between channels. Therefore, the output from each channel may use all four coefficients at some point during the dot product operation. During the operation, due to the sharing between channels within a slice, four accumulators (e.g., 0, 1, 2, and 3) may populate the result of the following calculation.

[0139] ACC[0]+=D[0][0]*C[0]+D[1][0]*C[1]+D[2][0]*C[2]+D[3][0]*C[3]

[0140] ACC[1]+=D[0][1]*C[0]+D[1][1]*C[1]+D[2][1]*C[2]+D[3][1]*C[3]

[0141] ACC[2]+=D[0][2]*C[0]+D[1][2]*C[1]+D[2][2]*C[2]+D[3][2]*C[3]

[0142] ACC[3]+=D[0][3]*C[0]+D[1][3]*C[1]+D[2][3]*C[2]+D[3][3]*C[3]

[0143] As another example operation that can benefit from the SIMD architecture of this disclosure, the following can be performed: Figure 3DTwo-point sorting is a two-point sorting operation. For two-point sorting, the payload is sorted using two values. This two-point sorting leverages communication between channel pairs within a slice and is useful in various computer vision applications, such as... For example, channel 0 has the key for entry 0, channel 1 has the corresponding payload, and so on, the payload can be sorted based on a comparison of the keys—for example, each key / payload pair in the following code:

[0144]

[0145] Now for reference Figure 3E Each block of method 300 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 300 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 300 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name just a few. Although a SIMD architecture has been described with respect to this disclosure, method 300 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0146] Figure 3E This includes a flowchart of method 300 for computing outputs using shared operands across channels in a SIMD architecture according to some embodiments of the present disclosure. In block B302, method 300 includes dividing the bit width of a processor into a plurality of data slices, each data slice including a second bit width less than the first bit width, each data slice including a plurality of channels, each channel including a third bit width less than the second bit width. For example, a vector processor may be divided into a number (e.g., 4) slices, and each slice may include a number of channels.

[0147] In box B304, method 300 includes loading a first vector into a first vector register, such that a first channel of a plurality of channels includes a first operand of the first vector, and a second channel of the plurality of channels includes a second operand of the first vector. For example, regarding Figure 3B The first data operand D[0] of the first vector can be loaded into the first channel, and the second data operand D[1] corresponding to the first vector can be loaded into the second channel.

[0148] In box B306, method 300 includes loading a second vector into a second vector register such that a first channel of a plurality of channels includes a third operand of the second vector, and a second channel of a plurality of channels includes a fourth operand of the second vector. For example, regarding Figure 3BThe first coefficient operand C[0] of the third vector can be loaded into the first channel, and the second coefficient operand C[1] corresponding to the third vector can be loaded into the second channel.

[0149] In box B308, method 300 includes using instructions to compute an output based at least in part on a first operand, a second operand, a third operand, and a fourth operand. For example, regarding Figure 3B The first accumulator (ACC[0]) can receive the result of calculating ACC[0] + = D[0] * C[0] + D[1] * C[1] + D[2] * C[2] + D[3] * C[3], including the values ​​of D[0], D[1], C[0], C[1], etc. This calculation may occur due to internal sharing and routing between channels of each slice.

[0150] In box B310, method 300 includes storing the output into a register. For example, regarding Figure 3B The calculated output can be stored in the accumulator register ACC[0], and then stored in memory.

[0151] Transposed load and storage operation with stride parameters

[0152] In traditional vector single-instruction multiple-data (SIMD) processors, the size of the local data memory can be adjusted to match the vector processing width. For example, for a 256-bit vector SIMD processor capable of handling 32 8-bit channels, 16 16-bit channels, or 8 32-bit channels, the local data memory could include a 256-bit wide memory or a 512-bit wide memory (e.g., twice the processing bit width). In such an example, the local data memory is organized as a single memory bank with full-width memory words. However, a wide vector SIMD processor with a single full-width memory bank can be inefficient—especially for unaligned memory accesses. For example, to load an array of 16-element 32-bit arrays at byte addresses 4 to 67, the processor might need two memory reads—for example, one read of addresses 0 to 63 (including addresses 0 to 3, which contain data not needed by the current operation) and a second read of addresses 64 to 127 (including addresses 68 to 127, which contain data not needed by the current operation). Therefore, without the grouped memory architecture disclosed herein, access patterns can be implemented through multiple loads or stores, which may result in slower compute kernels, reduced performance, and increased power consumption.

[0153] With this in mind, a single wide memory bank can be reorganized into multiple memory banks—for example, 16-bit memory banks (e.g., 32 16-bit memory banks, providing 512 bits of memory bandwidth per clock cycle). In this way, read and / or write operations can be performed within any 16-bit alignment range—thus reducing the number of redundant read / write operations, such as those described in the examples above. Using such a memory organization, reading addresses 4 through 67 might require only a single memory read. In addition to memory bank organization comprising smaller, individual memory banks, transpose load and / or storage functions can also be implemented. For example, the channel offset parameter K can be used to define the row address offset applied to each subsequent channel in the memory. The channel size might correspond to the data element size—e.g., 8-bit, 16-bit, 32-bit, etc. When a 2D array is stored in memory with a row spacing of W*K+1 elements, the interleaved access mode might be converted to a vertical mode, where K is the offset parameter and W is 64 / channel size (or data element size). For example, for 32-bit data elements, the row spacing could be 16*K+1. In some embodiments, a SIMD processor can be included as a component, and / or can be included with... Figures 13A-13D Example autonomous vehicles 1300 Figure 14 Example computing device 1400 and / or Figure 15 Similar components, features, and / or functions to the example data center 1500.

[0154] As an example, and regarding the figure as follows Figure 4A As shown, Table 400 may include a logical view of the transposed loads and an illustration of a memory group view with 17 row spacings exceeding 256 bits. Memory bars in the memory group view end with 18 separate 16-bit bars; this is for illustrative purposes only. For example, a memory group might be 256 bits in total, 512 bits in total, or some other total bit width—for example, each memory group might be 16 bits wide. In the memory group view using transposed loads, with a row spacing of 17, a single load operation can be performed to retrieve each highlighted value of the array.

[0155] While transposing the payload using this technique is beneficial for many operations, some algorithms—such as some computer vision algorithms—may require access even when using information about... Figure 4AThe described transpose load technique also cannot handle data patterns that can be accomplished with a single read and / or write. For example, it might be necessary to load a submatrix 8 high by 2 element-width, a matrix 4 high by 4 element-width, or other matrix or submatrix sizes, rather than a 16-high vertical vector. For example, in a dot product operation, the accumulation might be against two rows of 16 elements, 16 bits each time, so when storing the output, a T16 transpose store option with appropriate row spacing might be needed so that the two rows can be written out as a single memory write transaction. To address this, a stride parameter can be used with the transpose load and / or store. In some embodiments, the stride parameter can include a power stride of 2 (though this is not limiting), such as strides of 2, 4, 8, 32, etc., which can be referred to as T2, T4, T8, T32, etc. Examples of different transpose loads with stride parameters are provided in [the document / section]. Figure 4B Table 410 shows a logical view of the transposed load and a memory group view. Figure 4A Example, mirrored Figure 4B In this context, a stride parameter of 1 is included, but other stride parameters are multiples of 2. For example, T2 has a row spacing of 18, allowing a matrix of 2 elements wide by 8 high to be stored as a transposed load, so that each value can be retrieved using a single load transaction. Similarly, for T4, with a row spacing of 20 and a stride of 4, a matrix of 4 elements wide by 4 high can be stored, allowing each value to be retrieved using a single load transaction, and so on. Although described as a load transaction, this type of format can also be used for storage transactions, storing data in memory based on the transpose plus the stride parameter.

[0156] In such an example, the line spacing constraint can be adjusted based on the stride. For font T-transpose access, the line spacing could be 16K+1; for font T2-transpose access (e.g., for a stride of 2), the line spacing could be 16K+2; for font T4-transpose access (e.g., for a stride of 4), the line spacing could be 16K+4, and so on. Therefore, the line spacing could be equal to 16K + stride value, or 16K+1 + (T-1), where T is the stride parameter.

[0157] In operation, the VPU's VMEM architecture and the VPU's instruction set architecture (ISA) can be configured to perform transposed load and / or store operations, with or without a step parameter, to allow reading or writing data organized column-wise in a logical view within a single read operation. For example, the ISA can be configured to receive a starting address indicating whether data is to be read from or written to it (e.g., for reading or writing data from a register file), an indication of the write type (e.g., transposed write operation, with or without a step parameter), a row spacing value (e.g., the value of K in 16*K+1), and / or a step parameter value. It is important to note that the value of 16 corresponds to the number of data elements in a particular implementation, but the value of 16 (or W) may differ in different embodiments. Therefore, when writing data to memory according to a transposed write operation, the ISA can receive the starting address, row spacing, and / or step parameter to be written to the VMEM. As a result, instead of writing them out to a single memory bank in a single data column, values ​​can be written according to, for example... Figure 4A and 4B The transpose or offset shown is used to write data. When using a stride parameter, the first value of the stride can be written to memory, followed by the next number of elements corresponding to the stride, and then the line spacing can be applied to write the next set of values ​​to the memory bank, so that each value can be written to memory in one cycle. Similarly, during a read operation, depending on the transposed memory organization or the data being written, the ISA can receive the starting address, load type (e.g., transposed load, with or without a stride parameter), line spacing value (e.g., the value of K), and stride parameter value (e.g., data type indicator, such as byte, half-word, etc.). The ISA can then access data from the individual memory banks according to the transposed load instructions (and / or stride parameters) to retrieve one (or more) columns of data in a single read cycle. In this way, a single vector can be returned from a single read operation by retrieving one element from each memory bank.

[0158] Now for reference Figures 4C-4D Each block of methods 420 and 430 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Methods 420 and 430 may also be embodied as computer-usable instructions stored on a computer storage medium. Methods 420 and 430 may be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Although a SIMD architecture has been described with respect to this disclosure, methods 420 and 430 can be performed by any system or any combination of systems, including but not limited to those described herein.

[0159] Figure 4CThis includes a flowchart of a method 420 for configuring a transpose storage operation using stride parameters according to some embodiments of the present disclosure. In block B402, method 420 includes determining the dimensions of a matrix. For example, the width of the matrix may be determined.

[0160] In box B404, method 420 includes determining the stride parameter and row spacing for storing the matrix based on the dimension. For example, the row spacing can be determined using a stride value of 16K+, and the stride value can be determined based on the width of the matrix.

[0161] In box B406, method 420 includes using a stride parameter and row spacing such that the values ​​of the matrix are stored in memory. For example, once the row spacing and stride are determined, the values ​​of the matrix can be stored in memory such that the row spacing and stride parameter values ​​do not cause memory group conflicts when reading matrix values ​​from memory.

[0162] Now for reference Figure 4D , Figure 4D This includes a flowchart of a method 430 for configuring transpose memory operations using stride parameters according to some embodiments of the present disclosure. At block B408, method 430 includes receiving data representing row spacing and a starting memory address in a plurality of memory groups, the starting memory address corresponding to an element in a plurality of elements, the plurality of elements corresponding to columns in an array.

[0163] In box B410, method 430 includes reading multiple elements from multiple memory groups in a single read operation, at least in part based on row spacing, each of the multiple elements from the respective memory groups.

[0164] Loading with substitution and zero-insertion in a single instruction

[0165] In traditional processor instruction sets, load instructions can calculate a memory address using some indices, read the requested memory data from local memory, and store the memory data in a register. If the application requires additional data manipulation, additional instructions can be used to operate on the memory data in the register. In some cases, data manipulation may involve simple data rearrangement. In traditional processors, even such simple data manipulation in the register file requires additional instructions, thus incurring additional latency. For example, a traditional system might load data, perform substitution on the loaded data, and then use the rearranged data to perform one or more operations. If this data rearrangement capability is used to enhance the load instruction, some processing time can be saved, and computational cores can be executed with higher performance and lower power consumption.

[0166] To address these drawbacks, the systems and methods of this disclosure add a load with a permutation instruction that sends the permutation pattern along with a memory address to local memory. As a result, existing data routing and multiplexing for handling unaligned loads can be used to perform the permutation without a significant amount of additional logic. In addition to saving instructions that would otherwise be costly (e.g., a permutation using two vector inputs and two vector outputs would require five instructions), the overall latency of the permutation operation can be reduced. For example, there is no load-to-use latency and computation latency (e.g., for performing the permutation); the only latency is the load-to-use latency. In some embodiments, the load with permutation and / or zero-insertion described herein can be included in… Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 The components, features, and / or functions of the example data center 1500, or those similar to these components, features, and / or functions.

[0167] Therefore, loading with permutation features can be used to manipulate loaded data from memory into a desired format for operation. As an example, the coefficient data required for various filtering and dot product instructions can include specific repetition patterns, which can be implemented through loading and permutation. Regarding filtering operations, for example, regarding... Figure 3C As described, the coefficients 0, 1, 2, and 3 can be repeated over the vector width (e.g., 16 bits) – for example, as... Figure 5A As shown. In such an example, the write to the first register can start with D[0]-D

[15] , and then a sliding window of 4 can be used to start the next register with D[0]-D

[19] , and so on. In this filtered example, the coefficients C[0]-C[3] may repeat over the width of the vector, so using a permutation load may help to write the coefficients directly from the load in this order, rather than loading all the data, performing the permutation, and then writing the vector to the register. Therefore, in this example, the permutation pattern of the coefficient data could include {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3). In the same example, the permutation pattern of the data operands could be {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}. In this way, data operands and coefficient operands can be read according to the permutation order, rather than being read sequentially and then permuted before being written to the register for computation. As another example, for instance... Figure 5BAs shown, the filtering instruction may include two-vector coefficient operands, and therefore may include permutation patterns such as {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3,4,5,6,7,4,5,6,7,4,5,6,7,4,5,6,7,4,5,6,7}. The permutation pattern may be static or fixed, or it may be dynamically computed by an algorithm, allowing the permutation pattern to be flexible and dynamic. In the case of a repeating pattern, in an embodiment, the first instance of the repeating element may be loaded, then copied, and then written to the SIMD channel of the SIMD unit.

[0168] In some cases, it may be preferable to mask certain portions of the stored data with zero values. For example, to make visualization easier or consume less energy in software development (e.g., compared to retaining random data values), zeros can be inserted for unused entries. In other examples, zeros can be inserted to delineate blocks of data in a data structure, such as when the length of each data block is not fixed. In such examples, zero values ​​can indicate gaps between two data blocks. When processing a constant-size image block, for example, when extracting some variable-length information (e.g., the location of feature points) from each image block, zeros can be used to fill in any remaining data that does not correspond to the extracted information.

[0169] In practice, permutation indices can typically include 32 or 16 elements in the read—for example, in the ranges of 0-31 or 0-15, respectively. To include a zero value in the read, a permutation operation can be used to include a negative index value in the load so that a zero is written to the corresponding channel in the destination register. Therefore, during a write operation, for example, a negative value can be written to the corresponding channel in a SIMD architecture instead of zero.

[0170] As an example, a 30-by-30 image patch can be processed by a vector operation using 16 consecutive entries at a time. Since the width of 30 is not divisible by 16, each row can be processed by two vector operations: the first processing of the full vector width of 16 entries, and the second processing of the partial vector width of 14 entries. In such an example, it might be advantageous to use zero-padding for the loading of the second 14-entry vector to fill the last two vector channels, rather than using random data values ​​that might currently exist in memory.

[0171] In one or more embodiments, padding zeros can be inserted into the desired channel locations in the SIMD architecture, for example, to save the processing time required to write zeros to these channel locations. In the case of 16 channels, a normal permutation pattern might consist of 16 channel indices—for example, 0-15. In such an example, if the values ​​{100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115} exist, and the indexes are arranged in the pattern {0,1,2,3,4,5,6,7,8,9,10,11,12,13,-1,-1}, the final value loaded into the destination register should be {100,101,102,103,104,105,106,107,108,109,110,111,112,113,0,0}. Therefore, the two values ​​of -1 are automatically converted to 0 in the destination register based on the permutation pattern that includes negative values. In the previous method, -1 and -1 would respectively include 14 and 15, and the values ​​at memory locations 14 and 15 would be written to the register. However, these may include random values ​​that may require additional processing time compared to including a value of 0.

[0172] To implement loads with permutation characteristics, routing and multiplexing in memory logic can be used—for example, similar routing and logic used to perform unaligned memory loads. For instance, to support loading the full memory width (e.g., 32x16 bits) from any 16-bit address (or loading a 16x32-bit channel from any 32-bit address), the memory logic can include multiplexing logic to select any one of the 32 channels in the memory data to route to any destination register channel. For example, for unaligned memory loads, this can be driven by the following logic:

[0173] output_lane[0]=select(start_lane,memory_lane[0..31]);

[0174] output_lane[1]=select((start_lane+1)%32,memory_lane[0..31]);

[0175] output_lane[2]=select((start_lane+2)%32,memory_lane[0..31]);

[0176] …

[0177] output_lane

[31] =select((start_lane+31)%32,memory_lane[0..31]).

[0178] In an embodiment, a modulo operator (%) can be used to wrap around the total number of wrapped lanes. Therefore, for example, if the starting channel is channel 3, channels 3, 4, 5, ..., 31, 0, 1, 2 will be used as outputs to the register channel.

[0179] For loads with permutation features, this same logic can be largely reused, but it can include modified logic to perform the permutation operation. An example of the modified logic is as follows:

[0180] output_lane[0]=select((start_lane+permute[0])%32,memory_lane[0..31]);

[0181] output_lane[1]=select((start_lane+permute[1])%32,memory_lane[0..31]);

[0182] output_lane[2]=select((start_lane+permute[2])%32,memory_lane[0..31]);

[0183] …

[0184] output_lane

[31] =select((start_lane+permute

[31] )%32,memory_lane[0..31])

[0185] As an example, and regarding the figure as follows Figure 5CAs shown, a hardware structure 500 with a crossbar switch 510 (e.g., a crossbar switch) (which may be included as part of a VPU, SIMD unit, SoC, or other device type) can be used to retrieve data from any location in memory 512 and drive the data to any channel in the SIMD via corresponding multiplexers (muxes) 514A-514N. In this way, any of the 16 inputs (or other wide memory or registers) may be able to be written to any of the 16 output locations or channels. This can facilitate unaligned access, allowing load operations to start from any address and then be aligned downwards. For example, if data is read from memory at locations 2-18, data can be read from 2-18 but aligned with channels 0-16 (e.g., 2 goes into channel 0, 3 into channel 1, etc.). This is not possible in conventional systems where vector loads need to start from multiples of 16, such as 0, 16, 32, etc. Figure 5C As shown, since data from any memory index can be output to any channel in the SIMD unit, such as the VPU, permutation can also be performed. Multiplexer 518 can be used to inject or insert permutation control for each channel to inform multiplexer 514 of cross switch 510 from which memory location to read data based on the starting position (which can be aligned or unaligned) and the permutation pattern. Therefore, instead of simply extracting data from an aligned position, the permutation pattern can be used to update the memory read position, such that each multiplexer 514 sends the correct data to each channel of the SIMD unit. Furthermore, multiplexer 516 can be used to insert zeros for permutation patterns that include negative values ​​or other values ​​indicating zero insertion (e.g., in cases where values ​​other than negative values ​​are used to cause zero insertion). Thus, once the memory access position is sent from multiplexer 518 to cross switch 510, and the value from the memory access is sent to multiplexer 516 for zero insertion, the value corresponding to a negative value in the permutation pattern can be converted to a zero value to fill the value of the corresponding SIMD channel. Although in Figure 5C Only four groups of channels, multiplexers, and memory indexes are shown in this disclosure; this is not limiting and any number of groups may be included without departing from the scope of this disclosure.

[0186] Figure 5D The illustration shows an example use of the 500 hardware architecture. For example, Figure 5D The illustration in the image. 5D may be based on the following information:

[0187] crossbar_mode = 1;

[0188] start_lane = 2;

[0189] permute pattern={3,1,-1,…,2}={011b,001b,111b,…,010b};

[0190] mem read bus={100,101,102,…,103}

[0191] permute_low = {3, 1, 3, ..., 2}; / / Lower 2-position swap

[0192] permute_sign = {0, 0, 1, ..., 0}; / / Permuted bit 3

[0193] read data output={103,101,0,…,102}

[0194] Additionally, the following C code can describe Figure 5C and 5D The logic circuit of the hardware architecture:

[0195]

[0196] Thus, in Figure 5D In the example, bit 1 in multiplexer 518 can indicate that a load permutation value should be selected, and these values ​​{3,1,3,...,2} can be transmitted to the corresponding multiplexer 514 of crossbar switch 510. Thus, the value {103,101,103,...,102} can be read from memory and sent to multiplexer 516, where the permutation pattern can include a third value of -1, so the value of 103 can be converted to 0 by zero insertion. Therefore, the final value of {103,101,0,...,102} can be read back to the vector register.

[0197] Now refer to the diagram as follows Figure 5E As shown, each block of the method 550 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 550 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 550 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name just a few. Furthermore, by way of example, regarding... Figure 5C The hardware architecture described herein is for method 550. However, method 550 may additionally or alternatively be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0198] Figure 5EThis is a flowchart illustrating a method 550 for performing a loading operation using a substitution operation according to some embodiments of the present disclosure. In block B502, method 550 includes determining a substitution pattern for loading data from memory. For example, the substitution pattern can be static or dynamically calculated. The substitution pattern can be aligned (e.g., 0 to 16, or 0 to 32), unaligned (e.g., 2 to 18), repeated (e.g., 0, 1, 2, 3, 0, 1, 2, 3, ..., etc.) and / or other pattern types.

[0199] In block B504, method 550 includes determining the memory address location of each of the multiple channels based at least in part on a permutation pattern. For example, the permutation pattern may indicate from which memory address location data for a particular channel or register should be loaded. The permutation pattern may be implemented using multiplexer 518, so that the correct memory address according to the permutation pattern is sent to cross switch 512.

[0200] In block B506, method 550 includes loading values ​​into each of a plurality of channels, at least in part based on memory address locations. For example, based on memory address locations, multiplexer 514 from crossbar switch 512 can retrieve corresponding values ​​from memory to write to one or more channels within one or more vector registers. In some embodiments, multiplexer 516 can also be used to convert values ​​associated with negative values ​​in a permutation pattern (or other values ​​indicating zero-filling) to zero. Thus, in cases where the permutation pattern includes one or more negative values, values ​​loaded from memory can be converted to zero before being written to the vector registers.

[0201] In box B508, method 550 includes using these values ​​and at least one instruction to perform one or more operations within each of the multiple channels. For example, once a processing channel of a vector register or SIMD unit is filled, one or more operations—such as arithmetic instructions, logical instructions, shift / loop instructions, bit manipulation instructions, comparison instructions, conversion instructions, constant generation instructions, and / or similar—can be performed using one or more processing units corresponding to one or more processing channels.

[0202] Multi-point lookup for performing table lookups

[0203] In traditional processors with vector SIMD computation, local memory can include a bit width matching the bit width of the vector SIMD. As a result, these processors may typically only support read and / or write alignment and granularity corresponding to the bit width. However, table lookups are a common technique in embedded environments such as digital signal processing (DSP) and computer vision for implementing various nonlinear functions. For example, square root, logarithmic, sine, and cosine functions may require table lookups. To perform these functions, the input space can be uniformly sampled in a one-dimensional (1D) grid, and the output can be recorded at these input points in a one-dimensional table. However, when implementing nonlinear functions using table lookups, a trade-off is often required between table size (e.g., the number of entries in the table) and accuracy. To improve accuracy without requiring a large table size, interpolation lookups can be performed, where two points are searched around a fractional index for linear interpolation, or three points are searched around a fractional index for quadratic interpolation.

[0204] As an example, where a lookup table is used to implement the sine function, and the sine values ​​are tabulated in integer degrees, then table[0] = sin(0 degrees), table[1] = sin(1 degree), table[2] = sin(2 degrees), and so on. In such an example, if the evaluation is sin(1.7 degrees), then the score of table[1] * 0.3 + table[2] * 0.7 can be used to linearly interpolate between the two integer degree entries. In this example, the second entry of table[2] gets the score as the weight, and the first entry gets 1 minus the score, so the closer the score is to 1.0 or the position corresponding to the second entry, the higher the weight of the second entry.

[0205] As another example, an image or a patch of an image can be resampled. This could involve finding available pixels around a certain fractional pixel coordinate and then performing an interpolation lookup. In such an example, the table could include the image patch and could be two-dimensional. In this case, bilinear interpolation can be performed to interpolate in two dimensions, each of which is linear. For example, a patch at position Y = 5.1, X = 7.6 could be interpolated based on the following calculation:

[0206] (patch[5][7]*0.4+patch[5][8]*0.6)*0.9+(patch[6][7]*0.4+patch[6][8]*0.6)*0.1 However, performing this type of interpolation lookup in a traditional processor is expensive because a separate lookup needs to be performed for each value in each table. To speed up this process, the table can be copied to allow any number of lookups to be performed simultaneously using different instances of the table. For example, in the example above, when looking up patches at 5, 6, 7, and 8, the table may be copied at least 4 times to allow for parallel lookups in four tables. For example, in a processor (e.g., a VPU) that supports 32-way parallelism, the table may be copied 32 times. However, while copying the table may increase throughput per cycle, copying also requires additional memory capacity and usage, which may be unavailable or not optimal in some implementations.

[0207] With this in mind, the systems and methods described in this paper use two-point and / or 2x2-point lookup operations to increase throughput (or match, for example, the throughput of 32-way parallel lookup) while saving memory space. For example, using a per-memory-group address bus and associated logic and routing, two-point or 2x2-point (e.g., 4-point) parallel lookups can be performed with less memory usage. Thus, a single lookup of a table might result in two points in a two-point lookup or four points in a 2x2-point lookup. This can be accomplished based on hardware settings—e.g., group address, logic, routing, etc.—and the storage pattern in memory, allowing multiple reads of data without memory group conflicts. As mentioned above, without these features, implementing, for example, a 32-way parallel lookup would require copying the table 32 times. For example, this 32-way parallel lookup can be performed using the following C code:

[0208]

[0209] In this example, the lookup portion of the loop can perform 32 lookups per cycle, lasting two cycles (lookups and blending are performed in memory and vector math slots respectively, and pipelined to two cycles per iteration), and interpolate to produce 32 outputs. Therefore, the entire lookup / interpolation is 16 outputs per cycle and requires 32 copies of the table.

[0210] As a further example, and regarding the diagram reference Figure 6AThe diagram illustrates a 16-way parallel list organization for performing a single-point lookup with an index vector {0,1,2,3,4,5,4,3,...}. In such an example, using conventional architectures and memory layout techniques, the first and second lookups would need to be performed sequentially to read two entries from each memory bank. For example, the first memory bank T0 contains the values ​​at T0[0] and T0[1] to be read in the lookup operation, but because these values ​​are in the same memory bank, T0 (which may only contain a single read port) has the first value T0[0] read in the first pass and the second value T0[1] read in the second sequential pass. With such a memory layout, memory bank conflicts occur if two reads occur in the same memory bank, which can lead to processing delays and / or cause algorithms or other computations to fail to execute correctly.

[0211] However, using the architecture disclosed herein, the same 32 lookups may require only 16 table copies for a two-point lookup or only 8 for a 2x2 lookup. For example, for a two-point lookup, the same performance of 16 outputs per clock cycle can be achieved with 16 table copies, thus reducing memory footprint by a factor of 5. A 16-way parallel variant of the instruction can return a dual vector, with the first entry in the lower single vector and the second entry in the higher single vector. In C code, this 16-way parallel lookup and interpolation can be represented as follows:

[0212]

[0213] In such an example, the lookup and interpolation portions of the loop might only require a single clock cycle (lookup and interpolation are performed in memory and vector math slots respectively, and the pipeline iterates through one cycle), and interpolation produces 16 outputs. Therefore, the lookup / interpolation is 16 outputs per cycle. This is just one example, and regarding... Figure 6B This describes an 8-way parallel list organization for performing a two-point lookup with an index vector {0,1,2,3,4,5,4,3,...}. In such an example, because each memory bank T0, T1, T2, etc., contains only a single value to be read during the lookup operation, all 16 values ​​can be read in a single pass, instead of... Figure 6A In the example, Figure 6A In this scenario, due to potential memory group conflicts, only eight values ​​can be read in each of the two passes. Therefore, in this embodiment, the instructions for the lookup may include a single index and a pattern, which includes not only the search index but also the search index plus a position. Thus, the instructions may result in reading two values ​​for a two-point lookup, and the values ​​can be written to the lookup table in this format to allow the single read to be performed without memory group conflicts.

[0214] As an example, when performing vector operations, each channel of the VPU can process a set of pixel values ​​retrieved from memory. In some cases, a channel can process multiple values ​​from the same memory group, which can lead to memory group conflicts because the memory group may only include a single read port. Therefore, the methods and systems of this disclosure distribute values ​​across memory groups so that memory group conflicts do not occur, and each value for a single processing channel of the VPU can access each corresponding value read cycle in that single processing channel.

[0215] In a traditional system performing a 2D bilinear interpolation lookup, each output requires four lookups (e.g., 2x2), allowing for optimal throughput of 8 outputs and 32 table copies per clock cycle. Using a 2x2 point lookup, 8 outputs and 8 table copies per cycle (compared to 32) can be achieved, reducing the memory footprint required for parallel sub-tables by a factor of four. For example, for a 2x2 point lookup, two entries can be read from one row of the 2D table, and then two more from the next row. To avoid memory group conflicts in any memory banks, the row spacing in the 2D table can be limited to m*k+2, where m is the number of entries stored horizontally in each sub-table, and k is a row of any integer table sufficient to store the data. For an 8-way parallel 16-bit table, m = 32 (16-bit memory words) / 8 (parallelism) = 4. For a 2-way parallel 32-bit table, m = 16 (32-bit memory words) / 2 (parallelism) = 8.

[0216] As an example, and regarding Figures 6C-6D Line spacing constraints can be used to avoid memory contention. In such an example, a 2-way parallel word table is illustrated for 2x2 point lookup, with a line spacing of 10. The sub-table (m) contains 8 consecutive elements, where A[0][0…7] are placed consecutively in the sub-table, conforming to the formula 8k+2, where k can be any integer. Therefore, regardless of which index value is used as the starting point, the 2x2 point to be retrieved can be placed in different groups, which is mathematically guaranteed. For example, the group numbering of the 2x2 points relative to the sub-table is summarized as follows:

[0217] index%8,

[0218] (index+1)%8,

[0219] (index+line_pitch)%8=(index+8k+2)%8=(index+2)%8,

[0220] (index + line_pitch + 1) % 8 = (index + 8k + 2 + 1) % 8 = (index + 3) % 8. Typically, there are 4 entries to retrieve using a 2x2 lookup in the group number, relative to the subtable: index % m, (index + 1) % m, (index + 2) % m, (index + 3) % m. As long as m >= 4, there should be no memory group conflicts. Figures 6C-6D In the example, a lookup could include 2D indices such as (0,1) and (1,3), using Y then X as a convention for storing pixels in row-major order. Figure 6C The diagram shows a logical view of two two-dimensional tables, and... Figure 6D The diagram shows a memory layout view of the values ​​from the table. In the logical view, lookups are 2x2, as shown, and the memory layout view displays four points, each located in a different memory group (or a different column in the diagram), so that each of these values ​​can be read in a single memory cycle or pass. Based on instructions and read patterns using indexes (e.g., (0,1) and (1,3)), the values ​​in the table can be stored in memory in this way so that each value can be read from memory in a single pass. Therefore, using this memory layout and read instructions, the four entries for each sub-table can be returned in each cycle in the following format:

[0221] Single vectors with lower destinations: A[0][1], A[0][2], B[1][3], B[1][4], (the rest are padded with zeros)

[0222] Single vectors with higher destinations: A[1][1], A[1][2], B[2][3], B[2][4], (the rest are filled with zeros)

[0223] Despite Figure 6C The table is shown as two 2D tables, each with a width of 10 elements and a height of 3, such as table A and table B. This is not limiting; the tables can have any width and / or height, depending on the embodiment. Similarly, Figure 6D The memory layout includes a layout of 16 elements wide x 3 high, but this is not limiting, and the memory width and / or height can be any configuration depending on the embodiment.

[0224] In some implementations, such as when sampling an image patch, interpolation between portions of pixels can be performed. In some embodiments, to manipulate data by interpolating lookup values ​​without additional instructions, a vector horizontal interleaving blending (VHBlend_I) instruction can be executed, which can include horizontal blending with interleaved data to blend between channel pairs. For example, using this instruction, post-lookup bilinear interpolation can be performed in the same loop. This instruction can be based on... Figure 6EThe table layout is used to process each channel pair. Thus, Y0 and Y1 can be calculated as follows:

[0225] Y0 = x*(1–alpha0) + y*alpha0

[0226] Y1=z*(1–alpha1)+w*alpha1

[0227] Therefore, this instruction can cause horizontal mixing between channel pairs x and y, z and w, and can cause the output to be interleaved in the destination register. For example, the following C code snippet can be used to achieve optimal performance on an 8-way parallel table using a 2x2 point lookup.

[0228]

[0229] In this 8-way parallel list organization, sublists are designated A, B, ..., H, and loops can perform lookups and interpolations, resulting in 16 outputs per iteration. In such an example, the input can be organized as follows:

[0230] idx.lo = {idx0, idx1, idx2, idx3, idx4, idx5, idx6, idx7, (ignore the remainder)}

[0231] idx.hi = {idx8, idx9, idx10, idx11, idx12, idx13, idx14, idx15, (ignore the remainder)}

[0232] x_frac.lo = {xf0, xf0, xf1, xf1, ..., xf7, xf7} / / Note the repeating pattern

[0233] x_frac.hi = {xf8, xf8, xf9, xf9, ..., xf15, xf15} / / Note the repeating pattern

[0234] y_frac = {yf0, xf8, yf1, yf9, ..., yf15} / / Note the example of the intermediate and final results of this instruction in interleaved mode. Figure 6F The description includes arrows indicating the mixing and interleaving patterns of the data.

[0235] Now refer to the diagram as follows Figure 6GAs shown, each block of method 600 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 600 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 600 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service), or plug-in to another product, to name just a few. Furthermore, method 600 can be performed by any system, architecture, or component, or any combination of systems, architectures, or components, including but not limited to those described herein.

[0236] Figure 6G This illustrates some embodiments of the present disclosure for performing multi-point lookups (e.g., in a single clock cycle in a Decoupled Lookup Table (DLUT) accelerator, for example, regarding... Figures 9A-9C The flowchart of method 600 (described) is shown. In block B602, method 600 includes copying a table to memory to include a first value at a first physical address in a first memory group and a second value at a second physical address in a second memory group, the first and second values ​​being included in the same column in a logical memory view of the table. For example, a table can be copied to memory any number of times to take advantage of the system's memory access parallelism. The table could include a first value at a first logical address and a second value at a second logical address in the same column as the first value, which could potentially lead to memory group conflicts if stored in memory in this configuration, as the two values ​​might be stored in the same memory group. Therefore, when the table is copied to memory, a write instruction can write the first value to an adjacent first physical address—for example, in another memory group—as the second value, allowing both values ​​to be retrieved in the same cycle.

[0237] In box B604, method 600 includes determining a first index corresponding to a first physical address in memory. For example, a read operation may use an index that indicates a first location in memory from which a value is read.

[0238] In block B606, method 600 includes, at least in part, a read instruction corresponding to a multi-point lookup, reading a first value at a first physical address and a second value at a second physical address in a single cycle. For example, when a table is copied to memory, the table can be copied such that pairs of points in the same column or table (e.g., pixels corresponding to pixels in the same column of pixels) are stored in separate memory banks. Therefore, using a read instruction for a two-point lookup, which uses the index of the first point in the pair to read the first point and the adjacent second point stored in different memory banks, the first and second values ​​can be read in a single cycle from a first memory bank storing the first value and a second memory bank storing the second value. This operation can be performed on each pair of values ​​in each copied table to produce a high vector including the first value from each table and a low vector including the second value from each table, and these vectors can be used as vector registers in the VPU, as well as instructions for generating output (e.g., interpolation, etc.).

[0239] In box B608, method 600 includes performing one or more operations using a first value and a second value. For example, the first and second values ​​can be loaded into one or more channels of the VPU, and square root, logarithmic, sine, and cosine functions can be performed, linear or bilinear interpolation can be performed, and / or other types of operations can be performed. In the case of performing interpolation, a table is copied 16 times; for example, 16 two-point lookup operations may occur to produce 32 values—each vector channel of the VPU has 2 values—and interpolation can be performed on each channel to output 16 results. Therefore, each cycle can produce 16 interpolated outputs using only 16 copies of the table. This is likely a result of using two-point lookups, as the table containing the values ​​may only need to be copied half the time of a traditional single-point lookup operation (e.g., 16 times instead of 32 times) to allow for the same 32 values ​​with half the memory footprint throughput.

[0240] Each memory bank in the vector memory loads a cache.

[0241] In conventional processors, data caches can have a width of, for example, 32 bytes per cache line. A cache line is a data unit that is tracked by hardware. For example, hardware can track cache line usage information in tagged memory, including the full system address, whether the cache line has been written to, and the time of the last read of the cache line relative to other cache lines, to determine when to evict a cache line. In some implementations, the data cache is local memory, or a portion of local memory, used to temporarily map large data structures stored in external memory to local memory, allowing data to be processed without going through long-latency memory that directly processes external memory. This type of data cache is frequently used in traditional desktop or laptop computers.

[0242] Programmable vision accelerators and / or VPUs, as non-limiting examples, include embedded processors designed to run smaller, highly optimized sets of code. In such processor types, data caching may not be feasible because the programmer can manage the contents of local data memory. The systems and methods disclosed herein may include local memory managed by the programmer rather than cached, but may also include additional data caching capabilities in one or more (e.g., each) memory banks. Data caches can be narrow, such as, but not limited to, 16-bit wide, compared to more conventional data caches that include, for example, 32 bytes. Unlike conventional data caches, whose primary goal is to reduce latency, data caching can be primarily used to reduce power consumption.

[0243] For example, in computer vision processing, data access patterns often exhibit a degree of locality (e.g., lingering in a neighborhood for a period of time before moving to the next neighborhood). For instance, when performing 7x72D filtering using the VFilt4HHW instruction described here (compiling 4 taps at a time), the data read stream can read memory 3 times from one neighborhood, then move to another neighborhood and read 3 more times, and so on. In the coefficient reads of the operation, the same array of zero-padded values ​​(e.g., 7*2*4 = 56 half-words) can be used, advancing 4 half-words at a time until the last set of 4 half-words is read, then starting again from the beginning of the 56-half-word array until the filtering kernel completes.

[0244] Therefore, to leverage these local access patterns and reduce power consumption due to memory accesses, a data cache can be loaded into each memory bank with bidirectional set associativity (maintaining, for example, a total of 64 half-words). When the load cache is enabled, the most recently read set (e.g., most recent, two most recent, three most recent, etc.) of read data can be recorded in the cache, along with the row address and the most recently read entry in tag memory. As a result, when the same memory address is read again, a cache hit may occur, and the cache may serve the data instead of requiring a read from local memory again. In an embodiment, the load cache may be located between the memory logging logic and the memory itself, such that whenever a cache hit occurs, a memory read for that particular address or value will be stopped or not performed to save power.

[0245] Using this cache structure, and for the 7x7 2D filtering example above, loading the cache allows the system to skip almost two-thirds of data reads and almost all coefficient reads in a steady state. Instructions for using the data cache in each memory bank are in... Figures 7A-7CThe explanation is as follows. For example, the VFilt4HHW instruction can perform a 4-tap filtering task that may be larger and may consume two single half-word data vectors—e.g., data[0-15] and data[4-19]—and a single half-word vector coefficient—e.g., coef[0-3]—repeated four times to fill a single vector of 16 elements. In a 7x7 2D filter implementation using the VFilt4HHW instruction in two vector math slots, one can use... Figure 7A The data elements and coefficient arrays. Since the VPU of this disclosure can be configured to read bivectors, data[y][0-15] and data[y][16-31] can be read as bivectors. Similarly, data[y][4-19] and data[y][20-35], as well as data[y][8-23] and data[y][24-39], can be read as bivectors. Likewise, the data and coefficient reading patterns can correspond to... Figures 7B-7C For those, assume the row spacing of the data is 100 and the row spacing of the coefficients is 8.

[0246] Figure 7D Explain memory bank organization. For example, a 2-entry fully associative cache holds data values ​​at two locations in any superset, and data and coefficients can be placed in different supersets to allow the cache to work efficiently. In a coefficient read, memory bank 0-3 might first hold coefficient elements 0-3, add elements 32-35, and then read elements 64-67, which would evict elements 0-3. These elements will be repeated as a pattern in the next coefficient read. In a steady state with the load cache enabled, each scan of the coefficient read pattern can only read four memory banks. Therefore, saving memory entries by using the load cache for data might be (3*32-(32+4+4)) / (3*32) = 58.3%, and for coefficients it might be (14*16-4) / (14*16) = 98.2%.

[0247] Therefore, in some algorithms—such as computer vision algorithms with sliding windows—loading caches can be used to save power. For example, without a loading cache, each memory bank needs to be read in every cycle, even if most of the data is the same. In the example of reading 512 bits per iteration, the first 512 bits can be read, then another 512, and so on. For example, if the sliding window is only 8 bytes, then only 64 bits are new in each iteration, and the remaining 448 bits are the same. Without a data cache, these 448 bits need to be read from the data bank again. However, by using a data cache for each memory bank, these 448 bits can be retrieved from the loading cache, and only 64 new bits need to be read from other memory banks. Thus, the power required to read 448 bits from the memory bank is saved. Examples of algorithms that can benefit from using a loading cache are spatial filtering operations, deep learning inference operations (such as convolution operations), and so on.

[0248] about Figure 7E This illustrates a hardware architecture or logic for a memory bank with load caches. For example, support for unaligned access in memory (e.g., vector memory (VMEM)) can accelerate sliding window data access. This is a key memory access pattern in many computer vision algorithms, including filtering and convolution. For sliding window vector loading, most of the data from the random access memory (RAM) bank 702 remains unchanged. In such an example, when sliding 4B, only 4B of data changes in a 64B vector load, so only 4B of new data is read from RAM bank 702. To optimize the power consumption of the VMEM RAM, a mini-cache called a "load cache" can be attached to each group for each supergroup—therefore, each VMEM has a total of 3 supergroups x 32 groups = 96 load caches. In a non-limiting embodiment, the configuration of each load cache may include a two-row (2x2B = 4B) capacity, full associativity, and a pseudo-least recently used (pLRU) replacement strategy.

[0249] The data cache, which stores the most recently accessed data, is divided into two parts—tag storage 706 and data storage 704. Tag storage 706 can store the cache address and control information corresponding to the previous access, while data storage 704 can store the data from the previous access. The control information in tag storage 706 can include a validity flag (e.g., whether the entry is valid), a dirty flag (e.g., whether the entry has been modified and needs to be written back to memory), and / or a last-used flag (e.g., using a least recently used strategy to indicate the entry to be replaced if it needs to be replaced). Because the cache is a load cache, writing data may not update the cache, but the validity and last-used flags can be included in tag storage 706. The validity flag or bit can be used to qualify address matches, and any write should invalidate the entry. The last-used flag may be updated on each access.

[0250] As described herein, to make the caching scheme effective, the storage capacity of the load cache is much smaller than that of the memory or RAM library 702, in order to reduce access time and save power. In one embodiment, each load cache may correspond to a single RAM library 702, each of which may be a 2048x16-bit memory, and each load cache may be a 2x16-bit data storage 704 with a 23-bit tag storage 706 (e.g., 2 entries x (11-bit address + 1-bit valid) + 1-bit last used).

[0251] In operation, offset 722, row address 724, and increment 726 can be used to generate a memory address for memory access. This memory address can be delimited for comparison with tag storage 706—for example, with a number of previously accessed addresses (e.g., two previously accessed addresses). An arrow leading to the top of tag storage 706 can indicate a memory address. In some embodiments, tag storage 706 can use the entire memory address to compare with the memory address from a previously accessed storage. In other embodiments, a subset of address bits from the memory address can be used to address a subset of tags, so only a subset of tags is compared with the memory address. For example, in the case where a large number of previously accessed tags are stored in tag storage 706, only a subset of tags can be compared with a subset using memory address bits to reduce area and save power. In load cache designs with fewer tags—for example, corresponding to two previously accessed tags—the entire tag of a previous entry can be compared with the entire memory address. The “==?” decision box 720 compares the current memory address of RAM library 702 with the address stored in tag storage 706. When a miss occurs (e.g., a tag and memory address mismatch), a read of RAM library 702 can be selectively read and sent to hierarchical trigger 716 when read enable 708 and read data multiplexer (rd data mux) 712 are enabled. When a hit occurs (e.g., a tag and memory address match), data storage 704 can be addressed with 0 or 1 (in embodiments with two entries) to indicate which previous access the hit corresponds to. The corresponding entry in data memory can be sent to hierarchical trigger 716 via read data multiplexer 712. Hierarchical trigger 716 can then return the read-back data to the processor pipeline, ultimately routing it to the destination landmark or vector register of the load instruction.

[0252] The hierarchical trigger 714 can correspond to parity checking. For example, a sufficiently large memory may be needed to have parity bits (e.g., in parity terminal 710) to allow error detection and / or error correction. Error detection can be used in the memory (e.g., VMEM), and / or error correction logic can be implemented on the readback data.

[0253] Therefore, the load cache may include tag bits in tag storage 706 for way 0 and way 1, each tag bit including 11 address bits and 1 valid bit. The load cache may also include 1 pLRU bit and data bits in data storage 704 for way 0 and way 1, each data bit including 16 data bits and 2 parity bits. Once the load cache is enabled, lookups are possible in stage D1. To minimize power consumption, only the load caches of the RAM libraries 702 involved in the load may be enabled. For example, for a single vector load, only 16 of the 32 load caches may be looked up. Upon a load hit (e.g., when the load cache contains data to be accessed), read enable for a given RAM library 702 may be suppressed, preventing the RAM library 702 from being lit up. pLRU 720 may also be updated in stage D1. In stage D2, data and parity bits can be read from the load cache hit path and multiplexed with the RAM result.

[0254] When a cache miss occurs, in the D1 stage, in victim mode, the existing entry to be evicted to make room for the new entry can be determined based on the valid bits and pLRU. The victim path's label can then be updated with the missed address, and the read enable 708 of RAM library 702 can be enabled without suppressing it. In the D2 stage, data / parity from RAM library 702 is not only sent to the read data crossover switch, but also populated into the evicted cache line. Memory can also look up the load cache when enabled and engaged. A memory hit may invalidate the hit mode, and a memory miss may be ignored.

[0255] A cache hit saves power when reading RAM library 702. Conversely, a cache miss not only results in power consumption when reading RAM library 702, but also consumes power to search the cache to fill the victim path. Since not all types of memory access patterns achieve high cache hit rates—especially when accessing supergroups in indexed addressing mode—vector linear loads can only be searched in the cache.

[0256] When enabled, all memory can be looked up in the load cache to ensure that the load cache is never out of sync with data in, for example, the VMEMRAM library 702. For a given supergroup, the software can be used to disable the load cache for the RAM library 702 of that supergroup to minimize memory lookup capabilities, as described in more detail below.

[0257] For example, in some embodiments, the use of a data cache may not provide any benefit. For instance, in operations where access patterns do not repeat, a data cache may be useless, and performing the additional task of checking the cache before reading may waste time and / or effort, as the database may need to access the correct data. Therefore, a load cache can be enabled or disabled to reduce power loss due to access patterns with high load cache miss rates, and also allows the load cache to be used for access patterns where the data cache saves power. In some embodiments, the data cache can be programmed to be enabled or disabled using application code, so a programmer can program the code to enable the data cache when needed and disable it when not needed. In other embodiments, enabling or disabling can be performed by hardware analysis of read patterns and detection of overlapping patterns. For example, the hardware can enable the load cache for a threshold amount of overlap between consecutive read operations. However, the load cache can be disabled if the overlap is less than the threshold. As a non-limiting example, the threshold can be 25%, 40%, 50%, 75%, or different threshold overlap amounts between readings.

[0258] When loading the cache is disabled, and regarding the graph... Figure 7E As shown, the tag memory 706 may not be accessed, and the read enable 708 can be set to enable a read of the RAM library 702 for each read. Similarly, the data memory 704 may not be accessed, and the read data multiplexer 712 may always pass data from the RAM library 702 to the hierarchical flip-flop 716.

[0259] Furthermore, in some embodiments, the memory bank structure may include multiple supergroups—for example, three supergroups—and each supergroup may enable or disable a load cache based on specific access patterns within that supergroup. For example, in the case of using three superbanks, each superbank may include 32 RAM memory banks, and the data cache for each memory bank may include two entries, where each entry is a word, and therefore 16 bits. In the case of using two or more supergroups, the supergroups may be of any size, different sizes, the same size, or a combination thereof. For example, the first supergroup may be 128KB, the second supergroup may be 256KB, and the third supergroup may be 512KB.

[0260] Now refer to the diagram as follows Figure 7FAs shown, each block of the method 750 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 750 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 750 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name just a few. Furthermore, method 750 can be executed by any system, architecture, or component, or any combination of systems, architectures, or components, including but not limited to those described herein.

[0261] Figure 7F This is a flowchart illustrating a method 750 for using a data cache for a read operation according to some embodiments of the present disclosure. In block B702, method 750 includes receiving data representing a memory read address. For example, after a first read operation using some memory banks, a second read operation may be performed, which includes one or more memory banks in addition to one or more additional or other memory banks. Because the first read operation may have already included storing the output of the read in a data cache corresponding to each respective memory bank, these values ​​can be reused instead of requiring another read of the memory bank. Thus, a memory read address corresponding to the next read operation can be received, and a load cache—when enabled—can be accessed to determine if any data is stored in the load cache.

[0262] In block B704, method 750 includes comparing a memory read address with a load cache memory address corresponding to a previous memory read stored in the load cache. For example, after a previous memory read, data from the memory read may be stored in a load cache corresponding to a specific RAM library 702. To remember this information, tag memory 706 may include one or more previous memory addresses corresponding to reads from RAM library 702.

[0263] In box B706, method 750 includes determining that a memory read address at least partially overlaps with a load cache memory address. For example, the memory read address can be compared to a previous memory read address stored in tag memory 706. If a match is found, the load cache can be used to read at least some of the data corresponding to the memory read address of the current memory read.

[0264] In block B708, method 750 includes reading at least a portion of data corresponding to a memory read address from a load cache. For example, due to a hit in the load cache determined from tag storage 706, a portion of the data from the overlapped memory address can be read from the load cache, and the remainder of the data—if any—can be read from RAM library 702.

[0265] Decoupled configurable accelerator

[0266] To optimize processor performance for specific applications (such as real-time applications), the instruction set architecture (ISA) can be enhanced to create custom instructions to accelerate common operations. This allows the processor to reduce the number of cycles required to execute a specific task. The process of executing a custom ISA continues until the system's performance goals are met. However, these new instructions are added to manipulate data in the processor's register file or directly as memory operands, and these instructions are executed using the existing processor controller and existing memory addressing and access hardware. In such examples, it is desirable for the new instructions to fit the processor's register file read / write operand count (e.g., reusing existing ports), to fit the register file width (e.g., to fit the processor's data types), and to fit the processor pipeline stages. Due to these requirements for successfully adding instructions to an ISA, the flexibility of adding new instructions is limited. Furthermore, when creating an ISA for processing multi-stage pipelines (e.g., 30, 40, 50 stages, etc.), the configuration of the ISA becomes complex.

[0267] Furthermore, processors offer high flexibility at the cost of power consumption—because each added instruction requires fetching, decoding / dispatching, reading / writing register files and / or memory, etc. Therefore, adding additional functional units to implement these custom instructions increases the load on register file read / write ports, resulting in increased area requirements (e.g., potentially additional read / write ports) and power consumption (e.g., the additional load on the register file can be implemented). Additionally, the processing pipeline of embedded applications typically has multiple stages—the output of one stage feeds the input to the next stage. Techniques such as executing multiple threads in the processor (e.g., for different processing stages) can reduce scaling time, thus providing reduced latency. However, multithreading comes at the cost of hardware—instructions must be fetched / decoded / scheduled from multiple threads, state information for each state of each thread must be maintained (e.g., in register files), and control logic must be included to manage multiple threads in the processor. This leads to increased area and power requirements, while also making processor verification and programming more complex. Therefore, although various methods exist for reducing latency in the processing pipeline, existing methods require additional surface area of ​​the processor hardware, additional power consumption due to the additional hardware, and increase the complexity of programming the processor to perform various tasks.

[0268] To address the limitations of main processor configuration and the drawbacks of multi-threaded processors, the systems and methods disclosed herein utilize a main processor or one or more units of a main processor—such as a single-threaded processor like a VPU—except for domain-specific accelerators or coprocessors—such as vector memory (VMEM)—that are decoupled from the main processor and communicate with it via shared memory. In this way, accelerators can operate as sub-units of the main processor, but once configured, they can execute independently of the main processor's instructions, rather than requiring processor instructions for execution. For example, accelerator access instructions can be used to allow the main processor to configure and order accelerators, and shared memory can allow the sharing of inter-level data structures between the main processor and accelerators. Once the main processor starts or enables the accelerators (e.g., via a common accelerator interface and using one or more load / store instructions), the main processor is free to process different stages (thus providing the ability to work simultaneously at multiple stages of the processing pipeline and reduce runtime) or transition to a low-power or minimum-power state while waiting for the accelerators to complete processing (e.g., minimizing power consumption when not actively processing). Thus, once configured by the main processor, each of the one or more accelerators can operate independently and concurrently with the main processor. The host processor and accelerator can synchronize during processing via a handshake interface so that the host processor knows when the accelerator has finished processing and / or is ready to execute a new task, or vice versa. Shared memory can store configuration messages (e.g., for configuring the accelerator when configuration instructions cannot be effectively sent through the accelerator interface due to size limitations), input buffers (e.g., storing data for accelerator processing), and / or the accelerator's results (e.g., after processing is complete, data from the accelerator, such as a register file, can be stored back to the location in shared memory indicated by the host processor's configuration instructions). Therefore, once triggered, the accelerator can read configuration parameters and / or input data structures from shared memory and can write output result data structures to shared memory.

[0269] As a result, this combined system of main processor, shared memory, and decoupled accelerators allows for the flexibility of a programmable main processor while achieving the power levels of fixed-function hardware (e.g., because high computational processing stages of the processing pipeline can be implemented as accelerators) without significantly increasing the complexity of the main processor (e.g., because the main processor may only require additional accelerator configuration or access instructions to program the accelerators). For example, the accelerator's pipeline and data types (e.g., data width) can be independent of those of the main processor, allowing for further customization and optimization that might not be possible with a main processor alone, requiring instructions to adapt to the processor's register file read / write operand counts, register file widths, and pipeline stages.

[0270] In some embodiments, the accelerator and the main processor can be coupled at instruction execution time to achieve some power savings for the accelerator when coupling execution to the main processor pipeline. However, in such embodiments, the ability to concurrently process different stages of the pipeline is reduced because instructions are interleaved between the accelerator and the main processor. In one or more embodiments, the accelerator and the main processor can be coupled via a higher-level Level 2 (L2) memory connection instead of a shared memory connection. However, in such embodiments, higher-level decoupling (e.g., removing coupling from shared memory to a higher level) can increase communication overhead with the main processor.

[0271] Decoupling accelerators can be used for any task within any domain. For example, as a non-limiting example, they can perform 1D, 2D, etc., lookups as decoupling lookup table accelerators to detect and resolve memory group conflicts, perform 1D / 2D interpolation, etc., for computer vision algorithms such as feature tracking, object tracking, image warping, pyramid creation, etc., for sensor processing such as matrix multiplication or other operations on LiDAR, RADAR data, and / or similarly for machine learning or deep learning applications. Therefore, the topology described herein can be applied to any processing pipeline where a portion of the processing can be offloaded to the accelerator.

[0272] Depending on the implementation, there may be any number of decoupled accelerators on one or more chips, communicating with one or more main processors via shared memory. For example, a system-on-a-chip (SoC) or other integrated circuit (IC) may include a main processor and one or more accelerators. Programmers may be familiar with the various accelerators and write instructions or code that utilize the accelerators to improve the performance of the system for any variety of tasks. Although the main processor is primarily described as a VPU, this is not intended to be limiting, and the main processor may include any processor type, such as a CPU, GPU, DPU, or other processor, without departing from the scope of this disclosure.

[0273] Now for reference Figure 8A , Figure 8AA system 800 including one or more decoupled accelerators is illustrated according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, system 800 may be included and / or may include […]. Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 The components, features, and / or functions of the example data center 1500 are similar to those of other components, features, and / or functions.

[0274] System 800 may include a processor 802 (e.g., a main processor), such as a VPU, CPU, GPU, DPU, etc., a decoupling accelerator 804, and / or shared memory 806 (e.g., vector memory or VMEM). Processor 802 may be coupled to an instruction cache (I-cache) 810, which may cache instructions for execution by processor 802. Processor 802 may include general-purpose input / output (GPIO) 808 (e.g., digital signal pins on an IC that can be used as inputs, outputs, or both, and can be controllable at runtime), and an IC configurator 812. In some embodiments, as shown, processor 802 may use an Advanced Extensible Interface (AXI) for on-chip communication, such as, but not limited to, a 256-bit AXI interface. IC configurator 812 can be used to configure system 800.

[0275] Processor 802 can communicate directly with decoupled accelerator 804—for example, via a coprocessor or accelerator interface, such as an Advanced Peripheral Bus (APB) interface, and / or a handshake, programming, or event interface. For example, processor 802 can configure accelerator 804 using an accelerator interface (or configuration bus), initiate or trigger processing of accelerator 804 using an event interface, and synchronize with accelerator 804 using a handshake or event interface. Thus, each accelerator 804 can include mechanisms configured to communicate with processor 802 via a corresponding accelerator interface or configuration bus. For example, accelerator 804 can indicate the same to processor 802 via a handshake mechanism when processing is complete, or processor 802 can periodically poll accelerator 804 to request status or end time while processor 802 is waiting for accelerator 804 to complete processing. In some embodiments, the accelerator interface can include a 32-bit interface (or other smaller-sized interface) so that configuration instructions can be transmitted to accelerator 804. However, in some embodiments, the configuration message may be large (e.g., greater than 32 bits, or a multiple thereof), and the configuration message may instead be stored in shared memory 806, and the location of the configuration information in memory 806 may be sent to accelerator 804 via the accelerator interface to indicate where to retrieve the configuration information.

[0276] The configuration bus can therefore configure accelerator 804, and events (or programming interfaces) can be used to allow processor 802 to trigger or initiate processing on accelerator 804. Once triggered or initiated, accelerator 804 can operate autonomously, while processor 802 waits for processing to complete and / or executes different processing tasks or phases. For example, application programmers can program processor 802 and accelerator 804 to know what each is capable of, thereby dividing the application into multiple parts—some parts of processor 802 and some parts of accelerator 804. Therefore, in embodiments, processing can be performed in parallel between processor 802 and accelerator 804 to reduce runtime and improve efficiency. Configuration messages—shared via the accelerator interface and / or via shared memory 806—can be generated by processor 802 and used to indicate to accelerator 804 where in shared memory 806 the data to be processed begins, how much data to process, and where in shared memory 806 the results should be written back. Processor 802 can generate an input buffer at a specified location in shared memory 806, which includes data operations for accelerator 804. Once the configuration message is sent and the input buffer is stored in shared memory 806, accelerator 804 can receive a trigger signal from processor 802 via an event interface (e.g., a programming interface), and accelerator 804 may be processing data. Once accelerator 804 is triggered, processor 802 can then perform other work or enter a low-power state, and once accelerator 804 has finished processing, it can instruct processor 802 to do the same and can wait for additional work.

[0277] Processor 802 can set up input buffers or input data structures for accelerator 804 to process and store them in memory 806. Accelerator 804 can be configured by processor 802 using load / store operations; processor 802 is dedicated to configuring and communicating with accelerator 804. Configuration messages can configure various registers of accelerator 804 (e.g., 256 32-bit registers in one embodiment). For example, for a decoupled lookup table accelerator (as described more in detail herein), configuration information can indicate whether the lookup is for a 1D lookup with interpolation, a 2D lookup with bilinear interpolation, and / or another type of lookup. Once accelerator 804 knows a particular mode or function, it can configure registers to correctly read data from memory 806, process data, and write data back to memory 806.

[0278] In some embodiments, processor 802 may configure accelerator 804 to execute multiple tasks simultaneously to improve efficiency. For example, if accelerator 804 will execute various smaller tasks, configuring accelerator 804 individually might increase runtime because each task could complete quickly, requiring processor 802 to stop processing, configure accelerator 804 for another task, and so on. To this end, the first task message may include the address of the second task message, thereby allowing self-linking of multiple tasks. In this way, processor 802 can generate configuration messages for multiple tasks simultaneously, and generate configuration information and input buffers for each task, allowing accelerator 804 to execute various tasks sequentially before instructing processor 802 that processing is complete and accelerator 804 is ready to receive more work. Furthermore, to improve efficiency, accelerator 804 may be configured to overlap tasks, such that when the first task is nearing completion, accelerator 804 can begin decoding the next task and configuring registers for the next task. Finally, by including separate instructions for processor 802 and accelerator 804, accelerator 804 may be able to operate on data formats or types different from those supported by processor 802. This may be a result of the different architecture and layout of the accelerator 804's registers, which are specifically designed for a particular processing task.

[0279] In this embodiment, processor 802 can communicate with shared memory 806 via any number of memory interfaces (e.g., a 512-bit static random access memory (SRAM) interface). Similarly, as shown, accelerator 804 can communicate with shared memory 806 via any number of memory interfaces (e.g., a 512-bit SRAM interface). Arbitrator 814 can determine, in each cycle, which of processor 802 and / or accelerator 804 is allowed to access shared memory 806.

[0280] Now for reference Figure 8B Each block of the method 850 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 850 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 850 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service), or plug-in to another product, to name a few. Furthermore, regarding… Figure 8A System 800 describes method 850, which can be performed by any system, structure or component or any combination of systems, structures or components, including but not limited to those described herein.

[0281] Figure 8BThis is a flowchart illustrating a method 850 for using a decoupled accelerator according to some embodiments of the present disclosure. At block B802, method 850 includes receiving configuration information for one or more first processing tasks of a processing pipeline. For example, accelerator 804 may receive configuration information (e.g., configuration messages via an accelerator interface) from processor 802.

[0282] In block B804, method 850 includes configuring one or more registers of the accelerator based at least in part on configuration information. For example, accelerator 804 may configure one or more registers based on configuration information.

[0283] In block B806, method 850 includes reading data from an input buffer in memory based at least in part on an indication of the starting position of the input buffer included in configuration information. For example, the configuration information may include an indication of where the input buffer is stored in memory 806, and accelerator 804 may read data from the input buffer into a register.

[0284] In box B808, method 850 includes processing data from the input buffer to compute output data. For example, accelerator 804 may process data from the input buffer to generate or compute output.

[0285] In block B810, method 850 includes writing output data to memory at a location determined at least in part based on configuration information. For example, accelerator 804 may write computation results to memory 806 and may indicate to processor 802 that processing is complete. Processor 802 can then use the output data to execute one or more second processing tasks in the processing pipeline.

[0286] Decoupled lookup table accelerator

[0287] Parallel processing is used to accelerate many computational tasks, including but not limited to computer vision applications, deep learning applications, sensor processing applications, and / or other applications that benefit from parallelism (e.g., where the processing task is independent of other processing tasks). For example, vector processors can operate on multiple elements in the same operation to achieve the efficiency required to execute these types of parallel processing algorithms in real time, while consuming low power. A common operation in computer vision or deep learning tasks, for example, is performing a lookup from a lookup table, image patch, or surface based on an index or coordinate location. For this purpose, a single vector load or store operation can be used to access data from multiple elements. Unless the index being looked up is regular (e.g., consecutive or fixed integer steps in the horizontal, vertical, or depth directions), it results in random index access in memory.

[0288] To support regular but unaligned vector accesses from memory, the processor can construct vector memories using smaller RAM libraries. In this way, the hardware is able to create interesting addressing modes for vector memories by independently generating unique addresses for each RAM library. For unconventional indexed vector load operations in memory, this can lead to memory set conflicts in one or more memory banks of RAM, since the indices of different vector elements can be independent of each other. Memory set conflicts may not be statically determined because they are data-dependent, thus preventing the compiler from scheduling around memory set conflicts.

[0289] In some conventional systems, various architectural designs can be implemented to support unconventional indexed vector loading operations. For example, multiple read ports can be added to the RAM library. In such an example, if the hardware can handle 32 vectors, each group would require 32 read ports, increasing cost, area, and power consumption, and adding to the layout and wiring congestion around the RAM library. Another example involves reducing the throughput of index lookups to perform a single scalar lookup per load. However, this creates a bottleneck for vector execution and becomes a time-limiting factor. Yet another example involves making multi-level copies of the data structure in memory, allowing each vector channel to access data from a single group. While this example addresses some of the throughput issues of other approaches, memory capacity is limited by the space occupied by the data structure N times (where N is the number of entries to be accessed), which can lead to an overall performance degradation of the associated algorithm, in addition to the overhead of making copies. However, this approach is suitable for smaller data structures. In some examples, conflicts can be dynamically detected and resolved by serializing conflicting lookups. However, this can lead to increased hardware complexity, as memory group conflicts must be detected and resolved dynamically. Furthermore, these additional stages increase the loading and usage latency of these operations, thus affecting the compiler's ability to efficiently schedule code. Additionally, they may introduce data-dependent execution latency, which is a problem for efficient compiler scheduling. In some examples, combinations of these methods can be executed.

[0290] To address these shortcomings of other architectures, the systems and methods disclosed herein include a decoupled lookup table accelerator configured to support unconventional index vector loading operations. The decoupled lookup table accelerator may be included as an accelerator 804 of system 800 and may communicate with processor 802—e.g., a VPU—via shared memory 806. The decoupled lookup table (DLUT) may support multiple modes for performing table lookups, such as 1D lookup mode, 2D lookup mode, 2D conflict-free lookup mode, 1D lookup with interpolation mode, 2D lookup with interpolation mode, table reformatting mode, and / or other modes. In any lookup mode, the DLUT may accept an array of indexes in VMEM, which may be in 1D(x) or 2D(x,y) format. For example, each element may include 16 bits or 32 bits, and they may be unsigned. The DLUT may then perform prescribed index calculations, which may include 2D-to-1D mapping, truncation / rounding, integer / fraction splitting, and / or valid range detection, as non-limiting examples. For example, DLUT can detect or merge duplicate reads, detect memory group conflicts within an index, and issue read requests to VMEM to find the requested table entry. Each element can include 8 bits, 16 bits, or 32 bits, and they can be signed or unsigned. DLUT can then perform post-interpolation processing as configured and write the output back to VMEM. Each of these processing operations can be performed in the pipeline to increase throughput, reduce latency, and lower power consumption.

[0291] As a result, the DLUT accelerator overcomes the shortcomings of implementing dynamic conflict detection and resolution within the processor pipeline, allowing the compiler to efficiently schedule deterministic execution latency for all memory operations while avoiding the complexity of inline conflict detection. Because the accelerator operates as a tightly coupled accelerator—for example, via a VMEM shared with the VPU—the processor can configure and start the accelerator while continuing to process other independent parts or stages of the processing pipeline or algorithm. In some embodiments, the accelerator may include additional features to further reduce the load on the main processor, such as offloading index generation with patches having specific lookup patterns, performing optional 1D blending and 2D interpolation on the lookup data, and / or providing table reformatting support without lookups or interpolations. In practice, the entire system (including the processor 802 and accelerator 804 for performing lookups) has been shown to accelerate the processing of various computer vision algorithms (e.g., feature tracking, object tracking, image warping, pyramid creation, etc.) by one factor by two, while reducing energy consumption by more than 50% compared to executing the entire algorithm solely on the main processor.

[0292] Now for reference Figure 9A , Figure 9AA system 900 including a decoupled lookup table (DLUT) accelerator according to some embodiments of this disclosure is illustrated. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, system 900 may be included and / or may include with… Figure 8A System 800, Figures 13A-13D Example autonomous vehicles 1300 Figure 14 Example computing device 1400 and / or Figure 15 Similar components, features, and / or functions to the example data center 1500.

[0293] System 900 may include one or more processors 902 (which may correspond to...) Figure 8A The processor 802) and memory 904 (which may correspond to Figure 8A The shared memory 806) and the decoupled lookup table (DLUT) accelerator 906 (which can be used as) Figure 8AAccelerator 804 is included. In an embodiment, processor 902 may include a VPU, and memory 904 may include a VMEM. DLUT accelerator 906 (or "DLUT 906") may include a processing unit (PU) interface (I / F) 908 for communicating with processor 902, a controller 912 for communicating with processor 902, and a configurator 910 for configuring DLUT 906 based on information shared from processor 902 across PU interface 908 and / or from memory 904 based on indications from processor 902 regarding the location of configuration messages or information in memory 904. For example, PU interface 908 and controller 912 may correspond to the Advanced Peripheral Bus (APB) and the event or programming interface of system 800, respectively. The controller 912 may receive start or trigger commands or signals from the processor 902 (e.g., via an arrow marked "Start"), indicating that the DLUT 906 may begin processing, and / or may receive polling signals from the processor 902 to help synchronize the processor 902 with the DLUT 906. Furthermore, when the DLUT 906 completes processing one or more assigned tasks, the DLUT 906 may generate a signal to the processor 902 (e.g., via an arrow marked "Complete"), allowing the processor 902 to begin configuring the DLUT 906 for the next task.

[0294] During configuration, processor 902 can configure DLUT 906 directly via PU interface 908 and / or indirectly by indicating the location of configuration information in memory 904 via PU interface 908. In the latter example, DLUT 906 can retrieve configuration information from memory via, for example, a shared read port strm1_dm_rd, and can use the stored configuration information to configure DLUT 906 (e.g., configure subunits (e.g., IAU, CDRU, PPU, etc.) and / or other components of DLUT 906) to perform one or more tasks. For example, processor 902 can set up data structures in memory 904 required for DLUT 906 to perform one or more tasks. For example, for a 1000-coordinate lookup, processor 902 can set up data structures in memory 904 with each of the 1000 coordinates, and can further allocate buffers in memory 904 to which DLUT 906 writes its output. Processor 902 can also instruct DLUT 906 which operations to perform—e.g., 1D or 2D lookup, interpolation with or without, table reformatting, etc.—and DLUT 906 can use this information to configure sub-cells. The configuration information set by processor 902 can also include indications of the bit width of coordinate indices and the bit width of table entries, etc. Thus, once the input and output buffers are set in memory 904 and configuration information such as bit width and operation type is sent to DLUT 906, processor 902 can start or trigger DLUT 906 to begin processing. Therefore, compared to a system that relies solely on processor 902, processor 902 can perform other tasks while DLUT 906 performs lookups, interpolations, table reformatting, etc., thereby reducing runtime and improving efficiency.

[0295] In operation, DLUT 906 can receive a list of indices corresponding to coordinates from memory 904, and DLUT 906 can extract values ​​(e.g., where the values ​​are integer values) from the table corresponding to the indices and / or can pull surrounding values ​​of the fractional values ​​(e.g., left and right values ​​for a one-dimensional lookup or top-left, bottom-left, top-right, and bottom-right values ​​for a two-dimensional lookup), and perform interpolation or other operations on the surrounding values. Once the final values ​​are determined (e.g., directly through a lookup without post-processing, or after processing by the post-processing unit (PPU) 930), these values ​​can be written to the output buffer in memory 904, which corresponds one-to-one with the indices in the input buffer. To efficiently perform these tasks, in embodiments, the Index Address Unit (IAU) 922, Conflict Detection and Resolution Unit (CDRU) 924, Control-Time First-In-First-Out (FIFO) 928, Fractional (FRAC) FIFO 926, for example, the Post-Processing Unit (PPU) 930, Data Merging Unit (DCU) 932, and / or other components can be used.

[0296] For example, index (IDX) stream 916 may include an index stream read from memory 904 (e.g., via read port strm1_dm_rd) for lookup in one or more lookup tables, and the value corresponding to the index may be read from memory 904 (e.g., via read port strm0_dm_rd) via lookup table (LUT) stream 918. Output (OUT) stream 920 may be a value written back to memory 904 (e.g., via write port strm0_dm_wr) after processing using DLUT 906.

[0297] During configuration, processor 902 can instruct IDX stream 916 how to access the data structures used for indexing. For example, for a one-dimensional lookup, where the interface with memory 904 is 64 bytes wide, 64 bytes can be read in each cycle. In the case of performing a 1D lookup, a single coordinate can be read for each index value (e.g., the (x) value), while for a 2D lookup, two coordinate indices (e.g., (x, y) values) can be read for each index. In a non-limiting embodiment, each index can be 16 bits or 32 bits, so in each 64-byte read, there may be 8, 16, or 32 coordinates from IDX stream 916.

[0298] IDX stream 916 data can be sent to IAU 922 in its raw format as a raw index, and each coordinate can be an integer value or a fractional value. IAU 922 (where the index is a fractional value) can split the fractional value to provide fractional bits to FRACFIFO 926 to aid in finding surrounding values ​​using the PPU 930 mixed table. IAU 922 can then determine a set of indices to send to CDRU 924, where the number of indices sent can correspond to the number of lookups that LUT stream 918 can perform in a single cycle. For example, if LUT stream 918 can perform, for example, 32 lookups in a loop (based on the bit width of each value in the lookup table), then IAU 922 can send 32 indices to CDRU 924 on each iteration. In some examples, such as when the values ​​from IDX stream 916 to IAU 922 are integer values, IAU 922 can send each set of indices without any processing. However, when the value from the IDX stream 916 is a fractional value, the IAU 922 can determine which indices need to be looked up to obtain each surrounding value required (e.g., two indices for 1D interpolation or four indices for 2D interpolation) to perform interpolation or other operations, thereby obtaining a mixed value corresponding to the fractional value. For example, if the fractional value is (5.3, 6.2) corresponding to the (x, y) coordinates used for 2D lookup and interpolation, the IAU 922 can determine that the lookup will occur at (5, 6), (5, 7), (6, 6), and (6, 7), and then the PPU 930 can mix these values ​​to generate a final value corresponding to the index (5.3, 6.2). For example, these values ​​can be mixed with equal weight, or bilinear interpolation can be used to mix them such that values ​​closer to (5, 6) than (6, 7) are more heavily weighted to compute the final value of (5.3, 6.2).

[0299] The lookup set (e.g., 32 lookup indices for which LUT stream 918 can read 32 values ​​in each read cycle) can be sent to CDRU 924 in an appropriate order corresponding to the index order in the input buffer of memory 904 read using IDX stream 916. CDRU 924 then performs conflict detection and resolution by identifying memory group conflicts that would result if lookup table reads in LUT stream 918 occurred in the order received from IAU 922, and resolves memory group conflicts by changing the order of the indices. For example, if a lookup of an index set would cause a memory group conflict, and another set of indexes (e.g., later or earlier) is available for another lookup cycle, CDRU 924 can find a non-conflicting lookup from the other lookup cycle and swap the non-conflicting lookup with the conflicting lookup of that cycle. As a result, one or more memory group conflicts can be avoided, thereby increasing throughput. For example, if the IAU sends 32 indices per cycle, and 6 indices in a given cycle have memory group conflicts, the CDRU 924 can determine up to 6 indices from another lookup that will not cause the current lookup conflict, and these 32 lookups can be performed—for example, 26 lookups from the original 32 and 6 lookups from another set sent by the IAU 922. Once the lookups are determined (e.g., with or without alternatives to resolve the conflict), the lookup set can be read from memory 904 using the LUT stream 918.

[0300] To account for out-of-order lookups that may occur during replacements, CDRU 924 can use CTL FIFO 928 to indicate the order of lookups for each set of lookups from IAU 922 to the data merging unit. For example, for an initial set of 32 lookups, the DCU can determine that 8 lookups were performed in the first cycle, then 8 in another cycle, then 16 in yet another cycle, and then determine that the entire set of 32 lookups has been processed. The 32 lookups can then be pushed to PPU 930 for post-processing, if applicable, or they can be pushed directly to OUT stream 920 to be written to the output buffer in memory 904. This additional information indicating the actual order of lookups is determined by CDRU 924 and read from the indexes in LUT stream 918, which can then be transmitted to DCU 932 via CTL FIFO 928. Therefore, whatever changes CDRU 924 makes to the order of the indexes received from IAU 922, DCU 932 can take into account. CTL FIFO 928 can be useful because the number of cycles through IAU 922, CDRU 924, etc., is indeterminate and data-dependent. For example, since collisions are not known in advance (e.g., because the data may be nondeterministic) and are a result of programming, there is no solution that completely avoids collisions. Therefore, CTL FIFO 928 helps instruct DCU 932 to organize the search as a result of collision resolution.

[0301] When needed—for example, when additional operations need to be performed on lookup table values—PPU 930 can compute a final value for each index that can be read into memory 904. If no post-processing is required, PPU 930 may not need to collect results. For example, in the case of performing a normal 1D or 2D lookup on an index of an integer value that directly maps to a location in the lookup table, PPU 930 and FRAC FIFO 926 may not be used to perform additional processing. When performing interpolation (e.g., linear on a 1D lookup or bilinear on a 2D lookup) and / or other operations, PPU 930 and FRAC FIFO 926 can be used to convert the collected results into updated results or values ​​for writing to memory 904.

[0302] In some embodiments, DLUT 906 can be used in table reformatting modes. For example, IDX stream 916 and OUT stream 920 can be used to update addresses for access and / or transposition. In such an example, where a buffer exists in memory 904 and the indexes in the buffer are to be transposed, this operation can be offloaded to DLUT 906 (instead of having the address generation unit of processor 902 perform the transposition). Configuration information from processor 902—e.g., from the address generation unit—can indicate a read mode for reading from the buffer in memory 904 and a write mode for writing addresses back to memory 904 in different modes. For example, if a programmer knows that many conflicts will be caused by a particular access mode, the programmer can program processor 902 to configure DLUT 906 to perform table reformatting to shuffle the data so that fewer or no conflicts may occur.

[0303] As another example, DLUT 906 can be used to detect out-of-range sentinel return values ​​or predict out-of-range output writes. Therefore, for example, if coordinates in IDX stream 916 are outside a given image patch and corresponding values ​​should not be written, DLUT 906 can write a sentinel value, which can then instruct processor 902 when to process information in the output buffer that is not dependent on or uses the sentinel value in the processing. In some embodiments, the sentinel value can instruct processor 902 that these values ​​should not be written to memory, so values ​​identified as error values ​​may not be stored.

[0304] Therefore, the DLUT 906 can be implemented as a pipeline of sub-units that work together to perform specific tasks or operations. Each sub-unit can operate independently and communicate with other sub-units through a shared interface. About Figure 9B Table 940 illustrates the tasks of various sub-units of the DLUT906 during the processing of a specific operation.

[0305] Due to the DLUT accelerator described herein, the processor pipeline can maintain determinism by offloading dynamic conflict detection and resolution to a decoupled accelerator. Furthermore, the accelerator can operate independently and concurrently with the host processor (e.g., VPU), thus reducing runtime. The DLUT accelerator also allows 1D and / or 2D lookups from a common table, with conflict detection / resolution. The accelerator can perform various post-processing operations, such as 1D lookups with linear interpolation, 2D lookups with bilinear interpolation, out-of-range detection sentinel returns (1D and 2D), and / or out-of-range prediction shut-off output writes (1D and 2D). The DLUT accelerator can be configured to perform interpolation using configurable fractional bits and can support various indexing and data formats, such as 8, 16, and 32-bit signed and unsigned data formats, and 16 and 32-bit 1D and 2D coordinate indexing formats. The DLUT accelerator can also perform transformations between global and local coordinates using configurable X / Y offsets. The DLUT accelerator can also support dataflow units reading index buffers from the VMEM, performing lookups from the VMEM, and writing results (or lookups or interpolations) to the VMEM. The dataflow units can support up to 2D addressing for linear and transpose access. To optimize the number of cycles required for lookups / interpolations, lookup indexes may be out of order to minimize memory set conflicts—for example, if the VMEM supports N lookups, the accelerator can use an MxN index to maximize survival in conflict detection—and duplicate detection can be performed to filter out duplicate indexes that are guaranteed to cause conflicts. Furthermore, the DLUT accelerator's 2D lookup and interpolation modes can include indexes automatically generated within the accelerator based on several parameters (called automatic indexing mode), which is the opposite of programmer-provided index data blocks. This offloads index preparation work from the main processor to the accelerator.

[0306] Now for reference Figure 9C Each block of the method 950 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 950 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 950 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, regarding… Figure 9A System 900 describes method 950, which can be performed by any system, structure or component or any combination of systems, structures or components, including but not limited to those described herein.

[0307] Figure 9CThis is a flowchart illustrating a method 950 for using a decoupled lookup table accelerator according to some embodiments of the present disclosure. In block B902, method 950 includes configuring one or more sub-units of a DLUT accelerator based at least in part on configuration information generated by a processor. For example, DLUT 906 can configure sub-units of DLUT 906 using information received from processor 902 and / or retrieved from memory 904.

[0308] In box B904, method 950 includes reading a first set of indexes from a memory stream and determining a first subset of indexes that do not have memory group conflicts. For example, IAU 922 can generate a set of indexes for CDRU 924 to handle conflicts, and CDRU 924 can determine a subset of that set of indexes that does not have memory group conflicts.

[0309] In block B906, method 950 includes determining a second subset of indexes from a second set of indexes read from memory that do not conflict with the first subset of indexes in memory group. For example, IAU 922 may generate another set of indexes for CDRU 924 to handle conflicts, and CDRU 924 may determine to replace one or more conflicting indexes from the first set with one or more indexes from the second set of indexes that do not cause conflicts with the first set of indexes.

[0310] In block B908, method 950 includes performing a lookup of one or more lookup tables to retrieve multiple values ​​using a first subset and a second subset of an index in a single read cycle from memory. For example, DLUT 906 can read values ​​from memory 904 into LUT stream 918 using a subset of values ​​from the index set and values ​​from a second index set that is determined not to conflict with the subset of values ​​from the first index set.

[0311] In block B910, method 950 includes writing multiple values ​​to memory. For example, values ​​from LUT stream 918 may be written to memory 904 from output stream 920. Before being written out, DCU 932 may reorganize the data such that the data is in a one-to-one order with the indexes read from the input buffer of IDX stream 916. In some embodiments, PPU 930 may perform one or more operations, such as interpolation, on the retrieved values ​​before writing the final values ​​to memory 904 in OUT stream 920.

[0312] Hardware sequencer for direct memory access systems

[0313] Direct Memory Access (DMA) systems can be used to move data from different memory locations without requiring a central processing unit (CPU). For example, DMA can operate as a data movement engine, moving data from a source to a destination—for example, from a source such as external memory (e.g., DRAM) or internal memory such as an L2 buffer or vector memory (VMEM) of a vector processing unit (VPU) to a destination, such as the VPU. In practice, DMA systems can perform additional operations, such as, but not limited to, filling frame data, manipulating addresses, managing overlapping data, managing different traversal orders, and taking into account different frame sizes.

[0314] In digital signal processing, multiple DMA resources can be used to describe the movement of structured tile data between external memory and a processor (e.g., a VPU). These DMA resources can include descriptors, channels, flip-flops, and / or registers. For example, a descriptor can describe tile movement such as source location, destination location, line spacing, tile width, tile height, circular buffer arrangement, etc. However, the movement of tile data on image surfaces with spatial and temporal dependencies presents additional programming challenges for the user and requires many different DMA configuration resources. These tile data dependencies can also complicate control code and control sequences in the processor (e.g., VPU) code. For example, a typical processing operation can include filtering, such as 3x3 filtering. This type of operation introduces spatial dependencies because each output pixel will depend on the corresponding values ​​of the 3x3 pixels surrounding it. In such an operation, filtering can be performed using a 3x3 value matrix, and this operation can be called a spatially dependent operation. In practice, each tile in a frame may have the same size—e.g., 64x64—to reduce programming challenges. However, if a 3x3 filter is used on a 64x64 tile, adjacent tiles will require additional pixels up and down—for example, as shown below. Figure 10C The shaded area is shown. Therefore, this information needs to be encoded in DMA resources to allow data to be retrieved correctly across tiles—which results in additional programming overhead.

[0315] refer to Figure 10A-10G , Figure 10A-10G This illustrates various challenges in data movement when using DMA systems. For example, Figure 10AVisualization 1000 can correspond to filled frame data. Visualization 1000 can have nine sections: top left, top, top right, left, center, right, bottom left, bottom, and bottom right. In such an example, each section can include one or more tiles—for example, the top left section can include one tile, while the top section can include, for example, four tiles. Therefore, to accurately define this segmentation, in existing methods, nine descriptors (e.g., one per section), three channels (e.g., one for the left column, one for the center column, and one for the right column), and three triggers (e.g., one per channel) can be used.

[0316] Regarding padding, for example, due to spatial dependencies, when operations are performed on data near the boundaries of tiles or portions of a frame, the DMA system can pad values ​​or create values ​​for pixels outside the image boundaries. This is likely because in some implementations, requesting data outside a memory region for an image might trigger a fault. Therefore, DMA can be used to pad or create values ​​after acquiring image data from the corresponding memory region to avoid triggering a fault. Without padding, the structure of the data might not match the kernel size, for example, if filtering operations are performed. The acquired data with additional padding values ​​can then be sent to a destination (e.g., a VPU) so that the VPU can process the data according to its configuration and can process the data in the same way across the entire (padded) frame. When padding, zero padding can be used (e.g., where each new data point includes a zero value), repeated values ​​can be used (e.g., copying pixel values ​​from adjacent pixels from the acquired data), and / or another padding mechanism can be used. Furthermore, padding can be added to any side of the frame, and different padding can be added for different sides. For example, in Figure 10A In this context, the padding region 1002 on the right side can be larger than the left, top, or bottom of the frame. Padding increases the complexity of DMA programming when moving data from a source to a destination (e.g., from memory to VMEM), and also increases the complexity of VPU programming when dealing with larger padding frames.

[0317] Now for reference Figure 10B , Figure 10BThe visualization 1010 corresponds to address operations in a DMA system. For example, different descriptor addresses can be manipulated and programmed to retrieve consecutive frame data. For DMA to execute effectively, the address descriptions for data movement can be consecutive. Therefore, the address of each descriptor can be manipulated, and this manipulation must be passed from one descriptor to another. For example, when filling values ​​as shown in the figure, the starting address of each descriptor can be manipulated to make the retrieved data include the fill value. To do this, the programmer uses the starting address, the tile width, and the number of tiles in each section, and uses this information to generate the next descriptor address. For example, the first descriptor could result in data retrieval starting from the top left corner, then the top, then the top right corner, then the left, then the center, and so on. Figure 10B As shown by the arrow in the diagram. However, the starting descriptor address increases the complexity of DMA programming when moving data to a destination (such as VMEM).

[0318] As another example, and regarding Figure 10C To ensure continuous data processing, a DMA system may be needed to read vertically and horizontally overlapping data from adjacent tiles. For example, such as... Figure 10C As shown in the shaded area, it may be necessary to read overlapping data from a tile in the top left portion and its adjacent tile in the top portion within the same operation. Similarly, it may be necessary to read overlapping data from a tile in the top left portion and its adjacent tile in the left portion within the same operation. To do this, descriptors need to be updated or moved to include the overlapping portions. For example, the base descriptor might include the address at the top start, but to capture data from the adjacent tile in the top left portion, the top descriptor needs to be updated (e.g., moved to the left) to capture data from the tile in the top left corner. This updating requires additional programming complexity, especially as the number of descriptors increases.

[0319] In addition, regarding Figure 10D-10F DMA systems may need to support different traversal orders to read data from memory sequentially. For example, the associated traversal order may differ depending on whether filtering, convolution, matrix multiplication, and / or other operations are performed. With this in mind, various traversal orders can be supported, such as... Figure 10D Those shown include raster traversal orders starting from the top left (visualization 1030), raster traversal orders starting from the top right (visualization 1032), raster traversal orders starting from the bottom left (visualization 1034), and / or raster traversal orders starting from the bottom right (visualization 1036). Similarly, regarding Figure 10E For the visualization of 1038 cube images, the DMA system can support various cube traversal orders. Figure 10FVarious vertical mining traversal orders that can be supported by a DMA system are illustrated, such as a top-left vertical mining traversal order (Visualization 1040), a top-right vertical mining traversal order (Visualization 1042), a bottom-left vertical mining traversal order (Visualization 1046), and / or a bottom-right vertical mining traversal order (Visualization 1048). Supporting each of these different traversal orders for moving data to memory (e.g., VMEM) increases the complexity of DMA programming.

[0320] about Figure 10G DMA systems may also need to support different frame sizes, such as moving multiple frames of different sizes (e.g., Luma / Chroma composites or different pyramid levels). For example, a processor (e.g., a VPU) can process frames of different sizes to generate the final desired output. Figure 10A The illustration 1048 shows an example visualization of pyramid processing corresponding to a frame used for optical flow estimation. In such an example, pixel movement might first compute a smaller frame size, then use hints from the output of the smaller frame size to compute a larger frame size, then use hints from the larger frame size to compute an even larger frame size, and so on. Therefore, DMA systems can support acquiring frame data of various frame sizes, but this capability requires additional programming complexity from the DMA system. For example, descriptors must be programmed or updated for each different frame size.

[0321] To simplify the programming of the various operations supported by the DMA system, the DMA system and method disclosed herein can use a hardware sequencer combined with a DMA engine to solve the data movement problem. For example, the data movement of a complete image can be explicitly and fully described in a hardware sequencing mode, which has a simplified programming model for handling tile sorting (triggering), filling, overlapping (offset), traversal order, and different frame sizes (e.g., the image structure of the frames, such as...). Figure 10I (As shown). A hardware sequencer can reduce DMA resource usage (e.g., reduce the number of required descriptors, triggers, channels, etc.), offload control used for VPU control processing from the VPU, and reduce the complexity of DMA programming. This can be achieved by loading image or frame descriptor views from local programmable memory in the form of a sequence of commands (e.g., as shown). Figure 10I This is accomplished as shown. These hardware sequencer commands can combine each operation that increases programming complexity, as described herein—including image filling, tile overlap or offset, frame offset, image traversal order, and image size at the tile granularity. In addition to descriptor information (e.g., from image commands or from separate descriptor memory or SRAM), the hardware sequencer can also read image commands from memory and sort tile movements to traverse and draw the entire frame.

[0322] Now for reference Figure 10H , Figure 10H A DMA system 1050 including a hardware sequencer is illustrated according to some embodiments of the present disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, system 1050 may be included and / or may include with… Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 The components, features, and / or functions of the example data center 1500 are similar to those of other components, features, and / or functions.

[0323] System 1050 may include a DMA engine 1056, register control 1058, hardware (HW) sequencer controller 1060, descriptor SRAM 1052, and / or hardware (HW) sequencer command SRAM 1054. Existing systems may include only the DMA engine 1056 and the descriptor SRAM 1052 storing frame descriptors. Therefore, as described herein, when sending data from a source to a destination, the DMA engine 1056 needs to perform all padding, address manipulation, and other operations beforehand, and requires a VPU or other source to perform ordering via a handshake with the DMA system (e.g., with the VPU as the master node and the DMA as the slave node). In such an example, the DMA engine 1056 will process at the tile level, using descriptors for the various portions of a frame, each portion comprising one or more tiles, to retrieve one tile at a time for transmission to the destination, and subsequent tiles will be retrieved based on descriptors, which are based on instructions from the VPU to retrieve the next tile.

[0324] However, using Figure 10H The System 1050 can process frames at the frame level—for example, a single descriptor can be used... Figure 10HThe frame shown previously required nine descriptors. Therefore, in practice, when the DMA engine 1056 attempts to load descriptors from the descriptor SRAM 1052 (or more generally, the descriptor memory 1052), the HW sequencer control 1060 can intercept the descriptor loading and use a command sequence processing structure to handle multiple frames, tile rows / columns, and multiple descriptors. For this purpose, frame format 1070 ( Figure 10I This describes higher-level frames by processing tile rows / columns (depending on the traversal order) in hardware rather than at the tile level. For example, instead of filling tiles, an entire frame can be filled using frame format 1070, allowing many frames to be filled with a single fill command. Therefore, the entire frame can be understood, such as where to fill, where to overlap, how to automatically manipulate addresses, etc. Furthermore, since the DMA engine 1056 can directly extract descriptors from descriptor SRAM 1052 without intervention from hardware sequencer control 1060, legacy formats are still supported for operations that may not benefit from hardware sequencer control 1060.

[0325] The HW sequencer control 1060 can operate, for example, as a state machine that reads the HW sequencer command SRAM 1054 (or more generally, the HW sequencer command memory 1054), which stores a frame format 1070 including the sequencer commands. A processing controller—such as an R5 processor, CPU, ARM processor, etc.—can program or configure the hardware sequencer command SRAM 1054 and descriptor SRAM 1052 using programming code and / or settings from a higher-level engine.

[0326] The descriptor SRAM 1054 may include one or more descriptors that can define tile dimensions (e.g., tile width dx and tile height dy), the starting point of the image or frame (e.g., top left, bottom right, etc.), trigger type, and / or other micro-information about the scan type of the descriptor.

[0327] The HW sequencer command SRAM 1054 can store frame format 1070, frame size, frame padding, etc., that define a frame as a whole. For example, frame format 1070 may include frame headers for header control, offset control, and padding control, and may include column headers and / or row headers for the columns or rows of the frame (e.g., column headers for vertical scan mode and row headers for raster scan mode). Frame header control may include a frame repetition factor to identify how many times a particular frame will be repeated, and the number of descriptor rows and / or descriptor columns. Frame header offset control may include frame tile offsets (e.g., tile-to-tile offsets) and frame offsets (e.g., offsets between two or more frames that can be read using a single channel, such as YUV frames that can be processed to include three separate planes). The frame padding header may indicate how many rows or pixels of padding to add at the frame level (as opposed to per tile level in existing methods), such as filling the left side, top, right side, and / or bottom of the frame, thus filling the entire frame instead of filling each tile within each section of the frame at the tile level.

[0328] Column headers can be used for vertical traversal orders, while row headers can be used for raster or horizontal traversal orders. Column and / or row headers may include column or row offsets (e.g., offsets between each column or row), column or row repetition factors (e.g., how many times the same column or row is repeated across frames, such as N-1 times, where N is the number of times the column or row is processed), and the number of descriptors used for each column or row (e.g., a single descriptor can be used to repeat the same tile across rows or columns, or the first descriptor can be used to traverse a portion of a row, while the second descriptor can be used to traverse another portion, and so on). A descriptor ID can be described such that a descriptor—e.g., stored in descriptor SRAM 1052—can be pulled out and used to describe a row or column. For example, a descriptor ID may indicate which descriptor is used for a particular column and / or row, and how many times the descriptor is repeated (e.g., N-1 times, where N is the total number of times the descriptor is used). In an embodiment, there may be a set of descriptors (e.g., 64), and descriptor IDs can be used to determine which descriptor should be used for a particular column and / or row. In this way, the hardware sequencer controller 1060 views the upper-level structure of frames located above the base descriptors from the descriptor SRAM 1052, which allows for a simplification of the resources required by the DMA engine 1056 to achieve the same data transfer. Furthermore, the hardware sequencer control 1060 can prefetch tiles in advance (e.g., using register control 1058) to reduce latency, and the tile data can be immediately available when requested by the DMA engine 1056.

[0329] In operation, the HW sequencer control 1060 can read the image structure (e.g., frame format 1070) from the HW sequencer command SRAM 1054 and descriptor information from the descriptor SRAM 1052, and can combine this information to sort frames of the DMA engine 1056. Therefore, the HW sequencer control 1060 can read the image structure, pull in descriptors, and sort frames of the DMA engine 1056 with the correct descriptor format, instead of requiring separate encoding of each descriptor, trigger, channel, etc., for the DMA engine 1056. In embodiments, register control 1058 can help control traversal order, prefetching, and / or other frame addressing control. The HW sequencer control 1060 further simplifies the VPU code, so the VPU does not have to consider multiple channels. Instead, the VPU can request one tile, then the next tile, then the next tile, and so on. HW sequencer control 1060 understands the current position in the frame, and therefore knows the next tile to be extracted for DMA engine 1056, and DMA engine 1056 does not have to keep track of this information internally.

[0330] System 1050 is therefore backward compatible with previous methods because it can still support the use of various descriptors, triggers, channels, etc., but can also be understood at the frame level to reduce complexity. System 1050 can support image fill at all corners of frames with different pixel fill sizes, vertical and / or horizontal overlapping tiles to allow the VPU to access adjacent tiles for processing along tile boundaries, and traversing frames in different traversal orders. Furthermore, System 1050 can support automatic tile offset adjustment at the VMEM destination by the hardware sequencer control 1060. Because descriptors in a frame are hardware-linked, the user does not need to link or concatenate descriptors together. The hardware sequencer control 1060 can manage the address ordering of descriptors / tiles across frames without additional programming complexity, and the hardware sequencer control 1060 can prefetch tiles to improve performance.

[0331] In some embodiments, the descriptor may be included in the image or frame structure, rather than stored separately in the descriptor SRAM 1052. For example, without implementing legacy compatibility, the entire sorting structure and tile structure can be described within the frame structure. In such an example, Figure 10I The frame format can be used to include additional information about the descriptor, such as tile width, trigger type, etc., so that the same information is available to the HW sequencer control 1060 as when the descriptor is stored separately in the descriptor SRAM 1052.

[0332] refer to Figure 10J , Figure 10J This refers to the implementation of some embodiments of this disclosure for a raster scanning sequence. Figure 10IAn example of frame format 1070. For example, frame format 1070A is an example of a frame format in raster mode with frame address processing, using a single channel, a single trigger, and a single descriptor. In this example, the tile structure can be 16x8. Figure 10K This is an example of a tile structure with hardware ordering in a raster scan sequence according to some embodiments of the present disclosure, wherein frame address processing is performed using example frame format 1070A. For example, for each tile row, the same descriptor (e.g., tile dimension) (as indicated by “D1” in visualization 1072) can be used so that the same tiles are applied 16 times along each row (from C1 to C16), and then repeated for 8 rows from top to bottom (from R1 to R8). The sequence may include 20 bytes, as shown in frame format 1070A, and each row may have N*2+ bytes, where N represents the number of entries per row (e.g., ...). Figure 10J (As shown). Therefore, in order to sort frames as shown in visualization 1072, frame format 1070A may include frame padding of 3 rows of pixels (PL, PR, PT, PB) with no frame repetition, zero descriptor rows, no tile offset, no frame offset, and repeating 7 times (8 rows in total). The offset of each row may be the tile height (Ty) (so that each row is offset by the tile height). A descriptor may be used with descriptor ID D1, and the descriptor may be repeated 15 times in each row (16 times in total). Therefore, in practice, the HW sequencer control can use the descriptor corresponding to D1 (which includes the tile height and tile width) from descriptor SRAM 1052, and can use the image structure of frame format 1072 stored in HW sequencer control SRAM 1054 to sort image tiles piece by piece (16 tiles per row) and row by row (from R1 to R8) for the target processor (e.g., VPU). This allows the use of a single descriptor, a single trigger, and a single channel, thereby reducing programming complexity while also allowing the DMA system 1050 to become the primary or control component in the interaction between the DMA system 1050 and the VPU.

[0333] In some embodiments, as an extension of the HW sequencer control 1060, a DMA-triggered mode can be used to reduce software intervention in VPU programming by causing the DMA system 1050 command descriptors to be sequenced. For example, the DMA system 1050 can read an image from external memory, tile the image, and perform sequential processing on tiles for the VPU. To facilitate this, the VPU can expose start and finish signals. VPU startup can be driven by the DMA system 1050, and the VPU can send a finish signal to the DMA system 1050 when it has finished processing the instruction block. Thus, the DMA system 1050 (e.g., the hardware sequencer control 1060) and the VPU can participate in a handshake mechanism in which the DMA system 1050 is the primary node and the VPU is the secondary node. This DMA-triggered mode can minimize VPU tile control overhead and simplify the programming model of the DMA engine 1056. For example, specific code for double-buffered DMA data movement may not be required, and the DMA kernel code can be independent of the VPU kernel code. Therefore, DMA-triggered mode simplifies the VPU code because the DMA system uses the HW sequencer to control the 1060's tile sorting. The example code below illustrates the VPU code before and after DMA-triggered addition.

[0334]

[0335]

[0336] As a result, where previously the VPU had been requesting the tiles to be moved to the VMEM, now, because the HW sequencer control 1060 controls the ordering, the DMA system 1050 can trigger the movement of the tiles to both the VMEM and the VPU as targets. In this way, the DMA system 1050 can obtain the data to be processed by the VPU in advance, and when the VPU indicates that processing is complete, the DMA system 1050 can make the next data to be processed immediately available (e.g., in the VMEM) and can indicate the same content to the VPU.

[0337] When processing one or more frames, the HW sequencer control 1060 can retrieve one or more descriptors (which may indicate tile size, trigger type, etc.) from the descriptor SRAM 1052 and can retrieve the image structure from the hardware sequencer command SRAM 1054. The HW sequencer command 1060—in conjunction with register control 1058—can then begin traversing the first row or column based on the traversal order and using the first (and only in this embodiment) descriptor, and can then move to the second descriptor based on the number of repetitions (e.g., 1-N) encountered when using two or more descriptors, and so on. When each tile is determined, the DMA engine 1056 can retrieve tile data from the source data and write the tile data to the destination data (e.g., in VMEM). Once the data is written to the data destination, the processor (e.g., VPU) can be notified by the hardware sequencer control 1060 that the data is available for the processor to begin processing. Then, during processing, the DMA system 1050 can obtain the next data tile based on the sequence from the hardware sequencer control 1060 and write the data to the data destination, so that when the processor indicates that processing is complete, the hardware sequencer control 1060 can indicate to the VPU (via a handshake mechanism) that the next data to be processed is available, and so on, until processing is complete.

[0338] Now for reference Figure 10L Each block of the method 1080 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 1080 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 1080 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, method 1080 is about... Figure 10H The system described herein, method 1080, can be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0339] Figure 10L This is a flowchart of a method 1080 of a DMA system including a hardware sequencer according to some embodiments of the present disclosure. In block B1002, method 1080 includes retrieving a tile structure from a descriptor memory and retrieving a frame structure corresponding to a frame from a hardware sequencer command memory. For example, hardware sequencer control 1060 may retrieve descriptors from descriptor SRAM 1052.

[0340] In box B1004, method 1080 includes sorting the retrieval of tiles from a frame in source memory. For example, hardware sequencer control 1060—in combination with register control 1058 in an embodiment—can sort the retrieval of tiles from source memory by DMA engine 1056 based on the frame (or image) structure and tile descriptions from descriptors.

[0341] In block B1006, method 1080 includes writing retrieved data corresponding to a tile into a destination memory. For example, DMA engine 1056 may write the retrieved data corresponding to a tile into a target memory (e.g., VMEM) for processing by a target processor (e.g., VPU).

[0342] In box B1008, method 1080 includes providing a processor associated with the destination memory with an indication that the retrieved data is stored in the destination memory. For example, HW sequencer control 1060 may indicate to the processor that the data for the next tile is ready for processing.

[0343] In block B1010, method 1080 includes receiving an indication that processing of the retrieved data is complete. For example, after processing is complete, the processor may indicate to the DMA system 1050 that processing is complete, at which point the next data block may be loaded (or may have been preloaded) into the destination memory, and the DMA system 1050 may indicate the same to the processor.

[0344] Configure a DMA system for region-related data movement using a VPU.

[0345] When a known data pattern is acquired, the processing controller can configure a Direct Memory Access (DMA) system, and the processor (e.g., a Vector Processing Unit (VPU)) can trigger the DMA and sort it. However, when processing different data points or features of irregular or unknown data patterns, challenges may arise in reconfiguring data movement because feature or object locations are dynamically computed. For example, object tracking algorithms, feature tracking algorithms, object detection algorithms, deep learning algorithms using variable-sized regions of interest (ROIs), and / or other region-dependent data movement algorithms require dynamically adjusting address and data pairs so that the DMA system can retrieve the appropriate information for the processor (e.g., the VPU) to process. In traditional systems, when an unknown data pattern is acquired—e.g., in object tracking—the processing controller (e.g., an R5 processor core for controlling a programmable vision accelerator (PVA)) may need to interrupt the processing cycle to determine the updated information computed by the processor (e.g., the VPU) and reconfigure the DMA for the next iteration. Therefore, the processing controller introduces additional latency to, for example, tracking algorithms, requiring shorter response times.

[0346] To address the drawbacks of conventional systems requiring processor intervention, the systems and methods of this disclosure utilize a DMA and a processor (e.g., a VPU) to configure a tightly coupled processing loop that allows the DMA to reconfigure its descriptors based on the processor's output. Thus, the DMA can be dynamically reprogrammed at runtime to handle certain algorithms requiring region-dependent data movement. This VPU configuration mode can be used to update the DMA's descriptors to compute tracking feature data (including location) based on the runtime VPU. Therefore, the VPU can specify a list of address and data pairs in memory (e.g., VMEM) and then trigger the DMA to update its own descriptors to collect data from regions with the new computed addresses. By relying on the interface between the VPU and DMA, intervention from the processing controller (e.g., an R5 or ARM processing core) may not be necessary once the processing controller has initially configured the VPU and DMA to begin processing. This batch, fast, and synchronous MMIO access for updating feature descriptors thus reduces latency for object tracking, feature tracking, object detection, deep learning, and / or other algorithms with region-dependent data movement.

[0347] Now for reference Figure 11A , Figure 11A A data flow diagram 1100 is shown for configuring a Direct Memory Access (DMA) system using a Vector Processing Unit (VPU) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, process 1100 may be comprised of components including... Figures 13A-13D Exemplary autonomous vehicle 1300, Figure 14 Example computing device 1400 and / or Figure 15 The example data center 1500 has similar components, features, and / or functions to the system implementation.

[0348] The system executing process 1100 may include a processing controller 1102 (e.g., an R5 processor, an ARM processing core, an instruction set architecture (ISA), an x86 architecture, etc.), a direct memory access (DMA) system 1104, a vector processing unit (VPU) 1108 (or another processor type), a vector memory (VMEM) 1110 (or another memory type), and a descriptor RAM 1106. In practice, the VPU configuration mode can configure DMA to acquire descriptors by writing a series of non-contiguous address / data pairs to the DMA descriptor SRAM. Process 1100 can be described in terms of example features or object tracking algorithms. However, this is not intended to be limiting, and process 1100 and the underlying system can be used to execute any type of algorithm, such as those with region-dependent data movement.

[0349] For example, the first operation may include processing controller 1102 configuring DMA 1104 and VPU 1108 to perform processing on some data, and then triggering both DMA 1104 and VPU 1108 to perform processing. For example, processing controller 1102 may load descriptor RAM 1106 into memory for the starting point of processing, and may configure VPU 1108 registers for specific types of operations that VPU 1108 will perform on the data.

[0350] For the second operation, VPU 1108 can trigger DMA 1104 to read initial feature data points from VMEM 1110. For example, in order to start working, VPU 1108 needs data from DMA 1104, so VPU 1108 configures DMA 1104 to load data points into VMEM 1110 at the point where VPU 1108 knows to retrieve data for processing.

[0351] In the third operation, the VPU 1108 can process the current feature dataset and calculate the next tracked object or feature location. As a result, the VPU 1108 may now have calculated the new or updated location of the tracked feature or object.

[0352] In the fourth operation, the VPU 1108 can use the VPU configuration format (see...). Figure 11B (Description) Update VMEM 1110 with the updated location, then DMA 1104 can be triggered to update its descriptor in descriptor RAM 1106. For example, Figure 11B Table 1120 illustrates the VPU configuration format written by the VPU to the vector memory (VMEM) and read by the DMA system according to some embodiments of this disclosure. For example, for each address / data pair, the format may include four bytes of address and four bytes of data.

[0353] In the fifth operation, DMA 1104 may update the descriptors in descriptor RAM 1106 to retrieve appropriate data for the next processing iteration of VPU 1108. For example, DMA 1104 may read address / data pairs to form a VPU configuration format to patch the operation descriptor with the updated location. In embodiments, a one-to-one correspondence may exist between feature points and descriptors, such that each tracked feature, object, or point may include an associated descriptor. Thus, the address / data pair of each tracked feature, object, or point can be updated over time using a separate descriptor.

[0354] In the sixth operation, DMA 1104 can use the newly updated descriptor in descriptor RAM 1106 to obtain new feature data of the location. For example, DMA 1104 can indicate to VPU 1108 that the descriptor has been updated, and VPU 1108 can trigger DMA 1104 to read the new data into VMEM 1110, and so on.

[0355] As a result, after processing the controller's initial configuration operation, operations two through six can be repeated to form a tightly synchronized VPU configuration loop that requires controller intervention—thus reducing latency to address the short response times required by tracking or detection algorithms. Furthermore, because the DMA 1104 is overwriting addresses in memory with new update addresses, the DMA 1104 is updating code that it needs to look at to determine what to fetch next. By doing so, throughput is increased compared to traditional systems that rely on the control bus to update registers with addresses and data. Therefore, the benefits of defining an address / data protocol are realized, where variable address locations with variable amounts of data can be updated and how address / data pairs are updated. This allows the DMA 1104—whose width may be greater than the width of the control bus (e.g., 512 bits versus 32 bits, respectively)—to update (e.g., but not limited to) eight address / data pairs at a time (where each address / data pair is defined using eight bytes, such as...). Figure 11B (As shown).

[0356] Furthermore, although DMA is shown as being updated using the VPU configuration mode of process 1100, additional or replacement elements or components of the system can be updated. For example, the instruction cache of VPU 1108 can be updated using a similar method with the VPU. As another example, an updated hardware sequencer program can be written to update the hardware sequencer memory by providing address data. This would essentially involve writing a new program for the hardware sequencer RAM—for example, for... Figure 10H The hardware sequencer RAM 1054 of the hardware sequencer controller 1060.

[0357] Now for reference Figure 11C Each block of method 1150 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 1150 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 1150 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, regarding… Figure 11A The system describes method 1150, which can be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0358] Figure 11C This is a flowchart of a method 1150 for configuring a DMA system using a VPU according to some embodiments of the present disclosure. In block B1102, method 1150 includes using a processor and at least in part based on first data written to memory using the DMA system to calculate a first output corresponding to one or more first update positions of a tracked feature. For example, VPU 1108 can access data from VMEM 1110 written to VMEM 1110 using DMA 1104, and can process the data to calculate one or more object positions corresponding to tracked features, objects, points, etc.

[0359] In box B1104, method 1150 includes using a processor to update the memory to include second data representing one or more address / data pairs corresponding to one or more first update locations. For example, after calculating one or more locations, VPU 1108 can update VMEM 1110 with address / data pairs in a format such as regarding Figure 11B The format of the description.

[0360] In block B1106, method 1150 includes using a DMA system and updating one or more descriptors corresponding to the tracked feature based at least in part on the one or more address / data pairs. For example, DMA 1104 may access address / data pairs from VMEM 1110 and use the address / data pairs to update descriptors in descriptor RAM 1106 for the next read operation.

[0361] In block B1108, method 1150 includes writing third data to memory using a DMA system and based at least in part on the one or more descriptors. For example, DMA 1104 can write updated data from an address / data pair corresponding to an address identified using a descriptor to VMEM 1110.

[0362] In box B1110, method 1150 includes using a processor and at least partially based on third data to compute a second output corresponding to one or more second update locations of the tracked features. For example, once the updated data is in VMEM 1110, VPU 1108 can compute the next set of updated address / data pairs corresponding to the tracked features, objects, points, etc., and this process can be repeated until processing is complete.

[0363] Permanent Fault Detection in Programmable Vision Accelerators (PVA)

[0364] In safety-critical applications such as autonomous and semi-autonomous machine applications, there are stringent requirements for the detection and isolation of persistent faults. For example, when executing deep learning, computer vision, sensor processing, and / or other applications within a machine, persistent fault detection must be performed periodically within the allocated time budget to ensure accurate testing while also allowing the application to execute correctly—for example, with low latency. Regarding Automotive Safety Integrity Level (ASIL) D, applications executing in autonomous or semi-autonomous machines may require 90% or more coverage of persistent faults. This necessitates end-to-end coverage with low latency while meeting the runtime budget for each specific application. Traditional approaches use Built-in Self-Test (BIST) to identify faults, but these BIST techniques either lack sufficient coverage, introduce excessive latency into the system, and / or fail to meet the runtime budget for some applications.

[0365] To address the shortcomings of these conventional methods, this system and method can perform a Multiple-Input Signature Register (MISR) BIST—for example, performing fault detection on a Programmable Vision Accelerator (PVA) of a System-on-Chip (SoC). For example, in various embodiments of this disclosure, the PVA may include one or more DMA systems and one or more VPUs, controlled by one or more processing controllers (or control processors) (e.g., R5 and ARM processors, CPUs, and / or similar devices). Therefore, each component of the PVA may require testing, and this system and method perform MISR BIST to detect persistent faults in an end-to-end manner. In this way, persistent fault detection can be performed to cover end-to-end blocks of control and data logic, errors can be reported directly to a security processor to reduce latency, and can be tailored to specific applications to meet associated runtime budgets.

[0366] In various embodiments, the MISR can be used in a PVA to implement the software logic BIST for permanent fault detection. MISR hardware (as discussed here) Figure 12A(and / or 12B) may include Cyclic Redundancy Check (CRC) hardware, which is initialized (e.g., using a known seed value) using the processing controller. When executing a PVA application, the processing controller may allocate a portion of the timing budget (e.g., approximately 10% or less) to run a known software MISR test with known inputs having a deterministic pre-computed output with a correct signature or gold value. For example, in the case where the timing budget corresponds to 30 frames per second, a timing budget corresponding to 3 or fewer frames may be allocated to the MISR test. At the allocated time, the processing controller may initiate the MISR test and wait for the test to complete to terminate the MISR CRC calculation. Once the test is complete, the MISR hardware may read back the final CRC value and check it against the pre-computed gold value. In the event of a mismatch, the MISR hardware may report the error directly to the SoC's security processor for further action to handle the security error—e.g., causing the application's output to be ignored, resolving or fixing a permanent fault, etc.

[0367] Therefore, the MISR hardware within the DMA block can monitor one or more (e.g., all in the embodiment) transactions on the PVA's Advanced Extensible Interface (AXI) master port. By examining all output stages from the PVA, in the embodiment, the security integrity of the PVA can be checked against permanent defects (e.g., output information) that could corrupt the output stages and might be consumed by the PVA and / or another engine during application execution. The MISR hardware can thus detect errors across different blocks of the PVA (e.g., the processing controller, VPU, and DMA system) because these components collaborate and interact when producing output stages. The signatures computed in the MISR hardware can represent the state of these different PVA blocks during MISR testing.

[0368] In an embodiment, the MISR scheme may include CRC checks on both the write address (e.g., 40-bit control) and the write data (e.g., 512-bit data) leaving the AXI master port. This feature allows for the isolation of control path failures (e.g., addressing errors) from data path failures (e.g., calculation errors). Due to the configuration of the MISR hardware (as described herein), each DMAAXI port may be able to be checked. In an embodiment, control bits may be used to disable writes to the address and data outputs of all channels involved in the MISR calculation, thereby saving bandwidth consumption in the memory subsystem and during memory allocation. Furthermore, the MISR scheme may include control register bits for each channel to exclude or mask specific channels from the MISR calculation—e.g., to isolate insecure channels. In an embodiment, the DMA may use IEEE 802 and MPEG CRC-32 primitive polynomials to calculate the MISR CRC: X 32 +X 26 +X 23 +X 22 +X 16 +X 12 +X 11 +X 10 +X 8 +X 7 +X 5 +X 4 +X 2 +X+1. The MISR SET register can be used to set the initial CRC value (e.g., seed value) for address and data CRC calculations. The MISR REF register can be used to compare the CRC values ​​calculated for address and data.

[0369] To support MISR with 512-bit data, 8:1 bit compression can be applied—for example, each data byte can be compressed into 1 data bit using an 8>1 XOR operation to form 2x32-bit message data. To support MISR 40-bit addresses, the 9 most significant bits can be compressed—for example, the 9>1 XOR operation can be used to compress the 9 most significant bits to form a 32-bit message address. Variations in test patterns and instructions can be used to cover aliasing associated with compression. The likelihood of aliasing is likely low because error failures do not produce address CRC errors when an even number of errors exist in a byte on the output image. Furthermore, aliasing is likely unlikely because a reference CRC can be calculated on output images with the same pattern at the same even number of error bit positions throughout the MISR test. In experiments, aliasing was shown to cause an average coverage loss of 0.25%. In the embodiment, data compression with such low aliasing is valuable due to the bus width (e.g., 512 bits), and without compression, MISR testing may not meet the system's latency or runtime budget.

[0370] The MISR timer register can be used to time out the MISR calculation, and the MISR timer register decrements on each AXI clock cycle. The timeout function can be helpful if a fault causes the MISR test to hang, potentially preventing the MISR hardware from reporting errors. When the MISR test is complete, the processing controller can use a software event to stop the MISR calculation. The DMA system can compare the MISR REF value with the MISR VAL value of the data and address outputs of the MISR test, and the DMA hardware can update the MISR status register based on the comparison result. For example, the MISR status register can include one of the following values: 0: Idle; 1: Complete: Failed data; 3: Busy; 4: Complete: Both address and data failed; 5: Complete: Failed timeout; 6: RSVD; and 7: Complete: Passed. In the event of a MISR timeout error, the DMA can generate a timeout signal to the security processor, and in the event of a CRC check error in the data and / or address, the DMA can assert a security error to the security processor.

[0371] refer to Figure 12A , Figure 12A This is a system diagram of a built-in self-test (BIST) for performing cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the MISR hardware 1250 may include... Figure 10H DMA system 1050 Figures 13A-13D Example autonomous vehicles 1300 Figure 14 Example computing device 1400, and / or Figure 15 Similar components, features, and / or functions to the example data center 1500. For example, MISR hardware 1200 can be included in the DMA block of the PVA, such as... Figure 10H As shown. In this way, the MISR hardware 1200 can operate at the output stage of data movement, and addressing (or control) can be accessed from the output of the DMA engine 1056. For example... Figure 12AAs shown, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used depending on the embodiment.

[0372] In operation, the processing controller can control the DMA system 1050 and MISR hardware 1200—as well as one or more processing components of the system, such as VPUs. When performing MISR tests on the DMA system 1050, in embodiments, the test code may include sequences of all 0s, all 1s, alternating 0s and 1s, and / or random codes. In this way, high coverage of the DMA system 1050 can be achieved. For example, when testing the VPU, the test code may include application-specific or custom code. For example, during coverage testing for a specific application, the components or portions of the VPU used (e.g., registers, logic, etc.) can be determined, and test code can be generated so that those specific components or portions of the VPU are included in the execution of the test code. For example, random data with different instructions can be included in the test code to order the tests by different instructions to use different regions of the VPU logic. In this way, the overall coverage of the VPU is increased, particularly the coverage of the specific application executed on the VPU. By performing DMA and VPU tests in this manner, and because the processing controller is involved in the control and interaction between various components (e.g., the DMA system 1050 and the VPU), the processing controller can have high coverage, as the output and addressing of data movement are affected by the interactions of the processing controller.

[0373] During testing, when using different code patterns, the code patterns can be used alternately, or one code can be used for the first time frame (e.g., equivalent to 30fps), another code for the second time frame (e.g., equivalent to 30fps), another code for the third time frame (e.g., equivalent to 30fps), and so on. For example, in the DMA code example, the code 0 can be used for the first time frame, then the code 1 for the second time frame, then alternating between 0 and 1 codes (e.g., 0101010101...) for the third time frame, then a random code for the fourth time frame (e.g., 011100100010...), and then these four codes may repeat, and so on.

[0374] In practice, when testing DMA, for example, the processing controller can interact with the MISR controller 1206 to write set reference values ​​to the MISR dataset register 1210 and the MISR address set register 1216. These values ​​may differ for data and addresses and may be referred to as seed values ​​for CRC calculations. The processing controller can then initialize the channels in the DMA engine 1056 that perform data movements, and because the location of the test code in memory is known to the processing controller, a descriptor (e.g., configured by the processing controller in descriptor SRAM) 1052 can be used to sort the data through the MISR test for the DMA engine 1056. The processing controller can set a timer 1226 on the MISR hardware 1200 to enable the MISR test, and can then trigger the channels of the DMA engine 1056 to begin reading test data from the source destination and outputting the data to the MISR hardware 1200 for the MISR test. Therefore, when testing DMA, data movement is being tested (e.g., correct addressing and correct data in the addressed location), so the MISR hardware 1200 can access the output of the DMA engine 1056 while the DMA engine is performing data movements according to the test code. This tap that enters the output stage can... Figure 12AThe instruction, acting as external memory, can be funneled according to the processing controller's sequence (one data channel at a time, one address channel at a time). For example, for data channels, the processing controller can sort them by, for example, each of 16 data channels, and the corresponding AXI write data (wdata) for each channel can be fed through CH0-CH16 data CRC calculation 1202—e.g., concatenated. For example, the processing controller can configure the channel output register 1220 to pass through the channels one at a time according to the configuration order from the processing controller. In an embodiment, the channel mask register 1208 (e.g., programmed by the MISR controller 1206 based on interaction with the processing controller) can be configured by the processing controller to mask or remove various channels (e.g., channels not under test) from the CRC calculation. In an embodiment, an AND gate can be used to perform this masking. In the case of masking one or more channels, the gold value in the MISR data reference register 1222 (which can be provided by the processing controller to the MISR controller 1206) may only correspond to the CRC calculation of the unmasked channels. For each unmasked channel, the data on the channel (generated using test code read from memory) can be applied (e.g., compressed or uncompressed) to a polynomial in CRC data calculation 1202 to generate the MISR data value 1214 for that channel. Once the channel has completed its calculation, the processing controller can receive an instruction and may send the next data channel to CRC calculation 1202 to calculate the next MISR data value 1214, and so on, until each unmasked channel has a corresponding MISR data value 1214. Once each MISR data value 1214 for a particular iteration has been calculated, these values ​​1214 can be combined to generate a final MISR data value, which can be compared with a golden value in MISR data reference register 1222 to generate a MISR data state determination (e.g., which may include states corresponding to the values ​​0-7 mentioned above).

[0375] As another example, for address channels, the processing controller can sort, for example, each of 16 address or control channels, and feed the corresponding AXI write address (waddress) for each channel via CH0-CH16 CRC calculation 1204—e.g., concatenated. In an embodiment, channel mask register 1208 can be configured by the processing controller to mask or remove various channels from the CRC calculation—e.g., channels not under test. In an embodiment, this masking can be performed using an AND gate. In the case of masking one or more channels, the gold value in MISR data reference register 1224 may correspond only to the CRC calculation of the unmasked channels. For each unmasked channel, the address on the channel (generated using test code read from memory) can be applied (e.g., compressed or uncompressed) to a polynomial of CRC address calculation 1204 to generate the MISR address value 1218 for that channel. Once a channel has completed its calculation, the processing controller can receive an instruction and send the address data for the next channel to CRC calculation 1204 to calculate the next MISR address value 1218, and so on, until each unmasked channel has a corresponding MISR address value 1218. Once each MISR address value 1218 for a specific iteration has been calculated, these values ​​1218 can be combined to generate a final address MISR value, which can be compared with the golden value in the MISR reference register 1224 address to generate a MISR address state determination (e.g., which may include states corresponding to the values ​​0-7 mentioned above).

[0376] In some embodiments, MISR testing can be iterative, allowing the processing of first code, testing of the output, and then using the output for the next testable iteration, and so on. In such embodiments, MISR testing can include multiple phases, and a completed MISR test can include executing each phase.

[0377] When the MISR hardware 1200 is specifically used to test the VPU, for example, the DMA system 1050 can move test code into the VMEM, the VPU can process the test code and write the results back to the VMEM, and the DMA engine 1056 can read the results from the VMEM back to the destination location. When writing the results back to the destination location, the MISR hardware 1200 can access the DMA output and perform MISR on the data (e.g., including data and addresses), and perform a MISR similar to that discussed herein. In this way, the MISR hardware 1200 can be used to test the interaction between the VPU and the test code.

[0378] After completing the MISR test, the processing controller can receive an interrupt. For example, the processing controller can receive a completion interrupt and, if there is no error, can wait for the next MISR test cycle. In the case of an error interrupt, the type of error can be determined—e.g., failed data, failed address, both failed, etc.—and a security error can be asserted to the security processor. For example, in some embodiments where the MISR hardware 1200 is suspended or idle (e.g., with a timeout error), the DMA can generate a timeout signal to the SoC's security processor.

[0379] In some embodiments, to accelerate MISR calculations to compute CRCs on one or more channels (e.g., such as 16 in an embodiment), without serializing or hierarchically categorizing channel-MISR calculations, the channels can be demultiplexed based on the channel IDs present in the AXIID field to perform parallel channel calculations. For example, since CRC calculations are performed at different rates, therefore Figure 12A The method involves serial processing of one channel after another. However, using Figure 12B In systems like this, as described below, these calculations can be performed in parallel. For example, when the processing controller terminates the MISR calculation, the MISR controller can sort all channel outputs to calculate a final signature, which can be compared to reference or golden values ​​for the address and data outputs. This feature can accelerate the detection of permanent faults without requiring an additional programmer register interface—for example, because the same control registers can be used for all channels.

[0380] Similarly, and refer to Figure 12B , Figure 12B This is a system diagram of a built-in self-test (BIST) for parallel channel cyclic redundancy check (CRC) calculation in a programmable vision accelerator (PVA) according to some embodiments of this disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In some embodiments, the MISR hardware 1250 may include... Figure 10H DMA system 1050 Figures 13A-13D Example autonomous vehicles 1300 Figure 14 Example computing device 1400 and / or Figure 15Similar components, features, and / or functions to the example data center 1500. For example, MISR hardware 1250 can be included in the DMA block of the PVA, such as... Figure 10H As shown. In this way, the MISR hardware 1250 can operate at the output stage of data movement, and addressing (or control) can be accessed from the output of the DMA engine 1056. For example... Figure 12A As shown, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used depending on the embodiment.

[0381] MISR Hardware 1250 can be used with Figure 12A The MISR hardware 1200 operates similarly. In addition to the MISR hardware 1250 being configurable for parallel data channel and parallel address channel CRC calculations, the processing controller can, for example, configure the MISR dataset register 1256 to set a seed or reference value for each data CRC calculation 1260A-1260N (corresponding to AXI data channels 0-15 respectively), and can configure the MISR address set register 1258 to set a seed or reference value for each address CRC calculation 1262A-1262N (corresponding to AXI address channels 0-15 respectively). The processing controller, similar to... Figure 12A As described, the DMA system 1050 can then be triggered to move data (e.g., for DMA testing) and / or VPU processing (e.g., for VPU-specific testing), and the MISR hardware 1250 can be connected to the output stage for testing.

[0382] Therefore, the processing controller can send data from 16 channels to multiplexer (mux) 1252 and address data from 16 channels to multiplexer (mux) 1254. mux 1252 can then provide the data for the corresponding channel to the corresponding CRC calculations 1260A-1260N (e.g., channel 0 AXI data to channel 0 CRC calculation 1260, channel 1 data to channel 1 CRC calculation 1260B, etc.), and each CRC calculation 1260 can use this data along with a CRC polynomial with a reference value to calculate MISR data values ​​1284A-1284N (e.g., channel 0 CRC calculation 1260A can calculate MISR data 0 value 1284A, channel 1 CRC calculation 1260B can calculate MISR data 1 value 1284B, etc.). MISR data values ​​1284A-1284N can then be sorted from the multiplexer (mux) 1264 according to the MISR sequence from MISR control 1270 configured by the processing controller. In an embodiment, for example regarding... Figure 12AAs described, one or more channels may not be included in a particular MISR test, therefore the channel mask register 1268 can be configured by the processing controller to update the MISR sequence such that the MISR data values ​​1284 corresponding to one or more masked channels are not provided to the channel 0-16 data CRC calculation 1274 for calculating the final CRC value. For unmasked channels, the multiplexer 1264 can output the MISR data values ​​1284 according to the MISR sequence. In this way, taking into account the different channels and the different calculation times of the CRC calculation 1260, the MISR data values ​​1284 are forced to be output according to the MISR sequence, rather than being sent to the CRC calculation 1274 according to the timing of the CRC calculation being completed. Once the MISR sequence of the MISR data values ​​1284 is output by the multiplexer 1264 to the CRC calculation 1274, the CRC calculation 1274 can calculate the final CRC value and store the final CRC value in the VAL register 1276. The final CRC value in VAL register 1276 can then be compared with the gold value in MISR data reference register 1272 (configured by MISR control 1270 from the processing controller) to determine the MISR data status.

[0383] Similarly, the processing controller can send 16 address channels to a multiplexer (mux) 1254, which can then provide the corresponding address channels to the corresponding CRC calculations 1262A-1262N (e.g., channel 0 AXI address to channel 0 CRC calculation 1262, channel 1 address to channel 1 CRC calculation 1262B, etc.). Each CRC calculation 1262 can use the address and a CRC polynomial with a reference value to calculate the MISR address values ​​1286A-1286N (e.g., channel 0 CRC calculation 1262A can calculate MISR address 0 value 1286A, channel 1 CRC calculation 1262B can calculate MISR address 1 value 1286B, etc.). The MISR address values ​​1286A-1286N can then be sorted from the multiplexer (mux) 1266 according to the MISR sequence from the MISR control 1270, as configured by the processing controller. In an embodiment, for example regarding... Figure 12AAs described, one or more channels may not be included in a particular MISR test, therefore the channel mask register 1268 can be configured by the processing controller to update the MISR sequence such that the MISR address value 1286 corresponding to one or more masked channels is not provided to the channel 0-16 address CRC calculation 1280 for calculating the final CRC value. For unmasked channels, the MISR address value 1286 can be output by the multiplexer 1266 according to the MISR sequence. In this way, taking into account the different calculation times of different channels and CRC calculation 1262, the MISR address value 1286 is forced to be output according to the MISR sequence, rather than being sent to the CRC calculation 1280 according to the timing of the CRC calculation being completed. Once the multiplexer 1266 outputs the MISR sequence of the MISR address value 1286 to the CRC calculation 1280, the CRC calculation 1280 can calculate the final CRC value and store the final CRC value in the VAL register 1282. The final CRC value in VAL can then be compared with the gold value in MISR address reference register 1278 (configured by MISR control 1270 from the processing controller) to determine the MISR address status.

[0384] MISR data status and MISR address status can be similar to those described above. Figure 12A Use the description to check and use.

[0385] Now refer to the diagram as follows Figure 12C As shown, each block of method 1290 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. Method 1290 can also be embodied as computer-usable instructions stored on a computer storage medium. Method 1290 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service) or plug-in to another product, to name a few. Furthermore, method 1290 is about... Figure 12A The system described herein, method 1290, can be performed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0386] Figure 12C This is a flowchart of a BIST (Bound Execution of Information) method 1290 for permanent fault detection in a PVA according to some embodiments of this disclosure. In block B1202, method 1290 includes receiving multiple data channels from a DMA system one channel at a time and based on a sequence determined by the processing controller. For example, MISR hardware 1200 may receive one data channel (or one address data channel) at a time, depending on the order determined by the processing controller.

[0387] In box B1204, method 1290 includes calculating multiple MISR values ​​by performing a CRC calculation to calculate a MISR value using a polynomial of CRC calculation and corresponding data corresponding to that channel. For example, for each channel, CRC calculation 1202 (or address 1204) can use data (or address) from the channel and CRC calculation polynomial 1202 to calculate a MISR data value 1214 (or MISR address value 1216 of the address) (starting from a seed value in CRC MISR dataset register 1210 or MISR address set register 1216).

[0388] In box B1206, method 1290 includes calculating a final MISR value using multiple MISR values. For example, MISR data values ​​1214 from each channel (or MISR address values ​​from each channel) can be combined to generate the final MISR value.

[0389] In box B1208, method 1290 includes comparing the final MISR value with a signature value. For example, the final MISR value generated from the individual MISR values ​​1214 (or address value 1216) can be compared with the signature or gold value of the MISR data reference register 1222 (or the MISR address reference register 1224 of the address).

[0390] In block B1210, method 1290 includes outputting a MISR state based at least in part on a comparison. For example, based on the comparison in block B1208, a state—e.g., failed data, failed address, both failed, completed, etc.—can be determined, and this state can be used to notify the SoC's security processor of the error state.

[0391] Example autonomous vehicles

[0392] Figure 13AThis is an illustration of an example autonomous vehicle 1300 according to some embodiments of the present disclosure. The autonomous vehicle 1300 (or, alternatively, referred to herein as “vehicle 1300”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, drones, vehicles coupled to trailers, and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by a branch of the U.S. Department of Transportation—the National Highway Traffic Safety Administration (NHTSA)—and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of that standard). Vehicle 1300 may be able to perform one or more functions that meet the requirements of Level 3 to Level 5 autonomous driving. For example, depending on the embodiment, vehicle 1300 may be able to perform conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0393] Vehicle 1300 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1300 may include a propulsion system 1350, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1350 may be connected to the drivetrain of vehicle 1300, which may include a transmission, to enable propulsion of vehicle 1300. Propulsion system 1350 may be controlled in response to receiving a signal from throttle / accelerator 1352.

[0394] A steering system 1354, which may include a steering wheel, can be used to steer the vehicle 1300 (e.g., along a desired path or route) when the propulsion system 1350 is operating (e.g., when the vehicle is in motion). The steering system 1354 may receive signals from the steering actuator 1356. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0395] The brake sensor system 1346 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1348 and / or the brake sensor.

[0396] It can include one or more System-on-Chip (SoC) 1304 ( Figure 13C One or more controllers 1336, including and / or one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 1300. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1348, to operate steering system 1354 via one or more steering actuators 1356, and to operate propulsion system 1350 via one or more throttles / accelerators 1352. One or more controllers 1336 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1300. One or more controllers 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for functional safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1336 can handle two or more of the functions described above, and two or more controllers 1336 can handle a single function, and / or any combination thereof.

[0397] One or more controllers 1336 may provide signals for controlling one or more components and / or systems of vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, global navigation satellite system sensors 1358 (e.g., GPS sensors), RADAR sensors 1360, ultrasonic sensors 1362, LIDAR sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-angle cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1398, speed sensors 1344 (e.g., for measuring the rate of vehicle 1300), vibration sensors 1342, steering sensors 1340, braking sensors (e.g., as part of braking sensor system 1346), and / or other sensor types.

[0398] One or more of the controllers 1336 may receive inputs (e.g., represented by input data) from the instrument cluster 1332 of the vehicle 1300 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1334, an auditory signaling device, a speaker, and / or via other components of the vehicle 1300. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 13C Information such as the HD map 1322, location data (e.g., the location of vehicle 1300 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by the controller 1336, etc. For example, the HMI display 1334 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0399] The vehicle 1300 also includes a network interface 1324, which can communicate via one or more networks using one or more wireless antennas 1326 and / or a modem. For example, the network interface 1324 may be able to communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. One or more wireless antennas 1326 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0400] Figure 13B For use in accordance with some embodiments of this disclosure Figure 13A This is an example of the camera position and field of view of an example autonomous vehicle 1300. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 1300.

[0401] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 1300. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0402] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0403] One or more of the cameras can be mounted in mounting components such as custom-designed (3-D printed) parts to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0404] A camera with a field of view that includes the environment in front of the vehicle 1300 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1336 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0405] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 1370, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 13B The middle image shows only one wide-angle camera, but any number of wide-angle cameras 1370 can be present on the vehicle 1300. Furthermore, a remote camera 1398 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 1398 can also be used for object detection and classification, as well as basic object tracking.

[0406] One or more stereo cameras 1368 may also be included in a front-mounted configuration. The stereo camera 1368 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1368 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 may be used in addition to those described herein, or alternatively.

[0407] A camera (e.g., a side-view camera) having a field of view that includes the side of the vehicle 1300 can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, a surround camera 1374 (e.g., ...) Figure 13B The four surround cameras 1374 shown can be mounted on the vehicle 1300. The surround cameras 1374 can include a wide-angle camera 1370, a fisheye camera, a 360-degree camera, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1374 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0408] A camera with a field of view that includes the rear portion of the environment of the vehicle 1300 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front cameras as described herein (e.g., long-range and / or mid-range camera 1398, stereo camera 1368, infrared camera 1372, etc.).

[0409] Figure 13C For use in accordance with some embodiments of this disclosure Figure 13A The example autonomous vehicle 1300 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented in hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0410] Figure 13C Each component, feature, and system in vehicle 1300 is illustrated as being connected via bus 1302. Bus 1302 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 1300 used to assist in the control of various features and functions of vehicle 1300, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. CAN bus may be ASIL B compliant.

[0411] Although bus 1302 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 1302 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1302 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1302 may be used for a collision avoidance function, and a second bus 1302 may be used for drive control. In any example, each bus 1302 may communicate with any component of vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304, each controller 1336, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1300) and may be connected to a common bus such as the CAN bus.

[0412] Vehicle 1300 may include one or more controllers 1336, such as those described herein. Figure 13A The controllers described herein. Controller 1336 can be used for a wide variety of functions. Controller 1336 can be coupled to any other different components and systems of vehicle 1300 and can be used for the control of vehicle 1300, artificial intelligence of vehicle 1300, infotainment and / or the like for vehicle 1300.

[0413] Vehicle 1300 may include one or more System-on-Chip (SoC) 1304. SoC 1304 may include CPU 1306, GPU 1308, processor 1310, cache 1312, accelerator 1314, data storage 1316, and / or other components and features not shown. SoC 1304 can be used to control vehicle 1300 across a wide variety of platforms and systems. For example, one or more SoCs 1304 may be combined with an HD map 1322 in a system (e.g., the system of vehicle 1300), the HD map being accessible from one or more servers (e.g., via a network interface 1324). Figure 13D One or more servers (1378) receive map refresh and / or updates.

[0414] CPU 1306 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). CPU 1306 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1306 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1306 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1306 can be active at any given time.

[0415] The CPU 1306 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 1306 can further implement enhanced algorithms for managing power states, wherein allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0416] GPU 1308 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 1308 may be programmable and efficient for parallel workloads. In some examples, GPU 1308 may use an enhanced tensor instruction set. GPU 1308 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1308 may include at least eight streaming microprocessors. GPU 1308 may use a computation application programming interface (API). Furthermore, GPU 1308 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0417] In automotive and embedded applications, the GPU 1308 can be power-optimized for optimal performance. For example, the GPU 1308 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1308 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations to provide efficient execution of workloads. Streaming microprocessors may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors may include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0418] The GPU 1308 may include, in some examples, a high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem providing peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, synchronous graphics random access memory (SGRAM), such as fifth-generation graphics double data rate synchronous random access memory (GDDR5), may be used.

[0419] The GPU 1308 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 1308 to directly access the CPU 1306 page tables. In such an example, when the GPU 1308 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 1306. In response, the CPU 1306 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 1308. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1306 and the GPU 1308, simplifying GPU 1308 programming and porting applications to the GPU 1308.

[0420] In addition, the GPU 1308 may include access counters that track how frequently the GPU 1308 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0421] SoC 1304 may include any number of caches 1312, including those described herein. For example, cache 1312 may include an L3 cache available to both CPU 1306 and GPU 1308 (e.g., it is connected to both CPU 1306 and GPU 1308). Cache 1312 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0422] SoC 1304 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed on vehicle 1300, such as processing a DNN. Furthermore, SoC 1304 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 1304 may include one or more FPUs integrated as execution units within CPU 1306 and / or GPU 1308.

[0423] SoC 1304 may include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 1304 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 1308 and offload some tasks from GPU 1308 (e.g., freeing up more cycles of GPU 1308 to perform other tasks). As an example, accelerator 1314 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0424] Accelerator 1314 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU can perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0425] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0426] The DLA can perform any function of the GPU 1308, and by using inference accelerators, for example, designers can target either the DLA or the GPU 1308 for any function. For instance, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 1308 and / or other accelerators 1314.

[0427] Accelerator 1314 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0428] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0429] DMA enables PVA components to access system memory independently of the CPU 1306. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0430] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., a VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0431] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. Consequently, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a hardware accelerator cluster, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0432] Accelerator 1314 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1314. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks accessible by both PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, controller, and multiplexer. Any type of memory may be used. PVA and DLA may access memory via a backbone that provides high-speed memory access to PVA and DLA. The backbone may include (e.g., using APB) an on-chip computer vision network interconnecting PVA and DLA to memory.

[0433] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0434] In some examples, the SoC 1304 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0435] Accelerators 1314 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0436] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0437] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0438] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run neural networks to regress the confidence values. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g. from another subsystem), outputs from inertial measurement unit (IMU) sensors 1366 related to the vehicle's 1300 azimuth and distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1364 or RADAR sensor 1360), etc.

[0439] SoC 1304 may include one or more data storage units 1316 (e.g., memory). Data storage units 1316 may be on-chip memory of SoC 1304, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, data storage units 1316 may be large enough to store multiple instances of the neural network. Data storage units 1312 may include L2 or L3 cache 1312. References to data storage units 1316 may include references to memory associated with PVA, DLA, and / or other accelerators 1314 as described herein.

[0440] SoC 1304 may include one or more processors 1310 (e.g., embedded processors). Processor 1310 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 1304 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 1304 thermal and temperature sensor management, and / or SoC 1304 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 1304 may use the ring oscillator to detect the temperature of CPU 1306, GPU 1308, and / or accelerator 1314. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 1304 into a lower power state and / or place vehicle 1300 into a driver-safe parking mode (e.g., safely stop vehicle 1300).

[0441] The processor 1310 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0442] The processor 1310 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0443] The processor 1310 may also include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0444] The processor 1310 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0445] The processor 1310 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0446] Processor 1310 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 1370, the surround camera 1374, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0447] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0448] The video image compositer can also be configured to perform stereo correction on input stereo lens frames. When the operating system desktop is in use and the GPU 1308 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 1308 is powered on and activated, performing 3D rendering, the video image compositer can be used to offload the GPU 1308 to improve performance and responsiveness.

[0449] The SoC 1304 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 1304 may also include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0450] SoC 1304 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 1304 can be used to process data from cameras and sensors (e.g., LIDAR sensor 1364, RADAR sensor 1360, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 1302 (e.g., vehicle 1300 speed, steering wheel position, etc.), and data from GNSS sensor 1358 (connected via Ethernet or CAN bus). SoC 1304 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 1306 from routine data management tasks.

[0451] The SoC 1304 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 1304 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 1306, GPU 1308, and data storage 1316, the accelerator 1314 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0452] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0453] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1320) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0454] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with a light can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network that informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, informing the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 1308.

[0455] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1300. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 1404 provides security against theft and / or carjacking.

[0456] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 1396 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 1304 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to recognize the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 1358. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 1362, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0457] The vehicle may include a CPU 1318 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1304 via a high-speed interconnect (e.g., PCIe). The CPU 1318 may include, for example, an x86 processor. The CPU 1318 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 1304, and / or monitoring the status and health of the controller 1336 and / or the infotainment SoC 1330.

[0458] Vehicle 1300 may include a GPU 1320 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 1304 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1320 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 1300.

[0459] Vehicle 1300 may also include a network interface 1324, which may include one or more wireless antennas 1326 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 1324 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 1378 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 1300 with information about vehicles approaching vehicle 1300 (e.g., vehicles in front, to the side, and / or behind vehicle 1300). This functionality may be part of vehicle 1300's cooperative adaptive cruise control function.

[0460] Network interface 1324 may include a SoC that provides modulation and demodulation functions and enables controller 1336 to communicate via a wireless network. Network interface 1324 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, a...

Claims

1. A processor, comprising processing circuitry, the processing circuitry being configured to: The table is copied to memory to include a first value at a first physical address in a first memory group and a second value at a second physical address in a second memory group, the first value and the second value being included in the same column in the logical memory view of the table; Receive a read instruction corresponding to a multi-point lookup operation, the read instruction including at least a first index corresponding to a first physical address in the first memory group and an indication associated with a second physical address in the second memory group; Based at least on the read instruction, the first value located at the first physical address and the second value located at the second physical address are read in one cycle; and Use the first value and the second value to perform one or more operations.

2. The processor as claimed in claim 1, wherein: When the multi-point lookup includes a two-point lookup, the table is copied 0.5*N times, where N is the number of values ​​to be read in a given period of the multi-point lookup.

3. The processor of claim 1, wherein, When the multi-point lookup includes a 2x2 point lookup, the table is copied 0.25*M times, where M is the number of values ​​to be read in a given period of the 2x2 point lookup.

4. The processor of claim 1, wherein the one or more operations include an interpolation operation.

5. The processor of claim 1, wherein the processor is a vector processing unit (VPU) and the memory is a vector memory (VMEM).

6. The processor of claim 1, wherein the processor is included in a programmable vision accelerator of a system-on-a-chip (SoC).

7. The processor of claim 1, wherein the processor comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; A system used to perform deep learning operations; System-on-a-Chip (SoC); Systems including programmable vision accelerators (PVAs); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system that merges one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

8. A system comprising: Memory; as well as A processor, comprising processing circuitry, the processing circuitry being used to: An instance of the table is copied to memory such that a first value of the table instance is located at a first physical address in a first memory group, and a second value of the table instance is located at a second physical address in a second memory group, the first value and the second value being included in the same column in the logical memory view of the table; Determine the first index corresponding to the first physical address in memory; Based at least in part on the read instruction corresponding to multi-point lookup and the first index, the first value located at the first physical address and the second value located at the second physical address are read; and Use the first value and the second value to perform one or more operations.

9. The system of claim 8, wherein: When the multi-point lookup includes a two-point lookup, the table is copied 0.5*N times, where N is the number of values ​​to be read in a given period of the two-point lookup.

10. The system of claim 8, wherein, When the multi-point lookup includes a 2x2 point lookup, the table is copied 0.25*M times, where M is the number of values ​​to be read in a given period of the 2x2 point lookup.

11. The system of claim 8, wherein one or more operations include an interpolation operation.

12. The system of claim 8, wherein the processor is a vector processing unit (VPU) and the memory is a vector memory (VMEM).

13. The system of claim 8, wherein the processor is included in a programmable vision accelerator of a system-on-a-chip (SoC).

14. The system of claim 8, wherein the system comprises at least one of the following: Control systems for autonomous or semi-autonomous machines; Sensing systems for autonomous or semi-autonomous machines; A system used to perform simulation operations; A system used to perform deep learning operations; System-on-a-Chip (SoC); Systems including programmable vision accelerators (PVAs); A system including a vision processing unit; Systems implemented using edge devices; Systems implemented using robots; A system that merges one or more virtual machines (VMs); A system that is at least partially implemented in a data center; or A system that utilizes cloud computing resources at least in part.

15. A method comprising: An instance of the table is copied to memory to include a first portion of the table instance in a first memory group and a second portion of the table instance in a second memory group, the first portion and the second portion being included in the same column in the logical memory view of the table; Determine the first index corresponding to the first portion of the instance of the table; Based at least on the read instruction corresponding to a multi-point lookup and the first index, a first value associated with a first portion of an instance of the table is read from a first physical address in the first memory group, and a second value associated with a second portion of an instance of the table is read from a second physical address in the second memory group; and Use the first value and the second value to perform one or more operations.

16. The method of claim 15, wherein: When the multi-point lookup includes a two-point lookup, the table is copied 0.5*N times, where N is the number of values ​​to be read in a given period of the multi-point lookup.

17. The method of claim 15, wherein, When the multi-point lookup includes a 2x2 point lookup, the table is copied 0.25*M times, where M is the number of values ​​to be read in a given period of the 2x2 point lookup.

18. The method of claim 15, wherein the one or more operations include an interpolation operation.

19. The method of claim 15, wherein the method is performed using a vector processing unit (VPU) and the memory is a vector memory (VMEM).

20. The method of claim 15, wherein the method is performed using the vector processing unit (VPU) of the programmable vision accelerator of the system-on-chip (SoC).

Images