In a system-on-a-chip, performing multi-point table lookups in a single cycle.

Abstract

To provide a VPU and associated components that are optimized to improve VPU performance and throughput.SOLUTION: A VPU includes a min / max collector, an automatic store predication function, a SIMD data path configuration allowing inter-lane sharing, a transposed load / store including a stride parameter function, a load including a permute and zero insertion function, a hardware, logic device, and memory layout function allowing two point and two by two point lookups, and per memory bank load caching ability. Decoupled accelerators are used to offload VPU processing tasks, and a hardware sequencer is included in a DMA system. The DMA and VPU execute a VPU configuration mode that allows the VPU and DMA to operate without a processing controller for executing dynamic region-based data movement operations.SELECTED DRAWING: Figure 9A

Description

[Background technology]

[0001] A vector processing unit (VPU) is used to perform single-instruction multiple data (SIMD) operations in parallel. Common applications of VPUs include operations such as image processing, computer vision, signal processing, deep learning (e.g., convolution operations), and / or similar.

[0002] In some computer vision applications, for example, the dynamic range of intermediate values ​​is well understood. Therefore, a calculated value may be compared to this dynamic range to detect anomalies. However, conventional solutions for detecting and analyzing this minimum and maximum value involve writing all values ​​to memory and then performing analysis of the values ​​in memory, requiring additional processing cycles. Furthermore, to achieve high throughput despite latency from load to use, high-clock-rate processors may be made to perform software pipelines and / or loop unrolling. However, if the original number of iterations is not divisible by the unrolling factor, some iterations may remain after the completion of the unrolled loop, requiring additional extra loops to calculate the value in the final iteration. These extra loops increase the system's code size and latency, for example, because they cannot be unrolled to achieve optimal performance. In conventional single-instruction multiple data (SIMD) arithmetic, each SIMD unit can perform operations independently of each other in parallel on the data lanes of the SIMD unit itself. While some architectures may allow sharing between adjacent objects, this limited sharing is restrictive, and in many cases, the same operand needs to be copied to each data lane for processing. Additionally, in vector SIMD processors, all memory read operations may need to use standard or consistent units, such as equal to the vector processing width, which can be inefficient when memory banks are wide. For example, reading elements 4 through 67 with a memory width of 64 bytes might require two memory reads: one for 0 through 63 and another for 64 through 67. However, this would mean reading many additional values, such as values ​​0 through 3 and 68 through 127, even if these values ​​are not needed for the current operation. In traditional instruction sets, if additional data manipulation is required, the data can be read and stored in registers, and then additional instructions can be used to perform operations on the memory data in the registers.For example, this might require loading data, performing substitutions on the data, and then performing calculations using the reconstructed data. Therefore, data manipulation requires additional cycles, increasing latency. When performing table lookups using an existing VPU, tables may be duplicated so that each single value can be retrieved from the duplicated table, or additional read ports may be added to each memory bank so that multiple values ​​can be read from the same table within the same bank. However, duplicating a table per value requires additional memory and processing, and adding further read ports requires additional space on the chip. In traditional VPUs, a data cache may not be implemented because the VPU is programmed to run a smaller set of highly optimized code, allowing the programmer to manage the contents of local data memory. However, doing so requires reading values ​​from each memory bank with each access, even if the data for the next iteration contains overlap with one or more previously read calculations.

[0003] To optimize the performance of processors such as VPUs, the instruction set architecture (ISA) can be extended to create custom instructions that speed up commonly performed operations such as table lookups, convolutions, and / or similar operations. However, using the ISA in this way requires the processor itself to perform these operations, which means the processor becomes busy while these extended instructions are being executed.

[0004] In addition, the VPU can use a direct memory access (DMA) system to retrieve data to be processed on the VPU. Therefore, while the DMA system can act as a data movement engine, it can also perform additional operations such as image padding, address manipulation, overlapping data management, scan order management, frame size management, and / or similar. However, the programming complexity for programming the DMA system and VPU increases as the number of DMA resources, such as descriptors, channels, and triggers, increases. Dynamic updates of DMA resources become a processing load on the system when frame tiles contain spatial or temporal dependencies. In traditional DMA systems, when fetching unknown or data-dependent data, a processing controller (e.g., an R5 or ARM processing core) was required to intervene in the processing cycle to determine the updated information that would direct the next processing iteration. For example, in object or feature tracking, the VPU could calculate the next location of the object or feature, then the processing controller would intervene to update the memory addressing information, and then the DMA system would be triggered to use the updated information. However, the intervention of the processing controller increases latency, and computations using region-dependent data movement algorithms require more complex programming.

[0005] Furthermore, in safety-critical applications such as autonomous and semi-autonomous machine applications, there are stringent requirements for detecting and isolating permanent faults. For example, when running deep learning, computer vision, sensor processing, and / or other applications on machines, permanent fault detection must be performed regularly and within an allocated time budget, not only to enable accurate testing but also to allow the application to run properly, for example, with low latency. To do this, 100 percent coverage may be required, while meeting the time budget for each specific application to run with low latency. Traditional methods use built-in self-tests (BISTs) to identify faults, but these BIST techniques often have insufficient coverage, introduce excessive latency to the system, and / or fail to meet the time budget for running specific applications. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent Application No. 15 / 141,703 [Patent Document 2] U.S. Patent Application No. 16 / 101,232 [Overview of the project] [Means for solving the problem]

[0007] Embodiments of this disclosure relate to a vector processing unit (VPU), a decoupled accelerator that may be used to handle processing offloaded from the VPU, and improvements to a direct memory access (DMA) system that supports data movement between memory and the VPU. Considering the various shortcomings of conventional or existing solutions, the VPU of this disclosure may include a minimum / maximum hardware collector that is included in the data path from the VPU to memory, thereby allowing the minimum / maximum values ​​to be stored before they are stored in memory. In this way, the minimum / maximum values ​​are available as soon as the memory write operation is complete, thereby reducing the latency caused by determining the minimum / maximum after the values ​​have been stored in memory. In addition, the VPU may include an automatic predication feature that allows the use of a predication flag by setting a predication bit for each value computed in iterations beyond the final iteration. As a result, each set of iterations may contain the same number of iterations performed, but thanks to the predication flag, one or more values ​​from the final set of iterations may not be written to memory. Given the limitations of existing solutions in sharing data lanes, the SIMD architecture of this disclosure can define slices within the processor, each containing multiple lanes, and each lane can be configured to communicate with one another. Thus, an operand from one lane can be used by another lane, eliminating the need to copy each operand to each lane for processing. Given the inefficiency of loading from a single wide memory bank, the VPU may include multiple smaller memory banks that allow for smaller bit alignment, for example, 16-bit alignment if each memory bank is 16 bits. In this way, an example of reading values ​​from 4 to 67 can be performed with only one memory read instead of two memory reads, 0-63 and 64-127.In addition to this memory bank configuration, the VPU may include transposed load and / or store functions that can offset values ​​stored within a memory bank so that bank contention does not occur and more data can be read or written per cycle. Considering the shortcomings of data manipulation with conventional instruction sets, loads containing sort instructions can be used to send the sort pattern along with the memory address to local memory in order to retrieve data from memory according to sort or data manipulation patterns. Thus, data manipulation and data loading can be performed in the same cycle, thereby reducing latency. To address the drawbacks of duplicating tables for each value or adding read ports for table lookups, 2-point or 2x2-point lookups may be performed so that 2-point or 4-point lookups can be performed per table and per cycle, respectively. To achieve this, parallel 2-point or 4-point lookups may be possible using table offset storage patterns, per-memory bank address buses, and associated logical mechanisms and routing. In embodiments, each memory bank may include an associated data cache that can be enabled or disabled depending on a given operation. For example, in filtering operations where there is a lot of data overlap between iterations, a data cache can be used to store values ​​from one or more previous lookups so that only minimal reads are requested from each bank, thereby saving energy and power to the system.

[0008] To address the shortcomings of conventional ISAs in VPUs or other processor types, the systems and methods of the present disclosure may utilize decoupled accelerators configured by and communicating with the VPU via shared memory, but capable of performing specific tasks independently of the VPU, allowing the VPU to continue processing other tasks in parallel with the accelerators. For example, a decoupled lookup table (DLUT) accelerator can be used to improve system performance when performing table lookups. In this way, instead of the VPU performing line-by-line contention detection and resolution of memory banks, the DLUT accelerator can identify and avoid contentions, thereby improving system throughput.

[0009] In light of the shortcomings of conventional DMA systems, the systems and methods of this disclosure may include a hardware sequencer that performs operations on frame data, including a set of commands for the hardware sequencer. For example, the hardware sequencer can perform operations at the frame level rather than the tile level, executing a sequence for the DMA engine and eliminating the programming complexity of programming the DMA engine to perform the same operations, such as padding and addressing. In some embodiments, the DMA system may include a DMA trigger mode in which the DMA engine controls the movement of tiles to vector memory (VMEM) rather than requesting the VPU to trigger DMA to load the next tile. Thus, the command sequence is reversed, and DMA triggers the VPU. In light of the shortcomings of region-dependent data movement operations in DMA systems, the DMA system can operate in a tightly coupled loop using DMA and the VPU without requiring the intervention of a processing controller. For example, the VPU can update location information in VMEM for various features and / or objects being tracked, and the DMA can use this updated information to update descriptors in descriptor memory so that the next data supplied to the VPU for processing corresponds to the next location of the feature or object. This process is repeated until processing is complete, which can eliminate the need for intervention from the processing controller and reduce system latency.

[0010] In addition, considering the shortcomings of conventional methods for BIST, the System and Method can perform BIST in a multiple input signature register (MISR) to perform fault detection in, for example, a programmable vision accelerator (PVA) of a system-on-a-chip (SoC). For example, in various embodiments of the Disclosure, the PVA may include one or more DMA systems and one or more VPUs controlled using one or more processing controllers (or control processors), such as an R5 processor, ARM processor, CPU, and / or similar. Thus, each component of the PVA may require testing, and the System and Method perform BIST in the MISR to detect 100 percent of permanent faults. In this way, permanent fault detection can be performed to cover 100 percent of both control logic and data logic blocks, report errors directly to the safety processor to reduce latency, and customize specific applications to meet the associated runtime budget.

[0011] The system and method for improving the vector processing unit (VPU) are described in detail below with reference to the following figures in the attached drawings. [Brief explanation of the drawing]

[0012] [Figure 1A] This is a diagram illustrating an exemplary minimum / maximum collection system according to some embodiments of the present disclosure. [Figure 1B] This flowchart illustrates minimum / maximum collection methods according to some embodiments of the present disclosure. [Figure 2A] This is a diagram illustrating an exemplary system, including a processor having an address generation unit with an automatic predication function, according to some embodiments of the present disclosure. [Figure 2B] This table shows the sequence of state changes over time according to some embodiments of the present disclosure. [Figure 2C] A flowchart showing a method of automatic store prediction according to some embodiments of the present disclosure. [Figure 3A] An explanatory diagram of an exemplary single instruction multiple data (SIMD) data path configuration according to some embodiments of the present disclosure. [Figure 3B] A diagram showing the sharing of operands between slices of a SIMD architecture for filter processing operations according to some embodiments of the present disclosure. [Figure 3C] A diagram showing the sharing of operands between slices of a SIMD architecture for dot product operations according to some embodiments of the present disclosure. [Figure 3D] A diagram showing the sharing of operands between slices of a SIMD architecture for an operation of sorting a payload according to some embodiments of the present disclosure. [Figure 3E] A flowchart showing a method of calculating an output using an operand shared across lanes of a SIMD architecture according to some embodiments of the present disclosure. [Figure 4A] A logic diagram of a transpose load for reading from and writing to memory, and a diagram of a memory bank of the transpose load corresponding to the logic diagram according to some embodiments of the present disclosure. [Figure 4B] A logic diagram of a transpose load having various line pitch and stride parameters for reading from and writing to memory, and a diagram of a memory bank of the transpose load corresponding to the logic diagram according to some embodiments of the present disclosure. [Figure 4C] A flowchart showing a method of setting a write operation of a transpose load having a stride parameter according to some embodiments of the present disclosure. [Figure 4D] A flowchart showing a method of executing a write operation of a transpose load having a stride parameter according to some embodiments of the present disclosure. [Figure 5A]A table of data and coefficient layouts for various functions in a SIMD architecture, according to some embodiments of the present disclosure. [Figure 5B] A table of data and coefficient layouts for various functions in a SIMD architecture, according to some embodiments of the present disclosure. [Figure 5C] A diagram showing a hardware architecture for performing a load including permutation and zero insertion, according to some embodiments of the present disclosure. [Figure 5D] A diagram showing an exemplary usage of the hardware architecture of FIG. 5C, according to some embodiments of the present disclosure. [Figure 5E] A flowchart showing a method of load including permutation, according to some embodiments of the present disclosure. [Figure 6A] A diagram showing a 16-way parallel table configuration for 1-point lookup, according to some embodiments of the present disclosure. [Figure 6B] A diagram showing an 8-way parallel table configuration for 2-point lookup, according to some embodiments of the present disclosure. [Figure 6C] A logic diagram of a 2-way parallel word-type table for 2×2-point lookup, according to some embodiments of the present disclosure. [Figure 6D] A memory diagram of a 2-way parallel word-type table for 2×2-point lookup of FIG. 6C, according to some embodiments of the present disclosure. [Figure 6E] A diagram showing a layout for processing a pair of lanes using horizontal blending including interleaving operation of data, according to some embodiments of the present disclosure. [Figure 6F] A diagram showing intermediate and final results of horizontal blending including interleaving operation of data, according to some embodiments of the present disclosure. [Figure 6G] A flowchart showing a method of performing multi-point lookup, according to some embodiments of the present disclosure. [Figure 7A]This figure shows the elements of the array of data and coefficients according to some embodiments of the present disclosure. [Figure 7B] This figure shows the read operations required for a data operand using a data cache for a memory bank, according to some embodiments of the present disclosure. [Figure 7C] This figure shows the read operations required for coefficient operands using a data cache for a memory bank, according to some embodiments of the present disclosure. [Figure 7D] This figure shows the configuration of memory banks used with a load cache in some embodiments of the present disclosure. [Figure 7E] This figure shows a hardware architecture for using a memory bank data cache according to some embodiments of the present disclosure. [Figure 7F] This is a flowchart illustrating a method for using a data cache for a memory bank, according to some embodiments of the present disclosure. [Figure 8A] This figure shows a system comprising one or more separate accelerators according to some embodiments of the present disclosure. [Figure 8B] This is a flowchart illustrating a method of using a separate accelerator to perform one or more operations, according to some embodiments of the present disclosure. [Figure 9A] This figure shows a system including an accelerator for a separable lookup table, according to some embodiments of the present disclosure. [Figure 9B] This table shows the actions taken by various components of a separate lookup table accelerator when performing various operations according to some embodiments of the present disclosure. [Figure 9C] This is a flowchart illustrating a method of using a separate lookup table accelerator to perform one or more operations, according to some embodiments of the present disclosure. [Figure 10A] This is a visualized diagram illustrating how to pad a frame using a padding value, according to some embodiments of the present disclosure. [Figure 10B]This is a visualized diagram illustrating address operations on frame descriptors according to some embodiments of the present disclosure. [Figure 10C] This is a visualized diagram showing data overlapping between frame tiles, according to some embodiments of the present disclosure. [Figure 10D] This is a visualized diagram showing various raster scan sequences according to some embodiments of the present disclosure. [Figure 10E] This is a visualized diagram showing the scanning sequence of a cube according to some embodiments of the present disclosure. [Figure 10F] This is a visualized diagram showing various vertical mining scan sequences according to some embodiments of the present disclosure. [Figure 10G] This is a visualized diagram showing various image sizes in a pyramidal configuration according to some embodiments of the present disclosure. [Figure 10H] This is a diagram of a direct memory access (DMA) system including a hardware sequencer, according to some embodiments of the present disclosure. [Figure 10I] Figure 10H shows a frame format for storing sequencing commands for hardware sequencer control of a DMA system, according to some embodiments of the present disclosure. [Figure 10J] Figure 10I shows an example of a frame format for a raster scan sequence according to some embodiments of the present disclosure. [Figure 10K] Figure 10J shows an exemplary tile structure that processes frame addresses using the exemplary frame format of the present disclosure, which is sequenced in hardware during a raster scan sequence according to some embodiments of the present disclosure. [Figure 10L] This is a flowchart illustrating a method for using a hardware sequencer in a DMA system according to some embodiments of the present disclosure. [Figure 11A]This is a data flow diagram relating to the process of configuring a direct memory access (DMA) system using a vector processing unit (VPU) according to some embodiments of the present disclosure. [Figure 11B] This table shows the VPU configuration format, which is written to vector memory (VMEM) by the VPU and read by the DMA system, according to some embodiments of the present disclosure. [Figure 11C] This is a flowchart illustrating how to configure a DMA system using a VPU according to some embodiments of the present disclosure. [Figure 12A] This figure shows an internal self-test (BIST) system that performs cyclic redundancy check (CRC) calculations for a programmable vision accelerator (PVA), according to some embodiments of the present disclosure. [Figure 12B] This is a diagram of a BIST system for parallel channel CRC calculation of PVA according to some embodiments of the present disclosure. [Figure 12C] This is a flowchart of a method for performing BIST to detect permanent failures in a PVA, according to some embodiments of the present disclosure. [Figure 13A] This is a diagram of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13B] Figure 13A shows an example of camera position and field of view in an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13C] Figure 13A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 13D] This diagram illustrates a system for communication between a cloud-based server and an exemplary autonomous vehicle, as shown in Figure 13A, according to some embodiments of the present disclosure. [Figure 14] This is an exemplary block diagram of a computer processing device, suitable for use when carrying out some embodiments of the present disclosure. [Figure 15]This is an exemplary data center block diagram suitable for use when carrying out some embodiments of the present disclosure. [Modes for carrying out the invention]

[0013] Systems and methods relating to various components of a system-on-a-chip (SoC) are disclosed, such as vector processing units (VPUs), direct memory access (DMA) controllers, and hardware accelerators (for example, programmable vision accelerators (PVAs), such as a PVA that includes one or more pairs of VPUs and DMAs). For example, in various embodiments of this disclosure, the PVA may include one or more VPUs controlled using one or more processing controllers (or control processors), such as an R5 processor, an ARM processor, a CPU, and / or similar. The disclosure, relating to various components of an SoC, may be described in relation to an exemplary autonomous vehicle 1300 (or, as herein referred to as “vehicle 1300” or “self-aware vehicle 1300,” examples thereof, which are described in relation to Figures 13A–13D), but is not intended to be limited thereto. For example, the systems and methods described herein may be used in, but are not limited to, non-autonomous vehicles, semi-autonomous vehicles (for example, in one or more advanced driver assistance systems (ADAS)), piloted and unpiloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, airships, boats, shuttles, emergency response vehicles, motorcycles, electric or electric bicycles, aircraft, construction vehicles, underwater crafts, drones, and / or other vehicle types.In addition, this disclosure may be described in relation to computer vision, machine learning, artificial intelligence, image processing, and / or similar technologies, but is not intended to limit the scope of the systems and methods described herein. These systems and methods may be used in any other technological space where vector processing units (VPUs), direct memory access (DMA) systems, instruction set architectures (ISAs), programmable vision accelerators (PVAs), separate accelerators, separate lookup tables, hardware sequencers, single-input multiple data (SIMD) architectures, and / or other components of a SoC may be used. Furthermore, these components and related processes may be described in relation to a SoC, but is not intended to limit the scope of the SoC. These components may be implemented as standalone components, as separate components of a system, and / or as components integrated on a SoC. In some embodiments, the systems, components, features, functions, and / or methods of the present disclosure may be integrated into the exemplary autonomous vehicle 1300 of Figures 13A–13D, the exemplary computer processing device 1400 of Figure 14, and / or the exemplary data center 1500 of Figure 15.

[0014] Minimum / maximum hardware data collectors for anomaly detection For example, in computer vision applications, and especially in vision applications where safety is critical, calculating the dynamic range of intermediate results is a critical task. For instance, to detect noise or errors in intermediate calculations, values ​​outside the known or expected dynamic range can be used to identify values ​​that fall outside this range. In such cases, if a value falls outside the known or expected dynamic range, it may be flagged as corresponding to noise, error, and / or another problem. Therefore, it may be desirable to collect the minimum and maximum values ​​of intermediate results to detect data anomalies. In practice, these anomalies may be caused by, but are not limited to, noise in the image sensor, a corner case in the algorithm, or data corruption in memory or interconnects. Considering these issues, collecting minimum / maximum values ​​is an effective method for detecting these data anomalies. Minimum / maximum values ​​are also used in certain algorithms.

[0015] In certain applications of autonomous vehicles, runtime exceptions, such as infinity or non-numeric values, can result in invalid values ​​or errors, leading to failures or otherwise undesirable outcomes. Considering this, algorithms running as part of an autonomous vehicle platform may be evaluated to determine the range of possible values ​​obtained during processing, i.e., intermediate values ​​or other ranges. Once the range of values ​​is known, the actually calculated value is compared to the known range, and values ​​outside the minimum or maximum threshold may be flagged as errors. If an error is flagged, modifications to the processing may be implemented, such as ignoring data for a given iteration or identifying and correcting the problem. Thus, runtime exceptions are not tolerated, as they are considered and not relied upon by the autonomous vehicle.

[0016] As another example, minimum / maximum collection can be used in certain algorithms to normalize intermediate results to a specific numerical range, thereby improving the precision of processing, for example, block floating-point numbers. This normalization process may include a dynamic range collection step that collects the minimum and / or maximum values ​​of an array, and an adjustment step that applies a scaling factor to the array. However, in conventional processes, collecting minimum / maximum values ​​requires writing all values ​​to memory, then analyzing the values ​​for minimum / maximum, and adjusting them to scale.

[0017] Therefore, these conventional methods of evaluating minimums / maximums are performed in software and require an additional processing cycle. For example, the algorithm itself can run and calculate the values, and then the software can run to determine the minimum / maximum values ​​and compare them to a known range of values ​​to identify anomalies. This software needs to execute additional instructions, such as reading elements of an array of intermediate results and then performing the minimum / maximum calculation. As a result, the system runs longer to detect anomalies because it runs until the algorithm is complete, and then an additional process is performed to calculate the minimum / maximum of the algorithm's output. This can cause downstream processing to be delayed until the minimum / maximum values ​​are calculated and compared to a threshold, or downstream tasks may start calculating data containing errors while the minimum / maximum evaluation is taking place. This not only increases the execution time because this additional cycle is performed to identify anomaly data, but also increases the system's processing requirements and energy consumption.

[0018] Referring to Figure 1A, Figure 1A is an exemplary processor architecture 100 for minimum / maximum collection according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. The various functions described herein, performed by entities, may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, architecture 100 may include components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0019] In light of the shortcomings of conventional minimum / maximum evaluation processes as described herein, this disclosure includes systems and methods for minimum / maximum collection using hardware. For example, during computation, the calculated value may be written to memory 106, for example, local memory, and used in downstream computations in the same or another algorithm. To reduce execution time and processing, minimum / maximum collection hardware (e.g., minimum / maximum collector 104) can be used to capture the minimum / maximum value before or at the time the minimum / maximum value is written to memory 106, rather than waiting for the value to be read from memory 106 and then analyzing the minimum / maximum value. For example, an enable bit can be used to enable the minimum / maximum collection function of the minimum / maximum collector 104, which, when enabled, can update the minimum / maximum value as the value is calculated using the processor 102 and written to memory 106 (e.g., before or at the same time as it is stored in memory 106). In the embodiment, the enable bit may indicate the type of array being calculated, for example, signed or unsigned, so that the minimum / maximum collector 104 is configured to collect the minimum / maximum of a particular type of array. For example, the enable bit or another type of control feature may be used to disable the minimum / maximum collector 104 and / or configure it to collect unsigned minimum / maximum values ​​or signed minimum / maximum values. The data path for storing the data may include the minimum / maximum collection logic mechanism of the minimum / maximum collector 104, which reads values ​​so that they are calculated using the processor 102 to update or maintain minimum / maximum values ​​and stored in a register file.

[0020] For example, during computation, the current minimum and / or current maximum values ​​may be maintained in the minimum / maximum collector 104, and the current minimum and / or current maximum values ​​may be updated when a new, smaller minimum and / or a new, larger maximum value is written to memory 106. If the newly calculated value is greater than the minimum and / or less than the maximum, the current minimum and / or maximum values ​​may be maintained by the minimum / maximum collector 104. In this way, the minimum / maximum collector 104 can maintain the current minimum and / or maximum values ​​as each value is calculated completely over the entire computation. Once the computation over a given iteration is complete, the minimum / maximum values ​​become immediately available in the minimum / maximum collector 104, and these stored values ​​can be compared using software and / or hardware to minimum and / or maximum thresholds associated with a particular algorithm or computation performed to determine whether an anomaly exists. For example, this may include a mechanism that allows reading the collected minimum / maximum values ​​for evaluation. Thus, compared to previous methods, since the minimum / maximum values ​​are immediately available after the algorithm has been fully executed, no further cycle is required to calculate the minimum / maximum. Furthermore, in the embodiment, the minimum / maximum collector 104 (including, for example, hardware and / or logical mechanisms) can recognize store predications so that the minimum / maximum collector can exclude a particular data item if, for example, a store predication on a per-lane basis prevents that particular data item from being stored in memory 106. For example, if an address from an address generator contains a store predication flag, the calculated value may be canceled not only from being stored in memory 106, but also from being updated by the minimum / maximum collector 104.

[0021] In some embodiments, the minimum / maximum collector 104 may be implemented as a feature of the system, including an address generator, such as the address generator described in U.S. Patent Application No. 15 / 141,703, filed on April 28, 2016, which is incorporated herein by reference in its entirety. The address generator may be included in any type of processor or other processing unit, such as a vector processing unit (VPU), a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a data processing unit (DPU), and / or another processing unit type (such as those described with respect to Figures 13A-13D, Figure 14, and / or Figure 15). In some embodiments, one or more VPUs may be included in a programmable vision accelerator (PVA) and / or as part of a system-on-a-chip (SoC).

[0022] As a non-restrictive example, the input to a particular sensor type or algorithm may be limited to 16-bit units. To determine the dynamic range of this particular sensor and / or algorithm, the operations associated with the algorithm that processes the sensor input may be evaluated. In such an example, assuming the first operation is the addition of two 16-bit numbers, the first intermediate result would be a 17-bit number. The 17-bit number can then be multiplied by a 5-bit number to produce a 22-bit number. If this is the end of the algorithm, it may be determined that the output should not exceed 22 bits. Similarly, the minimum value may be evaluated. Therefore, during expansion, if the minimum / maximum value falls outside this known range (e.g., 22 bits), the output may be flagged.

[0023] In some embodiments, the path of stored data (for example, between processor 102 and memory 106) may include a saturation and / or rounding logic mechanism 108 that rounds the value stored in memory 106 to between a certain upper and lower limit or threshold, or based on some specific rule. Thus, in conventional methods, minimum / maximum evaluation may be performed after saturation and / or rounding. If an anomaly exists, saturation and / or rounding may mask the anomaly; for example, low and / or high values ​​may saturate between the upper and lower limits set by the saturation logic mechanism, so such conventional methods may not detect the anomaly.

[0024] However, in certain embodiments, unsaturated, unrounded, or absolute minimum / maximum values ​​may be valuable or desirable, in addition to, or instead of, saturated minimum / maximum values. Therefore, the minimum / maximum collector 104 of this disclosure can collect minimum / maximum values ​​from raw or unsaturated data (for example, before values ​​are manipulated using the saturation / rounding logic mechanism 108) for use in anomaly detection. In embodiments, collection of the average value of the data, or the average absolute value of the data, may be performed. The average value may be calculated, for example, by summing the elements, reading the sum back from the address generator's setting register, and dividing by the number of stored data items (which may be known to the application). In this way, the absolute minimum / maximum values, sum of values, and / or absolute sums may be added to the processor's store data path, and the resulting statistics may be set and collected, for example, by adding them to the address generator's setting feature set or by managing them separately. In some embodiments, the minimum / maximum collector 104 can collect values ​​before and / or after saturation, rounding, or other calculations that use the saturation / rounding logic mechanism 108.

[0025] Referring here to Figure 1B, each block of Method 110 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 110 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 110 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. Although Method 110 is described with respect to Architecture 100 in Figure 1A, it may be executed by any one system or any combination of systems, including, but not limited to, those described herein.

[0026] Figure 1B is a flowchart showing a minimum / maximum collection method 110 according to some embodiments of the present disclosure. Method 110 includes a step in block B102 of calculating one or more values. For example, if one or more algorithms are performed, such as a neural network, a computer vision algorithm, a filtering algorithm, etc., one or more values ​​may be calculated using processor 102.

[0027] Method 110 includes the step in block B104 of comparing a value among one or more values ​​with the currently stored minimum and currently stored maximum values. For example, the minimum / maximum collector 104 can compare each of any number of values ​​to be stored in memory 106, for example, a value in a register file, with the currently stored minimum and currently stored maximum values ​​(for example, currently stored in the hardware minimum / maximum collector 104). In such an example, before or simultaneously with the calculation and storage of a value in memory, the minimum / maximum collector can compare the value with the currently stored minimum and / or maximum values. In one or more embodiments, the minimum / maximum collector may be located in the middle of a data path between a hardware unit that calculates one or more values ​​and a memory unit that stores one or more values.

[0028] Method 110 includes the step of determining in block B106 whether a value is greater than the currently stored maximum value or less than the currently stored minimum value. For example, a system (e.g., a hardware minimum / maximum collector 104) can determine, based on the comparison in block B104, whether each value to be stored in memory is greater than the currently stored maximum value or less than the currently stored minimum value.

[0029] Method 110 includes the step in block B108 of updating the currently stored minimum value to a value smaller than the currently stored minimum value. For example, if the calculated value to be stored in memory is smaller than the minimum value currently stored in the hardware minimum / maximum collector, the hardware minimum / maximum collector may update the currently stored minimum value to the calculated value.

[0030] Method 110 includes the step in block B110 of updating the currently stored maximum value to a value greater than the currently stored maximum value. For example, if the calculated value to be stored in memory is greater than the maximum value currently stored in the hardware minimum / maximum collector, the hardware minimum / maximum collector may update the currently stored maximum value to the calculated value.

[0031] In this way, once several values ​​(for example, all) are stored, the minimum / maximum can be dynamically updated while values ​​are being stored so that they become immediately available by reading the minimum / maximum from the currently stored values ​​in the minimum / maximum collector.

[0032] Automated Store Prediction Even among high-clock-rate processors, a common embodiment is to set the processor into multiple pipeline stages. As a result, latency can occur between the issuance of an instruction to load a register from local memory and the time the register becomes available to perform another instruction—for example, load-to-use latency. Despite load-to-use latency, to achieve high throughput, compilers and application developers for processors may use software pipelines and / or loop unrolling. For example, software pipelines can be used to overlap the execution of multiple iterations of a loop, and loop unrolling can be used to expand a loop body by repeating its contents multiple times. These techniques can also enable the simultaneous execution of multiple iterations of a loop's contents, resulting in fewer, ideally zero, idle cycles in the schedule. When performing loop unrolling, the compiler can divide the number of loop interactions by the unrolling factor. For example, the compiler can assume that the original number of iterations is a multiple of the unrolling factor, so that the unrolled loop can be executed with equivalent functional behavior. In such a case, if the original number of iterations is 60 and the loop should be expanded by 6 times, the expanded loop may execute in 10 iterations. However, if the original number of iterations is 64 and in the case of normal integer division, 64 / 6 is also 10, the loop will not execute enough times (for example, the remaining 4 iterations may not be executed), resulting in different code behavior after expansion and potentially causing failures in the application. Some techniques add assertion statements to ensure that the number of iterations is actually and reliably a multiple of the expansion factor.

[0033] The set of steps or operations in the loop body can have a narrow range of optimal or desired expansion factors. For example, the expansion factor may be lower bounded by the minimum number of copies of the loop code to be scheduled to achieve best performance by filling gaps due to various latencies, and upper bounded by the maximum number of copies to be scheduled, due to limitations in the capacity of the register file, which may cause, for example, register spilling (saving to and restoring from the stack) and prevent scheduling from being optimized. As a further example, expansions in powers of two are acceptable in many applications because it is feasible to select combinations of tile width and tile height that allow for several powers of two (e.g., 2, 4, 8, etc.) of iterations. However, in some embodiments, the loop body may also be optimally expanded 6 or 7 times, while expansions in 4 or 8 times may be less efficient. In any case, loop expansion to achieve optimal scheduling may result in undesirable limitations on the number of iterations. Therefore, conventional techniques to address this problem may result in performance degradation and increased code size.

[0034] For example, because a limit on the number of iterations is inconvenient, a programmer can write two types of loops, such as a "multiple" loop and a "remainder" loop, when there should be no limit on the number of iterations. As an example, the following illustrative code segments illustrate the following: Code 1 - A vector addition loop without loop unrolling. Code 2 - The same loop with loop unrolling by 6, which can only work if the number of iterations is a multiple of 6. Code 3 - A two-loop solution that works for any number of iterations, but is less efficient because the remainder loop is not unrolled, and the code size is also larger due to the additional loop and iteration calculation.

[0035] Code 1

number

[0036] Code 2

number

[0037] Code 3

number

[0038] Using the vector processing unit (VPU) of the present disclosure, code 1 can achieve 6 cycles per iteration, code 2 can achieve 1 cycle per iteration, and the performance of code 3 may vary depending on the number of iterations. When the number of iterations (niter) is niter = 60 (no remainder is executed because it is a multiple of 6), code 3 can achieve 1.0 cycle per iteration, and when niter = 64 (the remainder loop is executed 4 times), code 3 can achieve an average of 1.3125 cycles per iteration (for example, (60*1+4*6) / 64=84 / 64=1.3125).

[0039] Referring to Figure 2A, Figure 2A is an exemplary system 200 including a processor 202 (such as a VPU) with an address generation unit having automatic predication capabilities, according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the processor 202 may include, and / or be included in, components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0040] In embodiments of this disclosure, loads and stores within a code segment may use an address generator 204 of a processor 202 (e.g., a VPU). For example, in each load and store, arguments (agen_a, agen_b, agen_c) of the address generator (agen) may be supplied to the load / store function. The arguments may identify registers of the address generator, including parameters that may be used to compute the address of a particular load and / or store operation, such as an address pointer, the number of iterations, and the current loop variable value. In some embodiments, the VPU may be designed so that each address generator register supports six (or other) addressing dimensions and therefore includes six (or other) iterations and six (or other) loop variables.

[0041] Considering the limitations on the number of iterations involving loop unrolling, the systems and methods of the present disclosure may include an address generator 204 having a logical mechanism for automatic predication of stores by the address generator 204 (e.g., a predication flag or bit 208). For example, predication can be used to provide conditional execution instructions, such as whether to do something (or not). The value of the predication bit 208 (e.g., 0 if to store, or 1 if to stop the store, or vice versa) can be used to indicate whether an instruction is executed. Execution cannot refer to the actual execution of an iteration, but it can refer to whether the value obtained by the execution of the iteration is stored in memory. Thus, in embodiments, an instruction or iteration that is not executed due to a predication flag may refer to an instruction or iteration that is executed, but whose execution result prevents or prevents it from changing the state of memory 206. Predication may include instruction-level predication and lane-level predication. Instruction-level predications can be used to indicate whether the entire instruction needs to be executed, while lane-level predications can be used to indicate which lane of data needs to be executed.

[0042] In some embodiments, after a loop variable has reached zero iterations, any subsequent store instruction execution automatically turns off predication, preventing further writes to memory 206. Thus, the automatic store predication feature can make code clearer by, for example, rounding up iterations that are not multiples of 6 (or another expansion factor) to the next multiple of 6, and not changing iterations that are not multiples of 6 (or another expansion factor). Although factor 6 is used, this is not intended to be limiting, and any expansion factor may be used without departing from the scope of this disclosure. Code 4 below includes an example of vector addition using automatic store predication.

[0043] Code 4

number

[0044] Code 4 allows the expanded loop to run 11 times, with 1.03125 cycles per iteration (e.g., 11 × 6 / 64 = 1.03125), assuming the original number of iterations (niter) is 64. Another way to account for the iteration limit, which is a multiple of the expansion factor, is to calculate the necessary predication flags in the loop and supply them in the store instruction. For example, Code 5 shown below illustrates an exemplary embodiment of predication flag calculation.

[0045] Code 5

number

[0046] Since Code 5 can be compiled to 1.5 cycles per iteration on the VPU of this disclosure, automatic predication may offer performance advantages over predications calculated in a loop. In the embodiment, the VPU may include a 7-way very long instruction word (VLIW) instruction scheme and may include two scalar slots per cycle for the scalar operations required to calculate the predication. If the loop has more vector operations per iteration, there may be enough scalar slots so that the calculation of the predication fits within the available slots and does not cause performance degradation. Even for computation loops where calculating the predication in real time has no performance impact, using automatic predication may still offer advantages in code size and energy consumption.

[0047] Therefore, software can be used to set up several iterations (e.g., N1-N6), and the software can cause the address generator-based load / store to occur, usually in a loop. The address generator hardware can maintain the loop variables (e.g., variables I1-I6) and advance the address pointer appropriately. When the address generator-based load / store has been performed over a predetermined number of iterations, the address pointer may get stuck at the last valid address, preventing subsequent stores to memory by turning off auto-predication (e.g., by setting the predication flag). Therefore, the address generator 204 may include an internal Boolean state of "auto-predication off," and the loop variable iteration logic mechanism may be configured to support auto-predication being turned off. For example, with respect to Figure 2B, when the address generator is initialized, in addition to the loop variables I1-I6, the value of the parameter auto-predication off ("auto_pred_off") (e.g., predication bit 208) may be initialized or reset to "0." After the loop variable has reached zero of the programmed iteration count, auto_pred_off may be updated to "1". Then, execution of any subsequent store instruction may automatically turn off predication as a result of the predication bit being "1", preventing further writes to memory.

[0048] In the example in Figure 2B, the number of iterations for registers N1-N6 of the address generator can be programmed as N1=4, N2=2, N3=N4=N5=N6=1. Therefore, the total number of programmed iterations can be 4*2*1*1*1*1=8, and as a result, the sequence shown in Figure 2B can be executed. As illustrated, the initial state and the following seven executions (e.g., the first eight iterations) can correspond to an auto_pred_off bit value of 0, and the eighth and ninth executions (e.g., the last two iterations) can correspond to an auto_pred_off bit value of 1, thus preventing the results of the ninth and tenth executions from being stored in memory.

[0049] In practice, a VPU can be configured to process several vector units simultaneously, such as 8 or 16, and therefore the VPU may require that an array be a multiple of the number of vector units. If the array is a multiple of the number of vector units, this setup works correctly. However, generally, arrays may not be a multiple of the number of vector units (for example, there is no guarantee that data will be computed according to arrays of the same size), and as a result, arrays are padded so that processing is always performed on batches of the same size. For example, the remaining iterations may be padded with the value "0", but this still requires an additional cycle in software to process the padded value. Thus, the added data becomes redundant computation and, as is a common problem in single-instruction multiple data (SIMD) software, padding can be inefficient because it complicates the software. Therefore, considering this problem, automatic store predication may be used.

[0050] In a non-restrictive example, if 16 batches are used, many of the 16 batches may be generated from the array, and the remaining values ​​may be included in the final batch along with the remainder or remaining space in the 16 batches where predication is turned off using the predication flag. In a specific example, if the array size is 82, 16 complete sets of 5 may be generated, the remaining 2 elements may be included in the last iteration, and the other 14 may be automatically predicated off, thus minimizing the redundant calculation of padding the batches with 14 values ​​and the unnecessary calculation of the padded data. As a further example, if the granularity of the vector processing includes a width of 32 and the array has 100 elements, three complete 32-element vectors may be processed, the remaining 4 elements may be processed in 4 of the 32 lanes (for example, where the predication flag may be turned on), and the other 28 lanes may be predicated off. In this way, programmers can vectorize arrays that are not multiples of the number of units in the sample. For example, hardware can calculate the number of elements to actually write to memory for every store and notify the store unit of this information. Therefore, even if it is possible to perform the math on padded or additional elements and store them, this extra calculation and storage is inefficient. This can lead to the setting of a predication flag to prevent the extra read from being unnecessary and to prevent (for example, to stop or prevent) the writing of the calculated value from the padded value to memory. This automatic predication can be performed at the instruction level, and software can be added to perform lane-level predication as well.

[0051] In addition, regarding automatic predication, the address generator can program several iterations, so additional information may not be necessary. Therefore, the address generator may have memory to support automatic predication, and additional software instructions may be added to automatically handle store predication and predication off. In this way, the hardware can determine when to store a complete result or a less-than-complete result by, for example, whether predication is off or signaled off in the last iteration, and this can be done at zero cost while maintaining performance. If only software is used, the process will require extra cycles and therefore slow down the process.

[0052] In some embodiments, predication may be used at a per-lane level, and as a result, these embodiments can not only use an iteration count that is not a multiple of the loop expansion factor, but can also efficiently handle any task size that is not a multiple of the vector width. In such embodiments, per-lane predication may be driven using a vector register, which can offer the advantage that the information is computed in real time, and by using the vector register, a shortcut may be realized by copying from the vector register to a scalar predication register, and then using the scalar predication register, which eliminates the need to use a per-lane predication flag. For example, per-lane predication may be performed by a vector register, which can be beneficial when the per-lane predication information is computed in a loop, and the computation may be vectorized.

[0053] For example, if a specific value in an array needs to be replaced, such as replacing any value greater than 100 with 999, the code could be written as follows:

number

[0054] This code may be functionally correct, but it could cause performance degradation. Therefore, the code can be vectorized with lane-by-lane predication by incorporating, for example, the following code:

number

[0055] If the calculation of predications is vectorized in this way, and the per-lane predications can only be transmitted via a scalar predication register, the execution time will be longer because the predication information needs to be copied from the vector register to the scalar predication register.

[0056] However, as shown in the code below, using lane-by-lane predication directly driven by the vector register features described herein, this example can achieve better performance than performing bit packing to move the predication mask from vector lane 0 to the scalar register.

number

[0057] Referring here to Figure 2C, each block of Method 220 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 220 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 220 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. Method 220 is described in relation to System 200 in Figure 2A, but can be executed by any one system or any combination of systems, including but not limited to those described herein.

[0058] Figure 2C is a flowchart illustrating an automated store predication method 220 according to some embodiments of the present disclosure. Method 220 includes a step in block B202 of determining the total number of iterations. For example, an address generator 204 may determine the total number of iterations to be performed for a given instruction.

[0059] Method 220 includes the step in block B204 of dividing the total number of iterations into multiple sets of iterations. For example, the address generator 204 may divide the iterations by an expansion factor to generate a loop body that includes the number of loop iterations.

[0060] Method 220 includes the step in block B206 of determining whether a set of iterations among multiple sets of iterations has a first number of iterations that is less than a second number of iterations corresponding to other sets of iterations among multiple sets of iterations. For example, after separating the iterations by the expansion factor, the address generator 204 can determine whether a set of iterations has fewer iterations than other sets. For example, if the expansion factor is 6 and the number of iterations is 62, there may be 11 sets of iterations. That is, 10 sets have 6 iterations and 1 set has 2 iterations. Therefore, the address generator 204 can determine that of the set of iterations with the remaining 2 iterations, 2 iterations need to be executed and the other 4 iterations need to be predicated off.

[0061] Method 220 includes the step in block B208 of generating a predication flag corresponding to at least one iteration of a set of iterations while the set of iterations is being executed. For example, if the address generator 204 determines that a set of iterations does not contain a complete set of iterations for the same number of times as other sets of iterations, it may enable the predication flag (by changing the value of the predication-off bit 208) to indicate that the results of the remaining iterations should be stored or written to memory.

[0062] Method 220 includes the step in block B210 of stopping the writing of a value corresponding to at least one iteration of a set of iterations into memory, at least in part based on a predication flag. For example, a calculated value may be stopped from being written to memory or may be calculated based on the fact that a predication flag is set.

[0063] Improved SIMD data path configuration for vector processors In conventional single-instruction multiple-data (SIMD) architectures, each SIMD processing unit operates in parallel and independently on its own data lane. In some machines, each SIMD processing unit can communicate directly with its neighbors (e.g., left and right neighbors in a linear array of processing units, or north, south, east, and west neighbors in a two-dimensional array or processing unit). However, communication only between adjacent data paths is limited and costly for implementations of operations requiring a significant number of input operands. One example is convolution, a common operation in image processing, computer vision, and machine learning. During convolution, various filters can be applied to adjacent pixels, such as a 3-tap one-dimensional filter requiring three data operands and three coefficient operands in a non-exclusive example. If these operands cannot be shared between data lanes in a SIMD architecture, six operands must be brought to each data lane to produce the result for that particular lane. Considering this, some common methods implement multiple read ports in the register file, but this requires not only additional surface area for SIMD architectures but also additional power for computation.

[0064] In light of the shortcomings of conventional SIMD architectures, the SIMD architecture of this disclosure enables communication between lanes by defining slices in a processor, such as a vector processing unit (VPU), which is composed of multiple lanes as a group. In a non-limiting example, in a processor, the SIMD lane configuration may include a hierarchical configuration containing a 384-bit data path, which may be divided into, for example, eight 48-bit (extended word) lanes, sixteen 24-bit (extended half-word) lanes, or thirty-two 12-bit (extended byte) lanes. In such an example, each byte may be extended by four bits. The first layer of communication on individual lanes may be called a SIMD slice, which may be 96 bits wide (for example, but not limited to this) and consist of two extended word lanes (for example, two 48-bit lanes), four extended half-word lanes (for example, four 24-bit lanes), or eight extended byte lanes (for example, eight 12-bit lanes). In non-limiting embodiments, the entire processor data path may include four SIMD slices, and the second layer of communication may be global between all four (or any other number) SIMD slices and all lanes. In this way, operand sharing may be achieved between the lanes of each slice, which may be useful in instructions such as filtering, dot product, and payload sorting. The SIMD architecture may be included in VPUs or other processor types, such as the processor of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0065] The SIMD instruction set architecture (ISA) allows for sharing between several lanes (e.g., eight) within a slice through the physical routing of the SIMD architecture. For example, as shown in Figure 3A, communication may be possible between 32-bit word data types, 16-bit half-word data types, and 8-bit byte data types within each slice. As a result, in an example such as the filtering operation shown in Figure 3B, if there are four input operands and four coefficients, an 8-bit x 8-bit multiplication and accumulation can be performed in half-word mode, and the coefficients can be shared with data from different lanes. In the SIMD architecture of this disclosure, to perform the same calculation that can be performed using only three input operands, a conventional SIMD architecture would require each lane to load all eight operands. Consequently, space and energy are saved for executing such instructions by requiring only three read ports, as each read port is associated with increased surface area and energy consumption. During the calculation, the following calculation results may be input to the four accumulators, for example, 0, 1, 2, and 3, as they are shared among the lanes within the slice. ACC[0] += D[0] * C[0] + D[1] * C[1] + D[2] * C[2] + D[3] * C[3] ACC[1] += D[1] * C[0] + D[2] * C[1] + D[3] * C[2] + D[4] * C[3] ACC[2] += D[2] * C[0] + D[3] * C[1] + D[4] * C[2] + D[5] * C[3] ACC[3] += D[3] * C[0] + D[4] * C[1] + D[5] * C[2] + D[6] * C[3]

[0066] As illustrated, for example, ACC[0] can access other lanes of src1a, including D[1], D[2], and D[3], and also other lanes of src2, including C[1], C[2], and C[3]. Similarly, other accumulators (ACCs) can access various lanes of src1 and src2. This type of operation would be impossible in conventional vector processors where lane sharing is limited or minimal. These calculations may include, for example, a sliding window technique, where each accumulator includes the result of moving a sliding window relative to the previous accumulator. For example, the first accumulator operates on D[0], D[1], D[2], and D[3], the second accumulator operates on D[1], D[2], D[3], and D[4], and so on. Each accumulator uses the same coefficients C[0], C[1], C[2], and C[3]. This is made possible by physical routing, which is shared between lanes in a SIMD architecture slice.

[0067] As another exemplary embodiment of the SIMD architecture of this disclosure, with respect to Figure 3C, the dot product in a vector multiplication operation may be performed using lane sharing. In such an example, two indices, for example D[0][0], indicate which lane the data belongs to and which set of outputs the data belongs to. In the calculation of the dot product, each lane uses only the data operand from its own lane, but the coefficients are shared between lanes. Thus, the output from each lane can use all four coefficients at some point during the dot product operation. Since they are shared between lanes in the slice during the operation, the following calculation results may be input to four accumulators, for example 0, 1, 2, and 3: ACC[0] += D[0][0] * C[0] + D[1][0] * C[1] + D[2][0] * C[2] + D[3][0] * C[3] ACC[1] += D[0][1] * C[0] + D[1][1] * C[1] + D[2][1] * C[2] + D[3][1] * C[3] ACC[2] += D[0][2] * C[0] + D[1][2] * C[1] + D[2][2] * C[2] + D[3][2] * C[3] ACC[3] += D[0][3] * C[0] + D[1][3] * C[1] + D[2][3] * C[2] + D[3][3] * C[3]

[0068] Another exemplary operation that can benefit from the SIMD architecture of this disclosure is the two-point sort operation shown in Figure 3D. In a two-point sort, payloads are sorted using two values. This two-point sort leverages communication between pairs of lanes in a slice and is useful in various computer vision applications, for example. For example, lane 0 may have the key item 0, and lane 1 may have the corresponding payload, and the payloads may be sorted according to key comparisons for each key / payload pair, as shown in the code below, for example.

number

[0069] Referring here to Figure 3E, each block of Method 300 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 300 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 300 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. Method 300 is described in relation to the SIMD architecture of this disclosure, but can be executed by any one system or any combination of systems, including but not limited to those described herein.

[0070] Figure 3E includes a flowchart of method 300 for computing an output using operands shared across lanes in a SIMD architecture, according to some embodiments of the present disclosure. Method 300 includes the step in block B302 of dividing the bit width of a processor into a plurality of data slices, each containing a second bit width smaller than a first bit width, and each of the plurality of data slices containing a plurality of lanes, each containing a third bit width smaller than a second bit width. For example, a vector processor may be divided into several numbers (e.g., four) of slices, and each slice may contain several numbers of lanes.

[0071] Method 300 includes the step of loading a first vector into a first vector register such that in block B304, a first lane of a plurality of lanes contains a first operand of the first vector, and a second lane of a plurality of lanes contains a second operand of the first vector. For example, with respect to Figure 3B, the first data operand D[0] of the first vector may be loaded into the first lane, and the second data operand D[1] corresponding to the first vector may be loaded into the second lane.

[0072] Method 300 includes the step of loading a second vector into a second vector register such that in block B306, a first lane of a plurality of lanes contains a third operand of the second vector, and a second lane of a plurality of lanes contains a fourth operand of the second vector. For example, with respect to Figure 3B, a third first coefficient operand C[0] may be loaded into the first lane, and a second coefficient operand C[1] corresponding to the third vector may be loaded into the second lane.

[0073] Method 300 includes the step in block B308 of calculating an output using instructions, at least partially based on a first operand, a second operand, a third operand, and a fourth operand. For example, with respect to Figure 3B, the first accumulator (ACC[0]) can receive the result of ACC[0]+=D[0]*C[0]+D[1]*C[1]+D[2]*C[2]+D[3]*C[3], which includes the values ​​of D[0], D[1], C[0], and C[1], in addition to the others. This calculation can be performed thanks to internal sharing and routing between lanes of each slice.

[0074] Method 300 includes the step of storing the output in a register in block B310. For example, with respect to Figure 3B, the calculated output may be stored in the accumulator register ACC[0] and then in memory.

[0075] Inverted load and store operations using stride parameters In conventional vector single-instruction multiplexed data (SIMD) processors, local data memory can be sized to match the vector processing width. For example, in a 256-bit vector SIMD processor capable of processing 32 lanes of 8 bits, 16 lanes of 16 bits, or 8 lanes of 32 bits, the local data memory may contain, for example, memory 256 bits wide or memory 512 bits wide (for example, twice the processing bit width). In such examples, the local data memory is configured as a single memory bank with full-width memory words. However, a wide vector SIMD processor with a single bank of full-width memory words can be inefficient, especially when memory access locations are misaligned. For example, to load a 32-bit array of 16 elements, with byte addresses from 4 to 67, the processor may require two memory reads, for example, one read addressing 0 to 63 (including addresses 0 to 3 containing data not needed for the current operation) and a second read addressing 64 to 127 (including addresses 68 to 127 containing data not needed for the current operation). Thus, without the bank-configured memory architecture of this disclosure, access patterns may be achieved by multiple loads or stores, which could result in slower computation kernel speed, reduced performance, and increased power consumption.

[0076] With this in mind, instead, a single wide memory bank can be configured as multiple memory banks, such as a 16-bit memory bank (for example, 32 banks of 16-bit memory providing a memory bandwidth of 512 bits per clock cycle). In this way, read and / or write operations are performed within an arbitrary 16-bit alignment range, thereby reducing the amount of extra read / write operations, as described in the example above. In such a memory configuration, only one memory read may be required to read addresses 4 through 67. In addition to memory bank configurations containing smaller individual memory banks, inverted load and / or store functionality can be implemented. For example, a lane offset parameter K can be used to specify the row address offset applied to each subsequent lane of memory. The lane size can correspond to the size of the data element, e.g., 8 bits, 16 bits, 32 bits, etc. If a 2D array is stored in memory with a line pitch of W*K+1 elements, the shifted access pattern can be converted to a vertical pattern, where K is the offset parameter and W is 64 / lane size (or size of data elements). For example, for 32-bit data elements, the line pitch could be 16*K+1. In some embodiments, a SIMD processor may be included, and / or may include, as one of the components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0077] As an example, with respect to Figure 4A, Table 400 may include an example of a logical diagram of an inverted load and a diagram of a memory bank of inverted loads with 17 line pitches exceeding 256 bits. The memory bank ends with 18 separate 16-bit banks in the memory bank diagram, but this is for illustrative purposes only. For example, the memory bank could total 256 bits, total 512 bits, or some other number of bits, if each memory bank could be 16 bits wide. In the diagram of a memory bank using inverted loads with 17 line pitches, only one load operation may be performed to obtain each of the highlighted values ​​in the array.

[0078] While inverted loading using this technique is beneficial for many operations, certain algorithms, such as some computer vision algorithms, may require access to data patterns that cannot be achieved with a single read and / or write, even using the inverted loading technique described with respect to Figure 4A. For example, there may be instances where loading a submatrix of 8x2 element width, a matrix of 4x4 element width, or another matrix or submatrix size is required, rather than loading a vertical vector of height 16. In dot product operations, for example, the accumulation may be directed 16 bits at a time across two rows of 16 elements, and therefore, when storing, a T16 inverted store option with an appropriate line pitch may be desirable so that the two rows can be written out as a single memory write transaction. With this in mind, a stride parameter may be used in inverted loading and / or storage. In some embodiments, the stride parameter may include strides that are powers of 2 (though this is not intended to be restrictive), such as 2, 4, 8, and 32, which may be called T2, T4, T8, T32, etc. Various examples of inverted loads using the stride parameter are illustrated in Table 410 of Figure 4B, which includes a logical diagram of the inverted load and a diagram of the memory bank. The example in Figure 4A reflected in Figure 4B includes a stride parameter of 1, but the other stride parameters are multiples of 2. For example, in T2 with a line pitch of 18, it is possible to store a 2-element wide × 8-height matrix as an inverted load so that each value can be retrieved using only one load transaction. Similarly, in T4 with a line pitch of 20 and a stride of 4, a 4-element wide × 4-height matrix can be stored so that each value can be retrieved using only one load transaction, and so on. Although described as a load transaction, this type of format can also be used for store transactions, which store data into memory according to inversion and stride parameters.

[0079] In such examples, the constraint on line pitch can be adjusted according to the stride. For transposed access of word type T, the line pitch may be 16K+1; for transposed access of word type T2 (e.g., with a stride of 2), the line pitch may be 16K+2; for transposed access of word type T4 (e.g., with a stride of 4), the line pitch may be 16K+4; and so on. Thus, the line pitch can be equal to 16K + stride value, or 16K+1 + (T-1), where T is the stride parameter.

[0080] During computation, the VPU's VMEM architecture and the VPU's instruction set architecture (ISA) may be configured to perform inverted load and / or store operations with or without a stride parameter, in order to enable data read or write operations to be organized into a logical diagram sequence in a single read operation. For example, the ISA may be configured to receive information indicating the starting address for reading data from or writing data to an instruction (for example, reading or writing data from a register file), specifying the write type (e.g., an inverted write operation with or without a stride parameter), the line pitch value (e.g., the value of K in 16*K+1), and / or the stride parameter value. Note that the value 16 corresponds to the number of data elements in a particular embodiment, but the value 16 (or W) may vary in different embodiments. Thus, when writing data to memory according to an inverted write operation, the ISA may receive the starting address, line pitch, and / or stride parameter to write to VMEM. As a result, when writing values, instead of writing the values ​​to just one column of data in a single memory bank, the data may be written according to transposition or offset, as illustrated, for example, in Figures 4A and 4B. When a stride parameter is used, a line pitch may be applied to write the next set of values ​​to the memory bank such that the first value of the stride, followed by a number of subsequent elements corresponding to the stride, and then each value may be written to memory in just one cycle. Similarly, during a read operation, if the data is configured or written according to an inverted store, the ISA may receive the starting address, the load type (e.g., an inverted load, with or without a stride parameter), the line pitch value (e.g., the value of K), and the stride parameter value (e.g., an indicator of the data type, such as bytes or halfwords).The ISA can then access data from various memory banks according to the transpose load instruction (and / or stride parameter) to obtain one column (or more columns) of data in just one read cycle. In this way, by obtaining one element from each memory bank, a single vector can be returned in just one read operation.

[0081] Referring here to Figures 4C to 4D, each block of Methods 420 and 430 described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Methods 420 and 430 can also be embodied as computer-usable instructions stored in a computer storage medium. Methods 420 and 430 may be provided, to name a few, by standalone applications, services or hosted services (standalone or in combination with another hosted service), or by plugging into another product. Although Methods 420 and 430 are described in relation to the SIMD architecture of this disclosure, they may be executed by any one system or any combination of systems, including but not limited to those described herein.

[0082] Figure 4C includes a flowchart of a method 420 for setting up inverted store operation using a stride parameter, according to some embodiments of the present disclosure. Method 420 includes a step in block B402 for determining the size of the matrix. For example, the width of the matrix may be determined.

[0083] Method 420 includes the step of determining a stride parameter and line pitch for storing a matrix in block B404 based on its size. For example, the line pitch may be determined using a stride value of 16K+, and the stride value may be determined based on the width of the matrix.

[0084] Method 420 includes the step in block B406 of using the stride parameter and line pitch to store the matrix values ​​in memory. For example, once the line pitch and stride are determined, the matrix values ​​may be stored in memory in such a way that when reading the matrix values ​​from memory, the line pitch and stride parameter values ​​do not result in bank contention.

[0085] Referring here to Figure 4D, which includes a flowchart of a method 430 for configuring inverted store operation using a stride parameter, according to some embodiments of the present disclosure. Method 430 includes the step in block B408 of receiving data representing a line pitch and a start memory address in one of a plurality of memory banks, the start memory address corresponding to one of a plurality of elements corresponding to a column of an array.

[0086] Method 430 includes a step in block B410 of reading multiple elements from multiple memory banks in a single read operation, wherein each of the multiple elements is read from the respective memory bank of the multiple memory banks, at least in part based on line pitch.

[0087] Loading, including sorting (permute) and zero insertion, in a single instruction. In traditional processor instruction sets, a load instruction can form a memory address through some indexing calculation, read the requested memory data from local memory, and place the memory data into a register. If the application requires additional data manipulation, additional instructions can be used to perform operations on the memory data in the register. In some cases, data operations may include simple data reconfiguration. In traditional processors, even this simple data operation in the register file requires additional instructions, which increases latency. For example, in a traditional system, data can be loaded, substitutions performed on the loaded data, and then one or more operations performed using the reconfigured data. If the load instruction is enhanced by this ability to reconfigure data, some processing time may be saved, and the computation kernel may run with higher performance and lower power consumption.

[0088] In consideration of these shortcomings, the systems and methods of the present disclosure add loads that include sorting instructions, which transmit sorting patterns along with memory addresses to local memory. As a result, substitutions can be performed without a large amount of additional logical mechanism, using existing data routing and multiplexing used to handle misaligned loads. In addition to saving instructions that would otherwise be spent, for example, five instructions used to perform sorting of double-vector inputs and double-vector outputs, the overall latency of the sorting operation can be reduced. For example, the only latency is the latency from load to use, rather than the latency from load to use and the computation latency (for example, to perform the substitution). In some embodiments, loads including sorting and / or zero insertion described herein may be included in or similar to the components, features, and / or functions of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0089] Therefore, loads that include substitution features can be useful for manipulating data loaded from memory into a desired format for computation. For example, coefficient data required for various filtering and dot product instructions may include specific repeating patterns that can be performed by loading and substitution. For a filtering operation as described with respect to Figure 3C, coefficients 0, 1, 2, and 3 may be repeated across a vector width (e.g., 16 bits), as illustrated in Figure 5A, for example. In such an example, the first write to registers can begin with D[0] to D

[15] , then, using a sliding window 4, the next registers can begin with D[0] to D

[19] , and so on. Since coefficients C[0] to C[3] can be repeated across the vector width in this filtering example, loads that include substitution can be useful when writing coefficients directly from the load in this order, rather than loading all the data, then performing the substitution, and then writing the vector to registers. Therefore, in this example, the substitution pattern for the coefficient data could include {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3). In this same example, the substitution pattern for the data operand could be {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19}. Thus, the data operands and coefficient operands may be read in the order of substitution rather than being read sequentially and then reordered before being written to the register for calculation. As illustrated in Figure 5B, another example of a filtering instruction may include coefficient operands of double vectors, and therefore may include substitution patterns such as {0,1,2,3,0,1,2,3,0,1,2,3,0,1,2,3,4,5,6,7,4,5,6,7,4,5,6,7,4,5,6,7}. The sorting pattern may be static or fixed, or it may be algorithmically calculated during operation to allow for flexible and dynamic sorting patterns.If the pattern is a repeating pattern, the first instance of the repeating element may be loaded, then duplicated, and in the embodiment, then written to the SIMD lane of the SIMD unit.

[0090] In some cases, it may be preferable to mask certain portions of memory data with zero values. For example, zeros may be inserted into unused entries to facilitate visualization in software development or to reduce energy consumption (compared to leaving irregular data values). In another example, zeros may be inserted to define chunks of data in a data structure, such as when the length of each chunk of data is not fixed. In such an example, the value of zero can indicate a gap between two chunks of data. When processing image patches of a fixed size, for example, if some variable-length information (e.g., the location of feature points) is extracted from each image patch, zeros can be used to pad the remaining data that does not correspond to the extracted information.

[0091] In practice, a substitution index can typically contain 32 or 16 elements in the range of 0 to 31 or 0 to 15, respectively, in the information being read. Negative index values ​​may be included in loads involving sort operations to include zero values ​​in the information being read, by writing zeros to the corresponding lanes of the destination register. Therefore, during a write operation, negative values, rather than zeros, may be written to the corresponding lanes in a SIMD architecture, for example.

[0092] As an example, a 30x30 image patch can be processed by a vector operation using 16 consecutive entries at a time. Since 30 is not divisible by 16, each row may be processed in two vector operations: the first to process the full vector width of 16 entries, and the second to process a partial vector width of 14 entries. In such an example, it may be beneficial for the second load of the 14-entry vector to be padded with zeros to fill the last two vector lanes in place of any irregular data values ​​that may currently exist in memory.

[0093] In one or more embodiments, padding zeros may be inserted at desired lane locations in the SIMD architecture, for example, to save the processing time required to write zeros to these lane locations. If 16 lanes are present, a typical substitution pattern may consist of the indices of the 16 lanes, for example, 0 to 15. In this example, if the values ​​{100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115} exist and the given indices as the substitution pattern are {0,1,2,3,4,5,6,7,8,9,10,11,12,13,-1,-1}, then the values ​​ultimately loaded into the destination register will be {100,101,102,103,104,105,106,107,108,109,110,111,112,113,0,0}. Therefore, the two -1 values ​​are automatically converted to 0 in the destination register based on the substitution pattern which includes negative values. In the previous method, -1 and -1 would contain 14 and 15 respectively, and the values ​​14 and 15 in memory would be written to the register. However, this could include irregular values, which may require additional processing time compared to cases where the value 0 is included.

[0094] To perform loads that include replacement features, routing and multiplexing in the memory logic mechanism, similar to those used to perform unaligned memory loads, may be used. For example, to support a load of the entire memory width (e.g., 32 x 16 bits) from an arbitrary 16-bit address (or 16 x 32-bit lanes from an arbitrary 32-bit address), the memory logic mechanism may include a multiplexing logic mechanism that selects one of the 32 lanes of memory data to route to any destination register lane. For example, in an unaligned memory load, the logic mechanism may be driven as follows:

number

[0095] In loads containing replacement features, this same logic mechanism can be effectively reused, but it may include a modified logic mechanism to perform the replacement operation. An example of a modified logic mechanism is included below.

number

[0096] As an example, with respect to Figure 5C, a hardware structure 500 (which may be included as part of a VPU, SIMD unit, SoC, or other device type) equipped with a crossbar 510 (e.g., a crossbar switch) can be used to retrieve data from any location in memory 512 and drive the data to any lane in SIMD via the respective multiplexers (mux) 514A-514N. In this way, it may be possible to write any of the 16 inputs (or other widths of memory or registers) to any of the 16 output locations or lanes. This can be useful in misaligned accesses, allowing the load operation to start from an arbitrary address and then be aligned downstream. For example, if data in memory is read from locations 2-18, the data is read from 2-18 but can be aligned to lanes 0-16 (e.g., 2 goes to lane 0, 3 goes to lane 1, and so on). This was not possible in conventional systems, where the vector load would need to start at a location that is a multiple of 16, such as 0, 16, 32, etc. When data from an arbitrary memory index can be output to any lane of a VPU's SIMD unit within this structure in Figure 5C, the substitution can also be completed. Using the multiplexer 518, sorting control can be introduced or inserted for each lane, informing the multiplexer 514 of the crossbar 510 which memory location to read data from based on the starting position (which may be aligned or misaligned) and the sorting pattern. Thus, rather than simply retrieving data from an aligned position, the sorting pattern can be used to update the memory read position so that each multiplexer 514 sends the appropriate data to each lane of the SIMD unit. In addition, the multiplexer 516 can be used to insert zeros into the sorting pattern, including negative values ​​or other values ​​indicating zero insertion (for example, if zero insertion is performed using a non-negative value).Therefore, when the memory access location is sent from the multiplexer 518 to the crossbar 510 and the value from the memory access is sent to the multiplexer 516 for zero insertion, the values ​​corresponding to the negative values ​​of the sorting pattern may be converted to zero values ​​to pad the values ​​for each SIMD lane. Although only four sets of lanes, multiplexers, and memory indices are shown in Figure 5C, this is not intended to be limiting and any number of sets may be included without departing from the scope of this disclosure.

[0097] Figure 5D illustrates an exemplary use of hardware architecture 500. For example, the example in Figure 5D can be derived from the following information:

number

[0098] In addition, the following C code can represent the logic circuits of the hardware architecture shown in Figures 5C and 5D.

number

[0099] Referring here to Figure 5E, each block of Method 550 described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 550 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 550 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 550 is described in relation to the hardware structure of Figure 5C as an example. However, Method 550 may be executed by any system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein, either additionally or alternatively.

[0100] Figure 5E is a flowchart showing a method 550 for performing a load including a sorting operation, according to some embodiments of the present disclosure. Method 550 includes the step in block B502 of determining a sorting pattern for loading data from memory. For example, the sorting pattern may be static or dynamically calculated. The sorting pattern may be aligned (e.g., 0 to 16 or 0 to 32), offset (e.g., 2 to 18), repeating (e.g., 0, 1, 2, 3, 0, 1, 2, 3, ... and so on), and / or other pattern types.

[0101] Method 550 includes determining the memory address location of each lane among several lanes, at least partially based on a sorting pattern, in block B504. For example, the sorting pattern may indicate from which memory address location data destined for a particular lane or register should be loaded. The sorting pattern may be implemented using a multiplexer 518 so that the appropriate memory addresses are sent to the crossbar 512 according to the sorting pattern.

[0102] Method 550 includes the step in block B506 of loading values ​​into each of a plurality of lanes based at least partially on memory address locations. For example, based on memory address locations, the multiplexer 514 in the crossbar 512 may retrieve the corresponding values ​​from memory for writing to one or more lanes in one or more vector registers. In some embodiments, the multiplexer 516 may also be used to convert values ​​associated with negative values ​​(or other values ​​indicating zero padding) in the sorting pattern to zero. Thus, if one or more negative values ​​are included in the sorting pattern, the values ​​loaded from memory may be converted to zero before being written to the vector registers.

[0103] Method 550 includes the step in block B508 of performing one or more operations within each of a plurality of lanes using a value and at least one instruction. For example, when a vector register or processing lane of a SIMD unit is input, one or more operations, such as arithmetic instructions, logical instructions, shift / rotate instructions, bitwise operation instructions, comparison instructions, conversion instructions, constant generation instructions, and / or similar, may be performed using one or more processing units corresponding to one or more processing lanes.

[0104] Perform table lookups, including multi-point lookups with blending. In conventional processors performing vector SIMD computations, local memory may contain bit widths that match the bit width of the vector SIMD. As a result, these processors can generally only support read and / or write alignment and granularity corresponding to the bit width. However, table lookup is a common technique in embedded computer vision environments, such as digital signal processing (DSP) and implementing various nonlinear functions. For example, square roots, logarithms, sine functions, and cosine functions may require table lookup. To perform these functions, the input space may be sampled uniformly in a one-dimensional (1D) grid, and the output may be recorded in a 1D table at these input points. However, when nonlinear functions are implemented with table lookup, there is often a trade-off between the size of the table (e.g., the number of entries in the table) and the precision. To improve accuracy without requiring a large table size, interpolation lookups can be performed. Linear interpolation looks up two points around a decimal index, or quadratic interpolation looks up three points around a fractional index.

[0105] As an example, if the sine function is implemented using a lookup table and the values ​​of the sine are tableed in integer degrees, then table[0]=sin(0 degrees), table[1]=sin(1 degree), table[2]=sin(2 degrees), and so on. In such an example, if the evaluation is sin(1.7 degrees), linear interpolation can be performed between the two integer degree entries using a decimal as table[1]*0.3+table[2]*0.7. In this example, the second entry, table[2], takes the decimal as its weight, and the first entry takes 1 minus the decimal, so the closer the decimal is to 1.0, i.e., the closer the second entry is to its corresponding position, the greater the weight given to the second entry.

[0106] As another example, an image or patch of an image may be resampled, which may require looking up the available pixels around a certain decimal pixel coordinate and then performing an interpolation lookup. In such an example, the table may contain an image patch, which may be two-dimensional. In this case, bilinear interpolation can be performed to interpolate linearly in both dimensions. As an example, a patch at position Y=5.1, X=7.6 may be interpolated according to the calculation below.

number

[0107] With this in mind, the systems and methods described herein use 2-point and / or 2x2-point lookup operations to increase throughput (or, for example, match the throughput of 32-way parallel processing) while saving memory space. For example, using a per-memory-bank address bus, as well as associated logic and routing, 2-point or 2x2-point (e.g., 4-point) parallel lookups can be performed with less memory usage. Thus, a single lookup to the table may yield 2 points in a 2-point lookup, or 4 points in a 2x2-point lookup. This can be achieved based on a hardware setup, such as bank addresses, logic, routing, etc., and patterns stored in memory that allow reading multiple data without bank contention. Without these features, for example, achieving a 32-way parallel lookup would require duplicating the table 32 times. For example, this 32-way parallel lookup can be performed using the following C code.

number

[0108] In this example, the lookup portion of the loop can be interpolated to perform 32 lookups per cycle and generate 32 outputs over two cycles (the lookup and blending are performed in memory and vector math slots, respectively, and pipelined over two cycles per iteration). Therefore, the overall lookup / interpolation has 16 outputs per cycle and requires 32 copies of the table.

[0109] As a further example, Figure 6A shows a 16-way parallel table configuration for performing a one-point lookup on the index vector {0,1,2,3,4,5,4,3,...}. In such an example, using conventional architecture and memory layout techniques, it would be necessary to perform the first and second lookups sequentially to read two entries from each memory bank. For example, the first memory bank T0 contains the values ​​T0[0] and T0[1] to be read in the lookup operation, but since both of these values ​​are in the same memory bank T0 (which may contain only one read port), the first value T0[0] is read in the first pass, and the second value T0[1] is read in the subsequent second pass. In such a memory layout, two reads from the same memory bank result in bank contention, leading to processing delays and / or the possibility of algorithms or other calculations being executed incorrectly.

[0110] However, using the architecture of this disclosure, the same number of table duplicates required for 32 lookups may be as few as 16 for a 2-point lookup or as few as 8 for a 2x2-point lookup. For example, with a 2-point lookup, the same performance of 16 outputs per clock cycle can be achieved with only 16 table duplicates, and therefore the memory occupied can be reduced by half. A 16-way parallel variation of the instruction can return a double vector in which the first entry is the lower part of a single vector and the second entry is the upper part of a single vector. In C code, this 16-way parallel lookup and interpolation may be represented as follows:

number

[0111] In such an example, the lookup and interpolation portion of the loop requires only one clock cycle (the lookup and blending are performed in mathematical slots in memory and vector, respectively, and pipelined into one cycle per iteration), and can be interpolated to produce 16 outputs. Thus, the lookup / interpolation has 16 outputs per cycle. As an example, Figure 6B shows an 8-way parallel table configuration for performing a 2-point lookup on the index vector {0,1,2,3,4,5,4,3,...}. In this example, since each memory bank T0, T1, T2, etc., contains only one value to be read during the lookup operation, all 16 values ​​can be read in a single pass, unlike the example in Figure 6A where only 8 values ​​can be read in each of the two passes due to the possibility of bank contention. To accomplish this, in the embodiment, the instruction for the lookup may include a pattern that includes obtaining the index, and not only the index, but also one additional position in addition to the index. Thus, the instruction may cause two values ​​to be read for a two-point lookup, and these values ​​may be written to the lookup table in this format, allowing this single read to be performed without bank contention.

[0112] As an example, when performing a vector operation, each lane of the VPU can process a set of pixel values ​​retrieved from memory. In some cases, a lane may process multiple values ​​from the same memory bank, which can result in bank contention, since a memory bank can only contain one read port. Therefore, the methods and systems of the present disclosure distribute values ​​across memory banks to prevent bank contention, so that, for example, each value in a single processing lane of the VPU can access its corresponding value in only one read cycle.

[0113] In conventional systems where 2D bilinear interpolation lookups are performed, 32 copies of the table require 4 lookups (e.g., 2x2) per output, enabling optimal throughput of 8 outputs per clock cycle. A 2x2 point lookup achieves 8 outputs per cycle with 8 table copies (compared to 32), thus reducing the memory occupied by parallel subtables to one-quarter. For example, a 2x2 point lookup may read 2 entries from one row of the 2D table, then 2 entries from the next row. To avoid any memory bank contention within any memory bank, the line pitch of a 2D table may be limited to m*k+2, where m is the number of entries in each subtable stored horizontally, and k is any integer sufficient to store in one row of the table. In an 8-way parallel 16-bit table, m = 32 (16-bit memory word) / 8 (parallel processing) = 4. In a two-way parallel 32-bit table, m = 16 (32-bit memory word) / 2 (parallel processing) = 8.

[0114] As an example, in Figures 6C-6D, memory contention can be avoided by using line pitch limitations. In such an example, a two-way parallel word-type table for 2x2 point lookups is shown with a line pitch of 10. The number of consecutive elements (m) in the subtable is 8, and A[0][0···7] are placed consecutively in the subtable, conforming to the formula 8k+2, where k can be any integer. Thus, no matter what index value is used to start, the 2x2 points to be retrieved can be placed in different banks, which is mathematically guaranteed. For example, the bank numbers for 2x2 points in the subtable are outlined below.

number

[0115] In Figure 6C, two 2D tables with a width of 10 elements × a height of 3 are shown, for example, Table A and Table B, but this is not intended to be limiting, and the tables may have any width and / or height depending on the embodiment. Similarly, the memory layout in Figure 6D includes a layout with a width of 16 elements × a height of 3, but this is not intended to be limiting, and the width and / or height of the memory may be set to any value depending on the embodiment.

[0116] In some embodiments, such as when sampling image patches, interpolation between a small number of pixels may be performed. In some embodiments, a vector horizontal blending (VHBlend_I) instruction including interleaving may be executed to interpolate the looked-up values ​​without additional instructions for manipulating the data, which may include horizontal blending that interleaves the data to blend within a pair of lanes. For example, this instruction may perform bilinear interpolation after the look-up within the same loop. This instruction can process each pair of lanes according to the table layout in Figure 6E. In this way, the calculations for Y0 and Y1 may be calculated as follows. Y0 = x * (1 - alpha0) + y * alpha0 Y1 = z * (1 - alpha1) + w * alpha1

[0117] Therefore, this instruction can be used to horizontally blend between lane pairs x and y, and between z and w, and to interleave the outputs within the destination register. For example, the following C code segment can be used to achieve optimal performance for an 8-way parallel table with a 2x2 point lookup.

number

[0118] In this 8-way parallel table configuration, where the subtables are designated A, B, ..., H, the loop can perform lookups and interpolations, yielding 16 outputs per iteration. In such an example, the inputs may be configured as follows:

number

[0119] Referring here to Figure 6G, each block of Method 600 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 600 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 600 may be provided, to name a few, as a standalone application, service or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 600 may be executed by any one system, structure or component, or any combination of systems, structures or components, including but not limited to those described herein.

[0120] Figure 6G is a flowchart illustrating a method 600 for performing a multi-point lookup (for example, in just one clock cycle in an accelerator for a separated lookup table (DLUT), as described with respect to Figures 9A-9C) according to some embodiments of the present disclosure. Method 600 includes the step in block B602 of replicating a table into memory such that it includes a first value at a first physical address in a first memory bank and a second value at a second physical address in a second memory bank, and the first and second values ​​are in the same column of the table's logical memory diagram. For example, the table may be replicated into memory any number of times to take advantage of parallel processing of system memory access. The table may include a first value at a first logical address and a second value at a second logical address in the same column as the first value, and when stored in memory in this configuration, the two values ​​may be stored in the same memory bank, which may cause bank contention. Therefore, when replicating the table into memory, a write instruction can write the first value as the second value to a first physical address in an adjacent memory bank, for example, another memory bank, so that both values ​​can be retrieved in the same cycle.

[0121] Method 600 includes the step in block B604 of determining a first index corresponding to a first physical address in memory. For example, a read operation may use an index that indicates a first location in memory from which to begin reading the value.

[0122] Method 600 includes, in block B606, the step of reading a first value located at a first physical address and a second value at a second physical address in just one cycle, at least in part on a read instruction corresponding to a multi-point lookup. For example, when replicating a table into memory, the table may be replicated such that pairs of points in the same column or table (e.g., corresponding to pixels in the same pixel column) are stored in separate memory banks. Thus, the first and second values ​​may be read in just one cycle from the first memory bank storing the first value and the second memory bank storing the second value, using a two-point lookup read instruction that uses the index of the first point in the pair of points to read the first point and the adjacent second point stored in different memory banks. This operation may be performed for each duplicated pair of values ​​from each table to generate a high-level vector containing the first value from each table and a low-level vector containing the second value from each table. These vectors can then be used as vector registers in the VPU, in addition to instructions that generate the output (e.g., interpolation).

[0123] Method 600 includes the step in block B608 of performing one or more operations using a first value and a second value. For example, the first value and the second value may be loaded into one or more lanes of the VPU, square root, logarithm, sine and cosine functions may be performed, linear or bilinear interpolation may be performed, and / or other types of operations may be performed. As an example, if interpolation is performed and the table is duplicated 16 times, 16 two-point lookup operations can be performed to generate 32 values, i.e., two values ​​per vector lane of the VPU, and interpolation can be performed on each lane to output 16 results. Thus, 16 interpolated outputs can be generated per cycle using only 16 duplicates of the table. This may be a result of using two-point lookups, which may enable the same throughput of 32 values ​​using half the memory occupied by conventional single-point lookup operations (e.g., 16 instead of 32).

[0124] Load cache per memory bank within vector memory In conventional processors, the data cache width can be, for example, 32 bytes per cache line. A cache line is a unit of data that hardware keeps track of. For example, hardware can keep track of cache line usage information in tag memory, including all system addresses, whether the cache line has been written to, and the time it was last read compared to other cache lines to determine when to delete the cache line. In some embodiments, the data cache is local memory or a part of local memory, mapping larger data structures that are temporarily stored in external memory to local memory, thereby allowing data to be processed without the long memory latency that would result from directly processing external memory. This type of data cache is often used in conventional desktop or laptop computers.

[0125] As a non-limiting example, programmable vision accelerators and / or VPUs include embedded processors for executing smaller sets of highly optimized code. Such processor types may not implement a data cache so that the programmer can manage the contents of local data memory. The systems and methods of this disclosure may include programmer-managed local memory rather than cached memory, but may also include additional data caching capabilities in one or more memory banks, for example. This data cache may be narrower, such as 16 bits wide, compared to, for example, 32 bytes in a more conventional data cache. This data cache may be used primarily to reduce power consumption, unlike the primary goal of conventional data caches, which is to reduce latency.

[0126] In computer vision processing, for example, data access patterns often exhibit a degree of locality (e.g., staying in a neighborhood for a while before moving to the next neighborhood). For example, when performing a 7x7 2D filter using the VFilt4HHW instruction (which computes 4 taps at a time) described herein, the data read stream may read from a neighborhood for 3 memory reads, then move to another neighborhood and read 3 more times, and so on. For the coefficient reads of the calculation, the same array of zero-padded values ​​may be used (e.g., 7*2*4=56 halfwords), advancing 4 halfwords at a time until the last group of 4 halfwords is read, then returning and starting again from the beginning of the 56 halfword array until the filtering kernel is complete.

[0127] Therefore, to leverage these local access patterns and reduce power consumption from memory access, load data caching can be implemented in each memory bank using a two-way set-associative scheme (for example, holding a total of 64 halfwords). When the load cache is enabled, the last read set of data (e.g., last, last two, last three, etc.) can be recorded in the cache along with the row address and the last entry read in tag memory. As a result, if the same memory address is read again, a cache hit may occur, and the cache can supply the data without needing to read the data again from local memory. In embodiments, the load cache may be located between the memory logging logic mechanism and the memory itself, so that whenever a cache hit occurs, the memory read of that particular address or value is stopped or not performed to avoid wasting power.

[0128] Using this cache structure, in the case of the 7x7 2D filtering example above, the load cache may allow the system to skip nearly two-thirds of the data reads and almost all of the steady-state coefficient reads. Examples of how the data cache is used in each bank are shown in Figures 7A-7C. For example, the VFilt4HHW instruction can perform a potentially larger, 4-tap filtering task and may use two single half-word vectors of data, e.g., data[0-15] and data[4-19], and one single half-word vector of coefficients, e.g., coef[0-3], repeated four times to fill a single vector of 16 elements. In the embodiment of the 7x7 2D filter using the VFilt4HHW instruction in both vector math slots, the data element and coefficient arrays shown in Figure 7A may be used. The VPU of this disclosure can be configured to read double vectors, so that data[y][0~15] and data[y][16~31] can be read as double vectors. Similarly, data[y][4~19] and data[y][20~35], and data[y][8~23] and data[y][24~39] can be read as double vectors. Thus, the data and coefficient reading patterns can correspond to those in Figures 7B and 7C, respectively, assuming that the line pitch of the data is 100 and the line pitch of the coefficients is 8.

[0129] Figure 7D illustrates the configuration of memory banks. For example, a 2-entry fully associative caching scheme holds two copies of data within any given superbank, and the data and coefficients are placed in different superbanks, allowing caching to function effectively. In a coefficient read, banks 0-3 can initially hold coefficient elements 0-3, then add elements 32-35, and then read elements 64-67, which removes elements 0-3. As the pattern repeats, elements 0-3 must be read again in the next coefficient read. In a steady state with the load cache enabled, only four banks can be read per sweep of the coefficient read pattern. As a result, by using a load cache for data, memory bank reads can be reduced by (3*32-(32+4+4)) / (3*32)=58.3%, and coefficients can be reduced by (14*16-4) / (14*16)=98.2%.

[0130] Therefore, in certain algorithms, such as computer vision algorithms that use sliding windows, load caches can be useful in preventing power waste. For example, without a load cache, it would be necessary to read each bank in each cycle, even though much of the data is the same. In an example where 512 bits are read in each iteration, the first 512 bits may be read, then another 512 bits may be read, and so on. For example, if the sliding window is only 8 bytes, only 64 bits are new in each iteration, with the remaining 448 bits being the same. Without a data cache, these 448 bits would need to be read again from the data bank. However, with a data cache for each memory bank, these 448 bits can be drawn from the load cache, and only the new 64 bits would need to be read from the other memory bank. Thus, the power required to read 448 bits from the memory bank is saved. Examples of algorithms that can benefit from the use of load caches include spatial filtering operations, deep learning inference operations (such as convolutions), and / or similar operations.

[0131] Figure 7E illustrates a hardware architecture or logic mechanism for a memory bank with a load cache. For example, with regard to supporting shifted access to memory (e.g., vector memory (VMEM)), sliding window data access can be accelerated. This is an important memory access pattern for many computer vision algorithms, including filtering and convolution. In a sliding window vector load, most of the data from random access memory (RAM) bank 702 remains the same. In such an example, if a 64B vector load slides by only 4B, only 4B of the data changes, so only 4B of the new data is read from RAM bank 702. To optimize the power consumption of VMEM RAM, a small cache called a "load cache" may be attached to each bank for each superbank, resulting in a total of 3 superbanks × 32 banks = 96 load caches per VMEM. Each load cache configuration may, in non-limiting embodiments, include a capacity of 2 lines (2 x 2B = 4B), a fully associative scheme, and a pseudo least recently used (pLRU) replacement policy.

[0132] The data cache, where the most recent access is stored, is divided into two parts: tag storage 706 and data storage 704. Tag storage 706 may store cached address and control information corresponding to previous accesses, while data storage 704 may store data from previous accesses. The control information in tag storage 706 may include a valid flag (e.g., whether the entry is valid), a dirty flag (e.g., whether the entry has been modified and needs to be written back to memory), and / or a last-used flag (e.g., indicating which entry to replace if an entry should be replaced with the least recently used policy). Since the cache is a load cache, written data may not update the cache, but the valid flag and last-used flag may be included in tag storage 706. The valid flag or bits can be used to add conditions to address matching, so that any write should invalidate the entry. The last-used flag may be updated with each access.

[0133] As described herein, in the case of an effective caching scheme, the storage capacity of the load cache is much lower than that of the memory or RAM bank 702 in order to reduce access time and save power. In one embodiment, each load cache may correspond to only one RAM bank 702, each of which may be 2048 × 16-bit memory, and each load cache may be 2 × 16-bit data storage units 704 with 23-bit tag storage units 706 (e.g., 2 entries × (11-bit address + 1-bit active) + 1-bit last used)).

[0134] During calculation, a memory address for memory access can be generated using offset 722, line address 724, and step size 726. This memory address can be retrieved for comparison with the tag storage 706, for example, to compare with several previously accessed addresses (e.g., two previous accesses). An arrow pointing to the top of the tag storage 706 can represent a memory address. In some embodiments, the tag storage 706 can use the entire memory address for comparison with the memory address stored by previous accesses. In other embodiments, a subset of tags can be addressed using a subset of address bits from the memory address, so that only a subset of tags is compared with the memory address. For example, if more previous access tags are stored in the tag storage 706, a subset of memory address bits may be used to compare only a subset of tags in order to reduce area and save power. In load cache designs with fewer tags, such as tags corresponding to two previous accesses, the entire tag of a previous entry may be compared with the entire memory address. The "==?" decision block 720 compares the current memory address of RAM bank 702 with the address stored in tag storage 706. If there is a mismatch (for example, the tag and memory address do not match), reading from RAM bank 702 may be enabled using read enable 708 and read data multiplexer (rd data mux) 712, and RAM bank 702 may be selected and read for transmission to staging flop 716. If there is a hit (for example, the tag and memory address match), data storage 704 may be addressed with 0 or 1 (in the two-entry embodiment) to indicate which prior access the hit corresponds to. The corresponding entry in data memory may be transmitted to staging flop 716 via rd data mux 712.The multi-stage flop 716 returns the read-back data to the processor pipeline, which can ultimately be routed to the destination scalar or vector register of the load instruction.

[0135] The multistage flop 714 can accommodate parity checking. For example, a sufficiently large memory may be required to have a parity bit (e.g., of the parity terminal 710) that enables error detection and / or error correction. Error detection may be used in memory (e.g., VMEM), and / or the logic for error correction may be performed on the read-back data.

[0136] Therefore, the load cache may include the tag bits of the tag storage units 706 for way 0 and way 1, each of which may include an 11-bit address and a 1-bit valid bit. The load cache may further include the data bits of the data storage units 704 for way 0 and way 1, each containing a 1-bit pLRU and 16 bits of data and 2 bits of parity. When enabled, the load cache can be looked up in the D1 stage. To minimize power consumption, only the load cache of the RAM bank 702 involved in the load may be enabled. For example, in a single vector load, only 16 of the 32 load caches may be looked up. When a load hits (for example, if the load cache contains data to be accessed), the read enable of a given RAM bank 702 is suppressed, thereby preventing the RAM bank 702 from rising. The pLRU 720 may also be updated in the D1 stage. In the D2 stage, data and parity bits are read from the hit way of the load cache and can be multiplexed with the RAM results.

[0137] In the event of a load cache miss, in the D1 stage, existing entries that should be deleted to make space for a new entry in the way where the victim occurs can be determined based on valid bits and pLRU. The tag of the way where the victim occurs may then be updated with the address of the miss, and the read enable 708 of RAM bank 702 may not be suppressed. In the D2 stage, data / parity from RAM bank 702 is not only sent to the crossbar of the read data, but the data is also used to fill the deleted cache line. Stores can also look up the load cache when enabled and involved. A store hit may invalidate the hit way, and a store miss may be ignored.

[0138] A load cache hit saves power that would otherwise be used to read RAM bank 702. Conversely, a load cache miss not only incurs power that would otherwise be used to read RAM bank 702, but also consumes power to look up the load cache to fill the way left by the victim. Not all types of memory access patterns will have a high hit rate in the load cache, especially when accessing the superbank in indexed addressing mode; therefore, only vector linear loads can be looked up in the load cache.

[0139] When all stores are enabled, they are looked up in the load cache to ensure that the load cache remains synchronized with the data in, for example, VMEM RAM bank 702. For a given superbank, in applications where a low hit rate of the load cache is expected, the load cache of RAM bank 702 for that superbank can be disabled using software, as described in more detail below, to minimize power consumption due to store lookups.

[0140] For example, in some embodiments, the use of a data cache may not be beneficial. For instance, in operations where access patterns are not repeated, a data cache may not be useful, and performing the extra task of checking the cache before reading may waste time and / or energy because it may be necessary to read the data bank to access the appropriate data. Therefore, load caching can be enabled or disabled, thereby reducing the power penalty caused by access patterns with a high miss rate of the load cache, but the load cache can also be used for access patterns where the data cache can be used to save power. In some embodiments, enabling or disabling can be programmed using application code, so that the programmer can program the code to enable the data cache when desired and disable it when undesirable. In other embodiments, enabling or disabling can be performed by hardware that analyzes read patterns and detects overlapping patterns. For example, hardware may enable the load cache with respect to a threshold amount of overlap between consecutive read operations. However, if the overlap is less than the threshold, the load cache may be disabled. The threshold could be, in non-restrictive examples, 25%, 40%, 50%, 75%, or various overlap threshold amounts between reads.

[0141] With respect to Figure 7E, when the load cache is disabled, the tag storage unit 706 cannot be accessed, and the read enable 708 can be set so that a read to RAM bank 702 is enabled for each read. Similarly, the data storage unit 704 cannot be accessed, and the read data multiplexer 712 can always pass the data from RAM bank 702 to the multi-stage flop 716.

[0142] Furthermore, in some embodiments, the memory bank structure may include multiple superbanks, for example, three superbanks, and each superbank may enable or disable a load cache depending on the specific access pattern within each superbank. For example, if three superbanks are used, each superbank may contain 32 RAM memory banks, and the data cache for each memory bank may contain two entries, if each entry is one word, i.e., 16 bits. If two or more superbanks are used, the superbanks may be of any size, different sizes, the same size, or a combination thereof. For example, the first superbank may be 128KB, the second superbank may be 256KB, and the third superbank may be 512KB.

[0143] Referring here to Figure 7F, each block of Method 750 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 750 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 750 may be provided, to name a few, as a standalone application, service or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 750 may be executed by any one system, structure or component, or any combination of systems, structures or components, including but not limited to those described herein.

[0144] Figure 7F is a flowchart illustrating a method 750 using a data cache for a read operation, according to some embodiments of the present disclosure. Method 750 includes the step in block B702 of receiving data representing a memory read address. For example, after a first read operation using several memory banks, a second read operation may be performed, involving one or more additional memory banks, as well as other memory banks, in addition to one or more of the memory banks. The first read operation may have included storing the read output in the data cache corresponding to each memory bank, so that this value can be reused rather than requesting another read of the memory bank. Thus, a memory read address may be received in response to a subsequent read operation, and the load cache may be accessed, if enabled, to determine whether any of the data is stored in the load cache.

[0145] Method 750 includes the step in block B704 of comparing the memory read address with a memory address in the load cache corresponding to a previous memory read, which is stored in the load cache. For example, after a previous memory read, the data from the memory read may be stored in the load cache corresponding to a particular RAM bank 702. To store this information, the tag storage unit 706 may contain memory addresses corresponding to one or more previous reads from RAM bank 702.

[0146] Method 750 includes the step of determining in block B706 whether the memory read address overlaps at least partially with a memory address in the load cache. For example, the memory read address may be compared to a previous memory read address of a previous read stored in the tag storage 706. If a hit occurs, the load cache can be used to read at least a portion of the data corresponding to the memory read address of the current memory read.

[0147] Method 750 includes the step in block B708 ​​of reading at least a portion of the data corresponding to the memory read address from the load cache. For example, a load cache hit determined by the tag storage 706 may cause a portion of the data from overlapping memory addresses to be read from the load cache, and any remaining data may be read from RAM bank 702.

[0148] Detachable, configurable accelerator To optimize processor performance in specific applications, such as real-time applications, the instruction set architecture (ISA) can be extended to create custom instructions that accelerate commonly performed operations. This allows the processor to reduce the number of cycles required to perform a particular task. The process of customizing the ISA continues until the system's performance goals are achieved. However, these new instructions are added directly to memory, either in the processor's register file or as operands, to perform calculations on data, and are executed using not only the existing processor controller but also existing memory addressing and access hardware. In such cases, it is desirable that the new instructions fit within the total number of read / write operands in the processor's register file (e.g., reusing existing ports), within the width of the register file (e.g., within the processor's data types), and within the processor's pipeline stages. These requirements for successfully adding instructions to the ISA limit the degree of flexibility in adding new instructions. In addition, the configuration of the ISA becomes complex when creating an ISA to handle pipelines with many stages (e.g., 30, 40, 50 stages, etc.).

[0149] Furthermore, processors enable a high degree of flexibility at the expense of power loss, because every additional instruction requires fetching, decoding / dispatching, reading from / writing to register files and / or memory. Therefore, adding additional functional units to implement these custom instructions puts pressure on register file read / write ports, resulting in increased required area (e.g., additional read / write ports may be needed) and power (e.g., extra loads to register files may be performed). Also, embedded application processing pipelines typically have multiple stages, where the output from one stage supplies input to the next. Techniques such as running multiple threads in the processor (e.g., in various processing stages) can reduce extension time, thereby lowering latency. However, multithreading comes at the expense of hardware; that is, multithreading requires not only fetching / decoding / dispatching instructions from multiple threads and maintaining state information for each thread (e.g., in register files), but also the inclusion of control logic to handle multiple threads in the processor. This increases the complexity of processor verification and programming, as well as the area and power requirements. Therefore, although various methods exist to reduce processing pipeline latency, existing methods require additional surface area in the processor hardware, additional power consumption by the additional hardware, and increased complexity in programming the processor to perform various tasks.

[0150] In consideration of the limitations in setting up primary processors and the shortcomings of multithreaded processors, the systems and methods of the present disclosure use one or more units of a primary or primary processor, such as a single-threaded processor like a VPU, in addition to a domain-specific accelerator or coprocessor that is isolated from the primary processor and communicates with the primary processor via shared memory such as vector memory (VMEM). Thus, the accelerator can perform operations as a subunit of the primary processor, but rather, once set up, it can execute independently of instructions from the primary processor, rather than requesting the execution of instructions from the processor. For example, access instructions for the accelerator may be used to enable the primary processor to set up and sequence the accelerator, and shared memory may be used to enable the sharing of inter-stage data structures between the primary processor and the accelerator. When the main processor kicks off or starts an accelerator (for example, via a common accelerator interface and using one or more load / store instructions), the main processor is free to either process another stage (thus enabling the ability to work simultaneously on multiple stages of the processing pipeline while reducing execution time) or transition to a low-power or minimum-power state (for example, minimizing power usage when not actively processing) while waiting for the accelerator to complete its processing. Thus, each of the one or more accelerators, once configured by the main processor, can operate independently and concurrently with the main processor. The main processor and accelerators can synchronize via a handshake interface during processing, so the main processor knows when an accelerator has finished processing and / or is ready for a new task, and vice versa.Shared memory can store configuration messages (for example, to configure the accelerator if configuration instructions cannot be efficiently sent via the accelerator interface due to size constraints), input buffers (for example, to store data for processing by the accelerator), and / or accelerator output results (for example, after processing is complete, data from the accelerator, for example, its register file, may be stored back in shared memory at the location indicated by the configuration instruction from the main processor). Thus, when triggered, the accelerator can read configuration parameters and / or input data structures from shared memory, and write output result data structures to shared memory.

[0151] As a result, this combination of a main processor, shared memory, and separate accelerators enables a flexible, programmable main processor without significantly increasing the complexity of the main processor (for example, the main processor may only require additional accelerator configuration or access instructions to program the accelerators) while achieving power consumption levels of hardware with fixed functionality (for example, large computational stages in the processing pipeline may be implemented as accelerators). For example, the accelerator's pipeline and data type (e.g., data width) may be independent of the main processor's pipeline and data type, allowing for further customization and optimization that may not be possible with the main processor alone, requiring the total number of read / write operands in the processor's register file, the register file width, and instructions that fit within the pipeline stages.

[0152] In some embodiments, the accelerator and the main processor may be coupled to achieve some power savings for the accelerator during instruction execution, while simultaneously coupling execution to the main processor's pipeline. However, in such embodiments, the ability to process different stages of the pipeline simultaneously will be reduced because instructions will be interleaved between the accelerator and the main processor. In one or more embodiments, the accelerator and the main processor may be coupled via a higher-level second-level (L2) memory rather than being connected via shared memory. However, in such embodiments, the higher-level isolation (e.g., discoupling from the higher level via shared memory) may increase the overhead of communication with the main processor.

[0153] Detached accelerators can be used for any task in any domain, such as performing 1D, 2D, and other lookups, performing 1D / 2D interpolation, and / or similar operations, for detecting and resolving bank conflicts, as accelerators for detached lookup tables, for sensor processing such as matrix multiplication or other operations on Lidar data, radar data, and / or similar, for machine learning or deep learning applications, as well as for computer vision algorithms such as feature tracking, object tracking, image warping, pyramid creation, and / or similar. Accordingly, the forms described herein can be applied to any processing pipeline in which part of the processing can be offloaded to the accelerator.

[0154] Depending on the embodiment, any number of isolated accelerators may be present on a chip that communicates with the main processor via shared memory. For example, a system-on-a-chip (SoC) or other integrated circuit (IC) may include a main processor and one or more accelerators, and a programmer can write instructions or code that recognizes various accelerators and uses accelerators that can enhance the performance of any of the various tasks of the system. The main processor is primarily described as a VPU, but this is not intended to limit it, and the main processor may include any processor type, such as a CPU, GPU, DPU, or other processor, without departing from the scope of this disclosure.

[0155] Referring here to Figure 8A, Figure 8A illustrates a system 800 including one or more separate accelerators according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the system 800 may include, and / or be comprised of, components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0156] System 800 may include a processor 802 (e.g., a main processor) such as a VPU, CPU, GPU, or DPU, a separate accelerator 804, and / or shared memory 806 (e.g., vector memory or VMEM). The processor 802 may be coupled to an instruction cache (I-cache) 810, which can cache instructions executed by the processor 802. The processor 802 may include general-purpose input / output (GPIO) 808 (e.g., digital signal pins on an IC that can be used as input, output, or both, and may be controllable at runtime), and an IC configurator 812. In some embodiments, as illustrated, the processor 802 may communicate on-chip using an AXI interface, such as, but not limited to, a 256-bit advanced extensible interface (AXI). System 800 can be configured using the IC configurator 812.

[0157] Processor 802 can communicate directly with the separate accelerator 804, for example, via a coprocessor or accelerator interface, such as an advanced peripheral bus (APB) interface, and / or via a handshake, programming, or event interface. For example, processor 802 can configure accelerator 804 using an accelerator interface (or configuration bus), kick off or trigger processing of accelerator 804 using an event interface, and synchronize with accelerator 804 using a handshake or event interface. Thus, each accelerator 804 may include a mechanism to be configured to communicate with processor 802 via its respective accelerator interface or configuration bus. For example, when accelerator 804 completes processing, it may indicate completion to processor 802 via a handshake mechanism, or while processor 802 is waiting for accelerator 804 to finish processing, processor 802 may periodically poll accelerator 804 to request status or completion time. In some embodiments, the accelerator interface may include a 32-bit interface (or other, smaller-sized interface) so that configuration instructions can be transmitted to accelerator 804. However, in some embodiments, configuration messages may be larger (e.g., larger than 32 bits, or several multiples of 32 bits), and instead, configuration messages may be stored in shared memory 806, and the location of the configuration information in memory 806 may be transmitted to accelerator 804 via the accelerator interface to indicate where to retrieve the configuration information.

[0158] Therefore, the configuration bus can configure the accelerator 804, and using events (or a programming interface), it may be possible to enable the processor 802 to trigger or start processing by the accelerator 804. Once triggered or kicked off, the accelerator 804 can perform calculations independently, and the processor 802 waits for the processing to complete and / or to execute another processing task or stage. For example, the programmer of an application can program the processor 802 and the accelerator 804 using knowledge of what each can do, and as a result, the application program can be divided into parts, i.e., some parts for the processor 802 and some parts for the accelerator 804. Thus, in this embodiment, processing can be executed in parallel between the processor 802 and the accelerator 804, reducing execution time and increasing efficiency. A configuration message, shared via the accelerator interface and / or shared memory 806, can be generated by processor 802 and used to indicate to accelerator 804 where the data to be processed begins in shared memory 806, how much data to process, and where in shared memory 806 the results should be written back. Processor 802 can generate an input buffer at a specified location in shared memory 806 containing the data that accelerator 804 will subsequently process. Once the configuration message is transmitted and the input buffer is stored in shared memory 806, accelerator 804 can receive a trigger signal from processor 802 via an event interface (e.g., a programming interface), and accelerator 804 can begin processing the data. When accelerator 804 is triggered, processor 802 can then perform other tasks or enter a low-power state, and when accelerator 804 has finished processing, accelerator 804 can indicate to processor 802 that it has finished processing and can wait for additional work.

[0159] The processor 802 can set up an input buffer or input data structure for processing by the accelerator 804, and can store the input buffer or input data structure in memory 806. The processor 802 can configure the accelerator 804 using load / store operations, in particular to configure the accelerator 804 and to communicate with the accelerator 804. Configuration messages can configure various registers of the accelerator 804 (for example, 256 32-bit registers in this embodiment). For example, in an accelerator with a separate lookup table (described in more detail herein), configuration information can indicate whether the lookup is a 1D lookup with interpolation, a 2D lookup with bilinear interpolation, and / or another type of lookup. The accelerator 804 can configure its registers to appropriately read and process data from memory 806 and write data back to memory 806 when it recognizes a particular mode or function.

[0160] In some embodiments, the processor 802 can configure the accelerator 804 to execute multiple tasks at once to increase efficiency. For example, if the accelerator 804 should execute various smaller tasks, configuring the accelerator 804 individually would increase execution time as each task could be completed quickly, which may prompt the processor 802 to stop processing, configure the accelerator 804 for another task, etc. To do this, the first task message may include the address of a second task message that enables self-chaining of multiple tasks. Thus, the processor 802 can immediately generate configuration messages for multiple tasks and generate configuration information and input buffers for each task so that the accelerator 804 can execute various tasks sequentially before indicating to the processor 802 that it has completed processing and is ready to accept further work. Furthermore, to improve efficiency, accelerator 804 may be configured to overlap tasks so that it can begin decoding the next task and setting up registers for the next task when the first task is nearing completion. Finally, accelerator 804 may be capable of performing operations on data formats or types different from those normally supported by processor 802, by including separate instructions for processor 802 and accelerator 804. This may be a result of the different, task-specific, register architecture and layout of accelerator 804.

[0161] In this embodiment, the processor 802 can communicate with the shared memory 806 via any number of memory interfaces (for example, a 512-bit static random access memory (SRAM) interface). Similarly, as shown in the figure, the accelerator 804 can communicate with the shared memory 806 via any number of memory interfaces (for example, a 512-bit SRAM interface). The arbiter 814 can determine, cycle by cycle, whether the processor 802 or / or the accelerator 804 is permitted to access the shared memory 806.

[0162] Referring here to Figure 8B, each block of Method 850 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 850 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 850 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 850 is described in relation to System 800 in Figure 8A, and Method 850 may be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0163] Figure 8B is a flowchart illustrating a method 850 using a separate accelerator according to some embodiments of the present disclosure. The method 850 includes, in block B802, a step step of receiving configuration information for one or more first processing tasks in a processing pipeline. For example, accelerator 804 may receive configuration information from processor 802 (e.g., configuration messages via the accelerator interface).

[0164] Method 850 includes the step in block B804 of configuring one or more registers of the accelerator based at least in part on configuration information. For example, accelerator 804 may configure one or more registers based on configuration information.

[0165] Method 850 includes the step in block B806 of reading data from an input buffer in memory based at least in part on an instruction for the starting position of the input buffer contained in the configuration information. For example, the configuration information may include an instruction for where in memory 806 the input buffer is stored, and the accelerator 804 can read data from the input buffer into a register.

[0166] Method 850 includes the step in block B808 of processing data from an input buffer to compute output data. For example, accelerator 804 can process data from an input buffer to generate or compute an output.

[0167] Method 850 includes the step in block B810 of writing output data to a location in memory determined at least in part based on configuration information. For example, accelerator 804 may write the result of the calculation to memory 806 and indicate to processor 802 that processing is complete. Processor 802 may then use the output data to execute one or more second processing tasks in the processing pipeline.

[0168] Accelerator for detachable look-up tables Parallel processing is used to speed up many computational tasks, including but not limited to computer vision applications, deep learning applications, sensor processing applications, and / or other applications that benefit from parallel processing (for example, when processing tasks are independent of other processing tasks). For example, vector processors can process multiple elements in the same operation and achieve the efficiency required to run these types of parallel processing algorithms in real time while being low power consumption. For example, a common operation in computer vision or deep learning tasks is to perform lookups from lookup tables, image patches, or faces based on indexes or coordinate positions. To do this, data from multiple elements may be accessed using single-vector load or store operations. If the index being looked up is not regular (for example, a continuous or fixed integer stride in the horizontal, vertical, or depth direction), it results in irregular indexed access from memory.

[0169] A processor can construct vector memory using smaller banks of RAM to support regular, but misaligned, vector access from memory. In this way, the hardware can create an addressing pattern of interest for vector memory by generating unique addresses individually for each RAM bank. The loading operation of irregularly indexed vectors from memory can cause bank contention in one or more banks of RAM, as the indices of different vector elements may be unrelated to each other. Because bank contention is data-dependent, it may not be statically determined, preventing the compiler from scheduling in a way that avoids bank contention.

[0170] In some conventional systems, various architectural designs can be implemented to support the loading operation of irregularly indexed vectors. For example, multiple read ports can be added to RAM banks. In such an example, if the hardware can handle 32 vectors, 32 read ports would be required for each bank, which would increase not only cost, area, and power, but also the density of space and routes around the RAM banks. Another example involves lowering the throughput of indexed lookups to perform only one scalar lookup per load. However, this creates a bottleneck in vector execution and becomes a limiting factor for execution time. A further example involves creating multipole copies of the data structure in memory so that each vector lane can access data from only one bank. While this example may solve some throughput problems of other techniques, memory capacity is limited by occupying N times (where N is the number of entries to be accessed) space for the data structure, which, in addition to the overhead of creating copies, can cause an overall performance degradation of the associated algorithms. However, this method is preferable when the data structure is small. In some cases, conflicts can be resolved by dynamically detecting and sequentially performing the conflicting lookups. However, this can result in increased hardware complexity because bank conflicts must be dynamically detected and resolved. Furthermore, these additional stages increase the latency from loading to using these operations, thereby affecting the compiler's ability to efficiently schedule code. In addition, data-dependent execution latency may be introduced, which is problematic for the compiler's efficient scheduling. In some cases, a combination of these methods may be implemented.

[0171] In consideration of these shortcomings of other architectures, the systems and methods of the present disclosure include an accelerator for a separated lookup table configured to support loading operations of irregularly indexed vectors. The accelerator for the separated lookup table may be included as accelerator 804 of system 800 and can communicate with a processor 802, such as a VPU, via shared memory 806. The separated lookup table (DLUT) may support multiple modes for performing table lookups, such as a 1D lookup mode, a 2D lookup mode, a 2D non-conflict lookup mode, a 1D lookup with interpolation mode, a 2D lookup with interpolation mode, a table reformatting mode, and / or other modes. In any lookup mode, the DLUT may receive an array of indices in a VMEM, which may be in 1D(x) format or 2D(x,y) format. Each element may contain, for example, 16 bits or 32 bits, which may be unsigned. The DLUT can then perform predetermined index calculations, which may include, in non-limiting examples, 2D-to-1D mapping, truncation / rounding, integer / fractional partitioning, and / or range detection. For example, the DLUT can detect or merge duplicate reads, detect bank contention in the index, and issue read requests to VMEM to look up entries in the requested table. Each element may contain 8 bits, 16 bits, or 32 bits, which may be signed or unsigned. The DLUT can then perform post-interpolation processing as configured and write the output back to VMEM. Each of these processing operations may be pipelined to improve throughput, reduce latency, and lower power consumption.

[0172] As a result, the DLUT accelerator overcomes the shortcomings of implementing dynamic conflict detection and resolution in the processor pipeline, enabling the compiler to efficiently schedule deterministic execution latency for all memory operations while avoiding the complexity of performing line conflict detection. Since the accelerator operates as a tightly coupled accelerator, for example via VMEM shared with the VPU, the processor can set up and kick off the accelerator while continuing to process other distinct parts or stages of the processing pipeline or algorithm. In some embodiments, the accelerator may include additional features to further reduce the load on the main processor, such as offloading index generation for patches with specific lookup patterns, performing optional 1D blending and 2D interpolation on the data to be looked up, and / or providing support for table reformatting without lookup or interpolation. In practice, the entire system, including processor 802 and accelerator 804 for performing lookups, has been demonstrated to twice as fast in processing various computer vision algorithms (e.g., feature tracking, object tracking, image warping, pyramid creation, etc.) while achieving a more than 50% reduction in energy consumption compared to running the entire algorithm entirely on the main processor.

[0173] Referring here to Figure 9A, which shows a system 900 including an accelerator for a Detachable Lookup Table (DLUT) according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, system 900 may include, and / or be comprised of, components, features, and / or functions similar to those of system 800 in Figure 8A, the exemplary autonomous vehicle 1300 in Figures 13A to 13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0174] System 900 may include one or more processors 902 (which may correspond to processor 802 in Figure 8A), memory 904 (which may correspond to shared memory 806 in Figure 8A), and a separate lookup table (DLUT) accelerator 906 (which may be included as one of the accelerators 804 in Figure 8A). In embodiments, the processor 902 may include a VPU, and the memory 904 may include VMEM. The DLUT accelerator 906 (or "DLUT906") may include a processing unit (PU) interface (I / F) 908 for communicating with the processor 902, a controller 912 for communicating with the processor 902, and a configurator 910 for configuring the DLUT906 based on information shared from the processor 902 via the PU interface 908 and / or instructions from the memory 904 about the location in memory 904 where configuration messages or information are stored. For example, the PU interface 908 and the controller 912 can correspond to the Advanced Peripheral Bus (APB) and the event or programming interface of the system 800, respectively. The controller 912 can receive kick-off or trigger commands or signals from the processor 902 (for example, via an arrow labeled "Start") to help synchronize the processor 902 with the DLUT 906, indicating that the DLUT 906 is ready to start processing and / or can receive polling signals from the processor 902. Furthermore, when the DLUT 906 has finished processing one or more assigned tasks, the controller 912 can generate a signal to the processor 902 (for example, via an arrow labeled "Complete") so that the processor 902 can begin setting up for the DLUT 906's next task.

[0175] During configuration, the processor 902 can configure the DLUT906 directly via the PU interface 908 and / or indirectly by indicating the location of configuration information in memory 904 via the PU interface 908. In the latter example, the DLUT906 can retrieve configuration information from memory, for example, via the shared read port strm1_dm_rd, and use the stored configuration information to configure the DLUT906 (for example, configure subunits (e.g., IAU, CDRU, PPU, etc.) and / or other components of the DLUT906) to perform one or more tasks. For example, the processor 902 can set up in memory 904 the data structures necessary for the DLUT906 to perform one or more tasks. For example, in a lookup of 1000 coordinates, the processor 902 can set up in memory 904 a data structure corresponding to each of the 1000 coordinates, and further allocate a buffer in memory 904 on which the DLUT906 should write output. The processor 902 can also indicate which operations should be performed by the DLUT 906, such as 1D or 2D lookups, with or without interpolation or table reformatting, and the DLUT 906 can use this information to configure its subunits. The configuration information set up by the processor 902 may also include instructions such as the bit width of the coordinate index and the bit width of the table entries. Thus, once the input and output buffers are set up in memory 904 and configuration information such as bit width and operation type is sent to the DLUT 906, the processor 902 can kick off or trigger the DLUT 906 to begin processing. As a result, in contrast to a system that relies solely on the processor 902, the processor 902 can perform other tasks while the DLUT 906 is performing lookups, interpolation, table reformatting, and / or similar operations, thereby reducing execution time and improving efficiency.

[0176] During the calculation, DLUT906 can receive a list of indices corresponding to coordinates from memory 904, and DLUT906 can retrieve values ​​from the table corresponding to the indices (for example, if the value is an integer) and / or values ​​around decimal values ​​(for example, left and right values ​​in the case of a 1D lookup, or top-left, bottom-left, top-right, and bottom-right values ​​in the case of a 2D lookup) and perform interpolation or other operations on the surrounding values. Once the final value is determined (for example, directly by the lookup if no post-processing is performed, or after processing by the post-processing unit (PPU) 930), the value may be written to an output buffer in memory 904 that corresponds one-to-one with the index from the input buffer. To efficiently perform these tasks, the embodiment may use an index address unit (IAU) 922, a conflict detection and resolution unit (CDRU) 924, a control (CTL) first-in, first-out (FIFO) 928, a fractional (FRAC) FIFO 926, a post-processing unit (PPU) 930, a data coalesce unit (DCU) 932, and / or other components.

[0177] For example, the index (IDX) stream 916 may contain a stream of indexes to be read from memory 904 (e.g., via a read port, strm1_dm_rd) to be looked up in one or more lookup tables, and the values ​​corresponding to the indexes may be read from memory 904 (e.g., via a read port, strm0_dm_rd) through a lookup table (LUT) stream 918. The output (OUT) stream 920 may be values ​​that are processed using DLUT 906 and then written back to memory 904 (e.g., via a write port, strm0_dm_wr).

[0178] When configuring, the processor 902 can indicate to the IDX stream 916 how to access the data structure related to the index. For example, in the case of a 1D lookup with a 64-byte wide interface to memory 904, 64 bytes may be read in each cycle. When a 1D lookup is performed, only one coordinate (e.g., one (x) value) may be read for each index value, while in the case of a 2D lookup, two coordinate indices (e.g., one (x,y) value) may be read for each index. In a non-limiting embodiment, each index can be 16 bits or 32 bits, so with 64-byte reads, it is possible to obtain 8, 16, or 32 coordinates from the IDX stream 916.

[0179] The data in IDX stream 916 can be sent to IAU922 in its raw format as raw indices, where each coordinate can be an integer or a decimal value. If the index is a decimal value, IAU922 can split the decimal value and feed the fractional bits to FRAC FIFO926, which can then use PPU930 to blend the surrounding values ​​looked up in the table. IAU922 can then determine the set of indices to send to CDRU924, where the number of indices sent may correspond to the number of lookups that LUT stream 918 can perform in just one cycle. For example, if LUT stream 918 can perform, say, 32 lookups in one cycle (based on the bit width of each value in the lookup table), IAU922 can send 32 indices to CDRU924 in each iteration. In some practical cases, such as when the values ​​from IDX stream 916 to IAU922 are integer values, IAU922 can send each set of indices without any processing. However, if the value from the IDX stream 916 is a decimal value, the IAU922 can determine which indices need to be lookup-ups (for example, two indices for 1D interpolation, or four indices for 2D interpolation) to obtain each of the surrounding values ​​necessary to perform interpolation or another operation to obtain a blended value corresponding to the decimal value. For example, if the decimal value corresponding to the (x,y) coordinates for 2D lookup and interpolation is (5.3,6.2), the IAU922 can determine that the lookups should be performed at (5,6), (5,7), (6,6), and (6,7), and the PPU930 can then blend the values ​​to produce the final value corresponding to index (5.3,6.2). For example, the values ​​may be blended with equal weighting, or they may be blended using bilinear interpolation so that values ​​closer to (5,6) than to (6,7) are weighted more heavily in calculating the final value of (5.3,6.2).

[0180] A set of lookups (for example, 32 lookup indices if the LUT stream 918 can read 32 values ​​in each read cycle) can be sent to the CDRU 924 in an appropriate order corresponding to the order of the indices in the input buffer of memory 904, as read using the IDX stream 916. The CDRU 924 then performs conflict detection and resolution by identifying bank conflicts that occur when the lookup table reads in the LUT stream 918 are performed in the order received from the IAU 922, and resolving bank conflicts by changing the order of the indices to avoid the bank conflict. For example, if a set of indices is looked up and causes a bank conflict, and another (for example, a later or earlier) set of indices is available in another lookup cycle, the CDRU 924 can find a non-conflicting lookup from the other lookup cycle and replace the conflicting lookup in that cycle with the non-conflicting lookup. As a result, one or more bank conflicts are avoided, thereby increasing throughput. In a practical example, if the IAU sends 32 indices in each cycle, and in a given cycle there is a bank conflict for 6 of the indices, the CDRU924 can determine up to 6 indices from another set of lookups that do not cause a conflict in the current lookup, and can perform these 32 lookups, for example, 26 lookups from the original 32 and 6 lookups from another set sent by the IAU922. Once the set of lookups is determined (for example, with or without substitution considering the conflict), the set of lookups can be read from memory 904 using the LUT stream 918.

[0181] When substitution is performed to account for out-of-order lookups, the CDRU924 can use the CTL FIFO928 to indicate the order of lookups for each set of lookups from the IAU922 to the data merging unit. For example, the DCU can determine that for an initial set of 32 lookups, 8 were performed in the first cycle, then 8 in another cycle, then 16 in yet another cycle, and then determine that the entire set of 32 has been processed, and then, if applicable, the 32 lookups can be pushed to the PPU930 for post-processing or directly to the OUT stream920 for writing to the output buffer of memory904. This additional information indicating the actual order of the lookups, determined by the CDRU924 and read into the LUT stream918, can be communicated to the DCU932 via the CTL FIFO928. Thus, the DCU932 can take into account any changes the CDRU924 may make to the order of the indices received from the IAU922. The CTL FIFO928 can be useful because the number of cycles that pass through IAU922, CDRU924, etc., is not deterministic but data-dependent. For example, since conflicts are unpredictable (for example, because the data cannot be deterministic) and are a result of programming, there is no solution to completely avoid conflicts, and the CTL FIFO928 can help show the lookup configuration resulting from conflict resolution to DCU932.

[0182] The PPU930 can calculate the final value per index that can be read into memory 904, as needed, such as when additional calculations need to be performed on the lookup table values. If no post-processing is required, the PPU930 may be unnecessary except for collecting the results. For example, if a normal 1D or 2D lookup is performed on an index that is an integer value that maps directly to a position in the lookup table, the PPU930 and FRAC FIFO926 will not be used to perform any additional processing. For example, if linear interpolation in a 1D lookup or bilinear interpolation in a 2D lookup, and / or other calculations are performed, the PPU930 and FRAC FIFO926 can be used to convert the collected results into updated results or values ​​for writing to memory 904.

[0183] In some embodiments, DLUT906 may be used in table reformatting mode. For example, IDX stream 916 and OUT stream 920 may be used to update addresses for access and / or transposition. In such examples, if there is a buffer in memory 904 and the index in the buffer should be transposed, this operation can be offloaded to DLUT906 (rather than having the address generation unit of processor 902 perform the transposition). Configuration information from processor 902, for example from the address generation unit, may indicate a read pattern for reading from the buffer in memory 904 and a write pattern for writing the addresses back to memory 904 in a different pattern. For example, if a programmer knows that a particular access pattern will cause many collisions, the programmer can program processor 902 to configure DLUT906 to reformat the table and shuffle the data so that collisions are less likely or none at all.

[0184] As a further example, DLUT906 can be used for sentinel return values ​​upon out-of-range detection, or for output writing with out-of-range predication off. Therefore, for example, if coordinates in IDX stream 916 are outside a given image patch and the corresponding value should not be written, DLUT906 can instead write a sentinel value, which can indicate to processor 902 that the sentinel value should not be trusted or used in processing when processing the information in the output buffer. In some embodiments, this sentinel value can indicate to processor 902 that the value should not be written to memory, and values ​​identified as error values ​​cannot be stored.

[0185] Therefore, the DLUT906 can be implemented as a pipeline of subunits that function together to perform specific tasks or calculations. Each subunit can perform calculations independently and communicate with other subunits via a shared interface. With respect to Figure 9B, Table 940 shows the tasks of various subunits of the DLUT906 while processing specific calculations.

[0186] As a result of the DLUT accelerator described herein, the processor pipeline remains deterministic by offloading dynamic conflict detection and resolution to a separate accelerator. In addition, execution time is reduced because the accelerator can perform operations independently of and simultaneously with the main processor (e.g., VPU). The DLUT accelerator can further enable 1D and / or 2D lookups from a single common table through conflict detection / resolution. The accelerator can perform various post-processing operations, including 1D lookups with linear interpolation, 2D lookups with bilinear interpolation, sentinel return with out-of-range detection (both 1D and 2D), and / or output writing with out-of-range predication off (both 1D and 2D). The DLUT accelerator can be configured to perform interpolation with a configurable number of fractional bits and can support various index and data formats, including 8, 16, and 32-bit signed and unsigned data formats, as well as 1D and 2D coordinates for 16 and 32-bit index formats. The DLUT accelerator may also be able to convert between global and local coordinates using a configurable X / Y offset. The DLUT accelerator further supports a data stream unit that can read the index buffer from VMEM, perform a lookup from VMEM, and write the result (or lookup or interpolation) to VMEM. The data stream unit can support linear and transposed access, addressing up to 2D. The lookup index may be out of order to minimize bank contention in order to optimize the number of cycles required for lookup / interpolation. For example, if VMEM supports N lookups, the accelerator may use M × N indices to maximize the number of indices that can bypass contention detection. Duplicate detection may also be performed to ensure that duplicate indices that would create contention are excluded.Furthermore, the 2D lookup and interpolation modes of DLUT accelerators may include an index that is automatically generated from several parameters within the accelerator, rather than the programmer providing a set of index data (this is called automatic indexing mode). This offloads the index preparation from the main processor to the accelerator.

[0187] Referring here to Figure 9C, each block of Method 950 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 950 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 950 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 950 is described in relation to System 900 in Figure 9A, and Method 950 may be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0188] Figure 9C is a flowchart illustrating a method 950 using a separate lookup table accelerator according to some embodiments of the present disclosure. Method 950 includes the step in block B902 of configuring one or more subunits of the DLUT accelerator based at least in part on configuration information generated using a processor. For example, DLUT 906 may be configured using information received from processor 902 and / or obtained from memory 904.

[0189] Method 950 includes the step in block B904 of determining a first subset of indexes that are free from bank conflicts from a first set of indexes in a stream of indexes read from memory. For example, IAU922 can generate a set of indexes for CDRU924 to handle conflicts, and CDRU924 can determine a subset of indexes that are free from bank conflicts.

[0190] Method 950 includes the step in block B906 of determining from a second set of indexes in a stream of indexes read from memory that there are no bank conflicts with the first subset of indexes. For example, IAU922 may generate another set of indexes for CDRU924 to handle conflicts, and CDRU924 may determine to replace one or more of the conflicting indexes from the first set with one or more indexes from the second set of indexes that do not conflict with the first set of indexes.

[0191] Method 950 includes the step in block B908 of performing a lookup in one or more lookup tables to retrieve multiple values ​​from memory in a single read cycle using a first subset of the index and a second subset of the index. For example, DLUT906 may read values ​​from memory 904 to LUT stream 918 using a subset of values ​​from the index set and values ​​from the second set of the index which are determined not to conflict with the subset of values ​​from the first set of the index.

[0192] Method 950 includes the step of writing multiple values ​​to memory in block B910. For example, values ​​from LUT stream 918 may be written to memory 904 in output stream 920. Before being written, DCU 932 may reorganize the data so that it is in a one-to-one correspondence with the indices read from the input buffer in IDX stream 916. In some embodiments, PPU 930 may perform one or more operations, such as interpolation, on the acquired values ​​before writing the final values ​​to memory 904 in OUT stream 920.

[0193] Hardware sequencer for direct memory access systems Direct Memory Access (DMA) systems allow data to be moved between different memory locations without the need for a central processing unit (CPU). For example, DMA can act as a data movement engine, moving data from a source such as external memory (e.g., DRAM) or internal memory such as an L2 buffer or vector memory (VMEM) of a vector processing unit (VPU) to a destination such as a VPU. DMA systems can perform additional operations, but are not limited to these, such as padding frame data, manipulating addresses, managing overlapping data, managing different scan orders, and considering different frame sizes.

[0194] In digital signal processing, multiple DMA resources can be used to describe the movement of structured tile data between external memory and a processor such as a VPU. For example, these DMA resources may include descriptors, channels, triggers, and / or registers. Descriptors can describe tile movement, such as source position, destination position, line pitch, tile width, tile height, circular buffer placement, and / or similar. However, the movement of image plane tile data with spatial and temporal dependencies presents users with extra programming model challenges and requires a variety of DMA configuration resources. These tile data dependencies can also complicate the control code and control sequences of the processor (e.g., a VPU). For example, a typical processing operation might include filtering, such as 3x3 filtering. This type of operation introduces spatial dependency because every output pixel depends on the corresponding values ​​of the 3x3 pixels surrounding the output pixel. Such an operation can perform filtering using a 3x3 matrix of values, and this operation can be called a spatially dependent operation. In practice, to mitigate programming challenges, all tiles in a frame can be the same size, for example, 64x64. However, if a 3x3 filter is used with 64x64 tiles, additional pixels will be needed above and below adjacent tiles, as shown in the shaded area in Figure 10C. Therefore, in order to properly fetch data across the entire tile, this information needs to be encoded in the DMA resource, which in turn creates an extra programming burden to make it complete.

[0195] Referring to Figures 10A to 10G, these figures illustrate various data movement challenges when using a DMA system. For example, the visualized Figure 1000 in Figure 10A can correspond to frame data padding. Visualized Figure 1000 can have nine sections: top-left section, top section, top-right section, left section, center section, right section, bottom-left section, bottom section, and bottom-right section. In such an example, each section may contain one or more tiles. For example, the top-left section may contain one tile, while the top section may contain, for example, four tiles. Therefore, to precisely define this sectioning, existing methods may describe this frame using nine descriptors (e.g., one per section), three channels (e.g., one in the left column, one in the center column, and one in the right column), and three triggers (e.g., one per channel).

[0196] Regarding padding, for example, when performing operations on data near the boundaries of a tile or frame section due to spatial dependency, a DMA system may pad or create values ​​for pixels outside the image boundaries. This may be because, in certain embodiments, requesting data outside the memory area of ​​the image can lead to failures. Therefore, DMA can be used to avoid failures by padding or creating values ​​after fetching the image data from the corresponding memory area. Without padding, the data structure may not match the kernel size, for example, when performing filtering operations. The fetched data, including the additional padded values, can then be sent to a destination, such as a VPU, which can process the data according to the configuration of the fetched data and process the data in the same way across the entire (padding) frame. When padding, zero padding may be used (e.g., each new data point contains the value zero), duplicated values ​​may be used (e.g., duplicate the pixel values ​​of adjacent pixels from the fetched data), and / or other padding mechanisms may be used. In addition, padding can be added to any side of a frame and may be added to different sides in various ways. For example, in Figure 10A, the padded area 1002 may be larger on the right side of the frame than on the left, top, or bottom side. Padding increases the complexity of DMA programming when moving data from source to destination, such as from memory to VMEM, and also increases the complexity of VPU programming when handling larger padded frames.

[0197] Referring to Figure 10B, the visualized Figure 1010 in Figure 10B corresponds to address manipulation by the DMA system. For example, to fetch consecutive frame data, the addresses of various descriptors can be manipulated and programmed. For DMA to work effectively, the address descriptions of data movement can be consecutive. Therefore, the address of each descriptor can be manipulated, and this manipulation must be passed from one descriptor to another. For example, when padding values ​​as illustrated, the starting address of each descriptor can be manipulated so that the value to be padded is included in the fetched data. To do this, the programmer uses the starting address and tile width, as well as the number of tiles in each section, and this information to generate the address of the next descriptor. For example, the first descriptor can fetch data starting from the top left, then the top, then the top right, then the left, then the center, and so on, as indicated by the arrows in Figure 10B. However, the starting address of a descriptor increases the complexity of DMA programming when moving data to a destination such as VMEM.

[0198] As another example, with respect to Figure 10C, a DMA system may need to read data that overlaps vertically and horizontally from adjacent tiles to ensure continuous data processing. For example, as shown in the shaded area of ​​Figure 10C, overlapping data from a tile in the top-left section and a tile in the adjacent upper section may need to be read using the same operation. Similarly, overlapping data from a tile in the top-left section and a tile in the adjacent left section may need to be read using the same operation. To achieve this, descriptors must be updated or moved to include the overlapping portion. For example, the base descriptor may contain the address of the beginning of the upper section, but to capture data from the adjacent top-left section tile, the descriptor of the upper section must be updated (for example, moved to the left) to capture the data from the top-left tile. This update requires more complex programming, especially as the number of descriptors increases.

[0199] In addition, with respect to Figures 10D to 10F, the DMA system may need to support various scan orders in order to read data from memory sequentially. For example, the relevant scan order may differ depending on whether filtering, convolution, matrix multiplication, and / or other operations are performed. Taking this into consideration, various scan orders may be supported, such as those shown in Figure 10D, including raster scan orders starting from the top left (visualized in Figure 1030), raster scan orders starting from the top right (visualized in Figure 1032), raster scan orders starting from the bottom left (visualized in Figure 1034), and / or raster scan orders starting from the bottom right (visualized in Figure 1036). Similarly, with respect to the visualized Figure 1038 in Figure 10E, for a cube image, various cube scan orders may be supported by the DMA system. Figure 10F shows various vertical mining scan sequences that can be supported by a DMA system, including a vertical mining scan sequence starting from the top left (visualized in Figure 1040), a vertical mining scan sequence starting from the top right (visualized in Figure 1042), a vertical mining scan sequence starting from the bottom left (visualized in Figure 1046), and / or a vertical mining scan sequence starting from the bottom right (visualized in Figure 1048). Supporting each of these various scan sequences to move data into memory (such as VMEM) increases the complexity of DMA programming.

[0200] Regarding Figure 10G, the DMA system may also need to support various frame sizes, such as moving multiple frames of varying sizes (e.g., luma / chroma complexes or various pyramidal levels). For example, a processor such as a VPU can process frames of various sizes to produce the final desired output. Figure 10A shows an exemplary visualized Figure 1048 corresponding to the pyramidal processing of frames for optical flow estimation calculations. In such an example, the movement of pixels may be calculated first for smaller frame sizes, then for larger frame sizes using a queue from the output of the smaller frame sizes, then for even larger frame sizes using a queue from the larger frame sizes, and so on. Thus, the DMA system can support fetching frame data of various frame sizes, but this functionality requires more complex programming of the DMA system. For example, descriptors need to be programmed or updated for each different frame size.

[0201] To simplify the programming of these various operations supported by the DMA system, the DMA system and method of the present disclosure may use a hardware sequencer in combination with the DMA engine, taking data movement into consideration. For example, the data movement of an entire image can be explicitly and completely described in a hardware sequencing mode using a simplified programming model (e.g., an image structure of frames, as shown in Figure 10I) that handles tile sequencing (triggers), padding, overlap (offset), scan order, and different frame sizes. The hardware sequencer can reduce the usage of DMA resources (e.g., by reducing the number of required descriptors, triggers, channels, etc.), offload control from the VPU for VPU control processing, and reduce the complexity of DMA programming. This can be achieved by loading a diagram of an image or frame descriptor (e.g., as shown in Figure 10I) in the form of a series of commands from programmable local memory. This hardware sequence command can incorporate any of the operations described herein that increase programming complexity, including image padding, tile overlap or offset, frame offset, image scan order, and image size in tile granularity. The hardware sequencer can read image commands from memory, sequence tile movements, scan and fill entire frames, in addition to descriptor information (e.g., from image commands or from separate descriptor memory or SRAM).

[0202] Referring here to Figure 10H, Figure 10H shows a DMA system 1050 including a hardware sequencer according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, system 1050 may include, and / or be comprised of, components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0203] System 1050 may include a DMA engine 1056, a register control unit 1058, a hardware sequencer controller 1060, a descriptor SRAM 1052, and / or a hardware sequencer command SRAM 1054. Existing systems may only include a DMA engine 1056 and a descriptor SRAM 1052 for storing frame descriptors. Therefore, as described herein, the DMA engine 1056 previously had to perform all calculations such as padding and addressing when sending data from source to destination, and the VPU or other source had to perform sequence processing via a handshake using the DMA system (for example, with a VPU as the primary node and a DMA as the secondary node). In such an example, the DMA engine 1056 would process at the tile level, using descriptors for various sections of the frame, each containing one or more tiles, to retrieve tiles one at a time and send them to the destination, with subsequent tiles being retrieved according to descriptors based on instructions from the VPU to retrieve the next tile.

[0204] However, using system 1050 in Figure 10H, frames can be processed at the frame level. For example, a frame shown in Figure 10C that previously required nine descriptors may now use only one descriptor. Thus, in practice, when the DMA engine 1056 attempts to load a descriptor from the descriptor SRAM 1052 (or more generally, the descriptor memory 1052), the HW sequencer control unit 1060 can intercept the descriptor load and use a command sequence processing structure to handle multiple frames, tile rows / columns, and multiple descriptors. To do this, a frame format 1070 (Figure 10I) can be used, which describes frames at a higher level by processing tile rows / columns (depending on the scan order) in hardware rather than at the tile level. For example, instead of padding tiles, the entire frame can be padded using frame format 1070, so many frames can be padded with just one padding command. Therefore, the entire frame can be understood, including where to pad, where to overlap, and how to automatically manipulate addresses. In addition, since the DMA engine 1056 can directly retrieve descriptors from the descriptor SRAM 1052 without intervention from the HW sequencer control unit 1060, conventional formats can still be supported for operations that may not benefit from the HW sequencer control unit 1060.

[0205] The HW sequencer control unit 1060 can operate as a state machine, for example, by reading the HW sequencer command SRAM 1054 (or more generally, the HW sequencer command memory 1054) in which a frame format 1070 containing sequence processing commands is stored. A processing controller such as an R5 processor, CPU, or ARM processor can program or set the HW sequencer command SRAM 1054 and descriptor SRAM 1052 using programmed code and / or settings from a higher-level engine.

[0206] The descriptor SRAM1054 may contain one or more descriptors that can specify tile dimensions (e.g., tile width dx and tile height dy), the starting point of the image or frame (e.g., top left, bottom right, etc.), the trigger type, and / or other microinformation relating to the descriptor's scan type.

[0207] The HW sequencer command SRAM 1054 can store a frame format 1070 that defines the entire frame, frame size, frame padding, etc. For example, the frame format 1070 may include a frame header for header control, offset control, and padding control, as well as column headers and / or row headers for the columns or rows of the frame (e.g., a column header for a vertical scan pattern and a row header for a raster scan pattern). Frame header control may include not only a frame repeat factor to identify how many times a particular frame should be repeated, but also the number of descriptor rows and / or descriptor columns. Frame header offset control may include frame tile offsets (e.g., offsets from tile to tile) and frame offsets (e.g., offsets between two or more frames that can be read using only one channel, such as when a YUV frame containing three distinct faces may be processed). A frame padding header can indicate the number of lines or pixels of padding to be added at the frame level (rather than at the tile level in previous methods), such as padding on the left side, top, right side, and / or bottom of the frame, so that the entire frame is padded rather than padding each tile within each section of the frame at the tile level.

[0208] Column headers can be used when the scanning order is vertical, and row headers can be used when the scanning order is raster or horizontal. The column headers and / or row headers can include the column or row offset (e.g., the magnitude of the offset between each column or each row), the column or row repetition factor (e.g., the number of times the same column or row processing type is repeated over a frame, where N is the number of times the column or row is processed), and the number of descriptors used per column or row (e.g., can the same tile be repeated over a row or column using just one descriptor, or can one part of the row be scanned using a first descriptor and another part of the row be scanned using a second descriptor, etc.). The descriptor ID can be described such that, for example, the descriptor stored in descriptor SRAM 1052 can be retrieved and used to describe a row or column. For example, the descriptor ID can indicate which descriptor to use for a particular column and / or row, and how many times the descriptor is repeated (e.g., N - 1 times, where N is the total number of times the descriptor is used). In an example, there can be a set number (e.g., 64) of descriptors, and the descriptor ID can be used to determine which descriptor should be used for a particular column and / or row. In this way, the hardware sequencer controller 1060 examines the higher-order structure of the frame above the basic descriptors from descriptor SRAM 1052, thereby enabling the simplification of the resources required by the DMA engine 1056 to perform the same data transfer. Additionally, the HW sequencer control unit 1060 can prefetch tiles in advance, for example, using the register control unit 1058, to shorten the latency, and in response to a request from the DMA engine 1056, the tile data can be immediately available.

[0209] During computation, the HW Sequencer Control Unit 1060 can read the image structure (e.g., frame format 1070) from the HW Sequencer Command SRAM 1054 along with the descriptor information from the descriptor SRAM 1052, and combine the information for the DMA Engine 1056 to sequence across frames. Thus, rather than requiring separate coding for each descriptor, trigger, channel, etc., in the DMA Engine 1056, the HW Sequencer Control Unit 1060 can read the image structure, capture the descriptors, and sequence across frames in the appropriate descriptor format for the DMA Engine 1056. The Register Control Unit 1058 can, in embodiments, help control the scan order, prefetch, and / or other frame addressing controls. The HW Sequencer Control Unit 1060 further simplifies the VPU code, so that the VPU does not need to consider multiple channels. Instead, the VPU can request tile, then the next tile, then the next tile, and so on. Since the HW sequencer control unit 1060 understands its current position within the frame, it understands the next tile to be fetched for the DMA engine 1056, and the DMA engine 1056 does not need to keep tracking this information internally.

[0210] Thus, system 1050 can be backward compatible with previous techniques because the system can still support the use of various descriptors, triggers, channels, etc., and can be understood at the frame level to reduce complexity. System 1050 can support padding of images with various pixel padding sizes at all corners of the frame, vertical and / or horizontal tile overlap that enables the VPU to access adjacent tiles for processing along tile boundaries, and scanning of frames in different scan orders. In addition, system 1050 can support automatic tile offset adjustment by HW sequencer control unit 1060 in the destination VMEM. Since the frame descriptors are linked via hardware, the user does not need to link or stitch the descriptors together. HW sequencer control unit 1060 can manage the sequence processing of descriptors / tile-wide addresses across the frame without further increasing programming complexity, and HW sequencer control unit 1060 can also prefetch tiles to improve performance.

[0211] In some embodiments, the descriptors may be included in the image or frame structure rather than being stored separately in descriptor SRAM 1052. For example, if compatibility with legacy products is not implemented, all of the sequence processing structure and tile structure can be described within the frame structure. In such an example, the frame format of FIG. 10I may include additional information for descriptors such as tile width, trigger type, etc., such that HW sequencer control unit 1060 can use the same information as when the descriptors are stored separately in descriptor SRAM 1052.

[0212] Referring to Figure 10J, which is an example of the frame format 1070 of Figure 10I when implemented for a raster scan sequence according to some embodiments of the present disclosure. For example, frame format 1070A is an example of a frame format that uses only one channel, only one trigger, and only one descriptor and performs frame addressing in raster mode. In this example, the tile structure may be 16 × 8. Figure 10K is an example of this tile structure that is sequenced in hardware in a raster scan sequence and performs frame addressing using exemplary frame format 1070A, according to some embodiments of the present disclosure. For example, the same descriptor (e.g., the tile dimensions) may be used for each row of tiles (indicated as "D1" in the visualized Figure 1072), so that the same tile is used 16 times along each row (C1 to C16) and repeated for 8 rows from top to bottom (R1 to R8). The sequence may contain 20 bytes, as shown in frame format 1070A, and each row may have N*2+ bytes, where N represents the number of entries per row (as shown in Figure 10J). Thus, in order to sequence the frames as shown in visualized Figure 1072, frame format 1070A may contain frame repetitions, the number of descriptor rows may be zero, no tile offset, no frame offset, frame padding with 3 rows for left pixel (PL), right pixel (PR), top pixel (PT), and bottom pixel (PB), the row may be repeated 7 times (for a total of 8 rows), the offset of each row is the height of the tile (Ty) (so that each row is offset by the height of the tile), one descriptor may be used with descriptor ID D1, the descriptor may be repeated 15 times for each row (16 times for a total).Therefore, in practice, the HW sequencer control unit can use a descriptor corresponding to D1 (including tile height and tile width) from the descriptor SRAM 1052, and use the image structure of frame format 1072 stored in the HW sequencer control SRAM 1054 to sequence each image tile (16 tiles per row) and each row (R1 to R8) for the destination processor (e.g., VPU). In this way, only one descriptor, only one trigger, and only one channel may be used, which reduces the complexity of programming while also making the DMA system 1050 the primary or controlling component in the interaction between the DMA system 1050 and the VPU.

[0213] In some embodiments, as an extension of the HW sequencer control unit 1060, a DMA trigger mode can be used to instruct the DMA system 1050 to sequence descriptors, thereby reducing software intervention when programming the VPU. For example, the DMA system 1050 can read an image from external memory, arrange the image in a tiled manner, and sequence the tiles for the VPU. To facilitate this, the VPU may have access to start and complete signals. The VPU may be started by the DMA system 1050, and when the VPU has finished processing an instruction block, it can send a complete signal to the DMA system 1050. Thus, the DMA system 1050 (e.g., the HW sequencer control unit 1060) and the VPU can be involved in a handshake mechanism, where the DMA system 1050 is the primary node and the VPU is the secondary node. This DMA trigger mode minimizes the overhead of tile control by the VPU and simplifies the programming model for the DMA engine 1056. For example, double buffering of DMA data movement may not require specific code, and the DMA kernel code may be independent of the VPU kernel code. Therefore, since tile sequencing is handled by the DMA system using the HW sequencer control unit 1060, the DMA trigger mode simplifies the VPU code. The example code below shows the VPU code before and after adding the DMA trigger.

[0214] before

number

[0215] rear

number

[0216] As a result, previously the VPU would request the movement of tiles to VMEM, but now the HW sequencer control unit 1060 controls the sequence processing, so the DMA system 1050 can target the VPU and trigger the movement of tiles to VMEM. In this way, the DMA system 1050 can fetch the data to be processed by the VPU in advance, and when the VPU indicates that processing is complete, the DMA system 1050 can immediately make the next data to be processed available (for example, in VMEM) and indicate this to the VPU as well.

[0217] When processing a frame, the HW Sequencer Control Unit 1060 can retrieve descriptors (which may indicate tile dimensions, trigger type, etc.) from the descriptor SRAM 1052, and can also retrieve image structures from the HW Sequencer Command SRAM 1054. The HW Sequencer Command 1060, together with the Register Control Unit 1058, can then begin scanning a first row or column according to the scan order and using the first (in this embodiment, only) descriptor, and then, if two or more descriptors are used, can move on to a second descriptor based on the number of repetitions (e.g., 1-N), and so on. Once each tile is determined, the DMA engine 1056 can retrieve the tile data from the source data and write the tile data to the destination data (e.g., in VMEM). Once the data is written to the data destination, the processor (e.g., VPU) may be notified by the HW Sequencer Control Unit 1060 that the data is available for the processor to begin processing. Next, the DMA system 1050 can, during processing, fetch the next tile of data based on the sequence from the HW sequencer control unit 1060 and write the data to the data destination. As a result, when the processor indicates that processing is complete, the HW sequencer control unit 1060 can indicate to the VPU (via the handshake mechanism) that the next data to be processed is available, and so on until processing is complete.

[0218] Referring here to Figure 10L, each block of Method 1080 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 1080 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 1080 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 1080 is described in relation to the system in Figure 10H, and Method 1080 may be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0219] Figure 10L is a flowchart of Method 1080 for a DMA system including a hardware sequencer, according to some embodiments of the present disclosure. Method 1080 includes the step in block B1002 of obtaining tile structures from descriptor memory and frame structures corresponding to frames from HW Sequencer Command Memory. For example, the HW Sequencer Control Unit 1060 can obtain descriptors from descriptor SRAM 1052.

[0220] Method 1080 includes the step of sequence-processing the acquisition of frame tiles from source memory in block B1004. For example, in this embodiment, the hardware sequencer control unit 1060, together with the register control unit 1058, can sequence-process the acquisition of tiles from source memory by the DMA engine 1056 according to the description of the tiles from the frame (or image) structure and descriptor.

[0221] Method 1080 includes the step in block B1006 of writing data corresponding to the acquired tile to the destination memory. For example, the DMA engine 1056 may write data corresponding to the acquired tile to the destination memory (e.g., VMEM) for processing by the destination processor (e.g., VPU).

[0222] Method 1080 includes the step in block B1008 of supplying an instruction to the processor associated with the destination memory that the acquired data has been stored in the destination memory. For example, the HW sequencer control unit 1060 may indicate to the processor that the data for the next tile is ready for processing.

[0223] Method 1080 includes the step in block B1010 of receiving an instruction that processing of the acquired data is complete. For example, once processing is complete, the processor may indicate to the DMA system 1050 that processing is complete, at which point the next tile of data may be loaded (or preloaded) into the destination memory, and the DMA system 1050 may indicate this to the processor.

[0224] Using a VPU to configure a DMA system for region-dependent data movement When a processing controller fetches a known data pattern, it can configure a direct memory access (DMA) system, and a processor (e.g., a vector processing unit (VPU)) can trigger and sequence the DMA. However, when processing various data points or features for irregular or unknown data patterns, the dynamic calculation of feature or object locations can present challenges in resetting data movement. For example, in object tracking algorithms, feature tracking algorithms, object detection algorithms, deep learning algorithms using variable-size regions of interest (ROIs), and / or other region-dependent data movement algorithms, the DMA system needs to dynamically adjust address-data pairs so that a processor, such as a VPU, can obtain the appropriate information for processing. In conventional systems, when fetching an unknown data pattern, such as in object tracking, the processing controller (e.g., an R5 processor core controlling a programmable vision accelerator (PVA)) may need to interrupt the processing cycle to determine the updated information calculated by the processor (e.g., a VPU) and reset the DMA for the next iteration. Therefore, the processing controller introduces extra latency to tracking algorithms that require short response times, for example.

[0225] Considering the shortcomings of conventional systems that require intervention by a processing controller, the systems and methods of the present disclosure use a DMA and a processor (e.g., a VPU) to set up a tightly coupled processing loop that allows the DMA to reconfigure its descriptor based on the processor's output. Thus, the DMA can handle specific algorithms that are dynamically reprogrammed at runtime and require region-dependent data movement. Using this VPU configuration mode, the DMA's descriptor can be updated to track feature data, including location, based on VPU calculations at runtime. For this purpose, the VPU can specify a list of address-data pairs in memory, such as VMEM, and then trigger the DMA to update its own descriptor and collect data from regions with newly calculated addresses. By relying on the interface between the VPU and the DMA, once the processing controller (e.g., an R5 or ARM processing core) has initially configured the VPU and DMA to begin processing, intervention by the processing controller may be unnecessary. Thus, this bulk, high-speed, synchronous MMIO access for updating feature descriptors reduces latency for object tracking, feature tracking, object detection, deep learning, and / or other algorithms involving region-dependent data movement.

[0226] Referring here to Figure 11A, Figure 11A shows a data flow diagram 1100 of the process of configuring a direct memory access (DMA) system using a vector processing unit (VPU) according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the process 1100 may be performed by a system including components, features, and / or functions similar to those of the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15.

[0227] A system for executing process 1100 may include a processing controller 1102 (e.g., an R5 processor, an ARM processing core, an instruction set architecture (ISA), an X86 architecture, etc.), a direct memory access (DMA) system 1104, a vector processing unit (VPU) 1108 (or another processor type), vector memory (VMEM) 1110 (or another memory type), and descriptor RAM 1106. In practice, the VPU configuration mode can set the DMA descriptor by writing a sequence of non - consecutive address / data pairs to the DMA descriptor SRAM. Process 1100 can be described with respect to an exemplary feature or object - tracking algorithm. However, this is not intended to be limiting, and process 1100 and the underlying system can be used to execute any type of algorithm involving region - dependent data movement.

[0228] For example, a first operation may include a processing controller 1102 that configures the DMA 1104 and the VPU 1108 to perform processing on some data and then triggers both the DMA 1104 and the VPU 1108 to perform the processing. For example, the processing controller 1102 can load the memory start point into the descriptor RAM 1106 for processing and can also set the registers of the VPU 1108 for a particular type of operation that the VPU 1108 will perform on the data.

[0229] In a second operation, the VPU 1108 can trigger the DMA 1104 to read an initial feature data point in the VMEM 1110. For example, since the VPU 1108 needs data from the DMA 1104 to start work, the VPU 1108 configures the DMA 1104 to load a data point at a location in the VMEM 1110 known to the VPU 1108 to obtain the processing data.

[0230] In the third operation, the VPU1108 can process the current feature data set and calculate the location of the next object or feature to be tracked. As a result, the VPU1108 can now have the calculated new, i.e., updated location of the feature or object to be tracked.

[0231] In a fourth operation, the VPU 1108 can update the VMEM 1110 at the updated location using the VPU configuration format (described in relation to Figure 11B), and then trigger the DMA 1104 to update the DMA descriptor in the descriptor RAM 1106. For example, Figure 11B is Table 1120 showing the VPU configuration format written to vector memory (VMEM) by the VPU and read by the DMA system according to some embodiments of the present disclosure. For example, the format may include 4 bytes for the address and 4 bytes for the data for each address / data pair.

[0232] In a fifth operation, the DMA1104 can update the descriptors in the descriptor RAM1106 to obtain the appropriate data for the next iteration of processing by the VPU1108. For example, the DMA1104 can read the address / data pair format VPU configuration format and patch the operation descriptor at the updated location. In the embodiment, there may be a one-to-one correspondence between feature points and descriptors, and therefore each tracked feature, object, or point may contain an associated descriptor. In this way, the address / data pairs for each tracked feature, object, or point may be updated over time using the individual descriptors.

[0233] In the sixth operation, DMA1104 can use the newly updated descriptor in descriptor RAM1106 to fetch new feature data for a location. For example, DMA1104 can indicate to VPU1108 that the descriptor has been updated, and VPU1108 can trigger DMA1104 to read the new data from VMEM1110, and so on.

[0234] As a result, after the first setup operation by the processing controller, operations 2 through 6 are repeated to form a tightly synchronized VPU setup loop that does not require intervention from the processing controller, thereby reducing latency by taking into account the short response time required by the tracking or detection algorithm. In addition, since the DMA1104 overwrites the addresses in memory with the new updated addresses, the DMA1104 updates the code that it needs to look into to understand what to fetch next. This improves throughput compared to conventional systems that relied on the control bus to update registers according to addresses and data. Thus, the advantages of defining an address / data protocol are realized when a variable amount of address location with a variable amount of data can be updated, along with how the address / data pair is updated. This allows the DMA1104, which can have a width wider than the control bus width (for example, 512 bits versus 32 bits each), to update up to eight address / data pairs at once (for example, but not limited to these) (assuming each address / data pair is defined using 8 bytes, as shown in Figure 11B).

[0235] Furthermore, although the DMA is shown to be updated using the VPU configuration mode of process 1100, additional or alternative elements or components of the system may be updated. For example, the instruction cache of VPU 1108 may be updated using the VPU and in a similar manner. As another example, the program of the hardware sequencer to be updated may be written to update the hardware sequencer memory by providing address data. This would essentially involve writing the new program to the hardware sequencer RAM, such as the hardware sequencer RAM 1054 for the hardware sequence controller 1060 in Figure 10H.

[0236] Referring here to Figure 11C, each block of Method 1150 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 1150 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 1150 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 1150 is described in relation to the system in Figure 11A, and Method 1150 may be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0237] Figure 11C is a flowchart of a method 1150 for configuring a DMA system using a VPU, according to some embodiments of the present disclosure. Method 1150 includes, in block B1102, a step of using a processor and at least in part on first data written to memory using a DMA system to compute a first output corresponding to one or more first updated locations of a tracked feature. For example, VPU 1108 can use DMA 1104 to access data from VMEM 1110 written to VMEM 1110 and process the data to compute the locations of one or more objects corresponding to a tracked feature, object, point, etc.

[0238] Method 1150 includes the step in block B1104 of updating memory using a processor to include second data representing one or more address / data pairs corresponding to one or more first updated locations. For example, after calculating one or more locations, VPU 1108 may update VMEM 1110 in a format such as the format described in relation to Figure 11B.

[0239] Method 1150 includes the step in block B1106 of updating one or more descriptors corresponding to a tracked feature using a DMA system and at least partially based on one or more address / data pairs. For example, DMA 1104 may access an address / data pair in VMEM 1110 and use the address / data pair to update a descriptor in descriptor RAM 1106 for the next read operation.

[0240] Method 1150 includes the step in block B1108 of writing third data to memory using a DMA system and at least partially based on one or more descriptors. For example, DMA 1104 may write updated data corresponding to address / data pairs identified using descriptors to VMEM 1110.

[0241] Method 1150 includes the step in block B1110 of computing a second output corresponding to one or more second updated locations of the tracked features, using a processor and at least partially based on third data. For example, when the updated data enters VMEM1110, VPU1108 can compute the next set of updated address / data pairs corresponding to the tracked features, objects, points, etc., and this process can be repeated until processing is complete.

[0242] Detection of Permanent Failures in Programmable Vision Accelerators (PVAs) In safety-critical applications such as autonomous and semi-autonomous machine applications, there are stringent requirements for detecting and isolating permanent faults. For example, when running applications in deep learning, computer vision, sensor processing, and / or other machines, permanent fault detection must be performed regularly and within an allocated time budget, not only to enable accurate testing but also to enable the application to run properly, for example, with low latency. For automotive safety integrity level (ASIL) D, applications running on autonomous or semi-autonomous machines may require coverage of 90% or more of permanent faults. To achieve this, 100% coverage may be required while meeting the time budget for each specific application to run with low latency. Traditional methods use built-in self-tests (BISTs) to identify faults, but these BIST techniques do not include sufficient coverage, result in excessively long latency for the system, and / or fail to meet the time budget for running specific applications.

[0243] In light of the shortcomings of these conventional methods, the System and Method can perform BIST on a Multi-Input Signature Register (MISR) to perform fault detection in, for example, a programmable vision accelerator (PVA) of a system-on-a-chip (SoC). For example, in various embodiments of the Disclosure, the PVA may include one or more DMA systems and one or more VPUs controlled using one or more processing controllers (or control processors), such as an R5 processor, ARM processor, CPU, and / or similar. Therefore, each component of the PVA may require testing, and the System and Method can perform BIST on a MISR to detect 100 percent of permanent faults. In this way, permanent fault detection can be performed that covers 100 percent of both control logic and data logic blocks, reports errors directly to safety processors to reduce latency, and customizes specific applications to meet the associated runtime budget.

[0244] In various embodiments, MISR can be used in a PVA to implement BIST, a software logic mechanism for permanent fault detection. The MISR hardware described herein with respect to Figure 12A and / or Figure 12B may include, for example, cyclic redundancy check (CRC) hardware initialized with a known seed value using a processing controller. When a PVA application is run, the processing controller can allocate a portion of the timing budget, for example, less than 10% of the timing budget, to run a known software MISR test with known inputs, which has a deterministically pre-calculated output of the correct signature or golden value. For example, if the timing budget corresponds to 30 frames per second, a timing budget equivalent to less than 3 frames may be allocated to the MISR test. The processing controller can start the MISR test at the allocated time, wait for the test to complete, and then terminate the MISR CRC calculation. Once the test is complete, the MISR hardware can read back the final CRC value and check the final CRC value against the pre-calculated golden value. In case of a mismatch, the MISR hardware can report the error directly to the SoC's safety processor, which can then take further steps to handle the safety error, such as ignoring the application output or addressing or avoiding a permanent failure.

[0245] Therefore, the MISR hardware within the DMA block can monitor one or more transactions (e.g., all in the embodiment) on one or more (e.g., all in the embodiment) of the PVA's highly extensible interface (AXI) master ports. In the embodiment, the security integrity of the PVA can be checked for permanent defects that could corrupt output stages, such as output information, which may be exhausted by the PVA and / or another engine during application execution. Thus, the MISR hardware can detect errors across various blocks of the PVA (e.g., processing controllers, VPUs, and DMA systems) because all these components cooperate and interact with each other in producing output stages. Signatures computed by the MISR hardware can represent the state of these various blocks of the PVA throughout the duration of the MISR test.

[0246] In embodiments, the MISR scheme may include CRC checks on both the write address (e.g., 40-bit control) and the write data (e.g., 512-bit data) leaving the AXI master port. This feature may allow control path failures (e.g., addressing errors) to be isolated from data path failures (e.g., calculation errors). The MISR hardware configuration described herein may allow each DMA to check the AXI port. In embodiments, control bits may be used to disable writes of address and data outputs for all channels involved in MISR calculations in order to conserve bandwidth consumption during memory allocation in the memory subsystem. Furthermore, the MISR scheme may include per-channel control register bits to exclude or mask specific channels from MISR calculations, for example, to isolate unsafe channels. In embodiments, the DMA uses IEEE 802 and MPEG CRC-32 primitive polynomials for MISR CRC:X 32 +X26 +X 23 +X 22 +X 16 +X 12 +X 11 +X 10 +X 8 +X 7 +X 5 +X 4 +X 2 +X + 1 can be calculated. The MISR SET register can be used to set the CRC initial value (e.g., the seed value) for both address and data CRC calculations. The MISR REF register can be used to compare the CRC values in both address and data CRC calculations.

[0247] To support a MISR for 512-bit data, 8:1 bit data compression can be applied. For example, each data byte can be compressed to 1 data bit through an 8>1 exclusive OR (XOR) operation, forming a 2×32-bit message data. To support a 40-bit address of the MISR, 9 most significant bits can be compressed. For example, the 9 most significant address bits can be compressed through a 9>1 XOR operation, forming a 32-bit message address. Using the diversity of test patterns and instructions, aliasing related to compression can be covered. When there are an even number of errors in a byte of the output image, the probability of aliasing can be low because the error does not cause an address CRC error. In addition, for an output image having the same pattern at the same even error bit locations over the entire period of the MISR test, a reference CRC can be calculated, so the possibility of aliasing can be low. Aliasing has been shown to cause an average loss of 0.25% in coverage during testing. Such data compression with low aliasing is valuable because the bus width is, for example, 512 bits in the embodiment, and without compression, the MISR test may not meet the system latency or execution time budget.

[0248] The MISR timer register can be used to time out the MISR calculation, and the MISR timer register can be decremented with each AXI clock. The timeout feature can be useful in case of failures where the MISR test may hang up, preventing the MISR hardware from reporting an error. The processing controller can use a software event to stop the MISR calculation when the MISR test is finished. The DMA system can compare the MISR REF value to the MISR VAL value of both the data output and address output of the MISR test, and the DMA hardware can update the MISR status register based on the comparison result. For example, the MISR status register may contain one of the following values: 0: Idle, 1: Complete, data failed, 3: Busy, 4: Complete, both address and data failed, 5: Complete, failed due to timeout, 6: Reserved, and 7: Complete, passed. In the event of a MISR timeout error, the DMA can generate a timeout signal to the safety processor, and in the event of a CRC check error of data and / or address, the DMA can assert a safety error to the safety processor.

[0249] Referring to Figure 12A, Figure 12A is a diagram of an internal self-test (BIST) system performing cyclic redundancy check (CRC) calculations of a programmable vision accelerator (PVA), according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the MISR hardware 1250 may include components, features, and / or functions similar to those of the DMA system 1050 in Figure 10H, the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15. For example, the MISR hardware 1200 may be included in the DMA block of a PVA, as shown in Figure 10H. Thus, the MISR hardware 1200 can operate in the output stage of data movement, and addressing (or control) can utilize (tapping into) the output of the DMA engine 1056. In the example in Figure 12A, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used depending on the embodiment.

[0250] During operation, the processing controller can control one or more processing components of the system, such as the DMA system 1050, the MISR hardware 1200, and the VPU. When performing MISR testing of the DMA system 1050, in an embodiment, the test code may include all zeros, all ones, alternating zeros and ones, and / or irregular code sequences. In this way, a high level of coverage can be achieved for the DMA system 1050. For example, when testing the VPU, the test code may include application-specific or customized code. For example, during a coverage test for a particular application, the components or parts of the VPU used (e.g., registers, logic mechanisms, etc.) may be determined, and the test code may be generated so that those specific components or parts of the VPU are included when the test code is executed. For example, this irregular data corresponding to various instructions may be included in the test code so that the test is sequenced by various instructions to use various areas of the VPU's logic mechanisms. In this way, the coverage of the VPU in general, and specifically for the specific application running on the VPU, is increased. By performing DMA and VPU tests in this manner, and because the processing controller is involved in the control and interaction between various components (e.g., DMA system 1050 and VPU), the processing controller can achieve high coverage because the output of data movement and addressing is affected by the interactions of the processing controller.

[0251] During testing, if various code patterns are used, the code patterns may be used in alternating patterns, or one code may be used in a first time frame (e.g., a time equivalent to 30 fps), another code may be used in a second time frame (e.g., a time equivalent to 30 fps), another code may be used in a third time frame (e.g., a time equivalent to 30 fps), and so on. For example, in the DMA code example, the code 0 may be used in the first time frame, then the code 1 may be used in the second time frame, then a code that alternates between 0 and 1 (e.g., 0101010101...) may be used in the third time frame, then an irregular code (e.g., 011100100010...) may be used in the fourth time frame, and so on.

[0252] In practice, for example, when testing DMA, the processing controller can interact with the MISR control unit 1206 to write set reference values ​​to the MISR data set register 1210 and the MISR address set register 1216. These values ​​may differ for data and addresses and may be called seed values ​​for CRC calculation. The processing controller can then initialize the channel performing data movement within the DMA engine 1056, and since the processing controller knows the location of the test code in memory, it can sequence the DMA engine 1056 with data for the MISR test using a descriptor (for example, set by the processing controller in the descriptor SRAM 1052). The processing controller can enable the MISR test by setting a timer 1226 to the MISR hardware 1200, and then trigger the channel of the DMA engine 1056 to start reading test data from the source destination and output the data to the MISR hardware 1200 for the MISR test. Therefore, when testing DMA, since data movement is being tested (e.g., proper addressing and accurate data at the addressing location), the MISR hardware 1200 can utilize the output of the DMA engine 1056 when executing the data movement of the test code. This output stage can be shown as external memory in Figure 12A, which, when sequenced by the processing controller, can be sent to one data channel and one address channel at a time. For example, in the data channels, the processing controller can sequence through, for example, 16 data channels, and the corresponding AXI write data (wdata) for each channel can be supplied, for example, sequentially through the data CRC calculation unit 1202 of CH0 to CH16. For example, the processing controller can configure the channel output register 1220 to sequence through the channels one at a time according to a sequence set by the processing controller.In the embodiment, the channel mask register 1208 (programmed by the MISR control unit 1206, for example, based on interaction with the processing controller) may be set by the processing controller to mask out or exclude various channels, such as untested channels, from the CRC calculation. This masking may be performed using AND gates in the embodiment. If one or more channels are masked out, the golden value in the MISR data reference register 1222 (which may be supplied to the MISR control unit 1206 by the processing controller) can only correspond to the CRC calculation of unmasked channels. For each unmasked channel, the channel data generated using the test code read from memory (for example, with or without compression) can be applied to the polynomial of the CRC data calculation unit 1202 to generate the MISR data value 1214 for that channel. Once channel calculation is complete, the processing controller can receive instructions to send the next channel of data to the CRC calculation unit 1202 and calculate the next MISR data value 1214, and so on, until each unmasked channel has a corresponding MISR data value 1214. Once each MISR data value 1214 for a particular iteration has been calculated, these values ​​1214 can be combined to generate a final MISR data value that can be compared to a golden value in the MISR data reference register 1222, and a final MISR data status can be generated (which may include a status corresponding to the values ​​0-7 above, for example).

[0253] As another example, with respect to an address channel, the processing controller can sequence-process, for example, through each of 16 address or control channels, and the corresponding AXI write address (waddress) for each channel can be supplied, for example, sequentially through the address CRC calculation unit 1204 for CH0 to CH16. In an embodiment, the channel mask register 1208 can be configured by the processing controller to mask out or exclude various channels, for example, untested channels, from the CRC calculation. This masking can be performed using AND gates in an embodiment. If one or more channels are masked out, the golden value in the MISR data reference register 1224 can only correspond to the CRC calculation of the unmasked channels. For each unmasked channel, the channel address generated using a test code read from memory (for example, with or without compression) can be applied to the polynomial in the CRC address calculation unit 1204 to generate the MISR address value 1218 for that channel. Once the channel calculation is complete, the processing controller can receive instructions to send the next channel of address data to the CRC calculation unit 1204 and calculate the next MISR address value 1218, and so on, until each unmasked channel has a corresponding MISR address value 1218. Once each MISR address value 1218 for a particular iteration has been calculated, these values ​​1218 can be combined to generate a final MISR address value that can be compared to a golden value in the MISR address reference register 1224, and a final MISR address status can be generated (which may include, for example, a status corresponding to the values ​​0-7 above).

[0254] In some embodiments, the MISR test may be iterative, such that the first code may be processed, the output may be tested, and then the output may be used for the next iteration which may be tested, and so on. In such embodiments, the MISR test may consist of multiple stages, and completion of the MISR test may consist of executing each stage.

[0255] When the MISR hardware 1200 is used specifically to test the VPU, for example, the DMA system 1050 can move the test code to VMEM, the VPU can process the test code and write the results back to VMEM, and the DMA engine 1056 can read the results from VMEM and return them to the destination location. When writing the results back to the destination location, the MISR hardware 1200 can utilize the DMA output to perform MISR on the data (e.g., including data and addresses) and perform MISR similar to that discussed herein. In this way, the interaction between the VPU and the test code can be tested using the MISR hardware 1200.

[0256] After the MISR test is complete, the processing controller can receive an interrupt. For example, the processing controller can receive a completion interrupt and, if there are no errors, can wait for the next MISR test cycle. If the interrupt is an error interrupt, the type of error, such as data failure, address failure, or both failures, can be determined, and a safety error can be asserted to the safety processor. In some embodiments, if the MISR hardware 1200 hangs up or becomes idle (for example, a time-out error occurs), the DMA can generate a time-out signal to the safety processor of the SoC.

[0257] In some embodiments, to speed up MISR calculation and to calculate the CRC on one or more channels (e.g., all 16 in the embodiment) without sequential or multi-stage channel-MISR calculations, channels can be multiplexed based on the channel ID present in the AXI ID field, and channel calculations can be parallelized. For example, because CRC calculations are completed at varying speeds, the method in Figure 12A involved sequential processing of channels. However, using the system in Figure 12B described below, these calculations can be completed in parallel. For example, once the processing controller has finished MISR calculations, the MISR controller can sequence all channel outputs to calculate a final signature that can be compared to a reference value or golden value for both the address output and the data output. This feature can speed up the detection of permanent failures without the need for an additional programmer register interface, for example, because the same control register may be used for all channels.

[0258] Referring therefore to Figure 12B, Figure 12B is a diagram of an internal self-test (BIST) system for parallel channel cyclic redundancy check (CRC) calculation of a programmable vision accelerator (PVA), according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are presented merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory. In some embodiments, the MISR hardware 1250 may include components, features, and / or functions similar to those of the DMA system 1050 in Figure 10H, the exemplary autonomous vehicle 1300 in Figures 13A–13D, the exemplary computer processing device 1400 in Figure 14, and / or the exemplary data center 1500 in Figure 15. For example, the MISR hardware 1250 may be included in the DMA block of a PVA, as shown in Figure 10H. Thus, the MISR hardware 1250 can operate in the output stage of data movement, and addressing (or control) can utilize the output of the DMA engine 1056. In the example in Figure 12A, there may be 16 AXI data channels and 16 AXI address (or control) channels. However, this is not intended to be limiting, and any number (and / or type) of data and / or address channels may be used depending on the embodiment.

[0259] MISR hardware 1250 can operate similarly to MISR hardware 1200 in Figure 12A, except that MISR hardware 1250 can be configured for CRC calculation of parallel data channels and parallel address channels. For example, the processing controller can configure MISR data set register 1256 to set seed values ​​or reference values ​​for each data CRC calculation unit 1260A to 1260N (each corresponding to AXI data channels 0 to 15), and MISR address set register 1258 to set seed values ​​or reference values ​​for each address CRC calculation unit 1262A to 1262N (each corresponding to AXI address channels 0 to 15). Then, the processing controller, similar to that described with respect to Figure 12A, can trigger data movement (e.g., for DMA testing) and / or VPU processing (e.g., for VPU-specific testing) for the DMA system 1050 to move data around, and MISR hardware 1250 can utilize the output stage to be tested.

[0260] Therefore, the processing controller can send 16 channels of data to the multiplexer (mux) 1252 and 16 channels of address data to the multiplexer (mux) 1254. The mux 1252 can then supply the data for the corresponding channel to the corresponding CRC calculation units 1260A to 1260N (for example, AXI data for channel 0 to the CRC calculation unit 1260 for channel 0, data for channel 1 to the CRC calculation unit 1260B for channel 1, and so on), and each CRC calculation unit 1260 can calculate the MISR data values ​​1284A to 1284N using a CRC polynomial with the data and reference values ​​(for example, the CRC calculation unit 1260A for channel 0 can calculate the value 1284A for MISR data 0, the CRC calculation unit 1260B for channel 1 can calculate the value 1284B for MISR data 1, and so on). Next, the MISR data values ​​1284A to 1284N can be sequenced and output from the multiplexer (mux) 1264 according to the MISR sequence from the MISR control unit 1270 set by the processing controller. In embodiments such as those described with reference to Figure 12A, one or more channels may not be included in a particular MISR test, so a channel mask register 1268 may be set by the processing controller to update the MISR sequence, and therefore, the MISR data values ​​1284 corresponding to one or more masked channels are not supplied to the data CRC calculation unit 1274 for channels 0 to 16 to calculate the final CRC value. For unmasked channels, the MISR data values ​​1284 can be output from the mux 1264 according to the MISR sequence. In this way, the MISR data values ​​1284 are forcibly output according to the MISR sequence, rather than being sent to the CRC calculation unit 1274 depending on when the CRC calculation is completed, so that different channels and different calculation times for the CRC calculation unit 1260 are taken into account. When the MISR sequence of MISR data value 1284 is output from mux1264 to the CRC calculation unit 1274, the CRC calculation unit 1274 calculates the final CRC value and can store the final CRC value in the VAL register 1276.Next, the final CRC value in the VAL register 1276 is compared with the golden value in the MISR data reference register 1272 (set by the processing controller via the MISR control unit 1270) to determine the status of the MISR data.

[0261] Similarly, the processing controller can have the 16 channels of addresses sent to the multiplexer (mux) 1254, which can then supply the corresponding address channels to the corresponding CRC calculation units 1262A to 1262N (for example, the AXI address of channel 0 to the CRC calculation unit 1262 of channel 0, the address of channel 1 to the CRC calculation unit 1262B of channel 1, and so on), and each CRC calculation unit 1262 can calculate the MISR address values ​​1286A to 1286N using a CRC polynomial with the address and reference value (for example, the CRC calculation unit 1262A of channel 0 can calculate the value 1286A of MISR address 0, the CRC calculation unit 1262B of channel 1 can calculate the value 1286B of MISR address 1, and so on). Next, the MISR address values ​​1286A to 1286N can be sequenced and output from the multiplexer (mux) 1266 according to the MISR sequence from the MISR control unit 1270 set by the processing controller. In embodiments such as those described with reference to Figure 12A, one or more channels may not be included in a particular MISR test, so the channel mask register 1268 may be set by the processing controller to update the MISR sequence, and as a result, the MISR address values ​​1286 corresponding to one or more masked channels are not supplied to the address CRC calculation unit 1280 for channels 0 to 16 to calculate the final CRC value. For unmasked channels, the MISR address values ​​1286 may be output from the multiplexer 1266 according to the MISR sequence. In this way, the MISR address values ​​1286 are forcibly output according to the MISR sequence, rather than being sent to the CRC calculation unit 1280 depending on when the CRC calculation is completed, so that different channels and different calculation times for the CRC calculation unit 1262 are taken into account. When the MISR sequence with MISR address value 1286 is output from the multiplexer 1266 to the CRC calculation unit 1280, the CRC calculation unit 1280 calculates the final CRC value and can store the final CRC value in the VAL register 1282.Next, the final CRC value in the VAL register 1282 is compared with the golden value in the MISR address reference register 1278 (set by the processing controller via the MISR control unit 1270) to determine the status of the MISR address.

[0262] The status of the MISR data and the status of the MISR address can be checked and used in the same manner as described above with respect to Figure 12A.

[0263] Referring here to Figure 12C, each block of Method 1290 as described herein includes a computer processing process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 1290 can also be embodied as computer-usable instructions stored in a computer storage medium. Method 1290 may be provided, to name a few, as a standalone application, service, or hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 1290 is described in relation to the system in Figure 12A, and Method 1290 may be executed by any one system, structure, or component, or any combination of systems, structures, or components, including but not limited to those described herein.

[0264] Figure 12C is a flowchart of an execution (BIST) method 1290 for detecting a permanent failure in a PVA, according to some embodiments of the present disclosure. Method 1290 includes, in block B1202, a step of receiving data from multiple channels of a DMA system, one channel at a time, based on sequence processing of a processing controller. For example, MISR hardware 1200 may receive data from one channel at a time (or address data for one channel) according to sequence processing determined by the processing controller.

[0265] Method 1290 includes the step of calculating multiple MISR values ​​in block B1204 by performing a CRC calculation for each channel using the CRC calculation polynomial and the respective data corresponding to the channel in order to calculate the MISR value. For example, for each channel, the CRC calculation unit 1202 (or 1204 in the case of an address) can calculate a MISR data value 1214 (or MISR address value 1216 in the case of an address) using the data (or address) from the channel and the polynomial of the CRC calculation unit 1202 (starting with a seed value from the CRC MISR data set register 1210 or MISR address set register 1216).

[0266] Method 1290 includes the step in block B1206 of calculating a final MISR value using multiple MISR values. For example, the MISR data values ​​1214 (or MISR address values ​​from each channel) from each channel can be combined to generate a final MISR value.

[0267] Method 1290 includes the step in block B1208 of comparing the final MISR value with a signature value. For example, the final MISR value generated from individual MISR values ​​1214 (or address value 1216) may be compared with the signature or golden value of the MISR data reference register 1222 (or, in the case of an address, the MISR address reference register 1224).

[0268] Method 1290 includes a step in block B1210 to output a MISR status based at least partially on the comparison. For example, based on the comparison in block B1208, a status such as data failure, address failure, both failures, complete, etc., may be determined and used to inform the SoC's safety processor where the error status was generated.

[0269] Example of an autonomous vehicle Figure 13A shows an exemplary autonomous vehicle 1300 according to some embodiments of the present disclosure. The autonomous vehicle 1300 (or, as referred to herein as "vehicle 1300") may include, but is not limited to, automobiles, trucks, buses, emergency response vehicles, shuttles, electric or electric bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, underwater aircraft, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels in the "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard), defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE). Vehicle 1300 may be capable of functioning according to one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, vehicle 1300 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0270] Vehicle 1300 may include components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18 wheels, etc.), tires, axles, and other components of the vehicle. Vehicle 1300 may include a propulsion system 1350 such as an internal combustion engine, a hybrid generator, an all-electric engine, and / or another propulsion system type. The propulsion system 1350 may be coupled to a drive train of vehicle 1300, which may include a transmission, enabling the propulsion of vehicle 1300. The propulsion system 1350 may be controlled in response to receiving a signal from a throttle / accelerator 1352.

[0271] A steering system 1354, which may include a steering wheel, can be used to move the vehicle 1300 (for example, along a desired path or route) when the propulsion system 1350 is operating (for example, when the vehicle is moving). The steering system 1354 can receive signals from the steering actuator 1356. In fully automated (level 5) functionality, the steering wheel may be optional.

[0272] The brake sensor system 1346 can be used to activate the vehicle brakes in response to receiving signals from the brake actuator 1348 and / or the brake sensor.

[0273] A controller 1336, which may include one or more system-on-a-chip (SoC) 1304 (Figure 13C) and / or a GPU, can supply signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1300. For example, the controller can transmit signals to activate the vehicle brakes using one or more brake actuators 1348, to activate the steering system 1354 using one or more steering actuators 1356, and to activate the propulsion system 1350 using one or more throttle / accelerators 1352. The controller 1336 may include one or more onboard (e.g., integrated) computer processing devices (e.g., supercomputers) that process sensor signals and can output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver when driving the vehicle 1300. The controller 1336 may include a first controller 1336 for autonomous driving functions, a second controller 1336 for practical safety functions, a third controller 1336 for artificial intelligence functions (e.g., computer vision), a fourth controller 1336 for infotainment functions, a fifth controller 1336 for redundancy in emergency situations, and / or other controllers. In some examples, just one controller 1336 may handle two or more of the above functions, or two or more controllers 1336 may handle just one function, and / or any combination of these functions.

[0274] The controller 1336 can supply signals to control one or more components and / or systems of the vehicle 1300 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, sensors for global navigation satellite systems 1358 (e.g., sensors for global positioning systems), radar sensors 1360, ultrasonic sensors 1362, lidar sensors 1364, inertial measurement unit (IMU) sensors 1366 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1396, stereo cameras 1368, wide-field cameras 1370 (e.g., fisheye cameras), infrared cameras 1372, surround cameras 1374 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1398, speed sensors 1344 (e.g., for measuring the speed of a vehicle 1300), vibration sensors 1342, steering sensors 1340, brake sensors (e.g., as part of a brake sensor system 1346), and / or other sensor types.

[0275] One or more controllers 1336 can receive inputs (represented, for example, by input data) from the instrument cluster 1332 of the vehicle 1300 and supply outputs (represented, for example, by output data, display data, etc.) via the human-machine interface (HMI) display 1334, an audible annunciator, a loudspeaker, and / or other components of the vehicle 1300. Outputs may include information such as vehicle speed, speed, time, map data (e.g., HD map 1322 in Figure 13C), location data (e.g., the location of the vehicle 1300 on the map, etc.), direction, the location of other vehicles (e.g., occupancy grid), and information about objects and the status of objects perceived by the controller 1336. For example, the HMI display 1334 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or information regarding driving operations performed, performed, or expected to be performed by the vehicle (e.g., changing lanes here, taking exit 34B in 3.22 km (2 miles), etc.).

[0276] Vehicle 1300 further includes a network interface 1324 that can use one or more radio antennas 1326 and / or modems to communicate over one or more networks. For example, the network interface 1324 may enable communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The radio antennas 1326 may also enable communication between objects in the environment (e.g., vehicles, portable devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWAN) such as LoRaWAN, SigFox.

[0277] Figure 13B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1300 of Figure 13A according to several embodiments of the present disclosure. The cameras and their respective fields of view in the figure are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be positioned at other locations on the vehicle 1300.

[0278] The camera type may include, but is not limited to, a digital camera that can be adapted for use with the components and / or systems of the vehicle 1300. The camera may operate at Automotive Safety Level (ASIL) B and / or another ASIL. Depending on the embodiment, the camera type may be capable of any image acquisition rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red, transparent, transparent, transparent (RCCC) color filter array, a red, transparent, transparent, blue (RCCB) color filter array, a red, blue, green, transparent (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochromatic sensor color filter array, and / or another type of color filter array. In some embodiments, transparent pixel cameras, such as cameras equipped with RCCC, RCCB, and / or RBGC color filter arrays, may be used to enhance light sensitivity.

[0279] In some applications, one or more cameras can be used to perform advanced driver assistance system (ADAS) functions (for example, as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to include functions such as lane departure warning, traffic sign assistance, and intelligent headlamp control. One or more of the cameras (for example, all of them) can simultaneously record and provide image data (for example, video).

[0280] To block stray light and reflections from inside the vehicle (e.g., reflections from the instrument panel reflected by the windshield mirror) that could interfere with the camera's image data acquisition function, one or more of the cameras may be mounted on a mounting assembly, such as a custom-designed (3D-printed) assembly. In the case of door mirror mounting assemblies, the door mirror assembly may be custom 3D-printed so that the camera mounting plate matches the shape of the door mirror. In some examples, the camera may be integrated into the door mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0281] A camera having a field of view that includes part of the environment in front of the vehicle 1300 (e.g., a front camera) may be used to obtain a surround view that helps identify the path and obstacles ahead, and further assists in generating an occupancy grid and / or providing important information for determining a preferred vehicle path, with the help of one or more controllers 1336 and / or control SoCs. Using the front camera, many of the same ADAS functions as Lida can be performed, including emergency braking, pedestrian detection, and collision avoidance. The front camera may also be used for ADAS functions and systems, including other functions such as Lane Departure Warnings (LDW), Autonomous Cruise Control (ACC), and / or traffic sign recognition.

[0282] For example, various cameras, including monocular camera platforms containing CMOS (complementary metal oxide semiconductor) color image sensors, may be used in a frontal setting. Another example is a wide-field camera 1370, which may be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, crosswalks, or bicycles). Although only one wide-field camera is shown in Figure 13B, any number of wide-field cameras 1370 may be present in the vehicle 1300. In addition, long-range cameras 1398 (e.g., a pair of long-view stereo cameras) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1398 may also be used for object detection and classification, as well as basic object tracking.

[0283] One or more stereo cameras 1368 may also be included in the front configuration. The stereo camera 1368 may include an integrated control unit comprising a scalable processing unit that may include a multi-core microprocessor with a programmable logic circuit (FPGA) and an integrated CAN or Ethernet® interface on a single chip. Using such a unit, a 3D map of the vehicle's environment can be generated, including distance estimation of all points in the image. An alternative stereo camera 1368 may include a small stereoscopic sensor, which may include two camera lenses (one on each side), and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1368 may be used in addition to, or instead of, those described herein.

[0284] Cameras on the sides of the vehicle 1300 that have a field of view including part of the environment (e.g., side-view cameras) may be used for surround view and provide information used not only to create and update the occupancy grid but also to generate side collision warnings. For example, surround cameras 1374 (e.g., four surround cameras 1374 shown in Figure 13B) may be positioned on the vehicle 1300. Surround cameras 1374 may include wide-field cameras 1370, fisheye cameras, 360-degree cameras, and / or similar. In the four-camera example, four fisheye cameras may be positioned at the front, rear, and sides of the vehicle. In an alternative configuration, the vehicle may use three surround cameras 1374 (e.g., left, right, and rear) and one or more other cameras (e.g., front cameras) may be utilized as a fourth surround-view camera.

[0285] A camera having a field of view that includes part of the environment behind the vehicle 1300 (e.g., a rear-view camera) may be used for parking assistance, surrounding view, rear-end collision warning, and creation and updating of occupancy grids. A wide variety of cameras may be used, including, but not limited to, cameras also suitable as front cameras as described herein (e.g., long-range and / or medium-range camera 1398, stereo camera 1368, infrared camera 1372, etc.).

[0286] Figure 13C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1300 of Figure 13A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented merely as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components, or in combination with other components, in any preferred combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0287] Each component, feature, and system of vehicle 1300 in Figure 13C is shown as being connected via bus 1302. Bus 1302 may include a Controller Area Network (CAN) data interface (or, as herein referred to, the “CAN bus”). CAN may be an internal network of vehicle 1300 used to assist in the control of various features and functions of vehicle 1300, such as brake operation, acceleration, braking, steering, and windshield wipers. The CAN bus may be configured to have dozens or even hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to determine steering wheel angle, ground speed, revolutions per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0288] Although bus 1302 is described herein as a CAN bus, this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or instead of, a CAN bus. Furthermore, although only one line is used to represent bus 1302, this is not intended to limit it. For example, there may be any number of buses 1302, including one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1302 may be used to perform different functions and / or for redundancy. For example, a first bus 1302 may be used for collision avoidance, and a second bus 1302 may be used for operation control. In any example, each bus 1302 may communicate with any of the components of vehicle 1300, and two or more buses 1302 may communicate with the same component. In some examples, each SoC 1304, each controller 1336, and / or each computer in the vehicle may have access to the same input data (e.g., input from sensors in vehicle 1300) and may be connected to a common bus such as a CAN bus.

[0289] The vehicle 1300 may include one or more controllers 1336, as described herein with respect to Figure 13A. The controllers 1336 may be used for a variety of functions. The controllers 1336 may be coupled to any of the various other components and systems of the vehicle 1300 and may be used for the control of the vehicle 1300, artificial intelligence of the vehicle 1300, infotainment for the vehicle 1300, and / or similar.

[0290] Vehicle 1300 may include a system-on-a-chip (SoC) 1304. The SoC 1304 may include a CPU 1306, a GPU 1308, a processor 1310, a cache 1312, an accelerator 1314, a data store 1316, and / or other components and features not shown. The SoC 1304 can be used to control vehicle 1300 on various platforms and systems. For example, the SoC 1304 may be combined with an HD map 1322 that can receive map refreshes and / or updates from one or more servers (for example, server 1378 in Figure 13D) via a network interface 1324 in a system (for example, the system of vehicle 1300).

[0291] The CPU 1306 may include a CPU cluster or CPU complex (or, as herein referred to, a "CCPLEX (CPU complex)"). The CPU 1306 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1306 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1306 may include four dual-core clusters, each having its own dedicated L2 cache (e.g., 2MB of L2 cache). The CPU 1306 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the clusters of the CPU 1306 to be active at any given time.

[0292] The CPU1306 can implement power management functions including one or more of the following features: Individual hardware blocks may be automatically clock-gated for dynamic power saving when idle. The clock of each core may be gated when a core is not actively executing instructions by executing WFI / WFE instructions. Each core may be power-gated separately. Each core cluster may be clock-gated separately when all cores are clock-gated or power-gated. And / or each core cluster may be power-gated separately when all cores are power-gated. The CPU1306 can further implement an extended algorithm to manage power states, where tolerable power states and expected wake-up times are specified, and the hardware / microcode determines the optimal power state for cores, clusters, and CCPLEX to enter. Processing cores may support a sequence of entering simplified power states in software where work is offloaded to the microcode.

[0293] The GPU1308 may include an integrated GPU (or, as referred to herein, an "iGPU"). The GPU1308 may be programmable and efficient with respect to parallel workloads. In some embodiments, the GPU1308 may be able to use an extended tensor instruction set. The GPU1308 may include one or more streaming microprocessors, each streaming microprocessor may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1308 may include at least eight streaming microprocessors. The GPU1308 may be able to use a Computational Application Programming Interface (API). In addition, the GPU1308 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0294] The GPU1308 can be power-optimized to deliver maximum performance for automotive and embedded applications. For example, the GPU1308 can be manufactured based on Fin field-effect transistors (FinFETs). However, this is not intended to be limiting, and the GPU1308 may be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor may incorporate several blocks of mixed-precision processing cores. For example, but not limited to this, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA TENSOR CORE for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, streaming microprocessors may include separate parallel integer and floating-point data paths that combine computation and address calculation to enable efficient execution of workloads. Streaming microprocessors may include separate thread scheduling capabilities that enable finer-grained synchronization and coordination between parallel threads. Streaming microprocessors may include a combination of an L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0295] In some examples, the GPU1308 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem, enabling a peak memory bandwidth of approximately 900 GB / s. In some examples, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used in addition to or instead of HBM memory.

[0296] The GPU1308 may include integrated memory technology, including access counters, which enable more accurate transfer of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of memory ranges shared between processors. In some examples, support for the address translation service (ATS) may be used to allow the GPU1308 to directly access the CPU1306's page table. In such examples, if a miss occurs in the GPU1308's memory management unit (MMU), an address translation request may be transmitted to the CPU1306. In response, the CPU1306 can look up its page table, which maps virtual addresses to physical addresses, and send this translation back to the GPU1308. Thus, integrated memory technology allows for a single unified virtual address space for both the CPU1306 and GPU1308's memory, thereby simplifying the programming of the GPU1308 and the porting of applications to the GPU1308.

[0297] In addition, the GPU1308 may include access counters that can track how often the GPU1308 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0298] The SoC1304 may include any number of caches 1312, including those described herein. For example, cache 1312 may include an L3 cache that is available to both the CPU 1306 and the GPU 1308 (e.g., connected to both the CPU 1306 and the GPU 1308). Cache 1312 may include a write-back cache that can keep track of line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, but smaller cache sizes may be used.

[0299] The SoC1304 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or calculations of the vehicle 1300, such as processing DNNs. In addition, the SoC1304 may include a floating-point unit (FPU) or other mathematical or numerical coprocessor type for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1306 and / or GPU1308.

[0300] The SoC1304 may include one or more accelerators 1314 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC1304 may include a hardware acceleration cluster that may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to complement the GPU1308 and offload some of the tasks of the GPU1308 (e.g., free up more cycles of the GPU1308 to perform other tasks). As an example, accelerator 1314 may be used for target workloads that are sufficiently stable and suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (for example, those used for object detection).

[0301] Accelerator 1314 (for example, a hardware acceleration cluster) may include a deep learning accelerator (DLA). A DLA may include one or more tensor processing units (TPUs) that can be configured to enable an additional 10 trillion operations per second for deep learning applications and inference. A TPU may be an accelerator configured and optimized to perform image processing functions (for example, for CNNs, RCNNs, etc.). A DLA may be further optimized for specific neural network types and sets of floating-point operations, as well as for inference. The design of a DLA can enable higher performance per millisecond than a general-purpose GPU and significantly outperform a CPU. A TPU can execute several functions, including post-processor functions as well as single-instance convolution functions, supporting data types INT8, INT16, and FP16 for both features and weights.

[0302] DLA can rapidly and efficiently run neural networks, particularly CNNs, on processed or raw data for any of a variety of functions, including, but not limited to, the following: CNNs for object recognition and detection using data from cameras and sensors; CNNs for distance estimation using data from cameras and sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from cameras and sensors; and / or CNNs for security and / or safety-related events.

[0303] DLA can perform any function of GPU1308, and designers can target either DLA or GPU1308 for any function, for example by using an inference accelerator. For example, a designer could concentrate CNN processing and floating-point operations on DLA, and offload other functions to GPU1308 and / or other accelerators 1314.

[0304] Accelerator 1314 (for example, a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver-assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can strike a balance between performance and flexibility. For example, each PVA may include, for example, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0305] A RISC core can interact with an image sensor (for example, one of the image sensors in the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may include memory of any capacity. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0306] DMA can enable PVA components to access system memory independently of the CPU 1306. DMA can support any number of features used to optimize the PVA, including but not limited to supporting multidimensional addressing and / or cyclic addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0307] A vector processor can be a programmable processor designed to efficiently and flexibly program computer vision algorithms and to implement signal processing functions. In some examples, a PVA may include a PVA core and partitions for two vector processing subsystems. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripheral devices. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction multiple data (SIMD) or very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can increase throughput and speed.

[0308] Each vector processor may include an instruction cache, which may be coupled to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to utilize data parallelism. For example, in some embodiments, multiple vector processors included in just one PVA may run the same computer vision algorithm, but in different areas of the image. In other examples, the vector processors included in a particular PVA may run different computer vision algorithms simultaneously on the same image, or even run different algorithms on consecutive images or parts of images. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, PVAs may include additional error correction code (ECC) memory to enhance the overall system safety.

[0309] Accelerator 1314 (for example, a hardware acceleration cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1314. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible from both the PVA and DLA, for example but not limited to these. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone network that provides high-speed access to the PVA and DLA's memory. The backbone network may include an on-chip computer vision network interconnecting the PVA and DLA to the memory (for example, using an APB).

[0310] An on-chip computer vision network may include an interface that allows both the PVA and DLA to determine whether to supply a ready and valid signal before transmitting any control signals / addresses / data. Such an interface can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0311] In some embodiments, the SoC1304 may include a real-time ray-tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray-tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of an object (e.g., within a world model) and generate real-time visualization simulations for purposes such as radar signal interpretation, sound propagation synthesis and / or analysis, sonar system simulation, general wave propagation simulation, comparison with Lidar data for self-localization and / or other functions, and / or other applications. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray-tracing related operations.

[0312] Accelerator 1314 (e.g., hardware accelerator cluster) has diverse applications for autonomous driving. PVA can be a programmable vision accelerator that can be used in key processing stages of ADAS and autonomous vehicles. PVA's capabilities are well-suited to algorithmic domains that require predictable processing with low power and low latency. In other words, PVA performs semi-dense or dense regular computations well even for small datasets that require predictable execution times with low latency and low power. Therefore, for autonomous vehicle platforms, PVA is designed to run typical computer vision algorithms because it is efficient in object detection and integer mathematics operations.

[0313] For example, according to one embodiment of the technology, PVA is used to perform computer stereoscopic vision. A semi-global matching-based algorithm may be used in some examples, but this is not intended to be limiting. Many applications of Level 3-5 autonomous driving require motion estimation / stereo matching in motion (e.g., 3D reconstruction based on motion (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereoscopic vision functions for input from two monocular cameras.

[0314] In some applications, PVA can be used to perform dense optical flow, following the processing of raw radar data (e.g., using 4D Fast Fourier Transform) to provide processed radar. In other applications, PVA is used to process time-of-flight depth, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0315] Using DLA, any type of network can be run to improve control and driving safety, for example, a neural network that outputs a confidence score for each object detection. Such confidence scores can be interpreted as probabilities, or the relative "weight" of each detection compared to other detections. These confidence scores allow the system to further determine which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections exceeding this threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically apply emergency brakes, which is clearly undesirable. Therefore, only the most reliable detections should be considered triggers for the AEB. DLA can run a neural network to regress confidence scores. The neural network may receive, as input to the neural network, a subset of at least some parameters, including, among other things, the dimensions of the bounding box, ground surface estimates obtained (e.g., from another subsystem), and the output of an inertial measurement unit (IMU) sensor 1366, which correlates with the orientation, distance, and 3D position estimates of the vehicle 1300 obtained from the neural network and / or other sensors (e.g., a lidar sensor 1364 or a radar sensor 1360).

[0316] The SoC1304 may include a data store 1316 (for example, memory). The data store 1316 may be on-chip memory of the SoC1304 that can store neural networks to be run on the GPU and / or DLA. In some examples, the data store 1316 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1312 may comprise an L2 or L3 cache 1312. References to the data store 1316 may include references to memory associated with the PVA, DLA, and / or other accelerators 1314, as described herein.

[0317] The SoC1304 may include one or more processors 1310 (e.g., embedded processors). The processors 1310 may include a boot and power management processor, which may be a dedicated processor and subsystem handling boot power and management functions, as well as enforcement of associated security. The boot and power management processor may be part of the SoC1304's boot sequence and can provide runtime power management services. The boot and power management processor may enable clock and voltage programming, assistance with low-power state transitions of the system, thermal and temperature sensor management of the SoC1304, and / or power state management of the SoC1304. Each temperature sensor may be implemented as a ring oscillator with an output frequency proportional to temperature, and the SoC1304 may use the ring oscillator to detect the temperatures of the CPU 1306, GPU 1308, and / or accelerator 1314. If it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine and put the SoC1304 into a lower power state and / or put the vehicle 1300 into a driver state leading to a safe stop mode (for example, safely stopping the vehicle 1300).

[0318] The processor 1310 may further include a set of embedded processors that can function as a speech processing engine. This speech processing engine could be a speech subsystem enabling full hardware support for multi-channel speech via multiple interfaces, and a broad and flexible range of speech I / O interfaces. In some examples, the speech processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0319] The processor 1310 may further include an always-on processor engine that can implement hardware features necessary to support cases using low-power sensor management and wake. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripheral devices (e.g., timer and interrupt controllers), various I / O controller peripheral devices, and routing logic mechanisms.

[0320] The processor 1310 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic mechanisms. In safety mode, two or more cores can function as a single core by operating in lockstep mode and by a comparison logic mechanism that detects any differences between the cores' operations.

[0321] The processor 1310 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0322] The processor 1310 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0323] The processor 1310 may include a video image synthesizer, which may be a processing block (for example, implemented on a microprocessor) that performs video post-processing functions required by a video playback application to generate the final image for the player window. The video image synthesizer can perform lens distortion correction for the wide-field camera 1370, the surround camera 1374, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and respond to events within the cabin. The in-cabin system can perform lip-reading to activate mobile communication services to make phone calls, dictate emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. The driver may have access to certain functions only when the vehicle is operating in autonomous mode, otherwise they are disabled.

[0324] Video image synthesizers may include enhanced temporal noise reduction to reduce both spatial and temporal noise. For example, if motion is present in the video, noise reduction will appropriately weight the spatial information and reduce the weight of information supplied from adjacent frames. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0325] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frames. Furthermore, the video image synthesizer can be used to synthesize user interfaces when a desktop operating system is used, eliminating the need for the GPU1308 to continuously render new surfaces. Even when the GPU1308 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1308, improving performance and responsiveness.

[0326] The SoC1304 may further include a camera serial interface, a high-speed interface, and / or a video input block and associated pixel input functions that can be used for the camera, for receiving video and input from the camera. The SoC1304 may further include an input / output controller that can be controlled by software and used to receive I / O signals that are not bound to a specific role.

[0327] The SoC1304 may further include a wide range of peripheral device interfaces that enable communication with peripheral devices, audio codecs, power management units, and / or other devices. The SoC1304 can be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1364, radar sensor 1360, etc., which may be connected via Ethernet®), data from bus 1302 (e.g., vehicle speed, steering wheel position, etc.), and data from GNSS sensor 1358 (e.g., connected via Ethernet® or CAN bus). The SoC1304 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free the CPU 1306 from routine data management tasks.

[0328] The SoC1304 is an end-to-end platform with a flexible architecture spanning automation levels 3 through 5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and offering a platform for a flexible and reliable driving software stack along with deep learning tools. The SoC1304 has the potential to be faster, more reliable, and more energy and space-efficient than conventional systems. For example, the accelerator 1314, when combined with the CPU 1306, GPU 1308, and data store 1316, can provide a fast and efficient platform for autonomous vehicles at levels 3 through 5.

[0329] Therefore, this technology offers capabilities and functions that cannot be achieved with conventional systems. For example, computer vision algorithms can be executed on a CPU, configured using high-level programming languages ​​such as C, and can execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute complex object detection algorithms in real time, which are requirements for in-vehicle ADAS applications and practical Level 3-5 autonomous vehicles.

[0330] In contrast to conventional systems, the technology described herein, by comprising a CPU complex, a GPU complex, and a hardware acceleration cluster, enables the simultaneous and / or sequential execution of multiple neural networks, combining their results to enable Level 3–5 autonomous driving capabilities. For example, a DLA or CNN running on a dGPU (e.g., GPU1320) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs that the neural network has not been specifically trained on. The DLA may further include a neural network that can identify, interpret, and provide a semantic understanding of signs, which can then pass on that semantic understanding to a route planning module running on the CPU complex.

[0331] As another example, multiple neural networks may run simultaneously when Level 3, 4, or 5 driving is required. For instance, a warning sign consisting of an electronic light reading "Caution: Flashing signals indicate frozen conditions" can be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing signals indicate frozen conditions" may be interpreted by a second deployed neural network, which then informs the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions exist if flashing signals are detected. Flashing signals may be identified by running a third deployed neural network over multiple frames, which then informs the vehicle's route planning software whether (or not) flashing signals are present. All three neural networks can run simultaneously within the DLA and / or on a GPU1308, for example.

[0332] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1300. Using an always-on sensor processing eng...

Claims

1. A processor comprising a processing circuit, wherein the processing circuit is The table is copied into memory such that the first value is placed at the first physical address of the first memory bank, and the second value is placed at the second physical address of the second memory bank, and the first value and the second value are placed in the same column of the logical memory diagram of the table. A read instruction that corresponds to a lookup of multiple points, receiving a read instruction that includes at least an instruction relating to a first index corresponding to the first physical address of the first memory bank and the second physical address of the second memory bank, Based at least the read command, read the first value located at the first physical address and the second value located at the second physical address within one cycle, and Using the first value and the second value, perform one or more operations. Processor.

2. If the aforementioned multi-point lookup includes a two-point lookup, the table is duplicated 0.5 * N times, where N is the number of values ​​read in a given cycle of the two-point lookup, or If the aforementioned multi-point lookup includes a 2x2 point lookup, the table is duplicated 0.25*M times, where M is the number of values ​​read in a given cycle of the 2x2 point lookup. The processor according to claim 1.

3. The processor according to claim 1, wherein the one or more operations include interpolation operations.

4. The processor according to claim 1, wherein the processor is a vector processing unit (VPU) and the memory is a vector memory (VMEM).

5. The processor according to claim 1, wherein the processor is included in a programmable vision accelerator of a system-on-a-chip (SoC).

6. The aforementioned processor, Control systems for autonomous or semi-autonomous machines, Perception systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning calculations. System-on-a-chip (SoC), A system including a programmable vision accelerator (PVA), A system including a vision processing unit, Systems implemented using edge devices, Systems implemented using robots, A system incorporating one or more virtual machines (VMs), A system that is at least partially implemented within a data center, or A system that is at least partially implemented using cloud computing resources. The processor according to claim 1, which is included in at least one of the following.

7. Memory and A processor equipped with a processing circuit A system comprising, the processing circuit, The instance of the table is copied to memory such that the first value of the instance of the table is at the first physical address of the first memory bank, the second value of the instance of the table is at the second physical address of the second memory bank, and the first value and the second value are included in the same column of the logical memory diagram of the table. Determine the first index corresponding to the first physical address of the first memory bank, A read command that corresponds to a lookup of multiple points, and reads the first value located at the first physical address and the second value located at the second physical address, at least partially based on the first index, and Using the first value and the second value, perform one or more operations. system.

8. If the aforementioned multi-point lookup includes a two-point lookup, the table is duplicated 0.5 * N times, where N is the number of values ​​read in a given cycle of the two-point lookup, or If the aforementioned multi-point lookup includes a 2x2 point lookup, the table is duplicated 0.25*M times, where M is the number of values ​​read in a given cycle of the 2x2 point lookup. The system according to claim 7.

9. The system according to claim 7, wherein the one or more operations include an interpolation operation.

10. The system according to claim 7, wherein the processor is a vector processing unit (VPU) and the memory is a vector memory (VMEM).

11. The system according to claim 7, wherein the processor is included in a programmable vision accelerator of a system-on-a-chip (SoC).

12. The aforementioned system Control systems for autonomous or semi-autonomous machines, Perception systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning calculations. System-on-a-chip (SoC), A system including a programmable vision accelerator (PVA), A system including a vision processing unit, Systems implemented using edge devices, Systems implemented using robots, A system incorporating one or more virtual machines (VMs), A system that is at least partially implemented within a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 7, which is included in at least one of the following.

13. A method performed by a processing circuit provided in a processor, A step of replicating an instance of a table into memory such that it includes a first portion of the instance of the table in a first memory bank and a second portion of the instance of the table in a second memory bank, wherein the first portion and the second portion are included in the same column of the logical memory diagram of the table. The steps include determining a first index corresponding to the first portion of the instance of the table, A read instruction corresponding to a multi-point lookup, and a step of reading a first value relating to the first portion of the instance of the table from a first physical address of the first memory bank, and a second value relating to the second portion of the instance of the table from a second physical address of the second memory bank, based at least on the first index, The steps of performing one or more operations using the first value and the second value, Methods that include...

14. If the aforementioned multi-point lookup includes a two-point lookup, the table is duplicated 0.5 * N times, where N is the number of values ​​read in a given cycle of the two-point lookup, or If the aforementioned multi-point lookup includes a 2x2 point lookup, the table is duplicated 0.25*M times, where M is the number of values ​​read in a given cycle of the 2x2 point lookup. The method according to claim 13.

15. The method according to claim 13, wherein the one or more operations include an interpolation operation.

16. The method according to claim 13, wherein the processor is a vector processing unit (VPU) and the memory is a vector memory (VMEM).

17. The method according to claim 13, wherein the processor is included in a programmable vision accelerator of a system-on-a-chip (SoC).

18. The processor according to claim 1, wherein one or more operations are performed using the first value and the second value during a single cycle.

19. The system according to claim 7, wherein the processing circuit further receives the read command corresponding to the lookup of the plurality of points, and the read command includes a pattern relating to obtaining the first index and at least one other index.

20. The system according to claim 7, wherein the first value is stored at the first physical address of the first memory bank and the second value is stored at the second physical address of the second memory bank, at least on the basis that the first value is adjacent to the second value in the same column of the logical memory diagram of the table.

Images