Computational Imaging Pipeline

Abstract

The present application generally relates to a parallel processing device. The parallel processing device may include a plurality of processing elements, a memory subsystem, and an interconnection system. The memory subsystem may include a plurality of memory slices, at least one of the plurality of memory slices being associated with one of the plurality of processing elements, and comprising a plurality of random access memory (RAM) tiles, each tile having separate read and write ports. The interconnection system is configured to couple the plurality of processing elements and the memory subsystem. The interconnection system includes a local interconnection and a global interconnection.

Description

[0001] This application is a divisional application of the application with application number 201480045179.0, application date August 6, 2014, and invention name “Computational Imaging Pipeline”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of an earlier priority date of U.S. patent application Ser. No. 14 / 082,396, filed Nov. 18, 2013, entitled “APPARATUS, SYSTEMS, AND METHODS FOR PROVIDING COMPUTATIONAL IMAGING PIPELINE,” which claims priority to Romanian patent application OSIM Registratura A / 00812, filed Nov. 6, 2013, entitled “APPARATUS, SYSTEMS, AND METHODS FOR PROVIDING CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGING PIPELINE,” and to UK patent application Ser. No. GB1314263.3, filed Aug. 8, 2013, entitled “CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGING PIPELINE.” This application also claims the benefit of an earlier priority date of U.S. Patent Application No. 14 / 082,645, filed on November 18, 2013, entitled “APPARATUS, SYSTEMS, AND METHODS FOR PROVIDING CONFIGURABLE COMPUTATIONAL IMAGING PIPELINE,” which claims priority to Romanian Patent Application OSIM Registratura A / 00812, filed on November 6, 2013, entitled “APPARATUS, SYSTEMS, AND METHODS FOR PROVIDING CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGING PIPELINE,” and claims priority to UK Patent Application No. GB1314263.3, filed on August 8, 2013, entitled “CONFIGURABLE AND COMPOSABLE COMPUTATIONAL IMAGING PIPELINE.” The entire contents of each of the applications are hereby incorporated herein by reference. Technical Field

[0004] The present application relates generally to processing devices suitable for image and video processing. Background Art

[0005] Computational image and video processing is very demanding in terms of memory bandwidth, since image resolutions and frame rates are high, with high aggregate pixel rates of hundreds of megapixels per second being common. Furthermore, since the field is in its relative infancy, algorithms are in constant flux. Therefore, it is difficult to implement them completely in hardware, since changes to the algorithms may mean that they cannot be adapted. At the same time, software approaches that rely solely on implementations in processors are unrealistic. Therefore, it is generally desirable to provide a flexible architecture / infrastructure that can adapt to processors and hardware accelerators.

[0006] At the same time, the demand for such video and image processing comes largely from portable electronic devices (e.g., tablet computers and mobile devices), where power consumption is a key consideration. Therefore, a flexible infrastructure is generally needed to couple programmable multi-core processors and hardware accelerators with high-bandwidth memory subsystems that allow them to deliver sustained data transfer rates at low power levels necessary for portable electronic devices. Summary of the invention

[0007] According to the disclosed subject matter, the present invention provides apparatus, systems, and methods for providing a configurable and composable computational imaging pipeline.

[0008] The disclosed subject matter includes a parallel processing device. The processing device includes a plurality of processing elements each configured to execute instructions, and a memory subsystem including a plurality of memory slices (including a first memory slice associated with one of the plurality of processing elements). The first memory slice includes a plurality of random access memory (RAM) tiles each having a separate read and write port. The parallel processing device may include an interconnection system configured to couple the plurality of processing elements and the memory subsystem. The interconnection system may include a local interconnection configured to couple the first memory slice and one of the plurality of processing elements, and a global interconnection configured to couple the first memory slice and the remaining plurality of processing elements.

[0009] In some embodiments, one of the plurality of RAM tiles is associated with an arbitration block, wherein the arbitration block is configured to receive a memory access request from one of the plurality of processing elements and to grant access to the one of the plurality of RAM tiles to the one of the plurality of processing elements.

[0010] In some embodiments, the arbitration block is configured to grant access to one of the plurality of RAM tiles in a round-robin manner.

[0011] In some embodiments, the arbitration block includes a conflict detector configured to monitor memory access requests to one of the plurality of RAM tiles and to determine whether two or more of the plurality of processing elements attempt to access the one of the plurality of RAM tiles simultaneously.

[0012] In some embodiments, a conflict detector is coupled to a plurality of address decoders, wherein each of the plurality of address decoders is coupled to one of the plurality of processing elements and is configured to determine whether one of the plurality of processing elements attempts to access one of the plurality of RAM tiles associated with an arbitration block.

[0013] In some embodiments, the plurality of processing elements includes at least one vector processor and at least one hardware accelerator.

[0014] In some embodiments, a parallel processing device includes a plurality of memory slice controllers each configured to provide access to one of a plurality of memory slices.

[0015] In some embodiments, an interconnect system includes a first bus configured to provide communication between at least one vector processor and a memory subsystem.

[0016] In some embodiments, the interconnect system includes a second bus system configured to provide communication between the at least one hardware accelerator and the memory subsystem.

[0017] In some embodiments, the second bus system includes a chip address request filter configured to mediate communications between the at least one hardware accelerator and the memory subsystem by receiving memory access requests from the at least one hardware accelerator and by granting requests to the memory subsystem to the at least one hardware accelerator.

[0018] In some embodiments, one of the plurality of processing devices includes a buffer for increasing throughput of a memory subsystem, wherein a number of elements in the buffer is greater than a number of cycles for retrieving data from the memory subsystem.

[0019] The disclosed subject matter includes a method for operating a parallel processing system. The method includes providing a plurality of processing elements including a first processing element and a second processing element, wherein each of the plurality of processing elements is configured to execute instructions. The method also includes providing a memory subsystem including a plurality of memory slices (including a first memory slice associated with a first processing element), wherein the first memory slice includes a plurality of random access memory (RAM) tiles each having separate read and write ports. The method also includes receiving a first memory access request from the first processing element via a local interconnect of an interconnect system through an arbitration block associated with one of the plurality of RAM tiles. The method further includes sending a first authorization message to the first processing element via a global interconnect through the arbitration block to authorize the first processing element to access one of the plurality of RAM tiles.

[0020] In some embodiments, the method also includes receiving a second memory access request from a second processing element via a global interconnect of the interconnect system through the arbitration block; and sending a second authorization message to the second processing element via the global interconnect through the arbitration block to authorize the second processing element to access one of the plurality of RAM tiles.

[0021] In some embodiments, the method further includes sending, by the arbitration block, a plurality of grant messages to the plurality of processing elements to grant access to one of the plurality of RAM tiles in a round-robin manner.

[0022] In some embodiments, the method further includes monitoring, by a conflict detector in the arbitration block, memory access requests to one of the plurality of RAM tiles, and determining whether two or more of the plurality of processing elements attempt to access one of the plurality of RAM tiles simultaneously.

[0023] In some embodiments, the plurality of processing elements includes at least one vector processor and at least one hardware accelerator.

[0024] In some embodiments, the method further includes providing a plurality of memory slice controllers each configured to provide access to one of the plurality of memory slices.

[0025] In some embodiments, the method further includes providing communication between the at least one vector processor and the memory subsystem via a first bus system of the interconnect system.

[0026] In some embodiments, the method further includes providing communication between the at least one hardware accelerator and the memory subsystem via a second bus system of the interconnect system.

[0027] In some embodiments, the second bus system includes a chip address request filter configured to mediate communications between the at least one hardware accelerator and the memory subsystem by receiving a memory access request from the at least one hardware accelerator and by granting access to the memory subsystem to the at least one hardware accelerator.

[0028] The disclosed subject matter includes an electronic device. The electronic device includes a parallel processing device. The processing device includes a plurality of processing elements each configured to execute instructions, and a memory subsystem including a plurality of memory slices (including a first memory slice associated with one of the plurality of processing elements). The first memory slice includes a plurality of random access memory (RAM) tiles each having a separate read and write port. The parallel processing device may include an interconnection system configured to couple the plurality of processing elements and the memory subsystem. The interconnection system may include a local interconnection configured to couple the first memory slice and one of the plurality of processing elements, and a global interconnection configured to couple the first memory slice and the remaining plurality of processing elements. The electronic device also includes a processor, which communicates with the parallel processing device and is configured to run a module stored in the memory. The module is configured to: receive a flowchart associated with a data processing process, wherein the flowchart includes a plurality of nodes and a plurality of edges connecting two or more of the plurality of nodes, wherein each node identifies an operation and each edge identifies a relationship between the connected nodes; and assign a first node of the plurality of nodes to a first processing element of a parallel processing device, and assign a second node of the plurality of nodes to a second processing element of the parallel processing device, thereby parallelizing the operations associated with the first node and the second node.

[0029] In some embodiments, the flow chart is formatted in Extensible Markup Language (XML).

[0030] In some embodiments, the module is configured to assign a first node of the plurality of nodes to the first processing element based on past performance of a memory subsystem in the parallel processing device.

[0031] In some embodiments, a memory subsystem of the parallel processing device includes a counter configured to count a number of memory conflicts within a predetermined period of time, and the past performance of the memory subsystem includes the number of memory conflicts measured by the counter.

[0032] In some embodiments, the module is configured to assign a first node of the plurality of nodes to a first processing element while the parallel processing device operates at least a portion of the flowchart.

[0033] In some embodiments, the module is configured to receive multiple flow charts and distribute all operations associated with the multiple flow charts to a single processing element in a parallel processing device.

[0034] In some embodiments, the module is configured to interleave memory accesses of the processing elements in order to reduce memory conflicts.

[0035] In some embodiments, the electronic device comprises a mobile device.

[0036] In some implementations, the flow graph is specified using an application programming interface (API) associated with the parallel processing device.

[0037] In some embodiments, the module is configured to provide the input image data to the plurality of processing elements by dividing the input image data into strips and providing one strip of the input image data to one of the plurality of processing elements.

[0038] In some embodiments, the number of strips of input image data is the same as the number of multiple processing elements.

[0039] The disclosed subject matter includes a method. The method includes receiving a flowchart associated with a data processing process at a processor in communication with a parallel processing device, wherein the flowchart includes a plurality of nodes and a plurality of edges connecting two or more of the plurality of nodes, wherein each node identifies an operation, and each edge identifies a relationship between the connected nodes. The method also includes assigning a first node of the plurality of nodes to a first processing element of the parallel processing device, and assigning a second node of the plurality of nodes to a second processing element of the parallel processing device, thereby parallelizing operations associated with the first node and the second node. The parallel processing device also includes a memory subsystem, the memory subsystem including a plurality of memory slices (including a first memory slice associated with a first processing element), wherein the first memory slice includes a plurality of random access memory (RAM) tiles each having a separate read and write port; and an interconnection system configured to couple the first processing element, the second processing element, and the memory subsystem. The interconnection system includes a local interconnect configured to couple the first memory slice and the first processing element, and a global interconnect configured to couple the first memory slice and the second processing element.

[0040] In some embodiments, assigning a first node of the plurality of nodes to a first processing element of the parallel processing device includes assigning the first node of the plurality of nodes to the first processing element based on past performance of a first memory slice in the parallel processing device.

[0041] In some embodiments, the method further includes counting, at a counter in the memory subsystem, a number of memory conflicts in the first memory slice within a predetermined time period, and the past performance of the first memory slice includes the number of memory conflicts in the first memory slice.

[0042] In some embodiments, assigning a first node of the plurality of nodes to a first processing element is performed while a parallel processing device is operating at least a portion of the flow chart.

[0043] In some embodiments, the method further includes interleaving memory accesses of the processing elements to the first memory slice in order to reduce memory conflicts.

[0044] In some implementations, the flow graph is specified using an application programming interface (API) associated with the parallel processing device.

[0045] In some embodiments, the method further comprises providing the input image data to the plurality of processing elements by dividing the input image data into a plurality of strips and providing one of the plurality of strips of the input image data to one of the plurality of processing elements.

[0046] In some embodiments, the number of multiple strips of input image data is the same as the number of multiple processing elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various objects, features, and advantages of the presently disclosed subject matter may be more fully understood by reference to the following detailed description of the presently disclosed subject matter when considered in conjunction with the following drawings, in which like reference numerals indicate like elements.

[0048] Figure 1 yes Figure 1 Describe the Chimera computational imaging platform.

[0049] Figure 2 Describe the multi-core architecture of the Cell processor.

[0050] Figure 3 Describes the efficient low-power microprocessor (ELM) architecture.

[0051] Figure 4 An improved memory subsystem according to some embodiments is shown.

[0052] Figure 5 Shown are sections of a parallel processing device according to some embodiments.

[0053] Figure 6 A centralized conflict detection system in tile control logic is shown according to some embodiments.

[0054] Figure 7A distributed conflict detection system in tile control logic is shown according to some embodiments.

[0055] Figure 8 An arbitration block for reporting conflicting signals to a requestor is shown according to some embodiments.

[0056] Fig. 9 A round-robin oriented arbitration block is shown according to some embodiments.

[0057] Fig.10 A mechanism for reducing memory access latency due to memory access arbitration is shown in accordance with some embodiments.

[0058] Fig.11 An application of scheduling software according to some embodiments is shown.

[0059] Fig.12 A hierarchical structure of a system having a parallel processing device according to some embodiments is provided.

[0060] Fig.13 A description showing how a directed acyclic graph (DAG) or flow chart may be used to control the operation of a parallel processing device according to some embodiments.

[0061] Figures 14A-14B The scheduling and issuance of tasks by a compiler and scheduler are shown according to some embodiments.

[0062] Fig.15 The operation of a real-time DAG compiler according to some embodiments is shown.

[0063] Fig.16 According to some embodiments, the schedule generated by the OpenCL scheduler is compared to the schedule generated by the online DAG scheduler of the present invention.

[0064] Fig.17 A blocking mechanism for synchronizing the operation of a processor and / or filter accelerator is shown according to some embodiments.

[0065] Fig.18 A parallel processing device with different types of processing elements is shown according to some embodiments.

[0066] Fig.19 A multi-core memory subsystem of the present invention is shown according to some embodiments.

[0067] Fig. 20 A single piece of a connection matrix (CMX) infrastructure is shown in accordance with some embodiments.

[0068] Fig.21An accelerator memory controller (AMC) crossbar architecture is shown in accordance with some embodiments.

[0069] Fig. 22 An AMC crossbar port controller is shown according to some embodiments.

[0070] Fig.23 A read operation using AMC is shown according to some embodiments.

[0071] Fig.24 A write operation using AMC is shown according to some embodiments.

[0072] Fig.25 A parallel processing apparatus according to some embodiments is shown.

[0073] Fig.26 An electronic device including a parallel processing device according to some embodiments is shown. DETAILED DESCRIPTION

[0074] In the following description, a number of specific details about the systems and methods of the subject matter of the present disclosure and the environments in which such systems and methods can operate, etc. are shown to provide a thorough understanding of the subject matter of the present disclosure. However, it will be apparent to those skilled in the art that the subject matter of the present disclosure can be practiced without such specific details, and certain features that are well known in the art are not described in detail to avoid complication of the subject matter of the present disclosure. In addition, it should be understood that the examples provided below are exemplary, and it is contemplated that there are other systems and methods within the scope of the present disclosure.

[0075] One possible way to interconnect such different processing resources (eg, processors and hardware accelerators) is to use a bus as outlined in the computational photography engine of Chimera developed by NVidia. Figure 1Chimera's computational photography engine is shown. The Chimera computational photography engine 100 includes multiple graphics processing unit (GPU) cores 102, which are connected to a multi-core ARM processor subsystem 104 and a hardware (HW) image signal processing (ISP) accelerator 106 via a flat-level bus infrastructure 108 (e.g., a single hierarchical bus system that connects all processing elements). The Chimera computational photography engine is generally presented as a software framework that abstracts the details of the basic GPU core 102, CPU 104, and ISP block 106 from the programmer. In addition, the Chimera computational photography engine 100 describes the data flow through their computational photography engine via two information buses 108-0, 108-1, the first bus 108-0 carrying image or frame data, and the second bus 108-1 carrying status information associated with each frame.

[0076] Using a flat bus infrastructure such as in Chimera can be cheap and convenient to implement. However, using a flat bus infrastructure as a means of interconnecting non-uniform processing elements such as GPU core 102, CPU 104, and ISP block 106 (e.g., various types of processing elements) can have a number of noteworthy disadvantages. First, using a bus to interconnect computing resources means that memory may be locally distributed throughout the system for each central processing unit (CPU) 104, graphics processing unit (GPU) 102, and / or image signal processor (ISP) block 106. Therefore, memory cannot be flexibly allocated within the processing pipeline according to the requirements of the computational photography pipeline that the programmer wishes to implement. This lack of flexibility may mean that certain image or video processing is difficult or impossible to implement, or that the implementation is limited in frame rate, image quality, or otherwise.

[0077] Second, using a flat bus infrastructure also means that different computing resources (CPU 104, GPU 102, and ISP block 106) must compete for bus bandwidth. This competition requires arbitration, which reduces the amount of available bus bandwidth. Therefore, a decreasing theoretical bandwidth is available for actual work. The reduction in bandwidth may mean that the processing pipeline fails to meet the performance requirements of the application in terms of frame rate, image quality, and / or power.

[0078] Third, insufficient memory near a particular computing resource may mean that data must be transferred back and forth between memory associated with a given GPU 102, CPU 104, or hardware ISP block 106 and another computing resource. This lack of locality means that additional bus bandwidth and arbitration overhead may be incurred. Furthermore, lack of locality also means that additional power is consumed. Therefore, it may be difficult or impossible to support a particular algorithm at a particular target frame rate.

[0079] Fourth, using a flat bus infrastructure can also compound difficulties in constructing pipelines from non-uniform processing elements, each of which may have different latency characteristics. For example, the GPU core 102 is designed to tolerate latency by running multiple overlapping processing threads in order to handle multiple outstanding accesses to memory (usually external DRAM) to cover latency, whereas the normal CPU 104 and hardware ISP block 106 are not designed to tolerate latency.

[0080] Another way to interconnect different processing resources is through Figure 2 2. The Cell Processor Architecture 200 is provided by the Cell Processor Architecture developed by IBM and shown in FIG. The Cell Processor Architecture 200 includes a Local Store (LS) 202, also known as a Synergistic Execution Unit (SXU), available to each processor 204. The Cell Processor 200 relies on an infrastructure of shared time and directs memory access (DMA) 206 transfers to programmatically schedule data transfers between the LS 202 of one processor and the LS 202 of another processor. A difficulty with the Cell Architecture 200 is the complexity that programmers face in explicitly scheduling background data transfers hundreds of cycles in advance (due to high latency in the Cell Architecture 200) to ensure that shared data is available to each processor 204 when needed. If the programmer does not explicitly schedule background data transfers, then the processor 204 will stall, which will degrade performance.

[0081] Another way to interconnect different processing resources is to use a shared multi-core memory subsystem to efficiently share data between processors in a multi-core processing system. This shared multi-core memory subsystem is used in efficient low power microprocessor (ELM) systems. Figure 3ELM system is shown. ELM system 300 includes ensemble 302, which is the main physical design unit for computing resources in ELM system. Ensemble 302 includes a cluster of four processors 304 loosely coupled. The cluster of four processors 304 share local resources such as ensemble memory 306 and interfaces to the interconnection network. Ensemble memory 306 captures instruction and data working sets close to processor 304, and is stored in order to allow local processor 304 and network interface controller to access it at the same time. Each processor 304 in ensemble 302 is assigned a preferred library in ensemble memory 306. Access to its preferred library by processor 304 is prioritized higher than (and will block access) access by other processors and network interfaces. Instructions and data that are private to a single processor 304 can be stored in its preferred library to provide a deterministic number of accesses. The arbiter that controls access to read and write ports is biased to establish affinity between processor 304 and memory library 306. This allows software to make stronger assumptions about bandwidth availability and latency when accessing data that may be shared by multiple processors.

[0082] However, due to physically larger random access memory (RAM) blocks, the ELM architecture 300 may consume a lot of power. In addition, the ELM architecture 300 may suffer from lower throughput where there is a lot of data sharing between the processors 304. Furthermore, no provision is made for data sharing between the processors 304 and the hardware accelerators, which may be advantageous in terms of power and performance in some cases.

[0083] The present disclosure relates to devices, systems and methods for enabling multiple processors and hardware accelerators to access shared data simultaneously with other processors and hardware accelerators. The present disclosure provides devices, systems and methods for accessing shared data simultaneously without being blocked by a local processor with a higher affinity (e.g., higher priority) for accessing local storage.

[0084] The apparatus, system and method of the present disclosure provide a large number of benefits over existing multi-core memory architectures.Existing multi-core memory architectures use a single monolithic RAM block per processor, which can limit the bandwidth of accessible data.The architecture of the present disclosure can provide a mechanism for accessing memory with a substantially higher bandwidth than the existing multi-core memory architecture using a single monolithic RAM block.The architecture of the present disclosure instantiates multiple physical RAM blocks by each processor, replacing each processor instantiating a single larger RAM block to obtain this higher bandwidth.Each RAM block may include a dedicated access arbitration block and surrounding infrastructure.Therefore, each RAM block in the memory subsystem can be accessed by multiple processing elements in the system independently of other RAM blocks, such as vector processors, reduced instruction set computing (RISC) processors, hardware accelerators, or DMA engines, etc.

[0085] Because the memory bank based on a single large RAM instance is more area-efficient than the memory bank based on multiple smaller RAM instances, it is counterintuitive to a certain extent that it is more advantageous to use multiple small RAM instances than to use a single large RAM instance. However, the power dissipation of smaller RAM instances is usually significantly lower than that of a single large RAM instance. In addition, if a single large physical RAM instance will obtain the same bandwidth as a multi-instance RAM block, then a single large physical RAM instance will cause substantially higher power than a RAM consisting of multiple physical RAM instances. Therefore, at least from a power dissipation perspective, a memory subsystem can benefit from using multiple physical RAM instances instead of using a single large RAM instance.

[0086] A memory subsystem with multiple physical RAM instances may have added advantages at the cost of each RAM access, such as memory access time or power consumption of smaller RAM blocks is generally much lower than that of larger RAM blocks. This is due to the shortened bit lines used to read / write data from the RAM. In addition, the access time for reading and writing smaller RAM blocks is also lower (due to the reduced resistance-capacitance (RC) time constant associated with the shorter bit lines). Therefore, processing elements coupled to a multi-RAM-based memory subsystem may operate at a higher frequency, which in turn reduces static power due to static leakage current. This can be particularly useful when processors and memories are isolated into power domains. For example, when a given processor or filter accelerator has completed its task, the power domain associated with the given processor or filter accelerator can be advantageously gated off. Therefore, the memory subsystem in the architecture of the present disclosure has excellent characteristics in terms of available bandwidth and power dissipation.

[0087] In addition, a memory subsystem with multiple RAM instances (each with arbitrated access) can provide many ways for data to be shared between processors and hardware accelerators without having to dedicate RAM blocks to specific processors by locking RAM blocks. In principle, if a larger RAM is subdivided into N sub-blocks, the available data bandwidth increases by about N times. This is based on the assumption that data can be divided in time by multiple processing elements to reduce simultaneous sharing (e.g., access conflicts). For example, when a consumer processor or consumer accelerator reads data from a data buffer filled by a producer processor or producer accelerator, there is simultaneous sharing of the data buffer, resulting in access conflicts.

[0088] In some embodiments, the disclosed architecture can provide a mechanism for reducing the simultaneous sharing of data. Specifically, the disclosed architecture can be responsible for reducing the simultaneous sharing via a static memory allocation mechanism and / or a dynamic memory allocation mechanism. For example, in a static memory allocation mechanism, before the program starts, such as during the program compilation phase, data is mapped to different parts of a memory to reduce the simultaneous sharing of data. On the other hand, in a dynamic memory allocation scheme, data is mapped to different parts of a memory during program execution. The static memory allocation mechanism provides a predictable mechanism for allocating memory to data, and it does not cause any substantial overhead in terms of power or performance.

[0089] As another example, the disclosed architecture may be used in conjunction with a runtime scheduler running on a controller (e.g., a supervisory RSIC processor) or one or more processors that mediate access to a data structure that is partitioned across multiple RAM blocks. The runtime scheduler may be configured to interleave the start times of different processing elements operating on portions (e.g., lines or tiles) of data (e.g., an image frame) in order to reduce simultaneous access to shared data.

[0090] In some embodiments, the run-time scheduler can be supplemented with a hardware arbitration block. For example, the hardware arbitration block can be configured to mediate shared memory accesses by processors (such as vector processors) via a shared deterministic interconnect designed to reduce stalls. In some cases, the hardware arbitration block can be configured to perform cycle-oriented scheduling. Cycle-oriented scheduling can include scheduling resource usage at clock cycle intervals, as opposed to scheduling resource usage at task level intervals, which can require multiple clock cycles. Scheduling resource allocation at clock cycle intervals can provide higher performance.

[0091] In other embodiments, the runtime scheduler may be supplemented with a plurality of hardware accelerators, each of which may include an input buffer and an output buffer to store data. The input and output buffers may be configured to absorb (or hide) latency variance in accessing external resources (such as external memory). The input and output buffers may include first-in-first-out (FIFO) buffers, and the FIFO buffers may include a sufficient number of slots to store a sufficient amount of data and / or instructions to absorb latency variance in accessing external resources.

[0092] In some embodiments, the apparatus, system and method of the present disclosure provide a parallel processing device. The parallel processing device may include multiple processors, such as parallel processors, each of which can execute instructions. The parallel processing device may also include multiple memory slices, each memory slice is associated with one h in the parallel processing device, and gives priority access to that processor over other processing devices in the parallel processing device. Each memory slice may include multiple RAM tiles, wherein each RAM tile may include a read port and a write port. In some cases, each memory slice may be provided with a memory slice controller for providing access to the relevant memory slice. The processor and the RAM tile may be coupled to each other via a bus. In some cases, the bus may couple any processor to any memory slice. Suitably, each RAM tile may include a tile control logic block for granting access to the tile. The tile control logic block is sometimes referred to as a tile control logic or arbitration block.

[0093] In some embodiments, the parallel processing device may also include at least one hardware accelerator configured to perform a predefined processing function (eg, image processing). In some cases, the predefined processing function may include a filtering operation.

[0094] In some embodiments, at least one hardware accelerator may be coupled to the memory slice via a separate bus. The separate bus may include an associated accelerator memory controller (AMC), which is configured to receive requests from at least one hardware accelerator and grant access to the memory slice to the hardware accelerator through an associated memory slice controller. It should be understood that the memory access path adopted by the hardware accelerator may be different from the path adopted by the vector processor. In some embodiments, at least one hardware accelerator may include an internal buffer (e.g., a FIFO memory) to compensate for delays in accessing the memory slice.

[0095] In some embodiments, the parallel processing device may include a host processor. The host processor may be configured to communicate with the AMC via a host bus. The parallel processing device may also be provided with an application programming interface (API). The API provides a high-level interface to the vector processor and / or hardware accelerator. In some embodiments, the parallel processing device may operate in conjunction with a compiler that provides instructions for the parallel processing device.

[0096] In some cases, the compiler is configured to run on a host processor that is different from a processing element, such as a vector processor or a hardware accelerator. In some cases, the compiler is configured to execute the image / video API 1206 ( Fig.12) receives the flow chart. The compiler may also be configured to map one or more aspects of the flow chart to one or more of the processing elements, such as a vector processor or a hardware accelerator. In some embodiments, the flow chart may include nodes and edges, wherein each node identifies an operation, and each edge identifies the relationship between nodes (e.g., operations), such as the order in which the operations are implemented. The compiler may be configured to assign a node (e.g., operation) to one of the processing elements in order to parallelize the calculation of the flow chart. In some embodiments, the flow chart may be provided in an extensible markup language (XML) format. In some embodiments, the compiler may be configured to assign multiple flow charts to a single processing element.

[0097] In some embodiments, the parallel processing device may be configured to measure its performance and provide information to the compiler. Thus, the compiler may use past performance information received from the parallel processing device to determine current task allocations to processing elements in the parallel processing device. In some embodiments, the performance information may indicate the number of access conflicts experienced by one or more processing elements in the processing device.

[0098] In some cases, the parallel processing device can be used in video applications, which are very computationally intensive. To address the computational requirements of video applications, the parallel processing device can configure its memory subsystem to reduce access conflicts between processing units during memory access. To this end, as previously discussed, the parallel processing device can subdivide the monolithic memory bank into multiple physical RAM instances instead of using the monolithic memory bank as a single physical memory block. Through this subdivision, each physical RAM instance can be arbitrated for read and write operations, thereby increasing the available bandwidth by the number of physical RAM instances in the memory bank.

[0099] In some embodiments, hardware loop-oriented arbitration may also provide multiple traffic classes and programmable scheduling masks. Multiple traffic classes and programmable scheduling masks may be controlled using a runtime scheduler. The hardware loop-oriented arbitration block may include a port arbitration block that may be configured to allocate multiple requestors of a single shared resource in a round-robin scheduling scheme. In a round-robin scheduling scheme, requestors (e.g., processing elements) are granted access to resources (e.g., memory) in the order in which they receive requests from the requestors. In some cases, a port arbitration block may increase a round-robin scheduling scheme to compensate for multiple traffic classes. A single shared resource may include RAM tiles, shared registers, or other resources accessible to vector processors, filter accelerators, and RISC processors to share data. In addition, the arbitration block may allow the round-robin resource allocation to be rewritten with a priority vector or super priority vector. The priority vector or super priority vector may be provided by the runtime scheduler to prioritize certain traffic classes (e.g., video traffic classes) as required by specific applications of interest.

[0100] In some embodiments, a processing element may include one or more of a processor (such as a vector processor or a streaming hybrid architecture vector engine processor), a hardware accelerator, and a hardware filter operator.

[0101] Figure 4 A parallel processing device with a memory subsystem is shown that allows multiple processors (e.g., Streaming Hybrid Architecture Vector Engine (SHAVE) processors) to share a multi-port memory subsystem according to some embodiments. Specifically, Figure 4 A parallel processing device 400 suitable for processing image and video data is shown. The processing device 400 includes a plurality of processing elements 402 (such as processors). Figure 4 In an exemplary configuration of , processing device 400 includes 8 processing elements (SHAVE 0 402-0–SHAVE 7 402-7). Each processing element 402 may include two load storage units 404, 406 (LSU0, LSU1) through which data can be loaded from and stored in memory 412. Each processing element 402 may also include an instruction unit 408 into which instructions can be loaded. In a specific embodiment in which the processor includes a SHAVE, the SHAVE may include one or more of a reduced instruction set computer (RISC), a digital signal processor (DSP), a very long instruction word (VLIW), and / or a graphics processing unit (GPU). Memory 412 includes a plurality of memory slices 412-0...412-7, referred to herein as connection matrix (CMX) slices. Each memory slice 412-N is associated with a corresponding processing element 402-7.

[0102] Parallel processing device 400 may also include an interconnect system 410 coupling processing elements 402 and memory slices 412. Interconnect system 410 is referred to herein as an inter-shave interconnect (ISI). ISI may include a bus through which processing element 402 can read data or write data to any portion of any of memory slices 412.

[0103] Figure 55 shows a section of a parallel processing device according to some embodiments. Section 500 includes a single processing element 402-N, a memory slice 412-N associated with the single processing element 402-N, an ISI 410 coupling the single processing element 402-N and other memory slices (not shown), and a tile control logic 506 for arbitrating communications between tiles in the memory slice 412-N and the processing element 402. As shown in section 500, the processing element 402-N can be configured to directly access the memory slice 412-N associated with the processing element 402-N, and the processing element 402-N can access other memory slices (not shown) via the ISI.

[0104] In some embodiments, each memory slice 412-N may include multiple RAM tiles or physical RAM blocks 502-0...502-N. For example, a memory slice 412-N having a size of 128kB may include 4 32kB single-port RAM tiles (e.g., physical RAM elements) organized into 4k x 32-bit words. In some embodiments, tile 502 may also be referred to as a logical RAM block. In some embodiments, tile 502 may include a single-port complementary metal oxide semiconductor (CMOS) RAM. The advantage of a single-port CMOS RAM is that it is generally available in most semiconductor processes. In other embodiments, tile 502 may include a multi-port CMOS RAM.

[0105] In some embodiments, each tile 502 may be associated with tile control logic 506. Tile control logic 506 is configured to receive requests from processing element 402 and provide access to separate read and write ports of the associated tile 502. For example, when processing element 402-N wishes to access data in RAM tile 502-0, processing element 402-N may send a memory access request to tile control logic 506-0 associated with RAM tile 502-0 before processing element 402-N sends a memory data request directly to RAM tile 502-0. The memory access request may include the memory address of the data requested by processing element 402-N. Tile control logic 506-0 may then analyze the memory access request and determine whether processing element 402-N can access the requested memory. If processing element 402-N can access the requested memory, tile control logic 506-0 may send an access granted message to processing element 402-N, and then processing element 402-N may send a memory data request to RAM tile 502-0.

[0106] Because there is the possibility of simultaneous access by multiple processing elements, in some embodiments, tile control logic 506 may include a conflict detector configured to detect instances in which two or more processing elements (such as processors or accelerators) attempt to access any one of the tiles in the memory slice. The conflict detector may monitor access to each tile 502 for attempted simultaneous accesses. The conflict detector may be configured to report to the runtime scheduler that an access conflict has occurred and needs to be resolved.

[0107] Figure 6 A centralized conflict detection system in a tile control logic according to some embodiments is shown. The conflict detection system may include a centralized arbitration block 608, which includes a plurality of conflict detectors 604 and a plurality of one-hot address encoders 602. In some embodiments, the one-hot address encoder 602 is configured to receive a memory access request from one of the processing elements 402 and determine whether the memory access request is for data stored in the RAM tile 502 associated with the one-hot address encoder 602. Each conflict detector 604 may be coupled to one or more one-hot address encoders 602, which are also coupled to one of the processing elements 402 that can access the tile 502 associated with the conflict detector 604. In some embodiments, the conflict detector 604 may be coupled to all one-hot address encoders 602 associated with a particular RAM tile 502.

[0108] If the memory access request is for data stored in the RAM tile 502 associated with the one-hot code address encoder 602, then the one-hot code address encoder 602 may provide a bit value of "1" to the conflict detector 604 of the specific RAM tile; if the memory access request is not for data stored in the RAM tile 502 associated with the one-hot code address encoder 602, then the one-hot code address encoder 602 may provide a bit value of "0" to the conflict detector 604 of the specific RAM tile.

[0109] In some embodiments, the one-hot code address encoder 602 is configured to determine whether the memory access request is for data stored in the RAM tile 502 associated with the one-hot code address encoder 602 by analyzing the target address of the memory access request. For example, when the RAM tile 502 associated with the one-hot code address encoder 602 is designed to have a memory address range of 0x0000 and 0x00ff, then the one-hot code address encoder 602 can determine whether the target address of the memory access request falls within the range of 0x0000 and 0x00ff. If so, the memory access request is for data stored in the RAM tile 502 associated with the one-hot code address encoder 602; if not, the memory access request is not for data stored in the RAM tile 502 associated with the one-hot code address encoder 602. In some cases, the one-hot code address encoder 602 can use a range comparison block to determine whether the target address of the memory access request falls within the address range associated with the RAM tile 502.

[0110] Once conflict detector 604 receives bit values ​​from all one-hot address encoders 602, conflict detector 604 may count the number of "1"s (e.g., the total bit value) in the received bit values ​​in order to determine if there is more than one processing element 402 currently requesting access to the same RAM tile 502. If there is more than one processing element currently requesting access to the same RAM tile 502, conflict detector 604 may report a conflict.

[0111] Figure 7 A distributed conflict detection system in a tile control logic according to some embodiments is shown. The distributed conflict detector system may include a distributed arbitrator 702, which includes multiple conflict detectors 704. The operation of the distributed conflict detection system is substantially similar to the operation of the centralized conflict detection system. In this case, the conflict detectors 704 are arranged in a distributed manner. Specifically, the distributed arbitrator 702 may include conflict detectors 704 arranged in a serial manner, wherein each conflict detector 704 is only coupled to a subset of the one-hot code address encoders 602 associated with a particular RAM tile 502. This arrangement is different from a centralized conflict detector system in which the conflict detectors 704 are coupled to all of the one-hot code address encoders 602 associated with a particular RAM tile 502.

[0112] For example, when a particular RAM tile 502 is accessible by 64 processing elements 402, the first conflict detector 704-0 may receive memory access requests from 32 processing elements, and the second conflict detector 704-1 may receive memory access requests from the remaining 32 processing elements. The first conflict detector 704-0 may be configured to analyze one or more memory access requests from the 32 processing elements coupled to itself, and determine a first number of elements of the 32 processing elements coupled to itself that request access to the particular RAM tile 502-0. In parallel, the second conflict detector 704-1 may be configured to analyze one or more memory access requests from the 32 processing elements coupled to itself, and determine a second number of elements of the 32 processing elements coupled to itself that request access to the particular RAM tile 502-0. Subsequently, the second conflict detector 704 may add the first number and the second number to determine how many of the 64 processing elements request access to the particular RAM tile 502-0.

[0113] Once the conflict detection system detects a conflict, the conflict detection system may send a stop signal to the processing element 402 . Figure 8 An arbitration block for reporting conflict signals to a requester according to some embodiments is shown. More specifically, an OR gate is used to combine the outputs of the range comparison blocks in the conflict detection system to generate a stop signal to the requester. The stop signal indicates that more than one processing element is attempting to access the same physical RAM sub-block within the memory slice associated with the requester. After receiving the stop signal, the requester can stop the memory access operation until the conflict has been cleared. In some embodiments, the conflict can be cleared by hardware independent of the program code.

[0114] In some embodiments, the arbitration block may operate at cycle intervals. In such embodiments, the arbitration block allocates resources at clock cycle intervals rather than at task level intervals, which may include multiple clock cycles. Such cycle-oriented scheduling can improve the performance of the system. The arbitration block may be implemented in hardware so that the arbitration block can perform cycle-oriented scheduling in real time. For example, at any particular instance, an arbitration block implemented in hardware can be configured to allocate resources for the next clock cycle.

[0115] Fig. 9A loop-directed arbitration block according to some embodiments is shown. The loop-directed arbitration block may include a port arbitration block 900. The port arbitration block 900 may include a first port selection block 930 and a second port selection block 932. The first port selection block 930 is configured to determine which of the memory access requests (identified as the bit positions in the client request vector) is assigned to slice port [0] for accessing the memory slice coupled to slice port [0], while the second selection block 932 is configured to determine which of the client request vectors is assigned to slice port [1] for accessing the memory slice coupled to slice port [1].

[0116] The first port selection block 930 includes a first leading one detector (LOD) 902-0 and a second LOD 902-1. The first LOD 902-0 is configured to receive a client request vector that may include multiple digits. Each digit in the client request vector indicates whether a message access request has been received from the requester associated with that digit position. In some cases, the client request vector operates in an "effective high" mode. Once the first LOD 902-0 receives the client request vector, the first LOD 902-0 is configured to detect the digit position (counting from left to right) at which the request first becomes non-zero, thereby identifying the first memory access request (counting from left to right) to the first port selection block 930. In parallel, the client request vector can be masked by an AND logic operator 912 to generate a masked client request vector using a mask generated by a mask register 906 and a mask left shifter 904. Mask register 906 may be set by a processor in communication with mask register 906, and mask left shifter 904 may be configured to left shift the mask represented by mask register 906. Second LOD 902-1 may receive the masked client request vector from AND logic operator 912 and detect leading ones in the masked client request vector.

[0117] The outputs from the first LOD 902-0 and the second LOD 902-1 are then provided to a port [0] winner selection block 908. The port [0] winner selection block 908 also receives two additional inputs: a priority vector and a super priority vector. The port [0] winner selection block 908 is configured to determine which of the received memory access requests should be assigned to the slice port [0] based on the priorities of the inputs. In some embodiments, the priorities of the inputs may be ranked as follows: starting with the super priority vector having the highest priority, the masked LOD vector is divided into priority vectors of priority requests and non-priority requests, followed by the unmasked LOD vector having the lowest priority. In other embodiments, other priorities may be specified.

[0118] While the first port selection block 930 may be configured to determine whether a client request vector may be assigned to slice port [0], the second port selection block 932 may be configured to determine whether a client request vector may be assigned to slice port [1]. The second port selection block 932 includes a first subsequent one detector (TOD) 912-0, a second TOD 912-1, a mask register 914, a mask right phase shifter 916, a port [1] winner selection block 918, and a masked AND logic block 920. The TOD 912 is configured to receive a client request vector that may include a plurality of bits, and detect the bit position at which the vector first becomes non-zero (counting from right to left). The operation of the second port selection block 932 is substantially similar to the first port selection block 930, except that it operates from the right to the left of the input vector, thereby using the subsequent one detector 912-0 to select the subsequent one in the input request vector.

[0119] The outputs of the port winner selection blocks 908, 918 are also provided to the same winner detection block 910, which is configured to determine whether the same memory access request has won access to both slice port [0] and slice port [1]. If the same client request vector has won access to both slice port [0] and slice port [1], then the same winner detection block 910 selects one or more slice ports to route the request and allocates another port to the next highest ranking request in the input vector. This avoids over-allocation of resources to specific requests, thereby improving resource allocation to competing requesters.

[0120] The operation of the port arbitration block 900 works by starting with the left hand side of the 32 bit client request vector and the masked LOD 902-1 outputs the position of the first masked request vector, if this masked request vector is not superseded by a higher priority input via the priority or super priority vector, then the requester corresponding to the LOD position wins and is granted access to port [0]. The LOD position is also used to advance the mask position via the 32 bit left shifter 904 and is also used to compare with the port 1 LOD assignment to check if the same requester has been given access to both ports and in this case only one of the ports is granted, where the flip-flop is switched to grant access on an alternating basis between ports 0 and 1 in the event of sequential identical winner detections. In this case, the LOD output from the masked detector 902-1 has been assigned priority via the corresponding one bit in the priority vector and the requesting client is granted 2 back-to-back cyclic access to port 0. In the absence of a leading one in the masked client request vector and no higher priority request exists, the unmasked LOD wins and is assigned access to port 0. In either case, a 1 bit in the super-priority vector will overwrite any of the previous requests and grant unrestricted access to port 0 to the requestor.

[0121] The logic in the lower portion of the figure starts at the right hand side of the request vector and otherwise operates in the same manner as the upper portion, which starts at the left hand side of the request vector. In this case, the operation of port 1 of the arbitration block is the same as port 0 of the logic portion in terms of priority, etc.

[0122] In some embodiments, processing element 402 may include a buffer to reduce memory access latency due to memory access arbitration. Fig.10A mechanism for reducing memory access latency due to memory access arbitration according to some embodiments is shown. In a typical memory access arbitration scheme, the memory access arbitration block is a pipeline that incurs a fixed overhead arbitration penalty when a shared resource (such as a RAM tile 502) is allocated to one of multiple processing elements (e.g., requestors). For example, when processing element 402 sends a memory access request to arbitration block 608 / 702, processing element 402 spends at least 4 cycles to receive an access grant message because it spends at least one cycle in each of the following steps: (1) analyzing the memory access request at one-hot code address encoder 602, (2) analyzing the output of one-hot code address encoder 602 at arbitration block 608 / 702, (3) sending an access grant message to one-hot code address encoder 602 via arbitration block 608 / 702, and (4) sending the access grant message to processing element 402 via one-hot code address encoder 602. Then subsequently, processing element 402 must send memory data requests to RAM tile 502, and receive data from RAM tile 502, each of which takes at least one cycle. Therefore, memory access operations have a latency of at least 6 cycles. This fixed penalty will reduce the bandwidth of the memory subsystem.

[0123] This latency issue can be solved using a memory access request buffer 1002 maintained in the processing element 402. For example, the memory access request buffer 1002 can receive memory access requests from the processing element every clock cycle, and store the received memory access requests until they are ready to be sent to the memory arbitration block 608 / 702. The effective buffer 1002 synchronizes the rate at which memory access requests are sent to the memory arbitration block 608 / 702 and the rate at which data is received from the memory subsystem. In some embodiments, the buffer may include a queue. The number of elements in the buffer 1002 (e.g., the depth of the buffer) may be greater than the number of cycles used to retrieve data from the memory subsystem. For example, when the RAM access latency is 6 cycles, the number of elements in the buffer 1002 may be 10. The buffer 1002 can reduce the arbitration latency penalty and improve the throughput of the memory subsystem. Through the memory access request buffer, in principle, up to 100% of the total memory bandwidth can be allocated between requesters.

[0124] It should be appreciated that a potential problem with using multiple RAM instances is that by allowing simultaneous access to sub-instances within a library by multiple processing elements, memory contention may result.

[0125] The present disclosure provides at least two methods for resolving memory contention. First, pay attention to software design, as will be described later, so as to avoid memory contention and / or memory conflicts by carefully arranging data in the memory subsystem so as to reduce contention and / or memory conflicts. In addition, software development tools associated with the parallel processing device may allow reporting of memory contention or memory conflicts during the software design stage. Therefore, by improving data arrangement in response to memory contention or memory conflicts reported during the software design stage, memory contention problems or memory conflict problems can be corrected.

[0126] Second, as further described below, an ISI block within the architecture is configured to detect port conflicts (contentions) in hardware and stall processing elements with lower priorities. For example, the ISI block is configured to analyze memory access requests from processing elements, service memory access request sequences, and route memory access requests according to priority order so that all data reads and writes from all processing elements are completed in priority order.

[0127] The priority order for the processing elements may be established in a number of ways. In some embodiments, the priority order may be statically defined at system design time. For example, the priority order may be encoded as a reset state of a system register so that when the system power is turned on, the system powers up with a set of pre-assigned priorities. In other embodiments, the priority order may be dynamically determined via a user programmable register.

[0128] In some embodiments, programmers can plan the data placement for their software applications to reduce contention for shared sub-blocks of memory within a memory slice. In some cases, the planning of data placement can be aided by an arbitration block. For example, the arbitration block can detect memory contention, grant access to memory to the processing element associated with the highest priority task on a priority basis, stop other processing elements that are contentious, and unwind the contention process by processing until the contention has been resolved.

[0129] Fig.11 The application of scheduling software according to some embodiments is shown. In the present application, the scheduling software can coordinate the implementation of the 3x3 fuzzy kernel in the processing pipeline. The scheduling software can determine the order of operations in the runtime and coordinate the operations through the processing elements. The flowchart 1100 for the pipeline includes elements 1-element 5, 1102-1110. Element 1 1102 may include an input buffer 1112, a processing block 1144 and an output buffer 1114. The input buffer 1112 and the output buffer 1114 can be implemented using triggers. In some embodiments, each of the other elements 1104-1110 may have a structure substantially similar to element 1 1102.

[0130] In some embodiments, element 2 1104 may include a processing element (e.g., a vector processor or hardware accelerator) that may filter an input with a 3x3 fuzzy filter. Element 2 1104 may be configured to receive input from a shared buffer 1118 that temporarily holds the output of element 1 1102. To apply a 3x3 fuzzy kernel to the input, element 2 1104 may receive at least 3 rows of data from the shared input buffer 1118 before it begins the operation. Thus, a SW scheduler 1120 that may run on a RISC processor 1122 may detect that the correct number of rows of data is contained in the shared buffer 1118 before sending a signal to element 2 1104 that it may begin the filtering operation.

[0131] After the initial signal that there are 3 lines of data, the SW scheduler 1120 can be configured to send a signal to element 2 1104 when each additional new line has been added to the rolling 3-line buffer 1118. In addition to row-by-row synchronization, cycle-by-cycle arbitration and synchronization are performed for each element in the pipeline. For example, element 1 1102 may include a hardware accelerator that produces one complete output pixel per cycle. To achieve this throughput, the hardware accelerator may keep the input buffer 1112 full so that the processing block 1144 has enough data to continue its operation. In this way, the processing block 1144 can generate enough output to keep the throughput of element 1102 as high as possible.

[0132] In some embodiments, the software tool chain can use the memory subsystem to predict memory conflicts from analyzing software programs. The software tool chain may include an integrated development environment (IDE) (e.g., an Eclipse-based IDE) based on a graphical user interface (GUI), from which developers can edit code, call compilers, assemblers, and perform source level debugging when needed. The software tool chain can be configured to predict memory conflicts using a system simulator via dynamic analysis of programs running on multiple processors, and the system simulator models all processing, buses, memory elements, and peripherals. The software tool chain may also be configured to record in a log file or display device whether different programs running on different processors or hardware resources attempt to access specific blocks of memory slices at the same time. The software tool chain may be configured to record on a cycle-by-cycle basis.

[0133] In some embodiments, pipeline 1100 may also include one or more hardware counters (e.g., one counter for each memory instance) that are incremented each time a memory conflict occurs. These counters can then be read by a hardware debugger (e.g., JTAG) and displayed on a screen or logged to a file. Subsequent analysis of the log file by a system programmer may allow memory accesses to be scheduled differently to reduce the likelihood of memory port conflicts.

[0134] For IBM's cell architecture (in Figure 2 A key difficulty for programmers of the disclosed architecture (shown in FIG. 1 ) is to programmably schedule data transfers hundreds of cycles in advance so that when the vector processor accesses the data, the data can be DMA controlled and stored in the local storage (LS). Some embodiments of the disclosed architecture can solve this problem by handling the arbitration and scheduling of accesses in hardware and recording conflicts in user-readable hardware counters. This allows the disclosed architecture to be used to create high-performance video / image processing pipelines.

[0135] Fig.12 A hierarchical structure of a system with a parallel processing device according to some embodiments is provided. The system 1200 may include a parallel computing system 1202 having a plurality of processing elements (such as filters) and a software application 1204 running on the parallel computing system 1204, an application programming interface (API) 1206 for enabling the application 1204 to interact with the parallel computing system 1202, a compiler 1208 for compiling the software application 1204 for running on the parallel computing system 1202, and a runtime scheduler 1210 for controlling the operation of the processing elements in the parallel computing system 1202.

[0136] In some embodiments, the parallel processing device of the present disclosure may be configured to operate in conjunction with a pipeline description tool (e.g., a software application) 1204, which allows the image processing pipeline to be described as a flow chart. The pipeline description tool 1204 can describe the image / visual processing pipeline in a flexible manner independent of the underlying hardware / software platform. Specifically, the flow chart used by the pipeline description tool allows the task to be described independently of the processing elements (e.g., processor and filter accelerator resources) that can be used to implement the flow chart. The output of the resulting pipeline description tool may include a description of a directed acyclic graph (DAG) or a flow chart. The description of the DAG or flow chart may be stored in a suitable format (such as XML).

[0137] In some embodiments, the description of the DAG or flow chart may be accessible to all other tools in the system 1200 and may be used to control operations of parallel processing devices according to the DAG. Fig.13A description showing how a DAG or flow chart may be used to control the operation of a parallel processing device according to some embodiments.

[0138] Prior to actual operation of the computing device, a compiler 1208 for the parallel computing system 1202 may perform (1) a description of the flowchart 1306 and (2) a description of the available resources 1302, and generate a task list 1304 indicating how to execute the DAG across multiple processing elements. For example, when a task cannot be executed on a single processing element, the compiler 1208 may split the task across multiple processing elements; when a task can be executed on a single processing element, the compiler 1208 may assign the task to a single processing element.

[0139] In some cases, when a task will only use a portion of the processing element's capabilities, compiler 1208 may fuse and schedule multiple tasks to be executed on a single processing element in a sequential manner, up to the limit that can be supported by the processing element. Fig.14A The scheduling and release of tasks by the compiler and scheduler according to some embodiments are shown. The advantage of using the compiler and scheduler to schedule tasks is that the compiler and scheduler can automatically schedule tasks based on the operations performed by the tasks. This is a great advantage over the prior art, in which the programmer must manually determine the schedule for the code running on a processing element or a group of processing elements running a specific task, including when to schedule data transfers from peripheral devices to CMX, from CMX to CMX blocks, and from CMX back to peripheral devices via DMA. This is a laborious and error-prone task, and the use of DFG allows this process to be automated, thereby saving time and increasing output.

[0140] During runtime of the computing device, the runtime scheduler 1210 can dynamically schedule tasks across available processing elements based on the task list 1304 generated by the compiler 1208. The runtime scheduler 1210 can operate on the host RSIC processor 1306 in a multi-core system and can schedule tasks across processing elements (such as multiple vector processors, filter accelerators, and direct memory access (DMA) engines) using statistics from the hardware performance monitor and timer 1308. In some embodiments, the hardware performance monitor and timer 1308 may include stall counters, CMX conflict counters, bus cycle counters (ISI, APB, and AXI), and cycle counters that can be read by the runtime scheduler 1210.

[0141] In some embodiments, runtime scheduler 1210 may assign tasks to available processing elements based on statistics from hardware performance monitor and timer 1308. Hardware performance monitor and timer 1308 may be used to increase the efficiency of processing elements, or to perform tasks using a smaller number of processing elements in order to save power or allow other tasks to be computed in parallel.

[0142] To this end, the hardware performance monitor and timer 1308 may provide a performance metric. The performance metric may be a number indicating the activity level of a processing element. The performance metric may be used to control the number of instantiated processing elements used to perform a task. For example, when the performance metric associated with a particular processing element is greater than a predetermined threshold, then the runtime scheduler 1210 may instantiate additional processing elements of the same type as the particular processing element, thereby allocating tasks on more processing elements. For another example, when the performance metric associated with a particular processing element is less than a predetermined threshold, then the runtime scheduler 1210 may remove one of the instantiated processing elements of the same type as the particular processing element, thereby reducing the number of processing elements that perform a task.

[0143] In some embodiments, the runtime scheduler 1210 may prioritize the use of processing elements. For example, the runtime scheduler 1210 may be configured to determine whether a task should be preferentially assigned to a processor or a hardware filter accelerator.

[0144] In some embodiments, the runtime scheduler 1210 may be configured to change the CMX buffer placement in the memory subsystem so that the system can comply with runtime configuration criteria. The runtime configuration criteria may include, for example, image processing throughput (frames per second), energy consumption, the amount of memory used by the system, the number of operating processors, and / or the number of operating filter accelerators.

[0145] The output buffer may be arranged in memory in one of several ways. In some cases, the output buffer may be physically contiguous in memory. In other cases, the output buffer may be "blocked" or "sliced". For example, the output buffer may be split into N vertical strips, where N is the number of processors assigned to the image processing application. Each strip is located in a different CMX slice. Since each processor can access the input and output buffers locally, this arrangement can help the processor. However, because such an arrangement can cause many conflicts for the filter accelerator, this arrangement can be harmful to the filter accelerator. Filter accelerators usually process data from left to right. Therefore, all filter accelerators will start their processing by accessing the first strip of the image, which will cause many conflicts from the beginning. In other cases, the output buffer may be interleaved. For example, the output buffer may be split across all 16 CMX slices, with an interleaving of a predetermined size. The predetermined size may be 128 bits. Because spreading access across CMX reduces the possibility of conflicts, the interleaving arrangement of the output buffer may help the filter accelerator.

[0146] In some embodiments, buffers (such as input buffers or output buffers) can be allocated based on whether their producers and consumers are hardware and / or software. Consumers are more important since they typically require more bandwidth (filters typically read multiple lines and output one line). Hardware filters are programmed corresponding to the arrangement of buffers (they support sequential, interleaved, and sliced ​​memory addressing).

[0147] Fig. 14B A process for automatically scheduling tasks using a compiler and a scheduler according to some embodiments is shown. The compiler determines a list of tasks to be executed by a parallel processing device based on a DAG. In step 1402, the runtime scheduler is configured to receive the task list and maintain the task list in separate queues. For example, when the task list includes (1) tasks to be executed by DMA, (2) tasks to be executed by a processor, and (3) tasks to be executed by a hardware filter, the runtime scheduler may store the tasks in three separate queues, such as a first queue for DMA, a second queue for the processor, and a third queue for the hardware filter.

[0148] In steps 1404-1408, when the associated hardware components become available for the new task, the runtime compiler is configured to issue the task to the associated hardware components. For example, in step 1404, when DMA becomes available for executing the new task, the runtime compiler is configured to dequeue the first queue for DMA and provide the dequeued task to DMA. Similarly, in step 1406, when the processor becomes available for executing the new task, the runtime compiler is configured to dequeue the second queue for the processor and provide the dequeued task to the processor. In addition, in step 1408, when the hardware filter becomes available for executing the new task, the runtime compiler is configured to dequeue the third queue for the hardware filter and provide the dequeued task to the hardware filter.

[0149] In some embodiments, the runtime scheduler 1210 can use the counter values ​​from the hardware performance monitor and timer 1308 to adjust the use of the processing elements, particularly in the case where more than one pipeline (e.g., software application 1204) is running simultaneously on the processing element array, since the pipelines do not need to be designed together. For example, if the effective bus bandwidth allocated to each pipeline is less than expected, and the number of conflicts occurring in accessing CMX memory is large, the runtime scheduler 1210 can use this information to interleave the execution of the two pipelines by modifying the order in which tasks are extracted from the queues of the two pipelines, thereby reducing memory conflicts.

[0150] In some implementations, the DAG compiler may operate in real-time (eg, online). Fig.15 The operation of the real-time DAG compiler according to some embodiments is shown. The real-time DAG compiler 1502 can be configured to receive the input XML description of the DAG, the description of the available processing elements, and any user-defined constraints (such as, the number of processors, frame rate, power dissipation target, etc.). Subsequently, the real-time DAG compiler 1502 can be configured to schedule DAG components across processing elements (including, for example, DMA engines, processors, hardware filters, and memories) to ensure that the DAG can meet the user-defined constraints when mapped to system resources as specified. In some embodiments, the real-time DAG compiler 1502 can determine whether the tasks in the DAG may be executed in parallel in a breadth-first manner. If the breadth of the DAG is greater than the number of processing elements that can be used to execute tasks in parallel (for example, the amount of available processing power is less than the parallelism of the DAG), the real-time DAG compiler 1502 can "fold" the tasks so that the tasks are executed sequentially on the available processing elements.

[0151] Fig.16According to some embodiments, the schedule generated by the OpenCL scheduler is compared with the schedule generated by the online DAG scheduler of the present invention. The schedule generated by the scheduler 1208 / 1502 of the present invention can eliminate the redundant copies and DMA transfers that exist in the typical OpenCL schedule. Because the GPU used to perform processing on the DAG tasks is far away from the processor that executes the schedule, these data transfers exist in the OpenCL schedule. In a typical application processor used in a mobile device, large blocks of data are transferred back and forth between the processor that executes the schedule and the GPU that performs the processing. In the design of the present invention, all processing elements share the same memory space, and therefore no copying back and forth is required, thereby saving considerable time, bandwidth, and power dissipation.

[0152] In some embodiments, when a task will only use a portion of the processing element's capabilities, the runtime scheduler 1210 may also be configured to fuse and schedule multiple tasks to be executed on a single processing element in a sequential manner, up to the limit that can be supported by the processing element, as shown in FIG. 14 .

[0153] In image processing applications, the scheduler may be configured to divide the processing tasks among the processors by dividing the image into strips. For example, the image may be divided into vertical strips or horizontal strips of predetermined width.

[0154] In some embodiments, the scheduler may predetermine the number of processors for a specific image processing application. This allows the scheduler to predetermine the number of bars for an image. In some embodiments, filtering operations may be performed by a series of processors. For example, when there are 5 software filters executed by an application, processing element 402 may be configured to execute a first software filter simultaneously at a first time instance, and to execute a second software filter simultaneously at a second time instance, etc. This means that the computational load is more evenly balanced in the processors assigned to the specific image processing application. This is because the processors are configured to execute the filters of the same list simultaneously in the same order.

[0155] When too many processors are assigned to image processing applications, the processors may spend a lot of time idling, waiting on the hardware filter accelerator to complete tasks. On the other hand, when too few processors are assigned to applications, the hardware filter accelerator may spend a lot of time idling. In some embodiments, the run scheduler 1210 may be configured to detect these situations and adapt accordingly. In some embodiments, the scheduler 1210 may be configured to over-allocate processors to specific image processing applications and allow the processor to shut down once the processor has completed its task before the hardware filter accelerator.

[0156] In some embodiments, the scheduler may use a blocking mechanism to synchronize processing elements, such as hardware filter accelerators and processors. The scheduler output may include a command stream. These commands may include (1) start commands for processing elements (such as hardware filter accelerators and processors) and (2) blocking commands. The blocking command indicates that the processing element should wait from proceeding to the next set of commands until all processing elements in the group have reached the blocking command, even if some of the processing elements have actually completed their tasks. In some embodiments, the scheduler may provide blocking commands based on dependencies between tasks performed by the processing elements.

[0157] Fig.17 A blocking mechanism for synchronizing processing elements according to some embodiments is shown. The command stream includes blocking commands (1702, 1712) and task commands (1704, 1706, 1708, 1710). Each task command may be associated with a processing element, and as indicated in the following figure, the task commands may be completed at different times. The scheduler may therefore insert blocking commands 1712 so that the processing element does not perform future tasks until the blocking commands 1712 are cleared. This blocking mechanism may be considered a time pipeline for parallel tasks.

[0158] In some embodiments, the blocking mechanism is implemented in hardware using interrupt signals 1714. For example, the scheduler can program a bit mask indicating which processing elements belong to a group. When a processing element completes an assigned task, the interrupt signal associated with the corresponding processing element is asserted. Once all interrupt signals associated with the processing elements in the group have been asserted, the controller of the processing element can receive a global interrupt signal indicating that all processing elements have reached the blocking command.

[0159] In some embodiments, the interrupt source may include a SHAVE vector processor, an RSIC processor, a hardware filter, or an external event. Specifically, the hardware filter may support various modes (including non-circular buffer modes in which the input / output buffer contains frames), and the filter may be configured to issue a single interrupt when it has processed an entire input frame or written a complete corresponding output frame. The filter is also programmable to operate on lines, slices, or tiles from a frame using appropriate settings for image size, buffer base address / line stride, etc.

[0160] An important challenge with complex parallel processing devices is how to program the processing elements in the parallel processing devices, especially for embedded systems that are very power sensitive and scarce in terms of resources like computer resources and memory. Computational imaging, video and image processing are very demanding in terms of performance, especially on embedded systems, since frame sizes and rates are very high and are strongly increasing year by year.

[0161] The solution to this problem in this article is to provide an application programming interface (API) 1206 that allows programmers to write applications at a high level without having to closely understand the details of the multi-core processor architecture 1202. Using the software API 1206, programmers can quickly create new image or video processing pipelines without having to understand the close details of the implementation, because the programmer does not need to consider the details of whether the function is implemented in software or hardware on a programmable processor. For example, an implementation of a fuzzy filter kernel is set as a reference software implementation running on one or more processors or hardware accelerator filters. The programmer can initially use a software fuzzy filter implementation, and can switch to using a software filter without changing the overall pipeline implementation because the ISI, AMC, and CMX arbitration blocks handle which processor and HW resources access the physical memory blocks and in which order, not the programmer.

[0162] Although the multi-port memory approach described above is adequate for sharing memory between the same processors with high bandwidth and low latency, it is not ideal for sharing bandwidth with other devices. These other devices may be hardware accelerators and other processors with different latency requirements, particularly in very high bandwidth applications such as computational video and image processing.

[0163] The architecture of the present disclosure can be used in conjunction with a multi-port memory subsystem to provide higher bandwidth to support more simultaneous accesses from multiple programmable VLIW processors with highly deterministic latency requirements, a large group of programmable image / video processing hardware filters and a bus interface to allow control and data access by conventional host processors and peripheral devices. Fig.18 A parallel processing device with different types of processing elements according to some embodiments is shown. The parallel processing device includes multiple processors 1802 and multiple filter accelerators 1804, and the multiple processors 1802 and the multiple filter accelerators 1804 can be coupled to the memory subsystem 412 via the ISI 410 and the accelerator memory controller (AMC) 1806, respectively.

[0164] The subsystems of the AMC 1086 and the multi-core memory (CMX) subsystem 412 provide on-chip storage to facilitate low-power streaming digital signal processing in software on the processor 1802 and hardware filter accelerators 1804 for specific image / video processing applications. In some embodiments, the CMX memory 412 is organized into 16 slices of 128kB (2MB total) organized into 64-bit words. Each processor 1802 may have direct access to a slice in the memory subsystem 412, and indirect (higher latency) access to all other slices in the memory system 412. The processor 1802 may use the CMX memory 412 for storing instructions or data, while the hardware filter accelerator 1804 uses the CMX memory 412 for storing data.

[0165] To facilitate data sharing between non-uniform processing elements, while allowing latency-intolerant processors 1802 to achieve high performance when accessing shared CMX memory 412 with HW filter accelerators 1804, the HW filter accelerators 1804 are designed to be latency-tolerant. This is achieved by providing each HW filter accelerator (filter) 1804 with a local FIFO that makes timing more flexible and a crossbar switch for shared access to CMX, thereby making ISI free to support inter-SHAVE communication without contention from the HW filter accelerators, as shown in Fig.10 In some embodiments shown.

[0166] In addition to external outgoing port conflicts, there may also be conflicts with incoming ISI 410 port accesses. Port conflicts may occur if more than one external slice attempts to access the same memory slice in any one cycle. The mapping of LSU ports to ISI interconnect ports is fixed, so without any conflicts, SHAVE 0 1802-0 may access slice 2 through LSU port 0, and SHAVE 11 (1702-11) may access slice 2 through LSU port 1. The ISI matrix may allow 8 x 2 ports x 64 bits of data to be transferred per cycle. For example, SHAVE N 1802 may access slice N+1 through LSU ports 0 and 1, and all 8 SHAVE processors may access simultaneously without any stalls.

[0167] In some embodiments, the memory subsystem 412 may be logically divided into slices (blocks). Fig.19 A multi-core memory subsystem of the present invention is shown according to some embodiments. Fig.19Detailed bus interconnections between AXI, AHB, SHAVE, ISI and CMX and filter accelerators, AMC and CMX are shown. The figure shows two AMC input and two AMC output ports, 2 ISI input and 2 ISI output ports, connections to L2 cache and mutually exclusive (mutually exclusive) blocks, and internal read / write arbitration and source multiplexing for addressing 4 memory tiles and FIFOs, and output destination selection for 4 memory block outputs to ISI and AMC.

[0168] Each slice can be connected to two of the 16 possible ISI input sources including 12 SHAVEs, DMA, Texture Management Unit (TMU), and AHB bus interface to the onboard host processor. Similarly, each slice has 2 output ISI ports, which allow the slice to send data to 2 of the 16 possible destinations including 12 SHAVEs, DMA, Texture Management Unit (TMU), and AXI and AHB bus interfaces to the onboard host processor. In the preferred embodiment, the slice contains 4 physical RAM blocks with input arbitration blocks that are in turn connected to the local SHAVE processors (2x LSU and 2x 64-bit instruction ports), 2 ISI input ports, 2 AMC input ports, and FIFOs for inter-SHAVE messaging, as well as message transfer FIFOs, L2 cache, and mutex blocks.

[0169] On the output path from the CMX slice, the input to the destination select block connects to 4 RAM instances, along with the L2 cache and hardware mutex blocks. Fig. 20 The output of the destination selection block shown as block 2002 is connected to 2 local LSU ports and instruction ports (SP_1 and SP_0) and 2x ISI output ports and 2x AMC output ports. The 2 ISI ports allow the local slice to be connected to two destinations from 12 possible processors, DMA, TMU AXI and AHB host buses. The processor is provided with access to the memory via the SHAVE Interconnect (ISI), which is connected to the 2x 64-bit ISI input and 2x 64-bit ISI output ports contained in the multi-core memory subsystem slice. The high-bandwidth, low-latency deterministic access provided by the ISI interconnect reduces stalls in the processor and provides high computational throughput.

[0170] Fig.21An AMC crossbar architecture according to some embodiments is shown. The AMC 1806 may be configured to connect a hardware image processing filter 1804 to an AMC port of a CMX multi-core memory slice 412. The AMC 1806 may include one or more slice port controllers 2102, preferably one controller for each CMX slice 412. The slice port controller 2102 is in turn connected to a slice address request filter (SARF) 2104. The SARF 2014 is in turn connected to an AMC client (in this embodiment, the AMC client is a hardware image processing accelerator). The SARF 2104 receives read / write requests from the filter accelerator and provides them with request or grant signals, thereby receiving data and addresses from SIPP blocks that have been granted write access, and providing read data to those that have been granted read access. In addition, the SARF provides an AXI master to the system host processor via the AXI host bus, which allows the host to query (read / write) the CMX memory via the AMC crossbar switch.

[0171] In some embodiments, the slice port controller 2102 is provided in the AMC 1806, which communicates with the 2x read and 2x write ports in the associated CMX memory slice 412, such as Fig.21 As shown. From the filter accelerator side of the CMX memory subsystem 412, each hardware filter is connected to a port on the accelerator memory controller (AMC) 1806. The AMC 1806 has a pair of 64-bit read ports, and a pair of 64-bit write ports, connecting it to each slice of the CMX memory 412 (there are 16 slices totaling 2MB in the preferred embodiment). Connecting the image processing hardware accelerator to the AMC 1806 via a read or write client interface and local buffers within the accelerator allows for more relaxed latency requirements, leaving more bandwidth available for ISI and processors with highly deterministic timing, allowing for reduced processor stalls.

[0172] Fig. 20 A single slice of the CMX infrastructure according to some embodiments is shown. The slice includes: an arbiter and source multiplexing that allows up to 4 of eight possible 64-bit sources to access 4 physical SRAM tiles in the CMX slice; a shared L2 cache; a shared mutex hardware block for inter-processor negotiation of mutually exclusive threads; and a 64-bit FIFO for low-bandwidth messaging between SHAVEs. The six input sources are: Fig.21] and slice_port[0] ports of the ACM shown in FIG; in addition, 2 ISI ports, ISIout1 and ISIout0; 2xLSU ports, LSU_1 and LSU_0; and finally, 2x instruction ports SP_1 and SP_0 that when combined allow 128-bit instructions to be read from the CMX. The arbiter and source multiplexer generate read / write addresses and 64-bit data to control the 4x SRAM tiles in response to priority access from 8 input sources. Although the input of the 64-bit inter-SHAVE communication FIFO is connected to the arbiter and source multiplexer, its output is readable only by the local processors in the CMX slice. In practice, the processors communicate with each other's message passing FIFOs via the ISIout1 and ISIout0 ports and the ISI infrastructure outside the CMX slice that connects the CMX, slices, and processors together.

[0173] Except Fig. 20 In addition to the arbitration between the 64 requestors on each of the 2 AMC ports of the slice shown in FIG, an additional 2:1 arbiter is provided to arbitrate between AMC ports 1 and 0. The purpose of this 2:1 arbiter is to prevent one or the other of the 2 AMC ports from saturating the entire AMC port bandwidth, which could result in excessive stalls on one of the requesting ports. This enhancement provides a more balanced resource allocation in the presence of multiple heavy port bandwidth requestors, and thus a higher sustained throughput for the overall architecture. Similarly, a 2:1 arbiter arbitrates between the 2 processor ports SP1 and SP0 for similar reasons.

[0174] Arbitration and multiplexing logic also controls access through the processors, either directly or via the ISI to the shared L2 cache, via a second arbitration level that shares access between 16 possible sources on a strict round-robin basis, with one 64-bit port connected between the second level arbiter and each of the 16 CMX slices. Similarly, the same logic allows access to 32 hardware mutexes that are used for mutually exclusive inter-processor negotiation among threads running on the 12 onboard processors and 2x 32-bit RISC processors (connected via the AHB and AXI buses on the ISI).

[0175] The priority in the preferred embodiment is that SP_1 and SP_0 have the highest priority, followed by LSU_1 and LSU_0, next ISIout1 and ISIout0, and finally AMCout1 and AMCout0, and finally FIFO has the lowest priority. The reason for this priority distribution is that SP_1 and SP_0 control program access to CMX by the processor, and if the next instruction is not available, then the processor will stall immediately, followed by LSU_1 and LSU_0 which will also cause the processor to stall; similarly ISIout1 and ISIout0 are from other processors and will cause other processors to stall if the data is not immediately available. The AMCout1 and AMCout0 ports will have the lowest priority because they have built-in FIFOs and can thus tolerate many delays before stalling. The processor FIFOs are only needed for low-bandwidth inter-processor messaging and thus have the lowest priority of all.

[0176] Once the arbiter has granted access to up to 4 sources of 4 SRAM tiles, L2 cache, mutex, and FIFO, output data from six read data sources (including 4 SRAM tiles, L2 cache, and mutex) is selected and directed to up to 4 of 8 possible 64-bit destination ports; 4 on the processor associated with the slice (SP_1, SP_0, LSU_1, and LSU_0); 2 associated with ISI (ISIout1 and ISIout0); and finally 2 associated with AMC (AMCout1 and AMCout0). Since only 4 64-bit sources need to be assigned to 8 destination ports, no priority order is required in the output multiplexer.

[0177] Fig. 22AMC cross port controller according to some embodiments is shown. AMC cross port controller 2202 includes round-robin arbiter 2204, which connects port controller 2202 to accelerator 1804 whose request has been filtered via processor. The arbiter can then push valid requests from AMC clients to port controller FIFO. In the case of a read request, the response to the request (read client ID and line index) is pushed to read TXID FIFO. The returned slice port data and valid signal are input into the port controller read logic, which takes out the read client ID and line index from Rd TX ID FIFO, and pushes the corresponding slice port read data and available signal to the Rd data FIFO that can be read from other AMC clients of the request. On the CMX side of FIFO, port stop logic takes out the request from FIFO, and slice port control is provided to 2x AMC input ports on the associated CMX memory slice.

[0178] The number of read and write client interfaces to CMX is individually configurable. Any client can address any slice (or slices) of CMX. With 16 slices in CMX, 2 ports per slice and a system clock frequency of 600MHz, the maximum total data memory bandwidth that can be supplied to the clients is 143GB / s: Maximum bandwidth = 600MHz*(64 / 8)*2*16 = 1.536e11 B / s = 143GB / s.

[0179] At a higher clock rate of 800MHZ, the bandwidth rises to 191GB / sec. The AMC arbitrates between simultaneous accesses on its client read / write interfaces connected to its hardware accelerator blocks. A maximum of two read / write accesses may be granted per chip per clock cycle, giving a maximum chip memory bandwidth of 8.9GB / s at a system clock frequency of 600MHz. Client accesses are not restricted to the CMX address space. Any access that falls outside the CMX address space is forwarded to the AMC's AXI bus master.

[0180] Fig.23A read operation using AMC 1806 is shown according to some embodiments. In this illustration, 4 data words are read from the address range A0-3. The AMC client (e.g., filter accelerator 1804) first asserts a request on the port controller input. The port controller 2202 responds by issuing a grant signal (gnt), which in turn causes the client to issue addresses A0, A1, A2, and finally A3. The corresponding rindex value appears on the rising edge of the clock (clk) corresponding to each grant. It can be seen that the timing on the client side can be very flexible when compared to the index address and data, which are the outputs from the CMX slice to the port controller. Deterministic timing on the CMX side of the port controller allows efficient sharing of access to the CMX between AMC clients and processors that are very sensitive to latency, while the FIFO allows and local storage in the AMC client allows the timing to be highly variable on the AMC client (e.g., filter accelerator) side of the CMX memory subsystem 412.

[0181] Fig.24 A write operation using AMC 1806 is shown according to some embodiments. In the timing diagram, 4 data words are shown to be transferred to CMX via AMC. A request is issued by the AMC client, and on the following clock (clk) rising edge, the grant signal (gnt) rises to transfer data word D0 associated with address A0 via AMC. The gnt signal then goes low for one clock cycle and rises high on the next rising edge of clk for 2 clock cycles, allowing D1 and D2 to address A1 and A2 respectively before gnt goes low again. On the next rising clk edge, gnt rises again to allow data word D3 to be transferred to address A3, so req and gnt go low on the next rising edge of clk to wait for the next read / write request.

[0182] Combination Fig.12 The Streaming Image Processing Pipeline (SIPP) software framework used in the exemplary arrangement of provides a flexible method for implementing an image processing pipeline using CMX memory 412 for scan line buffers, frame tiles (sub-segments of a frame), or in fact the entire frame at high resolution, attaching an external DRAM die in package wire bonding to a substrate to which the image / video processing die is attached. The SIPP framework takes care of complexities such as handling image boundaries (pixel duplication) and circular line buffer management, making the ISP (Image Signal Processing) function implemented in software (on the processor) simpler and more versatile.

[0183] Fig.25A parallel processing device 400 according to some embodiments is shown. The parallel processing device 400 may include a memory subsystem (CMX) 412, a plurality of filter accelerators 1804, and an AMC 1806 for arbitrating access to the memory subsystem 412. The memory subsystem (CMX) 412 is constructed to allow multiple processing elements 402 to access data and program code in parallel without stopping. These processing elements 402 may include, for example, a SHAVE (stream hybrid architecture vector engine) processor, an appropriate VLIW (very long instruction word) processor, parallel access to data and programs without stopping or a filter accelerator. In addition, the memory subsystem (CMX) 412 may make provisions for a host processor (not shown) to access the CMX memory subsystem 412 via a parallel bus such as AXI (not shown). In some embodiments, each processing element 402 can read / write up to 128 bits per cycle through its LSU port, and read up to 128 bits of program code per cycle through its instruction port. In addition to the ISI and AMC interfaces for the processor and filter accelerator, respectively, CMX 412 provides simultaneous read / write access to memory through AHB and AXI bus interfaces. AHB and AXI are standard ARM parallel interface buses that allow the processor, memory, and peripherals to be connected using the AMC 1806. The CMX memory subsystem 412 can be configured to handle a peak of 18 x 128-bit memory accesses per cycle.

[0184] The accelerator 1804 includes a collection of hardware image processing filters that can be used in the SIPP software framework. The accelerator 1804 can enable some of the most computationally intensive functions to be offloaded from the processing element 1802. The figure shows how multiple filter accelerators 1804 can be connected to the AMC 1804 that performs address filtering, arbitration, and multiplexing. Multiple MIPI camera serial interfaces 2502 can also be connected to the AMC 1804, and in the preferred embodiment, a total of 12 MIPI serial channels are connected in 6 groups of 2 channels. The AMC 1804 is also connected to the AXI and APB interfaces to allow 2 system RISC processors to access the CMX memory via the AMC in the reference embodiment. The final element in the figure is the CMX 412, and the AMC 1804 arbitrates access to the CMX 412, thereby allowing access to the physical RAM instance in the CMX memory 412 by multiple hardware filter accelerators 1804 at the same time. A reference filter accelerator 1804 is also shown, in this case a 5x5 2D filter core, which contains an fp16 (16-bit floating point format similar to IEEE754) arithmetic pipeline, an associated pipeline stall controller, a line buffer read client for storing line inputs to the fp16 pipeline, a line start control input and a line buffer write client for storing outputs from the fp16 pipeline. To allow accelerators to fit within the SIPP framework, they require high bandwidth access to CMX memory, which is provided by the Accelerator Memory Controller (AMC).

[0185] In some embodiments, CMX memory subsystem 412 may be split into 128kB blocks or slices associated with its adjacent processing elements 402 for high-speed, low-power access. Within the slice, the memory is organized into many smaller tiles, such as 3x32kB, 1x16kB, and 2x8KB independent SRAM blocks. The physical RAM size may be selected as a tradeoff between area utilization and configuration flexibility. Any processing element 402 may access physical RAM anywhere in memory subsystem (CMX) 412 with the same latency (3 cycles), but access outside the local processor slice is limited in bandwidth and will consume more power than accessing the local slice. In general, to reduce power consumption and increase performance, processing elements 402 may store data locally in dedicated memory slices.

[0186] In some embodiments, each physical RAM may be 64 bits wide. If more than one processing element 402 attempts to access a single physical RAM, conflicts may occur resulting in a processor stall. CMX will automatically arbitrate between port conflicts and ensure no data is lost. For each port conflict, a processing element 402 is stalled for one cycle resulting in lower throughput. With careful data placement within CMX 412 (by the programmer), port conflicts can be avoided and processor cycles better utilized.

[0187] In some embodiments, multiple processors are configured with accelerators and CMX memory.

[0188] from Fig.25 As can be seen from the image processing HW architecture shown in FIG, each filter accelerator 1804 may include at least one AMC read and / or write client interface to access the CMX memory 412. The number of read and / or write client interfaces on the AMC 1806 is appropriately configured. The AMC 1806 may include a pair of 64-bit ports to each slice of the CMX memory 412. The AMC 1806 routes requests from its clients to the appropriate CMX slice 412 (via partial address decoding). Simultaneous requests to the same slice from different clients may be arbitrated in a round-robin manner. Read data returned from the CMX 412 is routed back to the requesting AMC read client.

[0189] The AMC client (accelerator) 1804 presents the full 32-bit address to the AMC 1806. Accesses from clients not mapped into the CMX memory space are forwarded to the AMC's AXI master. Simultaneous accesses from different clients (outside the CMX memory space) are arbitrated in a round-robin fashion.

[0190] The AMC 1806 is not limited to providing CMX 412 access to filter accelerators 1804; any hardware accelerator or third-party element can use the AMC 1806 to access CMX and the wider platform memory space if its memory interface is properly adapted to the read / write client interface on the AMC.

[0191] The hardware image processing pipeline (SIPP) may include a filter accelerator 1804, an AMC 1806, a MIPI control 2502, APB and AXI interfaces and connections to the CMX multi-port memory 412, and an exemplary hardware 5x5 filter core. This arrangement allows multiple processors 1802 and hardware filter accelerators 1804 for image processing applications to share a memory subsystem 412 consisting of multiple single-port RAM (random access memory) physical blocks.

[0192] Using single-ported memory increases the power and area efficiency of the memory subsystem, but limits bandwidth. The arrangement of the present invention allows these RAM blocks to behave as a virtual multi-port memory subsystem capable of servicing multiple simultaneous read and write requests from multiple sources (processors and hardware blocks) by using multiple physical RAM instances and providing arbitrated access thereto in order to service multiple sources.

[0193] The use of application programming interfaces (APIs) and data partitioning at the application level is important to ensure that the physical RAM blocks used between the processor and the filter accelerators or the connections between the processors are reduced and thereby the data bandwidth delivered to the processor and hardware is increased for a given memory subsystem configuration.

[0194] In some implementations, parallel processing device 400 may reside in an electronic device. Fig.26 An electronic device including a parallel processing device according to some embodiments is shown. The electronic device 2600 may include a processor 2602 , a memory 2604 , one or more interfaces 2606 , and a parallel processor device 400 .

[0195] The electronic device 2600 may have a memory 2604, such as a computer-readable medium, a flash memory, a disk drive, an optical drive, a programmable read-only memory (PROM), and / or a read-only memory (ROM). The electronic device 2600 may be configured with one or more processors 2602 that process instructions and run software that may be stored in the memory 2604. The processor 2602 may also communicate with the memory 2604 and an interface 2606 to communicate with other devices. The processor 2602 may be any applicable processor, such as a system on a chip that combines a CPU, an application processor and flash memory, or a reduced instruction set computing (RISC) processor.

[0196] In some embodiments, the compiler 1208 and the runtime scheduler 1210 may be implemented in software stored in the memory 2604 and operated on the processor 2602. The memory 2604 may be a non-transitory computer readable medium, a flash memory, a disk drive, an optical drive, a programmable read-only memory (PROM), a read-only memory (ROM), or any other memory or combination of memories. The software may run on a processor capable of executing computer instructions or computer code. The processor may also be implemented in hardware using an application specific integrated circuit (ASIC), a programmable logic array (PLA), a field programmable gate array (FPGA), or any other integrated circuit.

[0197] In some implementations, compiler 1208 may be implemented in a separate computing device in conjunction with electronic device 2600 via interface 2060. For example, compiler 1208 may operate in conjunction with electronic device 2600 in a server.

[0198] Interface 2606 may be implemented in hardware or software. Interface 2606 may be used to receive data and control information from a network and a local source, such as a remote control of a television. The electronic device may also provide various user interfaces, such as a keyboard, a touch screen, a trackball, a touch pad, and / or a mouse. The electronic device may also include a speaker and a display device in some embodiments.

[0199] In some embodiments, the processing elements in parallel processing device 400 may include integrated chips capable of executing computer instructions or computer code. The processor may also be implemented in hardware using an application specific integrated circuit (ASIC), a programmable logic array (PLA), a field programmable gate array (FPGA), or any other integrated circuit.

[0200] In some embodiments, the parallel processing device 400 may be implemented as a system on a chip (SOC). In other embodiments, one or more blocks in the parallel processing device may be implemented as separate chips, and the parallel processing device may be packaged in a package system (SIP). In some embodiments, the parallel processing device 400 can be used for data processing applications. The data processing application may include an image processing application and / or a video processing application. The image processing application may include an image processing process (including an image filtering operation), and the video processing application may include a video decoding operation, a video encoding operation, and a video analysis operation for detecting motion or objects in a video. Additional applications of the present invention include machine learning and classification based on sequences of images, objects, or videos and augmented reality applications, including where a game application extracts geometry from multiple camera views (including a depth-enabled camera), and extracts features from multiple views from which wireframe geometry (e.g., via a point cloud) can be extracted for subsequent vertex shading by a GPU.

[0201] Electronic device 2600 may include a mobile device, such as a cellular phone. The mobile device may communicate with multiple radio access networks using multiple access technologies and wired communication networks. The mobile device may be a smart phone that provides enhanced capabilities (such as word processing, web browsing, games, e-book capabilities, operating systems, and full keyboards). The mobile device may run operating systems such as Symbian OS, iPhone OS, RIM's Blackberry, Windows Mobile, Linux, Palm WebOS, and Android. The screen may be a touch screen that may be used to input data to the mobile device, and the screen may be used in place of a full keyboard. The mobile device may have the ability to run applications or communicate with applications, which are provided by a server in a communication network. The mobile device may receive updates or other information from these applications on the network.

[0202] Electronic device 2600 may also encompass many other devices, such as televisions (TVs), video projectors, set-top boxes or set-top units, digital video recorders (DVRs), computers, netbooks, laptops, tablet computers, and any other audio / visual equipment that can communicate with a network. The electronic device may also maintain global positioning coordinates, profile information, or other location information in its stack or memory.

[0203] It will be appreciated that although several different arrangements have been described herein, features of each may be advantageously combined in various forms to achieve advantage.

[0204] In the previously described specification, the present application has been described with reference to specific examples. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention as set forth in the appended claims. For example, a connection may be any type of connection suitable for transmitting a signal from or to a corresponding node, unit or device (e.g., via an intermediate device). Accordingly, unless implied or otherwise indicated, a connection may be, for example, a direct connection or an indirect connection.

[0205] It should be understood that the architectures depicted herein are exemplary only, and that many other architectures that achieve the same functionality may in fact be implemented. In an abstract but still definite sense, any components that are used to achieve the same functionality are arranged to be effectively "associated" so that the desired functionality is achieved. Thus, any two components that are combined in this article to achieve a particular functionality may be considered to be "associated" with each other so that the desired functionality is achieved regardless of the architecture or intermediate components. Similarly, any two components that are so associated may also be considered to be "operably connected" or "operably coupled" to each other so as to achieve the desired functionality.

[0206] In addition, those skilled in the art will recognize that the boundaries between the functionality of the operations described above are illustrative only. The functionality of multiple operations can be combined into a single operation, and / or the functionality of a single operation can be distributed in additional operations. In addition, alternative embodiments can include multiple instances of a particular operation, and the order of the operations can be changed in various other embodiments.

[0207] However, other modifications, changes and alternatives are possible. Accordingly, the specification and drawings are to be construed in an illustrative rather than a restrictive sense.

[0208] In the claims, any reference symbol placed between brackets shall not be understood as a limitation to the claims. The word "comprising" does not exclude the existence of other elements or steps other than those listed in the claims. In addition, the term "a (kind) (a / an)" as used herein is defined as one or more than one. In addition, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be understood to imply that another claim element introduced by the indefinite article "a" or "an" will limit any specific claim containing such introduced claim elements to an invention containing only one such element, even when the same claim includes introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an". This is also applicable to the use of definite articles. Unless otherwise noted, terms such as "first" and "second" are used to arbitrarily distinguish between the elements described by such terms. Thus, such terms are not necessarily intended to indicate the time or other priority of such elements. The indisputable fact that certain measurements are described in mutually different claims does not indicate that the combination of these measurements cannot be used to obtain advantages.

Claims

1. A parallel processing device for accessing a memory, the parallel processing device comprising: a first processing element and a second processing element; Interconnection systems for: coupling a first address encoder to the first processing element; coupling a second address encoder to the second processing element; and routing a first memory access request from the first processing element to the first address encoder and routing a second memory access request from the second processing element to the second address encoder; conflict detector; as well as Centralized arbitration device, which is used to: coupling the first address encoder and the second address encoder to the conflict detector, the conflict detector being associated with a random access memory tile; determining a count of output values ​​from the first address encoder and the second address encoder via the collision detector; and A conflict of the first memory access request and the second memory access request to the random access memory tile is detected based on the count of the output values.

2. The parallel processing device according to claim 1, wherein The first address encoder is used to determine a first target address included in the first memory access request; and The second address encoder is used to determine a second target address included in the second memory access request.

3. The parallel processing device of claim 1 , wherein the output value comprises a first output value and a second output value, the first address encoder being used to generate the first output value when the first target address satisfies a range threshold, and the second address encoder being used to generate the second output value when the second target address satisfies the range threshold. 4 . The parallel processing apparatus of claim 1 , wherein the conflict detector is configured to calculate a count of the output value according to the first address encoder and the second address encoder to identify a conflict with the random access memory tile. 5 . The parallel processing apparatus of claim 1 , wherein the centralized arbitration device is configured to report the conflict when the count of the output values ​​is greater than 1. 6 . 6 . The parallel processing apparatus of claim 1 , wherein the conflict detector is configured to send a stall signal to one of the first processing element or the second processing element in response to detecting a conflict.

7. The parallel processing apparatus according to claim 1, further comprising: a first range comparison block associated with the random access memory tile; a second range comparison block associated with the random access memory tile; as well as An OR gate is communicatively connected to a first output of the first range comparison block and a second output of the second range comparison block, an output of the OR gate being communicatively connected to one of the first processing element and the second processing element to send a stop signal in response to a conflict.

8. The parallel processing device of claim 1 , wherein the centralized arbitration device is used to retrieve client request vectors associated with the first processing element and the second processing element, and the centralized arbitration device is used to determine the first memory access request and the second memory access request based on bit positions of the client request vectors.

9. The parallel processing device of claim 8, wherein the client request vector is associated with the first memory access request and the second memory access request, and at least one of the first memory access request or the second memory access request is a request for accessing one of a first memory slice or a second memory slice.

10. The parallel processing apparatus of claim 9, further comprising a port winner selection block for determining which one of the first memory access request or the second memory access request is to be allocated based on a priority vector.

11. The parallel processing device of claim 9, further comprising a winner detection block for selecting, in response to at least one of the first memory access request or the second memory access request allocated to the first memory slice and the second memory slice, a winner between the first memory slice and the second memory slice associated with a relatively highest input vector ranking to avoid over-allocation of resources.

12. A parallel processing system for accessing a memory, the parallel processing system comprising: a first processing element and a second processing element; means for processing element coupling, for coupling a first address encoder to said first processing element and a second address encoder to said second processing element; means for routing a first memory access request from the first processing element to the first address encoder and a second memory access request from the second processing element to the second address encoder; An apparatus for random access memory element coupling, for: coupling the first address encoder and the second address encoder to a conflict detector, the conflict detector being associated with a random access memory tile; and determining a count of output values ​​from the first address encoder and the second address encoder via the collision detector; as well as Means for detecting a conflict of the first memory access request and the second memory access request to the random access memory tile based on a count of output values.

13. The parallel processing system of claim 12, further comprising: identifying a first target address of the first memory access request according to the first address encoder; and A device for identifying a second target address of the second memory access request according to the second address encoder.

14. The parallel processing system of claim 13, wherein the identification device is configured to: generating a first grant bit in response to detecting that the first target address satisfies a range threshold; and A second grant bit is generated in response to detecting that the second target address satisfies a range threshold.

15. The parallel processing system of claim 14, wherein the identification device is to grant one of the first processing element or the second processing element and the random access memory tile when one of the first target address or the second target address satisfies a range threshold.

16. The parallel processing system of claim 12, wherein the means for detecting a conflict is for calculating a count of output values ​​from the first address encoder and the second address encoder to identify a conflict.

17. The parallel processing system of claim 12, wherein the means for detecting a conflict is for reporting the conflict when the count of output values ​​is greater than 1.

18. The parallel processing system of claim 12, wherein the means for detecting a conflict is for sending a stall signal to one of the first processing element or the second processing element in response to a conflict.

19. A computer readable medium comprising instructions which, when executed, cause one or more processors to at least: coupling a first address encoder to a first processing element; coupling a second address encoder to a second processing element; routing a first memory access request from the first processing element to the first address encoder and routing a second memory access request from the second processing element to the second address encoder; coupling the first address encoder and the second address encoder to a conflict detector, the conflict detector being associated with a random access memory tile; determining a count of output values ​​from the first address encoder and the second address encoder via the collision detector; and A conflict of the first memory access request and the second memory access request to the random access memory tile is detected based on the count of the output values.

20. The computer-readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to: identifying, according to the first address encoder, a first target address included in the first memory access request; and A second target address included in the second memory access request is identified according to the second address encoder.

21. The computer-readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to: generating, by the first address encoder, a first output value in response to detecting a first target address that satisfies a range threshold; and A second output value is generated by the second address encoder in response to detecting a second target address that satisfies a range threshold.

22. The computer-readable medium of claim 21, wherein the instructions, when executed, further cause the one or more processors to grant one of the first processing element or the second processing element and the random access memory tile when one of the first target address or the second target address satisfies a range threshold.

23. The computer readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to identify a conflict associated with the random access memory tile based on a count of output values ​​calculated by the first address encoder and the second address encoder.

24. The computer readable medium of claim 23, wherein the instructions, when executed, further cause the one or more processors to report the conflict when the count of output values ​​is greater than one.

25. The computer-readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to send a stall signal to one of the first processing element or the second processing element in response to detecting a conflict.

26. The computer-readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to: associating a first range comparison block with the random access memory tile; associating a second range comparison block with the random access memory tile; and An OR gate output communicatively connected to a first output of the first range comparison block and a second output of the second range comparison block is calculated to send a stop signal in response to a conflict.

27. The computer-readable medium of claim 19, wherein the instructions, when executed, further cause the one or more processors to retrieve a client request vector associated with the first processing element and the second processing element to determine the first memory access request and the second memory access request based on a bit position of the client request vector.

28. The computer-readable medium of claim 27, wherein the instructions, when executed, further cause the one or more processors to associate the client request vector with the first memory access request and the second memory access request, and at least one of the first memory access request or the second memory access request is a request to access one of a first memory slice or a second memory slice.

29. The computer-readable medium of claim 28, wherein the instructions, when executed, further cause the one or more processors to determine which of the first memory access request or the second memory access request to allocate based on a priority vector.

30. The computer-readable medium of claim 28, wherein the instructions, when executed, further cause the one or more processors to select a winner between the first memory slice and the second memory slice associated with a relatively highest input vector ranking to avoid over-allocation of resources in response to at least one of the first memory access request or the second memory access request allocated to the first memory slice and the second memory slice.

31. An integrated circuit for accessing a memory, the integrated circuit comprising: a first processing element having a first delay; a second processing element having a second delay; a first address encoder; A second address encoder; an interconnect system for: routing a first memory access request from the first processing element to the first address encoder, and routing a second memory access request from the second processing element to the second address encoder; a first conflict detector; a second conflict detector; as well as A distributed arbitration circuit for: coupling a first output of the first address encoder to the first collision detector; coupling a second output of the second address encoder to the second collision detector; determining a count of output values ​​in parallel based on the first output and the second output; and A conflict of the first memory access request and the second memory access request is detected based on the count of the output values.

32. An integrated circuit as claimed in claim 31, wherein The first address encoder is used to determine a first target address included in the first memory access request; and The second address encoder is used to determine a second target address included in the second memory access request.

33. The integrated circuit of claim 31 , wherein The first address encoder is configured to generate a first output value when the first target address satisfies a range threshold; and The second address encoder is configured to generate a second output value when the second target address satisfies a range threshold.

34. The integrated circuit of claim 31 , wherein to determine the count of output values: The first collision detector is used to determine a first count of output values ​​based on the first address encoder; and The second collision detector is used to determine a second count of output values ​​based on the second address encoder.

35. The integrated circuit of claim 31, wherein the distributed arbitration circuit is configured to report the conflict when the count of the output values ​​is greater than one.

36. The integrated circuit of claim 31, wherein the distributed arbitration circuit is to send a stall signal to a requestor in response to detecting a conflict.

37. The integrated circuit of claim 31 further comprising: a first range comparison block associated with a random access memory tile; a second range comparison block associated with the random access memory tile; as well as An OR gate is communicatively connected to a first output of the first range comparison block and a second output of the second range comparison block, an output of the OR gate being communicatively connected to one of the first processing element or the second processing element to send a stop signal in response to a conflict.

38. The integrated circuit of claim 31 , wherein the distributed arbitration circuit is used to retrieve a client request vector associated with the first processing element and the second processing element, the distributed arbitration circuit being used to determine the first memory access request and the second memory access request based on a bit position of the client request vector.

39. The integrated circuit of claim 38, wherein the client request vector is associated with the first memory access request and the second memory access request, and at least one of the first memory access request or the second memory access request is a request to access one of a first memory slice or a second memory slice.

40. The integrated circuit of claim 39 further comprises a winner detector for selecting, in response to at least one of the first memory access request or the second memory access request allocated to the first memory slice and the second memory slice, a winner in the first memory slice or the second memory slice associated with a relatively highest input vector ranking to avoid over-allocation of resources.

41. A computer readable medium comprising instructions which, when executed, cause one or more processors to at least: routing a first memory access request from a first processing element having a first latency to a first address encoder; routing a second memory access request from a second processing element having a second latency to a second address encoder; (A) coupling a first output of the first address encoder to a first collision detector and (B) coupling a second output of the second address encoder to a second collision detector; determining a count of output values ​​in parallel based on the first output and the second output; and A conflict of the first memory access request and the second memory access request is detected based on the count of the output values.

42. The computer readable medium of claim 41, wherein the instructions, when executed, further cause one or more processors to: identifying a first target address included in the first memory access request; and A second target address included in the second memory access request is identified.

43. The computer readable medium of claim 41, wherein the instructions, when executed, further cause one or more processors to: generating a first output value when the first target address satisfies a range threshold; and A second output value is generated when the second target address satisfies the range threshold.

44. The computer readable medium of claim 41, wherein to determine the count of output values, the instructions when executed further cause one or more processors to: determining a first count of output values ​​according to the first address encoder; and A second count of output values ​​is determined based on the second address encoder.

45. The computer readable medium of claim 44, wherein the instructions, when executed, further cause one or more processors to report the conflict when the count of output values ​​is greater than one.

46. ​​The computer-readable medium of claim 41, wherein the instructions, when executed, further cause one or more processors to send a stall signal to the requestor in response to detecting a conflict.

47. The computer readable medium of claim 41, wherein the instructions, when executed, further cause one or more processors to: associating the first range comparison block with the random access memory tile; associating a second range comparison block with the random access memory tile; and An OR gate communicatively connected to the first output of the first range comparison block and the second output of the second range comparison block is calculated to send a stop signal in response to a conflict.

48. The computer-readable medium of claim 41, wherein the instructions, when executed, further cause one or more processors to retrieve a client request vector associated with the first processing element and the second processing element to determine the first memory access request and the second memory access request based on a bit position of the client request vector.

49. The computer-readable medium of claim 48, wherein the instructions, when executed, further cause one or more processors to associate the client request vector with the first memory access request and the second memory access request, and at least one of the first memory access request and the second memory access request is a request to access one of a first memory slice or a second memory slice.

50. A method for accessing a memory, the method comprising: routing a first memory access request from a first processing element having a first latency to a first address encoder; routing a second memory access request from a second processing element having a second latency to a second address encoder; (A) coupling a first output of the first address encoder to a first collision detector and (B) coupling a second output of the second address encoder to a second collision detector; determining a count of output values ​​in parallel based on the first output and the second output; and A conflict of the first memory access request and the second memory access request is detected based on the count of the output values.

51. An integrated circuit for accessing a memory, the integrated circuit comprising: means for encoding a first address; means for encoding a second address; means for routing a first memory access request from a first processing element having a first latency to a device for encoding the first address and routing a second memory access request from a second processing element having a second latency to a device for encoding a second address; means for detecting a first conflict; means for detecting a second conflict; as well as Distributed devices for arbitration, used to: coupling a first output of a device for encoding the first address to a device for detecting the first collision; coupling a second output of the device for encoding the second address to the device for detecting the second collision; determining a count of output values ​​in parallel based on the first output and the second output; and A conflict of the first memory access request and the second memory access request is detected based on the count of the output values.

52. An integrated circuit as claimed in claim 51, wherein means for encoding said first address for determining a first target address included in said first memory access request; and The device for encoding the second address is used to determine a second target address included in the second memory access request.

53. An integrated circuit as claimed in claim 51, wherein means for encoding said first address for generating a first output value when the first target address satisfies a range threshold; and The device for encoding the second address is configured to generate a second output value when the second target address satisfies a range threshold.

54. The integrated circuit of claim 51, wherein to determine the count of output values: means for detecting the first collision for determining a first count of output values ​​from the means for encoding the first address; and The means for detecting the second collision is for determining a second count of output values ​​in accordance with the means for encoding the second address.

55. The integrated circuit of claim 54, wherein the distributed means for arbitrating is to report the conflict when the count of the output values ​​is greater than one.

56. The integrated circuit of claim 51, wherein the distributed device for arbitrating is to send a stall signal to the requestor in response to detecting a conflict.

57. The integrated circuit of claim 51 further comprising: means for comparing a first target address to a first address range, the means for comparing said first target address being associated with a random access memory tile; means for comparing a second target address to a second address range, the means for comparing the second target address being associated with the random access memory tile; as well as A device for generating a stop signal to send the stop signal in response to the conflict, the device for generating the stop signal is communicatively connected to a first output of the device for comparing the first target address and a second output of the device for comparing the second target address, and the output of the device for generating the stop signal is communicatively connected to one or more of the first processing element and the second processing element.

58. An integrated circuit as described in claim 51, wherein the distributed device for arbitration is used to retrieve a client request vector associated with the first processing element and the second processing element, and the distributed device for arbitration is used to determine at least one of the first memory access request or the second memory access request based on the bit position of the client request vector.

59. The integrated circuit of claim 58, wherein the client request vector is associated with the first memory access request and the second memory access request, and at least one of the first memory access request or the second memory access request is a request to access one of a first memory slice or a second memory slice.

60. The integrated circuit of claim 59, further comprising a device for detecting a winner, for selecting a winner memory slice of the first memory slice or the second memory slice associated with a higher input vector ranking in response to at least one of the first memory access request or the second memory access request allocated to the first memory slice and the second memory slice, thereby avoiding over-allocation of resources.

Images