A coarse-grained reconfigurable processor based on the ISP pipeline

Abstract

The present invention provides a coarse-grained reconfigurable processor based on an ISP pipeline. The processor includes: an ISP interaction interface unit, an execution control unit CU, a configuration management unit MU, a direct memory access unit DMA, an execution unit array, an array memory access unit, and a block memory. The present invention adds an on-line input / output interface to a typical ISP pipeline, transmits image data into the CGRA processor through the output interface, the CGRA executes the ISP software algorithm, and feeds back the calculation result to the inside of the ISP on-line through the ISP input interface, improving the image data processing speed and realizing real-time image processing of the ISP pipeline.

Description

Technical Field

[0001] The present invention belongs to the field of technology processors, and particularly relates to a coarse-grained reconfigurable processor and an electronic device based on an ISP (Image Signal Processing) pipeline. Background Art

[0002] Image sensors are widely used in terminal imaging devices, such as cameras, mobile phones, surveillance cameras, and automotive vision sensors. An image sensor is used to obtain the spatial variation of light intensity, and then uses image processing algorithms to reconstruct a color picture based on the data provided by the image sensor. A typical ISP algorithm pipeline includes deadpixel correction, black level compensation, lensshading correction, CFA interpolation, color correction matrix, AWB gain control, etc. In early terminal devices, the ISP algorithm pipeline could be implemented by a digital signal processor (DSP) or an application-specific integrated circuit (ASIC). However, as the requirements for imaging quality in devices such as mobile phones and surveillance increase, the DSP cannot provide the computing power required for real-time ISP processing. Modern terminal devices all adopt the ASIC solution, and the DSP is used to assist in implementing some pre-processing and post-processing operations of the ISP. However, the pre-processing and post-processing operations performed by the DSP are usually offline calculations because the DSP is not a real-time device and cannot interact with the ISP in real time online. Summary of the Invention

[0003] In order to solve the above problems existing in the prior art, the present invention provides a coarse-grained reconfigurable processor and an electronic device based on an ISP pipeline.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions.

[0005] The present invention provides a coarse-grained reconfigurable processor based on an ISP pipeline, including: an ISP interaction interface unit, an execution control unit CU (CU, Controller Unit), a configuration management unit MU (MU, Management Unit), a direct memory access unit DMA (DMA, Direct Memory Access), an execution unit array (PEA, Processing Element Array), an array memory access unit (LSU, Load / Store Unit), and a block memory;

[0006] The execution control unit CU is connected to and can control the configuration management unit MU and the ISP interaction interface unit; there is a row buffer array composed of multiple memories in the ISP interaction interface unit; the ISP interaction interface unit can interface with the ISP algorithm module.

[0007] The configuration management unit MU is connected to the execution unit array; the configuration management unit MU can call the direct memory access unit DMA; the configuration management unit MU can transfer the data stored in the ISP interaction interface unit to the memory of the execution unit array through DMA.

[0008] The execution unit array reads or stores processing data from / to the execution unit array memory access unit; the execution unit array memory access unit is connected to the block memory and can read or store processing data from / to the block memory;

[0009] The direct memory access unit DMA is connected to the block memory and can read the data to be processed from the block memory; the direct memory access unit DMA can store the data in the block memory.

[0010] In an embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the execution control unit CU is configured to be able to call and read the ISP interaction interface unit, close the functions of some algorithm modules of the ISP, and open the data output interfaces of the corresponding modules;

[0011] The execution control unit CU polls the ISP interaction interface unit. If there is data ready, the configuration management unit MU starts the direct memory access unit DMA to transfer the image data of the ISP stored in the ISP interaction interface unit from the row storage DVP_OUT line buffer to the block memory;

[0012] The execution control unit CU notifies the configuration management unit MU to configure the execution unit array and the array memory access unit, so that the execution unit array reads the data in the block memory through the execution unit array memory access unit, and after the calculation of the array, writes the calculation result back to the block memory through the execution unit array memory access unit;

[0013] If the execution unit array has completed its calculations, the execution control unit CU notifies the direct memory access unit DMA to transfer the data from the block memory to the ISP interface unit, so that the image data can return to the ISP pipeline for further calculations.

[0014] In one embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the ISP interface unit includes an image line buffer DVP_OUT line Buffer of the digital image interface; the execution control unit CU polls the image line buffer DVP_OUT line Buffer of the digital image interface to check if a line of data is ready.

[0015] In one embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the ISP interface unit includes a status register; the status register can feedback the number of image lines in the line buffer; it is determined whether there is data ready by the number of image lines in the line buffer.

[0016] In one embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, if the execution unit array has completed its calculations, the execution unit array reports the calculation completion information to the configuration management unit MU; the configuration management unit MU notifies the control unit CU, and the execution control unit CU notifies the direct memory access unit DMA to transfer the data from the block memory to the ISP interface unit.

[0017] In one embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, it is characterized in that: the execution unit array includes various types of heterogeneous operators connected by a fully interconnected network; the various types of heterogeneous operators include a branch operator SEL, a numerical operation operator ALU, an enhanced numerical operation operator EALU, a multiply-accumulate operator MAC, a special operator SPU, a buffer operator BUFFER, and a loop control operator LOOP; the fully interconnected network includes a Benes network, a Clos network, or a Crossbar network.

[0018] In one embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the fully interconnected network is an enhanced Benes network, including a first sub-Benes network and a second sub-Benes network both with N / 2 input and output terminals, and N switching units; the switching unit is provided with a first input terminal, a second input terminal, a first output terminal, and a second output terminal, and has four working modes: direct connection, switching, upper broadcast, and lower broadcast; where

[0019] N / 2 switching units are connected between the input ends In0 to In(N - 1) of the enhanced Benes network and the input ends of the first sub-Benes network and the second sub-Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network;

[0020] Another N / 2 switching units are connected between the output ends of the first sub-Benes network and the second sub-Benes network and the output ends Out0 to Out(N - 1) of the enhanced Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network.

[0021] In an implementation of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the execution unit array further includes a computing unit set Cluster, and the computing unit set Cluster includes one or more types of heterogeneous operators connected by an enhanced Benes network.

[0022] In an implementation of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the computing unit set Cluster includes 24 enhanced arithmetic logic units EALU, 48 arithmetic logic units ALU, 12 branch operators SEL, 2 loop control operators LOOP, and 48 buffer operators BUFFER connected by an enhanced Benes network.

[0023] In an implementation of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the execution unit array includes 4 computing unit sets Cluster, 50 multiply-accumulate operators MAC, 10 special operators SPU, 64 buffer operators BUFFER, and 6 loop control operators LOOP connected by an enhanced Benes network.

[0024] Compared with the prior art, the present invention has the following beneficial effects.

[0025] A coarse-grained reconfigurable processor based on the ISP pipeline proposed by the present invention can embed the processor into the ISP pipeline by configuring an ISP interaction interface unit, an execution control unit CU, a configuration management unit MU, a direct memory access unit DMA, an execution unit array, an array memory access unit, and a block memory. The present invention improves the image data processing speed significantly by setting a direct memory access unit DMA and an execution unit array for accelerating the execution of data stream codes, and realizes real-time image processing of the ISP pipeline by setting a configuration management unit MU for effective management and scheduling. Description of the Drawings

[0026] Figure 1 This is a block diagram of a coarse-grained reconfigurable processor based on the ISP pipeline according to an embodiment of the present invention.

[0027] Figure 2 This is a block diagram of a coarse-grained reconfigurable processor based on the ISP pipeline according to another embodiment of the present invention.

[0028] Figure 3 This is a block diagram of various types of heterogeneous operators connected by a fully interconnected network according to an embodiment of the present invention.

[0029] Figure 4 This is a schematic diagram of an enhanced Benes network structure.

[0030] Figure 5 This is a schematic diagram of the CGRA structure according to another embodiment of the present invention. Detailed Embodiment

[0031] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0032] The present invention provides a coarse-grained reconfigurable processor based on the ISP pipeline, as Figure 1 shown, which includes: an ISP interaction interface unit 101, an execution control unit CU 201, a configuration management unit MU 301, a direct memory access unit DMA 401, an execution unit array 501, an array memory access unit 601, and a block memory 701.

[0033] The execution control unit CU 201 is connected to and can control the configuration management unit MU 301 and the ISP interaction interface unit 101. There is a row buffer array composed of multiple memories in the ISP interaction interface unit 101. The ISP interaction interface unit can interface with an external ISP algorithm module.

[0034] The configuration management unit MU 301 is connected to the execution unit array 501. The configuration management unit MU 301 can call the direct memory access unit DMA 401. The configuration management unit MU 301 can transfer the data stored in the ISP interaction interface unit to the memory of the execution unit array 501 through the DMA 401.

[0035] The execution unit array 501 reads or stores processing data from / to the memory access unit of the execution unit array 501. The memory access unit of the execution unit array 501 is connected to the block memory 701 and is capable of reading or storing processing data from / to the block memory 701.

[0036] The direct memory access unit DMA401 is connected to the block memory 701 and is capable of reading the data to be processed from the block memory 701. The direct memory access unit DMA401 is capable of storing data into the block memory 701.

[0037] In this embodiment, by setting up the direct memory access unit DMA and the block memory 701 for acceleration, and by setting up the configuration management unit MU to effectively manage and schedule the direct memory access unit DMA and the block memory 701, the image data processing speed can be significantly improved, and real-time image processing of the ISP pipeline can be achieved.

[0038] In an embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline of the present invention, the execution control unit CU201 is configured to be able to call and read the ISP interaction interface unit 101, turn off some algorithm module functions of the ISP, and open the data output interfaces of the corresponding modules.

[0039] The execution control unit CU201 polls the ISP interaction interface unit 101. If there is data ready, the configuration management unit MU 301 starts the direct memory access unit DMA401 to move the image data of the ISP stored in the ISP interaction interface unit 101 from the row storage DVP_OUT linebuffer to the block memory 701.

[0040] The execution control unit CU201 notifies the configuration management unit MU 301 to configure the execution unit array 501 and the array memory access unit 601, so that the execution unit array 501 reads the data in the block memory 701 through the memory access unit of the execution unit array 501, and after the calculation of the array, writes the calculation result back to the block memory 701 through the memory access unit of the execution unit array 501.

[0041] If the execution unit array 501 has completed the calculation, the execution control unit CU201 notifies the direct memory access unit DMA401 to move the data from the block memory 701 to the ISP interaction interface unit 101, so that the image data returns to the ISP pipeline for continued calculation.

[0042] In an implementation of the ISP pipeline-based coarse-grained reconfigurable processor of the present invention, the ISP interface unit 101 includes an image line buffer DVP_OUT line Buffer of the digital image interface. The execution control unit CU201 polls the image line buffer DVP_OUT line Buffer of the digital image interface to check whether a line of data is ready.

[0043] In an implementation of the ISP pipeline-based coarse-grained reconfigurable processor of the present invention, the ISP interface unit 101 includes a status register. The status register can feedback the number of image lines in the line buffer. Whether data is ready is judged by the number of image lines in the line buffer.

[0044] In an implementation of the ISP pipeline-based coarse-grained reconfigurable processor of the present invention, if the execution unit array 501 finishes the calculation, the execution unit array 501 reports the calculation completion information to the configuration management unit MU 301. The configuration management unit MU 301 notifies the control unit CU, and the execution control unit CU201 notifies the direct memory access unit DMA401 to move the data from the block memory 701 to the ISP interface unit 101.

[0045] In an implementation of the ISP pipeline-based coarse-grained reconfigurable processor of the present invention, it is characterized in that: the execution unit array 501 includes various types of heterogeneous operators connected by a fully interconnected network. The various types of heterogeneous operators include a branch operator SEL, a numerical operation operator ALU, an enhanced numerical operation operator EALU, a multiply-accumulate operator MAC, a special operator SPU, a cache operator BUFFER, and a loop control operator LOOP. The fully interconnected network includes a Benes network, a Clos network, or a Crossbar network.

[0046] In an implementation of the ISP pipeline-based coarse-grained reconfigurable processor of the present invention, the fully interconnected network is an enhanced Benes network, including a first sub-Benes network and a second sub-Benes network with both the number of input and output ports being N / 2, and N switching units. The switching unit is provided with a first input end, a second input end, a first output end, and a second output end, and has four working modes: direct connection, switching, upper broadcast, and lower broadcast. Among them,

[0047] N / 2 switching units are connected between the input ends In0-In(N-1) of the enhanced Benes network and the input ends of the first sub-Benes network and the second sub-Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network.

[0048] Another N / 2 switching units are connected between the output ends of the first sub-Benes network and the second sub-Benes network and the output ends Out0 to Out(N-1) of the enhanced Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network.

[0049] In an embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline according to the present invention, the execution unit array 501 further includes a computing unit set Cluster, and the computing unit set Cluster includes one or more types of heterogeneous operators connected by an enhanced Benes network.

[0050] In an embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline according to the present invention, the computing unit set Cluster includes 24 enhanced arithmetic logic units (EALUs), 48 arithmetic logic units (ALUs), 12 branch operators (SELs), 2 loop control operators (LOOPs), and 48 buffer operators (BUFFERs) connected by an enhanced Benes network.

[0051] In an embodiment of the coarse-grained reconfigurable processor based on the ISP pipeline according to the present invention, the execution unit array 501 includes 4 computing unit sets Cluster, 50 multiply-accumulate operators (MACs), 10 special operators (SPUs), 64 buffer operators (BUFFERs), and 6 loop control operators (LOOPs) connected by an enhanced Benes network.

[0052] The present invention proposes a design and implementation of a coarse-grained reconfigurable architecture processor (CGRAProcessor) based on an ISP pipeline. An on-line input-output interface is added to a typical ISP pipeline, and image data is transmitted into the CGRA processor through the output interface. The CGRA executes the ISP software algorithm, and the calculation result is fed back into the ISP online through the ISP input interface.

[0053] Such as Figure 2As shown, this is an implementation of a coarse-grained reconfigurable processor based on the ISP pipeline in the present invention. Among them, the ISP pipeline is a typical ISP pipeline that supports typical ISP algorithms, the pipeline interface is an interface through which external data can be accessed between various ISP algorithm modules, and a typical ISP pipeline can support 1 to multiple Pipeline interfaces. The CGRA is a coarse-grained reconfigurable processor that can be embedded in the ISP pipeline, including CU, MU, PEA, LSU, DMA, and ISP_interface modules. Their functions are as follows:

[0054] CU (Controller Unit, execution control unit) is a general-purpose CPU Core. The execution tasks of the CU include three parts: the non-loop part of the program, the CSPM space management in the CGRA, and the space management of the DSPM in the CGRA. The typical usage process of the CU includes an instruction stream and a data stream, and in the typical usage process, it is stored in the ITCM (Instruction Tightly Coupled Memory, tightly coupled instruction cache) and DTCM (Data Tightly Coupled Memory, tightly coupled data cache):

[0055] Among them, the ITCM is mapped to the system address space, and the actual stored content is determined by the compiler. The DTCM is mapped to the system address space, and the actual stored content is jointly managed by the compiler and the programmer. The stack area is the responsibility of the programmer, and other parts are managed by the compiler.

[0056] MU (Management Unit, configuration management unit), used to manage and schedule acceleration resources such as PEA / DMA, mainly for task distribution and task synchronization of acceleration resources. The typical usage process of the MU uses CSPM (Context Scatch-pad-memory, configuration information storage) and PB (Parameter Buffer, parameter cache). The main functions of the MU include: commands issued by the CU will be queued and synchronized through the MU. The configuration information of the PEA and LSU is in the CSPM inside the MU, and when the MU receives the execution command of the PEA, it will read the content of the CSPM and configure it into the PEA and LSU.

[0057] musr: MU control and status register, MU-Status-Register. The CU controls the MU to complete certain functions by operating the MUSR, and views the status of the MU through the MUSR.

[0058] multi-cmdqueue: The multiple command queue consists of three command queues, which respectively correspond to the control commands of three functions: RDMA, PEA, and WDMA. The CU writes commands to the MU, and the MU adds them to the three queues of RMDA, PEA, and WDMA according to the type of the commands. The commands in the same queue are executed strictly in the queue order, and the commands in different queues can be executed in parallel.

[0059] CSPM is a section of the PEA / LSU Context buffer (the PEA / LSU Context are the configuration files required for the execution of PEA / LSU, namely "operator configuration", "network configuration", and "read-only parameters", and the MU can configure the PEA / LSU Context into the PEA / LSU through a group of high-width interfaces). PB is a section of real-time parameter space, and the CU transmits the "variable parameters" required for the TASK execution to the PEA / LSU array through PB / MU for execution.

[0060] PEA (Processing Element Array, execution unit array) is mainly responsible for accelerating the execution of data flow code. The PEA mainly includes core component units such as heterogeneous PEs and NETWORK, and the specific functions are as follows:

[0061] The Network network is a fully interconnected network that connects all PEs together. The fully interconnected network is a Benes network with enhanced functions. By configuring different configuration information, Network, LSU, and PE execute data flow calculations together.

[0062] LSU (Load / Store Unit, memory access unit), the LSU has M read / write ports connected to the PEA. Each read / write port calculates the internal cache space address according to the configuration mode (including: type of memory access [read / write], bit width of memory access [8 / 16 / 32bit], memory access mode [immediate addressing / register addressing / base relative addressing]), reads the corresponding data from the cache, and returns the data to the computing array. The characteristics that the LSU unit needs to possess are as follows:

[0063] The LSU unit can receive at most M memory access requests (read or write) from the computing unit simultaneously. It supports multiple addressing modes, including: immediate addressing Load / Store[Offset], where Offset comes from the configuration information of Load / Store; register addressing Load / Store[Base], where Base comes from the computing array; base relative addressing Load / Store[Base+Offset], where Base comes from the computing array and Offset comes from the configuration information of Load / Store.

[0064] Support memory access conflict detection and arbitration. Assume that the LSU unit initiates M requests to access N SPM - Banks simultaneously. LD / ST needs to add the M requests to the N memory access queues and complete the conflicting requests in sequence. After all M requests are completed, the results are returned to the computing array simultaneously. During the sequential execution of memory access by the LSU, the LSU needs to feedback the BANK conflict signal to the PEA, and the PEA pauses the execution of the entire array to ensure that the LSU and the PEA are always executed synchronously.

[0065] Support data format conversion. When the data in the DSPM is stored as an array with a granularity of byte or half - word, the LSU needs to convert the byte or half - word into a Word and send it to the PEA for calculation.

[0066] DMA (Direct Memory Access) is used for the CGRA to implement data exchange between internal caches (DSPM / ITCM / DTCM / CSPM) and external Memory, as well as between internal caches (DSPM←→DTCM, DSPM←→DSPM, lineBuffer←→DSPM).

[0067] The ISP interface is a multi - group interface for interacting with the ISP, usually the DVP interface (Digital Video Port). Inside, there is a Linebuffer composed of SRAM to cache the data input from or output to the ISP pipeline for the image row buffer of the image.

[0068] In an embodiment of the present invention, a processing method for a coarse - grained reconfigurable processor based on an ISP pipeline includes:

[0069] Step S101, the CU configures the ISP, turns off some algorithm module functions of the ISP, and opens the data output interfaces of the corresponding modules.

[0070] Step S102, the CU polls the image row buffer of the DVP_OUT line Buffer digital image interface of the ISP to check if there is a row of data ready. If there is a row of data ready, the mu starts the DMA to move the image data of the ISP from the DVP_OUT line buffer with status register to the Multiple Bank SRAM and can feedback the number of image rows cached in the line buffer.

[0071] In step S103, the CU notifies the MU to configure the PEA / LSU to process the data in the Multiple-bank SRAM. The PEA reads the data in the multi-bank SRAM through the LSU, performs calculations through the array, and writes the calculation results back to the multi-bank SRAM through the LSU. The configured information includes: 00: the operator function of each PE in the PEA, the interconnection information of the Benes network, and the operator function in the LSU.

[0072] In step S104, after the PEA calculation is completed, the CU notifies the DMA to move the data from the Multiple-Bank SRAM to the DVP_INline buffer, and the image data returns to the ISP pipeline for continued calculation.

[0073] Figure 3 This is a block diagram of various types of heterogeneous operators connected by a fully connected network according to an embodiment of the present invention. The coarse-grained reconfigurable array includes various types of heterogeneous operators 2 connected by a fully connected network 1.

[0074] This embodiment proposes a coarse-grained reconfigurable array CGRA, which is mainly composed of various types of heterogeneous operators 2. Its feature is that the various types of heterogeneous operators 2 are connected by a fully connected network 1. The heterogeneous operator 2 generally refers to numerical operation operators such as an arithmetic logic unit (ALU) and a multiply-accumulate unit (MAC). The fully connected network 1 can connect any one of its input terminals to any one of its output terminals, thereby realizing the interconnection between any two heterogeneous operators 2. Existing CGRAs are generally composed of homogeneous operators, and the homogeneous operators are mostly connected by a mesh structure. The port sharing of homogeneous operators (PEs) can introduce logical waste within the operator. For example, a PE has the capabilities of all operators such as an ALU, an enhanced arithmetic logic unit (EALU), a loop control (LOOP), a MAC, and a special unit (SPU). When the PE performs an ALU operation, it cannot perform other operations such as EALU and LOOP, resulting in waste of internal resources of the PE. To eliminate the internal resource waste caused by homogeneous operators, this embodiment uses heterogeneous operators 2. Since there are not too many duplicate complex functions inside each operator, a large amount of internal logic waste will not be caused. Connecting each operator by a mesh structure is another important reason for the resource waste problem existing in existing CGRAs.

[0075] Mapping the data flow graph of an algorithm onto a CGRA not only requires establishing a corresponding relationship between the operations of the data flow graph and the PEs, but also requires establishing routing channels on the MESH graph for the data transfer relationships in the data flow graph. Often, due to the constraints of MESH resources, many PEs cannot be used to map the operations of the data flow graph. To eliminate the resource waste caused by the operator connection structure, this embodiment uses a fully connected network to connect each operator. Since a fully connected network can achieve the interconnection between any two operators, there is no situation where the operations in the data flow graph cannot be mapped to the operators due to insufficient interconnection resources.

[0076] As an optional embodiment, the array includes six heterogeneous operators 2, namely the branch operator SEL, the numerical operation operator ALU, the enhanced numerical operation operator EALU, the multiply-accumulate operator MAC, the special operator SPU, the buffer operator BUFFER, and the loop control operator LOOP.

[0077] This embodiment gives the types of the heterogeneous operators 2. The CGRA array of this embodiment includes six heterogeneous operators 2, namely SEL, ALU, EALU, MAC, SPU, BUFFER, and LOOP. Each operator completes different functions. For example, MAC can implement multiply-add, multiply-accumulate, multiplication, and accumulation operations. The functions of the six heterogeneous operators 2 are shown in Table 1.

[0078] Table 1 Functional description of heterogeneous operators

[0079]

[0080] As an optional embodiment, the fully connected network 1 includes a Benes network or a Clos network or a Crossbar network.

[0081] This embodiment gives several network structures of the fully connected network 1. Benes, Clos, and Crossbar are three common network structures, and the fully connected network 1 of this embodiment can be composed of any one of them. The Benes network is a switching network that can achieve the interconnection between any two points. It was first proposed by Bell Labs for use in telecommunications systems and is still widely used in current switch and router networks. Crossbar is also a common point-to-point network. The interconnection cost introduced to achieve point-to-point interconnection is N 2 . While the cost of Benes is only N*Log2N. The technical principles of these three network structures belong to mature existing technologies and will not be introduced in detail separately here.

[0082] As an alternative embodiment, the fully interconnected network 1 is an enhanced Benes network, including a first sub-Benes network and a second sub-Benes network both with N / 2 input / output ports and N switching units. Each switching unit is provided with a first input end, a second input end, a first output end and a second output end, and has four operating modes: straight-through, switching, up-broadcasting and down-broadcasting. Among them,

[0083] N / 2 switching units are connected between the input ends In0~In(N - 1) of the enhanced Benes network and the input ends of the first sub-Benes network and the second sub-Benes network. The first output ends of the first N / 4 switching units are respectively connected to the first N / 4 input ends of the first sub-Benes network, and the second output ends of the first N / 4 switching units are respectively connected to the first N / 4 input ends of the second sub-Benes network. The first output ends of the last N / 4 switching units are respectively connected to the last N / 4 input ends of the first sub-Benes network, and the second output ends of the last N / 4 switching units are respectively connected to the last N / 4 input ends of the second sub-Benes network.

[0084] The other N / 2 switching units are connected between the output ends of the first sub-Benes network and the second sub-Benes network and the output ends Out0~Out(N - 1) of the enhanced sub-Benes network. The first input ends of the first N / 4 switching units are respectively connected to the first N / 4 output ends of the first sub-Benes network, and the second input ends of the first N / 4 switching units are respectively connected to the first N / 4 output ends of the second sub-Benes network. The first input ends of the last N / 4 switching units are respectively connected to the last N / 4 output ends of the first sub-Benes network, and the second input ends of the last N / 4 switching units are respectively connected to the last N / 4 output ends of the second sub-Benes network.

[0085] This embodiment provides a specific fully interconnected network 1. The fully interconnected network 1 in this embodiment adopts an enhanced Benes network structure, as Figure 4 shown. The Benes network is composed of 2×2 (2 inputs, 2 outputs) switching units. The construction method of an N×N (N inputs, N outputs) Benes network is as follows: there are N / 2 switching units on each side, a total of N switching units, and in the middle are two N / 2×N / 2 Benes networks (i.e., the first sub-Benes network and the second sub-Benes network), and the specific connection method is also as Figure 4As shown in the figure. Then, the middle two Benes networks are continuously decomposed in the above method until there is only one switching unit in the middle sub-network. The difference between the enhanced Benes network of this embodiment and the general Benes network is the switching unit that makes up the network: the switching unit of the general Benes network only has two working modes: direct connection and switching, while the switching unit of the enhanced Benes network has four working modes: direct connection, switching, upper broadcast, and lower broadcast. Therefore, the enhanced Benes network can not only connect any input end of the network to any output end, but also connect one input end to multiple output ends at the same time. Its function is enhanced compared with the general Benes network, so it is called the enhanced Benes network.

[0086] As an alternative embodiment, the array further includes a set of computing units Cluster, and the set of computing units Cluster includes one or more types of heterogeneous operators 2 connected by an enhanced Benes network.

[0087] This embodiment presents another technical solution of the array. In this embodiment, the array not only includes heterogeneous operators 2, but also includes a set of computing units Cluster composed of one or more types of heterogeneous operators 2. Since the interconnection cost in the fully interconnected network structure rises sharply with the increase in the number of resources, such as the enhanced Benes network grows as N*Log2N, this embodiment reduces the number of heterogeneous operators 2 by forming a cluster of one or more types of heterogeneous operators 2, thereby reducing the interconnection cost. In order to reduce resource waste, the heterogeneous operators 2 that make up the Cluster are still connected by the fully interconnected network 1, that is, the enhanced Benes network.

[0088] As an alternative embodiment, the set of computing units Cluster includes 24 enhanced arithmetic logic units EALU, 48 arithmetic logic units ALU, 12 branch operators SEL, 2 loop control operators LOOP, and 48 buffer operators BUFFER connected by an enhanced Benes network.

[0089] This embodiment presents a specific Cluster structure. The Cluster of this embodiment is composed of 5 types of heterogeneous operators, namely 24 EALU, 48 ALU, 12 SEL, 2 LOOP, and 48 BUFFER. These heterogeneous operators are connected by an enhanced Benes network. Since so many heterogeneous operators form a Cluster and are connected to other operators as one operator, the interconnection cost can be reduced.

[0090] As an alternative embodiment, the number of input and output ends of the enhanced Benes network is N = 256.

[0091] This embodiment is an alternative embodiment of the previous embodiment, which further defines the enhanced Benes network in the specific Cluster of the previous embodiment, and the number of input and output ports is N = 256 for both.

[0092] As an alternative embodiment, the array includes 4 sets of computing units Cluster connected by an enhanced Benes network, 50 multiply-accumulate operators MAC, 10 special operators SPU, 64 buffer operators BUFFER, and 6 loop control operators LOOP.

[0093] This embodiment gives a specific structure of the array. In addition to including various types of heterogeneous operators 2 (50 MACs, 10 SPUs, 64 LOOPS), the array of this embodiment also includes 4 Clusters. These heterogeneous operators 2 and Clusters are connected together by an enhanced Benes network, and its schematic diagram is as Figure 5 shown.

[0094] As an alternative embodiment, the number of input and output ports of the enhanced Benes network is N = 512 for both.

[0095] This embodiment is an alternative embodiment of the previous embodiment, which further defines the enhanced Benes network of the previous embodiment, and the number of input and output ports is N = 512 for both.

[0096] The embodiment of the present invention also provides a processor, which includes the coarse-grained reconfigurable array based on the fully interconnected network in any of the previous embodiments.

[0097] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A coarse-grained reconfigurable processor based on an ISP pipeline, characterized in that Including: ISP interaction interface unit, execution control unit CU, configuration management unit MU, direct memory access unit DMA, execution unit array, array memory access unit, and block memory; The execution control unit CU is connected to and can control the configuration management unit MU and the ISP interaction interface unit; in the ISP interaction interface unit, there is a row buffer array composed of multiple memories; the ISP interaction interface unit can interface with an external ISP algorithm module; The configuration management unit MU is connected to the execution unit array; The configuration management unit MU can call the direct memory access unit DMA; the configuration management unit MU can transfer the data stored in the ISP interaction interface unit to the memory of the execution unit array through DMA; The execution unit array reads or stores processing data from / to the execution unit array memory access unit; The execution unit array memory access unit is connected to the block memory and can read or store processing data from / to the block memory; The direct memory access unit DMA is connected to the block memory and can read the data to be processed from the block memory; The direct memory access unit DMA can store data in the block memory; The execution control unit CU is configured to be able to call and read the ISP interaction interface unit, close some algorithm module functions of the ISP, and open the data output interface of the corresponding module; The execution control unit CU polls the ISP interaction interface unit. If data is ready; then the configuration management unit MU starts the direct memory access unit DMA to transfer the image data of the ISP stored in the ISP interaction interface unit from the row storage DVP_OUT line buffer to the block memory; The execution control unit CU notifies the configuration management unit MU to configure the execution unit array and the array memory access unit, so that the execution unit array reads the data in the block memory through the execution unit array memory access unit, and after the calculation of the array, writes the calculation result back to the block memory through the execution unit array memory access unit; If the execution unit array calculation is completed, the execution control unit CU notifies the direct memory access unit DMA to transfer the data from the block memory to the ISP interaction interface unit, so that the image data returns to the ISP pipeline for continued calculation.

2. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 1, characterized in that The ISP interaction interface unit includes an image row buffer DVP_OUT line Buffer of a digital image interface; the execution control unit CU polls the image row buffer DVP_OUT line Buffer of the digital image interface to check whether a row of data is ready.

3. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 1, wherein The ISP interaction interface unit includes a status register; the status register can feedback the number of image rows in the row buffer; it is judged whether data is ready by the number of image rows in the row buffer.

4. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 1, wherein If the execution unit array has completed the calculation, the execution unit array reports the calculation completion information to the configuration management unit MU; the configuration management unit MU notifies the control unit CU, and the execution control unit CU notifies the direct memory access unit DMA to move the data from the block memory to the ISP interaction interface unit.

5. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 1, wherein: The execution unit array includes multiple types of heterogeneous operators connected by a fully interconnected network; the multiple types of heterogeneous operators include a branch operator SEL, a numerical operation operator ALU, an enhanced numerical operation operator EALU, a multiply-accumulate operator MAC, a special operator SPU, a cache operator BUFFER, and a loop control operator LOOP; the fully interconnected network includes a Benes network, a Clos network, or a Crossbar network.

6. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 5, wherein The fully interconnected network is an enhanced Benes network, including a first sub-Benes network and a second sub-Benes network with both the number of input and output ports being N / 2, and N switching units; each switching unit is provided with a first input end, a second input end, a first output end, and a second output end, and has four working modes: direct connection, switching, upper broadcast, and lower broadcast; where N / 2 switching units are connected between the input ends In0~In(N-1) of the enhanced Benes network and the input ends of the first sub-Benes network and the second sub-Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network; Another N / 2 switching units are connected between the output ends of the first sub-Benes network and the second sub-Benes network and the output ends Out0~Out(N-1) of the enhanced Benes network. The first output ports of the N / 2 switching units are connected to the input ports of the first sub-Benes network, and the second output ports of the N / 2 switching units are connected to the input ports of the second sub-Benes network.

7. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 5, characterized in that The execution unit array further includes a calculation unit set Cluster, and the calculation unit set Cluster includes one or more types of heterogeneous operators connected by an enhanced Benes network.

8. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 7, characterized in that The calculation unit set Cluster includes 24 enhanced numerical operation operators EALU, 48 numerical operation operators ALU, 12 branch operators SEL, 2 loop control operators LOOP, and 48 cache operators BUFFER connected by an enhanced Benes network.

9. The coarse-grained reconfigurable processor based on the ISP pipeline according to claim 5, characterized in that, The execution unit array includes 4 calculation unit sets Cluster, 50 multiply-accumulate operators MAC, 10 special operators SPU, 64 cache operators BUFFER, and 6 loop control operators LOOP connected by an enhanced Benes network.

Images