Accelerator framework of visual hardware and nonlinear continuous address access engine

Abstract

The invention provides an accelerator framework of visual hardware and a nonlinear continuous address access engine. The accelerator frame comprises: a system controller configured with a first interface for performing read-write control and a second interface for performing SRAM data read-write; the visual accelerator is connected with the system controller and the SRAM, and is configured to read and calculate DMA data from the SRAM according to a calculation instruction obtained from the system controller, and output a calculation result to the SRAM; and the SRAM controller is connected with the system controller, the visual accelerator and the plurality of memory banks of the SRAM, and is configured to arbitrate access requests acquired from the system controller and the visual accelerator according to a predefined priority so as to perform parallel access of each memory bank of the SRAM. By adopting the configuration, the flexibility of the accelerator framework can be improved, so that the method can adapt to rich SOC application scenes in the field of computer vision.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision (CV), and in particular to an accelerator framework of visual hardware and a non-linear continuous address access engine (NLAE). Background Art

[0002] In the field of computer vision technology, both offline neural network model training and online neural network reasoning involve a large amount of data calculation. General-purpose processor chips SOC (System on Chip) usually take a lot of time and energy to perform such calculations. For this reason, this field usually needs to develop specific hardware accelerators to provide higher performance and lower energy consumption to meet the task requirements of offline training and online reasoning.

[0003] However, in the existing hardware accelerator framework, the external transmission control protocol of read and write control is usually coupled with the transmission data service, which requires extensive modification of the application environment of the hardware accelerator when changing the interface type, data width and cache size, and requires technical personnel to re-develop and verify. Therefore, there are problems such as long development cycle, high modification difficulty, single application scenario and lack of flexibility.

[0004] In order to overcome the above-mentioned defects of the prior art, there is an urgent need in the art for a visual hardware acceleration technology to improve the flexibility of the accelerator framework to adapt to the rich SOC application scenarios in the field of computer vision technology. Summary of the invention

[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or decisive elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an accelerator framework for visual hardware, which can improve the flexibility of the accelerator framework by decoupling the external transmission control protocol and the transmission data service of read and write control, so as to adapt to the rich SOC application scenarios in the field of computer vision technology.

[0007] Specifically, the accelerator framework of the above-mentioned visual hardware provided according to the first aspect of the present invention includes a system controller, a vision accelerator (Vision Accelerator, VACC) and an SRAM (Static Random Access Memory) controller. The system controller is configured with a first interface for read and write control, and a second interface for reading and writing SRAM data. The vision accelerator is connected to the system controller and the SRAM, and is configured to: read and calculate DMA (Direct Memory Access) data from the SRAM according to the calculation instructions obtained from the system controller, and output the calculation results to the SRAM. The SRAM controller is connected to the system controller, the vision accelerator and multiple memory groups of the SRAM, and is configured to: arbitrate access requests obtained from the system controller and the vision accelerator according to a predefined priority, so as to perform parallel access to each of the memory groups of the SRAM.

[0008] Furthermore, in one embodiment of the present invention, the system controller is configured with a CSR (Control / Status Register) read / write control module, a WDMA (DMA Writing) module and an RDMA (DMA Reading) module. The first interface is configured in the CSR read / write control module, including a first bus interface and a first processor interface. The CSR read / write control module performs CSR access via the first bus interface, and performs the read / write control via the first processor interface. The second interface includes a second bus interface configured in the WDMA module, and a third bus interface configured in the RDMA module. The WDMA module writes out SRAM data via the second bus interface. The RDMA module reads in SRAM data from the outside via the third bus interface.

[0009] Furthermore, in one embodiment of the present invention, the system controller is also configured with an actuator. The actuator is configured with a read sequencer, an execution sequencer and a write sequencer, and is configured to: obtain a start control signal via the CSR read / write control module to generate a read / write instruction and / or a calculation instruction; generate an image block read request via the read sequencer according to the read / write instruction to start the RDMA module to read SRAM data from the outside; in response to a completion signal returned by the RDMA module, store the calculation instruction in the instruction read queue of the execution sequencer to sequentially send the calculation instruction to the visual accelerator; in response to a feedback signal returned by the visual accelerator, add an image block write instruction to the output task queue of the write sequencer; and output the image block write instruction to the WDMA module in sequence via the write sequencer to start the WDMA module to write SRAM data to the SRAM.

[0010] Furthermore, in one embodiment of the present invention, the RDMA module is connected to the read sequencer and is configured to: obtain the image block read request via the read sequencer to generate a corresponding external data read address and an SRAM data write address; output a DMA read request regarding the external data read address to the outside via the third bus interface to obtain DMA data returned from the outside; store the DMA data in a first cache queue of the RDMA module; and write the cache data in the first cache queue to the SRAM data write address in sequence according to the maximum row cache length supported by the SRAM.

[0011] Furthermore, in one embodiment of the present invention, the WDMA module is connected to the write sequencer and is configured to: obtain the image block write request via the write sequencer to generate a corresponding external data write address and an SRAM data read address; read the corresponding SRAM data from the SRAM according to the SRAM data read address, and store it in a second cache queue of the WDMA module; and output the DMA write request and DMA data regarding the external data write address to the outside in sequence via the second bus interface according to the cache data in the second cache queue.

[0012] Furthermore, in one embodiment of the present invention, an interface protocol adapter module is also configured in the system controller. The control interface front end of the interface protocol adapter module is connected to the CSR read-write control module, which is used to convert external read-write requests into CSR read-write requests that comply with the first bus interface protocol, so as to read and write CSR registers. The external write data front end of the interface protocol adapter module is connected to the WDMA module, which is used to convert the external data write address into a DMA write request that complies with the second bus interface protocol, so as to write DMA data to the outside. The external read data front end of the interface protocol adapter module is connected to the RDMA module, which is used to convert the external data read address into a DMA read request that complies with the third bus interface protocol, so as to obtain the DMA data returned from the outside.

[0013] Further, in one embodiment of the present invention, the first bus interface, the second bus interface and the third bus interface are selected from AXIL interface and / or APB interface, and / or the first processor interface is selected from VCIX interface, TIEQ interface and / or ROCC interface.

[0014] Furthermore, in one embodiment of the present invention, the step of reading and calculating DMA data from the SRAM according to the calculation instructions obtained from the system controller, and outputting the calculation results to the SRAM includes: establishing a slicing strategy regarding slicing offset, image block width and / or image block height according to the initialization parameters of the M matrix; in response to the calculation instructions obtained from the system controller, reading DMA data of multiple image blocks from the SRAM one by one according to the slicing strategy to perform a warp calculation; and outputting the result data of the warp calculation to the SRAM.

[0015] Furthermore, in one embodiment of the present invention, the accelerator framework further includes a non-linear continuous address access engine (NLAE). The visual accelerator is also connected to the non-linear continuous address access engine and is configured to: read and calculate DMA data from the non-linear continuous address access engine according to the calculation instruction obtained from the system controller, and output the calculation result to the SRAM. The non-linear continuous address access engine is configured to: determine the sequence access request of multiple memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator, so as to concurrently access each of the memory groups; read out the corresponding storage data from each of the memory groups, and merge them based on the element position; and return the merged DMA data to the visual accelerator.

[0016] Furthermore, in one embodiment of the present invention, the nonlinear continuous address access engine is configured with a plurality of merging units and a plurality of BSM (Bank State Machine) units respectively corresponding to each of the memory groups. The step of determining the sequence access request of the plurality of memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator to concurrently access each of the memory groups comprises: determining the memory groups accessed by each of the address offset elements according to the offset address information of the plurality of address offset elements of at least one address offset vector in the vector read request; merging the accesses of the plurality of address offset elements corresponding to each of the memory groups bank_i respectively through the merging unit merge_i corresponding to each of the memory groups bank_i, and sequentially outputting the addresses for accessing each of the memory groups bank_i through the corresponding BSM unit bsm_i; and reading out the corresponding DMA data from each of the memory groups bank_i in parallel according to the addresses output by each of the BSM units.

[0017] Furthermore, in one embodiment of the present invention, the nonlinear continuous address access engine is further configured with a plurality of data return units corresponding to each of the memory banks. The step of reading the corresponding storage data from each of the memory banks and merging the data based on the element position comprises: performing a logical operation on the depth data read from each of the memory banks bank_i and a mask generated based on the position of the corresponding address offset element in the memory return data to determine the number of offset bits, and performing a bit shift on the depth data, wherein the mask indicates whether the current address offset vector hits the current memory bank; outputting the bit shift result to the position of all address offset elements whose offset address information conforms to the corresponding data return unit outVecB_i; and performing a merging operation on the bit shift results output to each of the data return units outVecB_i to obtain merged depth image data.

[0018] Furthermore, in an embodiment of the present invention, the SRAM controller is configured with a plurality of pairs of arbitration units corresponding to the memory groups. The SRAM controller connects the corresponding memory groups to the first read / write port of the system controller, the second read / write port of the visual accelerator, and the third read / write port of the non-linear continuous address access engine through each pair of arbitration units, so as to arbitrate access requests obtained from the system controller, the visual accelerator, and the non-linear continuous address access engine according to a predefined priority, and perform parallel access to the memory groups.

[0019] Furthermore, in one embodiment of the present invention, the SRAM controller represents the SRAM corresponding to each of the memory banks via an array consisting of numBanks SyncReadMem parameters, and represents the arbitration unit corresponding to each of the memory banks via an array consisting of numBanks arbiter_ids.

[0020] Furthermore, in one embodiment of the present invention, the SRAM controller is further configured with a plurality of demultiplexing logic units corresponding to the memory groups. The SRAM controller connects the corresponding memory groups to the first read / write port of the system controller, the second read / write port of the visual accelerator, and the third read / write port of the nonlinear continuous address access engine through the demultiplexing logic units, so as to demultiplex the output data of the memory groups according to the predefined priority and transmit the output data to the corresponding first read / write port, the second read / write port, and / or the third read / write port.

[0021] Furthermore, in an embodiment of the present invention, the demultiplexing logic unit uses a when-elsewhen statement to demultiplex the output data of each of the memory groups.

[0022] In addition, the nonlinear continuous address access engine (NLAE) provided according to the second aspect of the present invention is respectively connected to the visual accelerator and SRAM provided according to the first aspect of the present invention, and is configured to: determine the sequence access request of multiple memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator, so as to concurrently access each of the memory groups; read out the corresponding storage data from each of the memory groups, and merge them based on the element position; and return the merged DMA data to the visual accelerator. By adopting the above configuration, the nonlinear continuous address access engine (NLAE) can merge the corresponding storage data read out from each memory group based on the element position, and return the merged DMA data to the visual accelerator, so as to adjust and configure the specific accelerator data access pattern, thereby supporting pipeline access of vector offset, memory group access merging, and parallel operation of multiple memory groups.

[0023] Furthermore, in one embodiment of the present invention, the nonlinear continuous address access engine is configured with a plurality of merging units and a plurality of BSM (Bank State Machine) units respectively corresponding to each of the memory groups. The step of determining the sequence access request of the plurality of memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator to concurrently access each of the memory groups comprises: determining the memory groups accessed by each of the address offset elements according to the offset address information of the plurality of address offset elements of at least one address offset vector in the vector read request; merging the accesses of the plurality of address offset elements corresponding to each of the memory groups bank_i respectively through the merging unit merge_i corresponding to each of the memory groups bank_i, and sequentially outputting the addresses for accessing each of the memory groups bank_i through the corresponding BSM unit bsm_i; and reading out the corresponding DMA data from each of the memory groups bank_i in parallel according to the addresses output by each of the BSM units.

[0024] Furthermore, in one embodiment of the present invention, the nonlinear continuous address access engine is further configured with a plurality of data return units corresponding to each of the memory banks. The step of reading the corresponding storage data from each of the memory banks and merging the data based on the element position comprises: performing a logical operation on the depth data read from each of the memory banks bank_i and a mask generated based on the position of the corresponding address offset element in the memory return data to determine the number of offset bits, and performing a bit shift on the depth data, wherein the mask indicates whether the current address offset vector hits the current memory bank; outputting the bit shift result to the position of all address offset elements whose offset address information conforms to the corresponding data return unit outVecB_i; and performing a merging operation on the bit shift results output to each of the data return units outVecB_i to obtain merged depth image data.

[0025] Furthermore, in an embodiment of the present invention, the SRAM controller is configured with a plurality of pairs of arbitration units corresponding to the memory groups. The SRAM controller connects the corresponding memory groups to the first read / write port of the system controller, the second read / write port of the visual accelerator, and the third read / write port of the non-linear continuous address access engine through each pair of arbitration units, so as to arbitrate access requests obtained from the system controller, the visual accelerator, and the non-linear continuous address access engine according to a predefined priority, and perform parallel access to the memory groups.

[0026] Furthermore, in one embodiment of the present invention, the SRAM controller represents the SRAM corresponding to each of the memory banks via an array consisting of numBanks SyncReadMem parameters, and represents the arbitration unit corresponding to each of the memory banks via an array consisting of numBanks arbiter_ids.

[0027] Furthermore, in one embodiment of the present invention, the SRAM controller is further configured with a plurality of demultiplexing logic units corresponding to the memory groups. The SRAM controller connects the corresponding memory groups to the first read / write port of the system controller, the second read / write port of the visual accelerator, and the third read / write port of the nonlinear continuous address access engine through the demultiplexing logic units, so as to demultiplex the output data of the memory groups according to the predefined priority and transmit the output data to the corresponding first read / write port, the second read / write port, and / or the third read / write port.

[0028] Furthermore, in an embodiment of the present invention, the demultiplexing logic unit uses a when-elsewhen statement to demultiplex the output data of each of the memory groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0030] Figure 1 A schematic diagram of the structure of an accelerator framework of visual hardware provided according to some embodiments of the present invention is shown.

[0031] Figure 2 A schematic structural diagram of a system controller provided according to some embodiments of the present invention is shown.

[0032] Figure 3 A flowchart of running a visual accelerator according to some embodiments of the present invention is shown.

[0033] Figure 4 A schematic diagram of the structure of a non-linear continuous address access engine provided according to some embodiments of the present invention is shown.

[0034] Figure 5 A schematic structural diagram of an SRAM controller provided according to some embodiments of the present invention is shown.

[0035] Figures 6A to 6D A schematic diagram of an application scenario of an accelerator framework for visual hardware provided according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0036] The following specific embodiments illustrate the implementation of the present invention, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, the following description will include many specific details. The present invention can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present invention, some specific details will be omitted in the description.

[0037] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0039] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0040] As mentioned above, in the existing hardware accelerator framework, the external transmission control protocol of read and write control is usually coupled to the transmission data service, which requires extensive modification of the application environment of the hardware accelerator when changing the interface type, data width and cache size, and requires technical personnel to re-develop and verify. Therefore, there are problems such as long development cycle, high modification difficulty, single application scenario and lack of flexibility.

[0041] In order to overcome the above-mentioned defects of the prior art, the present invention provides an accelerator framework for visual hardware, which can improve the flexibility of the accelerator framework by decoupling the external transmission control protocol and the transmission data service of the read-write control, so as to adapt to the rich SOC (System on Chip) application scenarios in the field of computer vision technology. In addition, the present invention also provides a nonlinear continuous address access engine, which can merge the corresponding storage data read from each memory group based on the element position, and return the merged DMA data to the visual accelerator to adjust and configure the specific accelerator data access pattern, so as to support pipeline access of vector offset, memory group access merging, and parallel operation of multiple memory groups.

[0042] First, please refer to Figure 1 , Figure 1 A schematic diagram of the structure of an accelerator framework of visual hardware provided according to some embodiments of the present invention is shown.

[0043] exist Figure 1 In the illustrated embodiment, the accelerator framework 100 of the visual hardware provided by the first aspect of the present invention is configured with a system controller 11, a vision accelerator (VACC) 12 and an SRAM (Static Random-Access Memory) controller 13. The system controller 11 is configured with a first interface 111 for read and write control, and second interfaces 112-113 for reading and writing SRAM data. The vision accelerator 12 is connected to the system controller 11 and an external static random access memory (SRAM) 21, respectively, and can read and calculate direct memory access (DMA) data from a target memory bank 211-214 of the SRAM 21 according to a calculation instruction obtained from the system controller 11, and output the calculation result back to the specified memory bank 211-214 of the SRAM 21. The SRAM controller 13 is connected to the system controller 11 , the visual accelerator 12 and multiple memory groups 211 - 214 of the SRAM 21 , respectively, and can arbitrate access requests obtained from the system controller 11 and the visual accelerator 12 according to a predefined priority to perform parallel access to each memory group 211 - 214 of the SRAM 21 .

[0044] Specifically, the system controller 11 may be further configured with a control / status register (CSR) read / write control module, a writing direct memory access (WDMA) module, and a reading direct memory access (RDMA) module. The first interface 111 is configured in the CSR read / write control module 11, and includes a first bus interface and a first processor interface, wherein the first bus interface can be selected from an AXIL interface and / or an APB interface, and the first processor interface can be selected from a VCIX interface, a TIEQ interface, and / or a ROCC interface. The CSR read / write control module performs CSR access via the first bus interface, and performs read / write control via the first processor interface. The second interfaces 112-113 include a second bus interface 112 configured in the WDMA module, and a third bus interface 113 configured in the RDMA module, wherein the second bus interface 112 and the third bus interface 113 are respectively selected from the AXIL interface and / or the APB interface. The WDMA module writes out the SRAM data via the second bus interface 112. The RDMA module reads in the SRAM data from the outside via the third bus interface 113.

[0045] Please refer to Figure 1 and Figure 2 , Figure 2 A schematic structural diagram of a system controller provided according to some embodiments of the present invention is shown.

[0046] like Figure 1 and Figure 2 As shown, in addition to the CSR read / write control module, the WDMA module and the RDMA module, the system controller 10 may also be preferably configured with an interface protocol adapter module and an actuator Xactor. Here, the interface protocol adapter module may be composed of a control interface front end cFrontend, an external write data front end wFrontend and an external read data front end rFrontend. The actuator 14 may be implemented by a hardened software module, which can not only provide a hardened structure at the macro level, but also provide the user with an option of automatic execution after being fully triggered, so as to increase the flexibility of the accelerator framework.

[0047] Specifically, the control interface front end cFrontend is connected to the CSR read / write control module, and is used to first convert the external read / write request into bus information bus, and then convert the bus information bus into a CSR read / write request csr that complies with the first bus interface protocol, so as to read and write the CSR register.

[0048] The actuator Xactor is configured with a reader sequencer 141, an executive sequencer 142, a writer sequencer 143 and a buffer manager module 144, which can obtain a start control signal csr via a CSR read / write control module to generate a read / write instruction and / or a calculation instruction, and generate an image block read request xTileReqIn according to the generated read / write instruction via the reader sequencer 141 to start the RDMA module to read SRAM data from the outside. Here, the image block read request xTileReqIn may include information describing the image block width, image block height, pixel position, and data width of at least one square image block in the input image.

[0049] The RDMA module is connected to the read sequencer 141, and preferably to the external read data front end rFrontend, not only supporting the target read and write of the SRAM 21, but also preferably supporting the read and write of multiple memory groups 211-214 of different rows, so as to increase the flexibility of the hardware computing logic support. Specifically, in response to the image block read request xTileReqIn obtained from the read sequencer 141, the RDMA module can first generate the corresponding external data read address and SRAM data write address, and preferably cut the external data read address across 4K through the splitter 1141, and then convert the external data read address into a DMA read request dma.read that complies with the third bus interface protocol through the external read data front end rFrontend, so as to output the DMA read request dma.read about the external data read address to the outside through the third bus interface 113, and obtain the external returned DMA data through the third bus interface 113. Here, the DMA data can be a burst data of one column. The RDMA module can first store the acquired DMA data into the local first cache queue 1142 via the data input interface dataIn, and then transmit the SRAM write instruction sram.write.1 to the corresponding memory groups 211~214 of SRAM 21 via the sXactWr unit, so as to write the cache data in the first cache queue 1142 to the corresponding SRAM data write address in sequence according to the maximum line cache length lineBuffer supported by SRAM 21.

[0050] Afterwards, in response to completing the SRAM write of the cache data in the first cache queue 1142, the RDMA module may return a completion signal finished.1 to the actuator Xactor. In response to the completion signal finished.1, the actuator Xactor may store the previously generated computing instructions into the instruction read queue of the execution sequencer 142 to sequentially send the computing instructions exec.cmd to the visual accelerator 12, thereby controlling the visual accelerator 12 to read and calculate the DMA data from the SRAM 21.

[0051] Afterwards, in response to the feedback signal exec.feedback returned by the visual accelerator 12, the actuator Xactor may add the image block write instruction xTileReqOut to the output task queue of the write sequencer 143, and sequentially output the image block write instruction xTileReqOut to the WDMA module via the write sequencer 143 to start the WDMA module to write the SRAM data to the SRAM 21. Specifically, the write sequencer 143 may keep detecting the state of its output task queue to ensure that the WDMA module can be started to perform calculations and output calculation results as soon as there is a task, until the actuator Xactor completes the writing of all image blocks of the input image.

[0052] The WDMA module is connected to the write sequencer 143 of the actuator Xactor, and preferably connected to the above-mentioned external write data front end wFrontend, which can not only realize direct memory access (DMA) based on the second bus interface protocol, but also further support the data writing of multiple memory groups 211-214 in different rows of SRAM 21, thereby realizing the decoupling of the processing logic and the handling logic of the second bus interface protocol through SRAM 21 to increase the flexibility of the operation logic. Specifically, in response to the image block write request xTileReqOut obtained from the write sequencer 143, the WDMA module can first generate the corresponding external data write address and SRAM data read address, and preferably cut the external data read address across 4K through the splitter unit (Splitter) 1151, and then transmit the SRAM read instruction sram.read.1 to the corresponding memory group 211-214 of SRAM 21 through the sXactRd unit under the control of the state machine, so as to read the corresponding SRAM data from SRAM 21 according to the SRAM data read address. Afterwards, the WDMA module can store the read SRAM data into the local second cache queue 1152, and output DMA data to the outside in sequence via the dataOut interface according to the cache data in the second cache queue 1152, and convert the external data write address into a DMA write request dma.write that complies with the second bus interface protocol via the external write data front end wFrontend, so as to write DMA data to the outside in sequence. Afterwards, in response to completing the DMA write of the cache data in the second cache queue 1152, the RDMA module can return a completion signal finished.2 to the actuator Xactor.

[0053] In this way, by respectively configuring a first interface 111 for read and write control and a second interface 112-113 for SRAM data reading and writing in the system controller 11, and further configuring the interface forms of a bus interface and / or a processor interface for them respectively, the present invention can achieve the decoupling of the external transmission control protocol and the transmission data service of the accelerator framework read and write control, thereby improving the flexibility of the entire accelerator framework to adapt to the current rich SOC application scenario requirements.

[0054] Please refer to Figure 1 and Figure 3 , Figure 3 A flowchart of running a visual accelerator according to some embodiments of the present invention is shown.

[0055] like Figure 1 and Figure 3As shown, in the process of visual acceleration processing of the input image, the visual accelerator 12 can first obtain the initialization parameters of the M matrix through the first interface 111, and establish the corresponding M matrix accordingly to determine the image tiling strategy for the tile offset, image block width and / or image block height of the input image. Here, the M matrix is ​​an n*n square matrix, and each element a ii >0, each element a ij ≤0(i≠j), and its inverse matrix is ​​a non-negative matrix.

[0056] Afterwards, in response to the calculation instruction exec.cmd obtained from the system controller 11, the visual accelerator 12 can first transmit the SRAM read instruction sram.Read.2 to the SRAM 21, read the DMA data of multiple image blocks (Tile block) of the input image one by one from the target memory groups 211 to 214 of the SRAM 21 according to the slicing strategy indicated by the M matrix, so as to perform the warp calculation, and then transmit the SRAM write instruction sram.Write.2 to the SRAM 21 to return the result data of the warp calculation to the corresponding position of the SRAM 21, until the DMA data of each image block of the input image is all read and calculated. In this way, the visual accelerator 12 can not only support the acceleration logic executed by the user in a command interactive manner through the calculation instruction exec.cmd, but also automatically execute the complete image frame (Frame) through the queue of the system controller 11 in an offline state.

[0057] Furthermore, in some embodiments of the present invention, the SRAM controller 13 may preferably be configured with a plurality of pairs of arbitration units (Arbiters) corresponding to the memory groups 211 to 214 of the SRAM 21, respectively. Figure 1As shown, in response to the SRAM read / write instructions sram.Read.1, sram.Write.1 provided by the system controller 11, and the SRAM read / write instructions sram.Read.2, sram.Write.2 provided by the visual accelerator 12, the SRAM controller 13 can first arbitrate the access requests obtained from the system controller 11 and the visual accelerator 12 through the multiple pairs of arbitration units according to a predefined priority, and then perform parallel access to each memory group 211~214 of the SRAM 21 according to the arbitration result. For example, each arbitration unit may be configured with an access priority of the visual accelerator 12 that is higher than that of the system controller 11, so that when multiple SRAM read / write instructions are received at the same time, the SRAM read / write instructions sram.Read.2 and sram.Write.2 provided by the visual accelerator 12 are executed first, and then the SRAM read / write instructions sram.Read.1 and sram.Write.1 provided by the system controller 11 are executed, thereby improving the throughput of the entire accelerator framework and reducing access conflicts when accessing the memory groups 211 to 214 of the SRAM 21 in parallel.

[0058] Furthermore, considering that the visual accelerator 12 will generate a large number of vector access requirements for nonlinear continuous addresses during image processing such as warping calculations, and that multiple SRAM access requests for these vectors are likely to cause access conflicts among the memory groups 211 to 214, the accelerator framework of the above-mentioned visual hardware provided in the first aspect of the present invention may also preferably be configured with the above-mentioned nonlinear continuous address access engine (Non Linear Access Engine, NLAE) provided in the second aspect of the present invention.

[0059] Please refer to Figure 1 and Figure 4 , Figure 4 A schematic diagram of the structure of a non-linear continuous address access engine provided according to some embodiments of the present invention is shown.

[0060] exist Figure 1In the illustrated embodiment, the nonlinear continuous address access engine 14 can be connected to the visual accelerator 12 and the SRAM 21, respectively. In response to the calculation instruction exec.cmd obtained from the system controller 11, the visual accelerator 12 can perform warping calculations on the DMA data of multiple image blocks of the input image one by one as described above, and transmit a vector read request of a non-continuous address to the nonlinear continuous address access engine 14. In response to the vector read request of the non-continuous address obtained from the visual accelerator 12, the nonlinear continuous address access engine 14 can correspondingly determine the sequence access request of multiple memory groups 211-214 in the SRAM 21, so as to concurrently access each memory group 211-214, read out the corresponding storage data from each memory group 211-214, and merge them based on the element position, and return the merged DMA data to the visual accelerator 12 to support it in performing image processing such as warping calculation.

[0061] Specifically, if Figure 4 As shown, the non-linear continuous address access engine 14 may be configured with a plurality of merging units merge_0 to merge_3 and a plurality of BSM (Bank State Machine) units bsm_0 to bsm_3 corresponding to each memory bank_0 to bank_3 respectively. In the process of concurrently accessing the plurality of memory banks bank_0 to bank_3 in the SRAM 21, the non-linear continuous address access engine 14 may first determine the target memory bank bank_i to be accessed by each address offset element according to the offset address information of vecLen address offset elements idxVec of at least one address offset vector (Vector) in the vector read request of the non-continuous address obtained from the visual accelerator 12.

[0062] Afterwards, the non-linear continuous address access engine 14 can merge the accesses of multiple address offset elements idxVec_i0~idxVec_ij corresponding to each target memory group bank_i through the merging unit Merger_i corresponding to each target memory group bank_i, and output the address addr_i for accessing each memory group bank_i in sequence through the corresponding BSM unit bsm_i, and read the corresponding DMA data from each memory group bank_i in parallel with the address addr_i.

[0063] In addition, Figure 4In the illustrated embodiment, the non-linear continuous address access engine 14 may also be preferably configured with a plurality of data return units outVecB_0 to outVecB_3 corresponding to each memory bank_0 to bank_3, respectively. In response to reading out the corresponding storage data (e.g., depth data) data_i from each target memory bank_i, the non-linear continuous address access engine 14 may first perform a logical operation on the depth data data_i read out from each memory bank_i and the mask mask_i generated based on the position elemIdx_i of the corresponding address offset element idxVec_i0 to idxVec_ij in the memory return data to determine the corresponding offset bit number bit_i, wherein the mask mask_i indicates whether the current address offset vector hits the current memory bank_i. Afterwards, the non-linear continuous address access engine 14 may perform a bit shift data_i on the acquired depth data according to the offset bit number bit_i, so as to output the bit shift result to the position elemIdx_i of all address offset elements whose offset address information conforms to the corresponding data return unit outVecB_i. Afterwards, the non-linear continuous address access engine 14 may perform a merging operation on the bit shift results output to each data return unit outVecB_i to obtain the merged depth image data outVec.

[0064] Compared with the existing access engine that uses scalar sequential access or simply aggregates each memory bank, has many cycles, poor performance, and often ignores repeated identical access items, the above-mentioned nonlinear, universal nonlinear continuous address access engine 14 provided by the present invention can not only be adjusted and configured for specific accelerator data access modes, but also perform concurrent access to vectors with non-continuous, random, and sparse addresses, realize merged access to the same memory bank entry, and combine the merged access of multiple memory bank entries in parallel with the same memory bank offset, thereby maximizing the avoidance of address access conflicts through a pipeline and improving the throughput of the accelerator framework.

[0065] In addition, please refer to Figure 5 . Figure 5 A schematic structural diagram of an SRAM controller provided according to some embodiments of the present invention is shown.

[0066] exist Figure 5In the illustrated embodiment, corresponding to the embodiment including the non-linear continuous address access engine 14, the SRAM controller 13 may be configured with a plurality of pairs of arbitration units arb_01-arb_22 corresponding to each memory bank_0-bank_2, and each memory bank_0-bank_2 is connected to the first read / write port dmaSRAM of the system controller 11, the second read / write port vaccSRAM of the visual accelerator 12, and the third read / write port nlaeSRAM of the non-linear continuous address access engine 14 through the plurality of pairs of arbitration units arb_01-arb_22, so as to arbitrate access requests obtained from the system controller 11, the visual accelerator 12, and the non-linear continuous address access engine 14 according to a predefined priority, and perform parallel access to each memory bank_0-bank_2. Here, the arbitration units arb_01-arb_22 may be standard arbiter implementation circuits. Each arbitration unit arb_01~arb_22 can be configured with an access priority of the visual accelerator 12 higher than the non-linear continuous address access engine 14, and the non-linear continuous address access engine 14 higher than the system controller 11. Therefore, when multiple SRAM read / write instructions are received at the same time, the SRAM read / write instructions sram.Read.2 and sram.Write.2 provided by the visual accelerator 12 are executed first, and then the serial access request provided by the non-linear continuous address access engine 14 is executed, and then the SRAM read / write instructions sram.Read.1 and sram.Write.1 provided by the system controller 11 are executed, thereby further improving the throughput of the entire accelerator framework and reducing access conflicts when accessing the memory groups 211~214 of the SRAM 21 in parallel.

[0067] Further, in some embodiments, the SRAM controller 13 can represent the SRAM 21 corresponding to each memory group bank_0~bank_2 via an array consisting of numBanks SyncReadMem parameters, and represent the arbitration units arb_01~arb_22 corresponding to each memory group bank_0~bank_2 via an array consisting of numBanks arbiter_id, thereby parameterizing the combination of multiple memory groups bank_0~bank_2 corresponding to the SRAM 21 and each arbitration unit arb_01~arb_22.

[0068] In addition, if Figure 5As shown, the SRAM controller 13 may also be preferably configured with a plurality of demultiplexing logic units dem_0 to dem_2 corresponding to each memory bank_0 to bank_2 respectively. The SRAM controller 13 may also connect each memory bank_0 to bank_2 to the first read / write port dmaSRAM of the system controller 11, the second read / write port vaccSRAM of the visual accelerator 12, and the third read / write port nlaeSRAM of the nonlinear continuous address access engine 14 through each demultiplexing logic unit dem_0 to dem_2, so as to demultiplex the output data of each memory bank_0 to bank_2 according to a predefined priority, and then transmit the demultiplexed output data to the corresponding first read / write port dmaSRAM, the second read / write port vaccSRAM, and / or the third read / write port nlaeSRAM respectively. In some embodiments, the demultiplexing logic units dem_0 to dem_2 may use a when-elsewhen statement to parse the Mux priority logic, so as to demultiplex and obtain the output data of each memory bank_i.

[0069] In this way, compared with the existing technology that cancels the fixed framework interface structure in order to realize acceleration logic, the SRAM controller 13 provided by the present invention can not only realize parallel read and write operations on numBanks SRAM memory groups, but also solve the problem of needing to connect multiple sources for concurrent access to the same memory group through the configured arbitration units arb_01~arb_22 and demultiplexing logic units dem_0~dem_2, and supports flexible configuration of the number of memory groups numBanks and the priority order of different sources.

[0070] Furthermore, in a preferred embodiment, the present invention can also implement the circuit construction of the above-mentioned system controller 11, visual accelerator 12, SRAM controller and / or non-linear continuous address access engine 14 through Chisel (Constructing Hardware In a Scala Embedded Language). By using Chisel's parameterized language to construct the circuit, the present invention can flexibly generate implementation circuits corresponding to different numbers of memory groups according to configuration parameters through Scala's Simple Build Tool (SBT), thereby improving the flexibility of the accelerator framework to adapt to a variety of SOC application scenarios.

[0071] In addition, by configuring the above-mentioned nonlinear continuous address access engine 14, the present invention can also support pipeline access of offset vectors, memory group access merging, and parallel access of multiple memory groups, support flexible configuration of memory group width, element width, memory group depth, and data depth, and can combine data access application mode (Pattern) to weigh the memory group data width and the number of memory groups to maximize performance and reduce storage space waste. For example, the present invention can support pipeline processing of visual signals such as RGB / iToF (indirect Time of Flight) / dToF (direct Time of Flight), support hardware acceleration of extended CPU / DSP with deep coupling of micro-architecture, and support acceleration of fully offline computing clusters (cluster).

[0072] Please refer to Figures 6A to 6D , Figures 6A to 6D A schematic diagram of an application scenario of an accelerator framework for visual hardware provided according to some embodiments of the present invention is shown.

[0073] exist Fig. 6A In the illustrated embodiment, the nonlinear continuous address access engine of the visual hardware accelerator framework can be configured as a local APB control interface and an SRAM-based row cache data interface, and the parameters of each level of computing module FVACC_i are configured through the MCU 610 and the local bus (Local bus) 611 such as APB, and the output of the engine and / or other sensors 612 are used to complete the interrupt signal to synchronously trigger each FVACC_i, so as to realize iToF / dTof, and even online multi-level pipeline processing of the image signal processor (Image Signal Processor, ISP), thereby realizing more complex image processing functions.

[0074] exist Figure 6B In the illustrated embodiment, the nonlinear continuous address access engine of the visual hardware accelerator framework can be configured as a control interface of a local APB (the data interface can be implemented through buses of the type AXI4, AMBA, etc.), and the parameters of the various levels of computing modules FVACC_i are configured through the MCU620 and the local bus (Local bus) 621 such as APB, and the output of the engine and / or other sensors 622 are used to complete the interrupt signal to synchronously trigger each FVACC_i. At the same time, each FVACC_i can start DMA transmission, read and write larger frame data stored in the system memory space through the system bus (System bus) 623, so as to realize iToF / dTof, and even offline multi-stage pipeline processing of the image signal processor (ISP), thereby realizing more complex image processing functions.

[0075] exist Figure 6C In the illustrated embodiment, the nonlinear continuous address access engine of the visual hardware accelerator framework can be configured as a control interface of an instruction extension interface 631. The data interface can be optionally implemented through a local SRAM interface, or a bus of the type AXI4, AMBA, etc. Various processors such as NPU, DSP, RISC / CPU can quickly configure the parameters of the various levels of computing modules FVACC_i through extended instructions, thereby starting the hardware acceleration module with extremely low latency to execute and complete the computing task.

[0076] exist Fig.6D In the illustrated embodiment, the nonlinear continuous address access engine of the visual hardware accelerator framework can be configured as a control interface based on an external bus 641. The data interface can be optionally implemented through a local SRAM interface, or a bus of the type AXI4, AMBA, etc. The processor 642 of ARM, RISCV, and other different ISA architectures can configure the parameters of various computing modules FVACC_i by reading and writing IO registers through the bus in a universal, broad, and easy-to-port manner to execute and complete computing tasks.

[0077] In summary, the accelerator framework of the above-mentioned visual hardware provided by the present invention can decouple transmission control, transmission transactions, transmission protocols and computing logic, and realize flexible configuration of different hardware interfaces and transmission bus protocols through layering of transmission protocol interface and primitive transmission control (raw xfer), layering of system controller queue (controller sequence) and visual accelerator (vacc) computing logic, SRAM flexible interface, etc., so that it is easier to adapt to different SOC platform environments.

[0078] In addition, by adopting the Chisel design method flow to construct the circuits of the above-mentioned system controller 11, visual accelerator 12, SRAM controller and / or non-linear continuous address access engine 14, the present invention can more easily realize the flexible configuration of system parameters, and generate different circuits through different operating parameters to realize the register transfer level (RTL), thereby more easily realizing high-level logic function verification and more quickly realizing the optimization configuration exploration of the verification system.

[0079] In addition, compared to the existing technology of reading and writing the IO registers of the accelerator through the peripheral drivers of the operating system such as MMIO (for example, in the Linux system, such operations are usually implemented through the Linux Kernel and HAL layers, and are also affected by the context of the Linux task, and the scheduling usually has an uncertain and large delay), the present invention also supports flexible configuration of the bus data width, and supports the processor local APB bus (Local APB Bus), RISC-V processor instruction extension interface, system AXI and AXI stream interface, Cadenc TIE Queue and other extended processor interfaces, which can quickly start the hardware and is not affected by the uncertain high delay caused by the host HAL, Kernel Driver, etc.

[0080] In addition, if Figure 6A to Figure 6D As shown, the present invention also supports offline and interactive processing modes, and can implement a vector reading mechanism and circuit for discontinuous addresses in the SRAM storage space based on this, so that the SRAM access module can further implement vector reading for irregular discontinuous spaces while implementing vector reading for regular continuous addresses.

[0081] In addition, the present invention can support deep coupling of the pipeline of the processor, and can also support multiple levels of pipe interconnection through line buffer data to achieve greater functions. For example, the present invention supports DMA interface connection and transmission of external SRAM space, and can realize multi-level cascading through ping-pong sharing of SRAM output and input of different modules, thereby realizing pipeline interconnection.

[0082] In addition, the present invention can also realize loose offline use through system bus, local bus, etc. to reduce the load of the host. For example, the present invention can connect the AXI interface to the SOC bus, and after the bus MMIO starts the control, it can independently execute and realize offline work.

[0083] In addition, the present invention can also realize high-throughput data reading through a queue buffer structure.

[0084] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0085] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0086] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0087] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An accelerator framework for visual hardware, characterized in that: include: A system controller is configured with a first interface for read and write control and a second interface for reading and writing SRAM data; A visual accelerator connected to the system controller and the SRAM, and configured to: read and calculate DMA data from the SRAM according to a calculation instruction obtained from the system controller, and output the calculation result to the SRAM; as well as The SRAM controller is connected to the system controller, the visual accelerator and the plurality of memory groups of the SRAM, and is configured to arbitrate access requests obtained from the system controller and the visual accelerator according to a predefined priority level to perform parallel access to each of the memory groups of the SRAM.

2. The accelerator frame according to claim 1, characterized in that: The system controller is configured with a CSR read-write control module, a WDMA module and an RDMA module, wherein: The first interface is configured in the CSR read-write control module, including a first bus interface and a first processor interface, wherein the CSR read-write control module performs CSR access via the first bus interface and performs the read-write control via the first processor interface. The second interface includes a second bus interface configured on the WDMA module and a third bus interface configured on the RDMA module, wherein the WDMA module writes SRAM data externally via the second bus interface, and the RDMA module reads SRAM data from the outside via the third bus interface.

3. The accelerator frame according to claim 2, characterized in that: The system controller is also provided with an actuator, and the actuator is provided with a read sequencer, an execution sequencer and a write sequencer, and is configured as follows: Obtaining a start control signal via the CSR read / write control module to generate a read / write instruction and / or a calculation instruction; According to the read / write instruction, an image block read request is generated via the read sequencer to start the RDMA module to read SRAM data from the outside; In response to a completion signal returned by the RDMA module, storing the computing instruction into an instruction read queue of the execution sequencer, so as to sequentially send the computing instruction to the visual accelerator; In response to a feedback signal returned by the visual accelerator, adding an image block write instruction to an output task queue of the write sequencer; as well as The image block write instructions are sequentially output to the WDMA module via the write sequencer to enable the WDMA module to write the SRAM data to the SRAM.

4. The accelerator frame according to claim 3, characterized in that: The RDMA module is connected to the read sequencer and is configured to: Obtaining the image block read request via the read sequencer to generate a corresponding external data read address and SRAM data write address; Outputting a DMA read request about the external data read address to the outside via the third bus interface to obtain DMA data returned from the outside; Storing the DMA data in a first cache queue of the RDMA module; as well as The cache data in the first cache queue is written out to the SRAM data write address in sequence according to the maximum row cache length supported by the SRAM.

5. The accelerator frame according to claim 3, characterized in that: The WDMA module is connected to the write sequencer and is configured as follows: Obtaining the image block write request via the write sequencer to generate a corresponding external data write address and SRAM data read address; Reading corresponding SRAM data from the SRAM according to the SRAM data read address, and storing the data in a second cache queue of the WDMA module; as well as According to the cache data in the second cache queue, a DMA write request and DMA data regarding the external data write address are sequentially output to the outside via the second bus interface.

6. The accelerator frame according to claim 2, characterized in that: The system controller is also configured with an interface protocol adaptation module, wherein: The control interface front end of the interface protocol adapter module is connected to the CSR read-write control module, and is used to convert external read-write requests into CSR read-write requests that comply with the first bus interface protocol, so as to read and write the CSR register. The external data writing front end of the interface protocol adapter module is connected to the WDMA module, which is used to convert the external data writing address into a DMA write request that complies with the second bus interface protocol to write DMA data to the outside. The external data reading front end of the interface protocol adapter module is connected to the RDMA module and is used to convert the external data reading address into a DMA read request that complies with the third bus interface protocol to obtain the external returned DMA data.

7. The accelerator frame according to claim 2, characterized in that: The first bus interface, the second bus interface and the third bus interface are selected from an AXIL interface and / or an APB interface, and / or The first processor interface is selected from a VCIX interface, a TIEQ interface and / or a ROCC interface.

8. The accelerator frame according to claim 1, characterized in that: The step of reading and calculating DMA data from the SRAM according to the calculation instruction obtained from the system controller, and outputting the calculation result to the SRAM comprises: Establishing a slicing strategy regarding a slicing offset, an image block width, and / or an image block height according to initialization parameters of the M matrix; In response to a calculation instruction obtained from the system controller, reading DMA data of a plurality of image blocks one by one from the SRAM according to the slicing strategy to perform warping calculation; and The result data of the curl calculation is output to the SRAM.

9. The accelerator frame according to claim 1, characterized in that: Also included is a non-linear sequential address access engine, wherein: The visual accelerator is also connected to the non-linear continuous address access engine and is configured to: read and calculate DMA data from the non-linear continuous address access engine according to a calculation instruction obtained from the system controller, and output the calculation result to the SRAM, The nonlinear continuous address access engine is configured to: determine the sequence access request of multiple memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator, so as to concurrently access each of the memory groups; read out the corresponding storage data from each of the memory groups and merge them based on the element position; and return the merged DMA data to the visual accelerator.

10. The accelerator frame according to claim 9, characterized in that: The non-linear continuous address access engine is configured with a plurality of merging units and a plurality of BSM units corresponding to each of the memory groups respectively. The step of determining the sequence access request of the plurality of memory groups in the SRAM according to the vector read request of the non-continuous address obtained from the visual accelerator to concurrently access each of the memory groups comprises: Determine, according to offset address information of multiple address offset elements of at least one address offset vector in the vector read request, the memory groups accessed by each of the address offset elements; Merging accesses of a plurality of address offset elements corresponding to each memory bank_i respectively via a merging unit merge_i corresponding to each memory bank_i, and sequentially outputting addresses for accessing each memory bank_i via a corresponding BSM unit bsm_i; and According to the addresses output by the BSM units, corresponding DMA data are read out from the memory banks bank_i in parallel.

11. The accelerator frame according to claim 10, characterized in that: The non-linear continuous address access engine is further configured with a plurality of data return units corresponding to each of the memory groups, and the steps of reading corresponding storage data from each of the memory groups and merging the corresponding storage data based on element positions include: Performing a logical operation on the depth data read from each of the memory banks bank_i and a mask generated based on the position of the corresponding address offset element in the memory return data to determine the number of offset bits, and performing a bit shift on the depth data, wherein the mask indicates whether the current address offset vector hits the current memory bank; Output the bit shift result to the position of all address offset elements of the corresponding data return unit outVecB_i according to the offset address information; and A merging operation is performed on the bit shift results output to each of the data return units outVecB_i to obtain merged depth image data.

12. The accelerator frame according to claim 9, characterized in that: The SRAM controller is provided with a plurality of pairs of arbitration units corresponding to the memory groups respectively, wherein: The SRAM controller connects the corresponding memory groups to the first read / write port of the system controller, the second read / write port of the visual accelerator, and the third read / write port of the non-linear continuous address access engine via each pair of the arbitration units, so as to arbitrate access requests obtained from the system controller, the visual accelerator, and the non-linear continuous address access engine according to a predefined priority, and perform parallel access to each of the memory groups.

13. The accelerator frame according to claim 12, characterized in that: The SRAM controller represents the SRAM corresponding to each of the memory banks via an array consisting of numBanks SyncReadMem parameters, and represents the arbitration unit corresponding to each of the memory banks via an array consisting of numBanks arbiter_ids.

14. The accelerator frame according to claim 12, characterized in that: The SRAM controller is also configured with a plurality of demultiplexing logic units corresponding to the memory groups respectively, wherein: The SRAM controller connects the corresponding memory groups to the first read-write port of the system controller, the second read-write port of the visual accelerator, and the third read-write port of the non-linear continuous address access engine via each demultiplexing logic unit, so as to demultiplex the output data of each memory group according to the predefined priority and transmit it to the corresponding first read-write port, the second read-write port and / or the third read-write port.

15. The accelerator frame according to claim 14, characterized in that: The demultiplexing logic unit demultiplexes the output data of each of the memory groups using a when-elsewhen statement.

16. A non-linear continuous address access engine, characterized in that: The non-linear continuous address access engine is connected to the visual accelerator and the SRAM respectively, and is configured as follows: Determine, based on a vector read request of a non-contiguous address obtained from the visual accelerator, a sequence access request of a plurality of memory banks in the SRAM to concurrently access each of the memory banks; Reading out corresponding stored data from each of the memory groups, and merging the data based on element positions; as well as The combined DMA data is returned to the vision accelerator.

Images