A universal and configurable image pipeline processing array architecture

Abstract

The present invention belongs to the field of stream processor computing, and discloses a universal and configurable image pipeline processing array architecture, including several pixel processing modules, a program segment storage module, a data segment storage module, a row data storage module, and a lookup table storage module; the present invention solves the shortcomings of poor flexibility of traditional customized image processing modules and excessively high throughput requirements of general multi-core architectures by designing a universal pipelined DSP array architecture suitable for image processing. On this basis, a multi-DSP shared storage architecture is proposed, which can flexibly adapt to the requirements of different pipelines of various image processing services and achieve better throughput and performance.

Description

Technical Field

[0001] The invention belongs to the field of stream processor computing, and in particular relates to a universal and configurable image pipeline processing array architecture. Background Art

[0002] The current field of digital image processing technology mainly processes the pixel array that constitutes the entire image. Many industries and applications use digital image processing technology, such as video, printers, camera imaging processing, medical equipment, etc. Digital image processing has a wide range of applications and diverse needs. The characteristics are relatively high processing accuracy and generally large amounts of data to be processed, so the requirements for processing speed are relatively high and have a certain degree of parallelism; at the same time, due to the continuous iteration and update of the algorithm, it is also required to improve the algorithm software to optimize the processing effect. The existing technology mainly uses pipelines to achieve real-time processing of large amounts of data.

[0003] There are two main types of traditional image processing modules. The first type is to design a dedicated hardware accelerator for proprietary business scenarios and workflows, accelerate the image processing algorithm through a large amount of parallel computing, and design the hardware interconnection architecture for its specific business pipeline to achieve the required performance requirements. This architecture can achieve extremely high processing speeds, and can also save power consumption caused by redundant hardware circuits to a certain extent, but its programmability and configurability are very low. If the algorithm is updated, it cannot be iterated. This method is suitable for scenarios where the algorithm is relatively certain, the possibility of future updates is small, and the performance requirements are extremely high.

[0004] like Figure 1 As shown in the figure, the second case combines the characteristics of the stream processor and configures each processor core to implement different algorithms, or multiple processor cores implement the same algorithm in parallel to meet specific image processing algorithm business requirements. Figure 1 As shown, data is input from the memory to the processor array through the data input control module. Processor 0 implements a specific algorithm according to specific business needs. After processing 1 frame of data, it passes the frame of data to processor 1. Processor 1 implements another algorithm and passes the frame of data to processor 2, and so on. This method can iterate and update the algorithm by continuously iterating the firmware in each processor core, and has a high degree of flexibility. The characteristics of the stream processor architecture can also achieve customized pipelines to achieve different business scenarios. However, this method has a relatively high demand for throughput, and it is difficult to achieve high performance within limited resources; at the same time, since the stream processor architecture is generally a homogeneous multi-core, the depth and number of its pipelines are fixed, and the types of pipelines that can be implemented and the customization requirements of each pipeline are still limited. Summary of the invention

[0005] The purpose of the present invention is to provide a universal and configurable image pipeline processing array architecture to solve the technical problems of poor flexibility of traditional customized image processing modules and excessively high throughput requirements of universal multi-core architectures.

[0006] In order to solve the above technical problems, the specific technical solution of a universal and configurable image pipeline processing array architecture of the present invention is as follows:

[0007] A universal and configurable image pipeline processing array architecture includes a plurality of pixel processing modules, a program segment storage module, a data segment storage module, a row data storage module and a lookup table storage module;

[0008] The pixel processing module contains multiple DSP processor cores to form a stream processing architecture, and different image algorithms can be configured through software. Multiple pixel processing modules can be processed in series or in parallel;

[0009] The program segment storage module is used to store the program segments of each DSP core; it has the function of flexibly configuring the size of each DSP core;

[0010] The data segment storage module is used to store the data segments of each DSP core, and has the function of flexible configuration of the size of each DSP core;

[0011] The lookup table storage module is used to store the lookup table of each DSP core;

[0012] The row data storage module is used to store the first N-1 rows of image data required for window calculation.

[0013] Furthermore, each module adopts a different interconnection mode according to different access requirements; the interconnection mode in the pixel processing module can be configured by software, and each computing node in the pipeline adopts a customized DSP.

[0014] Furthermore, the pixel processing module includes an input-output processing module and a switching matrix. The input-output processing module processes image data flow control, takes out image data from a memory or other source address, and inputs it into a DSP inside the image processing module; or sends data processed by the data processing module to an external memory or other destination address; the switching matrix sends data from a certain input-output processing module to any DSP according to the destination address through address decoding of the data packet, and can also send data processed by a certain DSP to any input-output processing module, i.e., an end-to-end fully interconnected structure.

[0015] Furthermore, the switching matrix is ​​divided into a host and a slave, the host is a sending end, the slave is a receiving end, each host is connected to all slaves, and the address of the sent data packet specifies the destination slave number; the slave receiving interface is divided into a read channel and a write channel, and the read channel and the write channel respectively use a selector to select a specified host from the host interface connected thereto according to the address configured by the software, and receive the data packet.

[0016] Furthermore, the input / output processing module port and the DSP or selector port can both serve as a host or a slave, and the switching matrix is ​​bidirectional, the number of DSPs can be configured in hardware, and are interconnected in a configurable chain form, the output of each DSP can be used as the input of the next-level DSP, connected to the input selector of the next-level DSP, and can also be sent to the switching matrix as a host and output to a certain input / output processing module; at the same time, the input of each DSP can be selected from the output of the previous-level DSP or the switching matrix, that is, a certain input / output processing module, through a selector, and the selector can be configured through software.

[0017] Furthermore, a single-chain structure based on the pipeline formed by the DSP in the pixel processing module can form an n-level image processing pipeline, and each level of the pipeline can perform different image algorithm calculation operations. The architecture can also be configured through software, and the input end selector of one or more DSPs can be configured to select the switching matrix, that is, the output end of a certain input-output processing module, and the output end of one or more DSPs is output to the switching matrix, that is, the input end of a certain input-output processing module. In this way, the single chain of the n-level DSP pipeline can be divided into multiple chains of different lengths or the same length.

[0018] Furthermore, the lookup table storage module is composed of a switching matrix and a storage unit, the storage unit is divided into different blocks, and the switching matrix has the same architecture as the switching matrix of the pixel processing module.

[0019] Furthermore, the row data storage module, program segment storage module and data segment storage module architecture use a semi-switching matrix as the data exchange bus architecture of the two modules; the host end of the semi-switching matrix parses the address through a decoding module and sends a request to the corresponding slave; the slave end uses a selector to select the corresponding host for communication through software configuration address.

[0020] Furthermore, the number of pixel processing modules required and the number of DSPs in each pixel processing module can be configured through hardware; when the number of hardware is fixed, some DSPs can be selected and unnecessary DSPs can be turned off through software configuration.

[0021] The universal and configurable image pipeline processing array architecture of the present invention has the following advantages: the present invention solves the shortcomings of poor flexibility of traditional customized image processing modules and excessively high throughput requirements of general multi-core architectures by designing a universal pipelined DSP array architecture suitable for image processing. On this basis, a multi-DSP shared storage architecture is proposed, which can flexibly adapt to the requirements of different pipelines of various image processing services and achieve better throughput and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of image pipeline processing under the existing homogeneous multi-core architecture;

[0023] Figure 2 This is a general image processing pipeline subsystem architecture diagram of the present invention;

[0024] Figure 3 This is a schematic diagram of the pixel processing module of the present invention;

[0025] Figure 4 The switching matrix architecture diagram of the present invention;

[0026] Figure 5 DSP default single-chain pipeline interconnection structure diagram of the present invention;

[0027] Figure 6 It is a DSP multi-chain pipeline interconnection structure diagram of the present invention;

[0028] Figure 7 It is a diagram of the interconnection structure of the DSP single-chain and multi-chain combined pipeline of the present invention;

[0029] Figure 8 This is a diagram of the architecture of the lookup table / data segment / program segment / row data storage module of the present invention;

[0030] Fig. 9 FIG. 4 is a diagram of a semi-switching matrix architecture of the present invention. DETAILED DESCRIPTION

[0031] In order to better understand the purpose, structure and function of the present invention, a universal and configurable image pipeline processing array architecture of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0032] A universal and configurable image pipeline processing array architecture comprises a pixel processing module, a program segment storage module, a data segment storage module, a lookup table storage module and a row data storage module.

[0033] The pixel processing module contains multiple DSP processor cores to form a stream processing architecture. Different image algorithms can be configured through software. Multiple pixel processing modules can form a longer image processing pipeline in series, or achieve higher data processing performance through parallel processing.

[0034] The program segment storage module is used to store the program segments of each DSP core. Since each DSP may implement different algorithms and different algorithms have different requirements for program segment size, a shared storage module is designed to improve storage utilization. At the same time, the storage size requirements of each DSP core can also be flexibly configured.

[0035] The data segment storage module is used to store the data segments of each DSP core, and also has the function of flexibly configuring the size of each DSP core.

[0036] The lookup table storage module is used to store the lookup table (LUT) of each DSP core. Since some image algorithms do not require LUT, but some image algorithms require a large LUT, it is also set as a shared storage unit, which allows each DSP core to flexibly configure the required LUT space and improve the utilization of storage space.

[0037] The row data storage module is used to store the first N-1 rows of image data required for window calculation. Each algorithm running on the core has different requirements for the size of the window, so the number of rows to be stored is also different. Designing it as a shared storage unit can increase the flexibility and utilization of resources.

[0038] At the same time, the different interconnection technologies used by each module can improve the overall performance of resource scheduling, and different interconnection methods can be adopted for different access requirements.

[0039] The interconnection mode in the pixel processing module can be flexibly configured through software, and different pipeline architectures and communication modes can be implemented according to the needs of different image services, thereby improving the versatility of the overall image processing module. At the same time, each computing node in the pipeline uses a customized DSP, which can configure different firmware for different algorithms, realizing flexible configuration of each algorithm in the pipeline and subsequent update support, which is more flexible and configurable.

[0040] Specifically, Figure 2 As shown, the general image processing pipeline subsystem architecture includes several pixel processing modules, the hardware quantity of which can be configured, a program segment storage module, a data segment storage module, a row data storage module and a lookup table storage module.

[0041] Each pixel processing module contains several DSP cores, the number of which can be configured by hardware, and they are interconnected through a flexible architecture, such as Figure 3As shown. The input and output processing module processes the image data flow control, mainly taking the image data from the memory or other source address and inputting it into the DSP inside the image processing module; or sending the data processed by the data processing module to the external memory or other destination address. Figure 3 As shown, the switching matrix decodes the address of the data packet and sends the data of a certain input / output processing module to any DSP according to the destination address. At the same time, it can send the data processed by a certain DSP to any input / output processing module, that is, an end-to-end fully interconnected structure.

[0042] like Figure 4 As shown in the figure, the switching matrix is ​​divided into a host and a slave. The host is the transmitter and the slave is the receiver. Each host is connected to all slaves. The address of the data packet sent specifies the destination slave number. On the slave side, the receiving interface is divided into a read channel and a write channel. The read and write channels use a selector to select a specified host from the host interface connected to it according to the address configured by the software and receive the data packet. Figure 3 In the architecture shown, the input / output processing module ports and DSP (or selector) ports can be used as hosts or slaves, and the switching matrix is ​​bidirectional. The number of DSPs can be configured in hardware and interconnected in a configurable chain form, mainly in that the output of each DSP can be used as the input of the next-level DSP and connected to the input selector of the next-level DSP, and can also be sent to the switching matrix as a host and output to a certain input / output processing module; at the same time, the input of each DSP can be selected from the output of the previous-level DSP or the switching matrix, that is, a certain input / output processing module, through the selector, which can be configured through software.

[0043] The interconnection architecture is very flexible. Figure 5 As shown in the figure, under the default configuration, the pipeline formed by the DSP in a pixel processing module is a single chain structure based on the pipeline, which can form an n-level image processing pipeline, and each level of the pipeline can perform different image algorithm calculation operations. The architecture can also be configured through software, and the input selector of one or more DSPs is configured to select the switching matrix, that is, the output of a certain input and output processing module, and output the output of one or more DSPs to the switching matrix, that is, the input of a certain input and output processing module. In this way, the single chain of the n-level DSP pipeline can be divided into multiple chains of different lengths or the same length, such as Figure 6 As shown, k, l, and m are all configurable and less than n. The multi-chain pipeline can also be combined in various ways to form an inflexible and diverse pipeline architecture, such as Figure 7 As shown in the figure, a single chain is combined with two parallel single chains to form a serial-parallel pipeline structure. This structure is very flexible and adaptable, and is suitable for the needs of the current image processing business and the continuous iteration and update of algorithms.

[0044] Specifically, the architecture of the lookup table storage module is as follows: Figure 8 As shown, it consists of a switching matrix and a storage unit, where the storage unit is divided into different blocks. Figure 4 Same architecture as in .

[0045] Similarly, the architecture of the row data storage module, program segment storage module and data segment storage module is as follows Figure 8 As shown in FIG. 1 , a semi-switching matrix is ​​used as the data exchange bus architecture of the two modules. The architecture of the semi-switching matrix is ​​shown in FIG. Fig. 9 As shown, the host side parses the address through the decoding module and sends a request to the corresponding slave; the slave side uses a selector to select the corresponding host for communication through software configuration address.

[0046] For this general image processing architecture, the number of pixel processing modules and the number of DSPs in each pixel processing module can be configured by hardware. When the number of hardware is fixed, some DSPs can be selected and unnecessary DSPs can be turned off by software configuration.

[0047] First, according to the needs of the algorithm pipeline, the software configures the interconnection method in each pixel processing module and configures it into the pipeline form required by the business scenario or demand. After the pipeline is configured, the corresponding algorithm in each DSP is configured according to the pipeline requirements and written into the program segment storage module. Each DSP will load the algorithm program from its corresponding storage space in the program segment storage module. Among them, the algorithm program can access the data segment / row data / lookup table storage module through different instructions, and through reasonable space allocation in the early stage, different space sizes are allocated for different needs of each DSP algorithm. At the same time, the actual and end addresses of the corresponding allocated space can be modified by software, so that the space allocated to each DSP is dynamically adjustable.

[0048] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A universal and configurable image pipeline processing array architecture, comprising a plurality of pixel processing modules, a program segment storage module, a data segment storage module, a row data storage module and a lookup table storage module; characterized in that: The pixel processing module contains multiple DSP processor cores to form a stream processing architecture, and different image algorithms can be configured through software. Multiple pixel processing modules can be processed in series or in parallel; The pixel processing module includes an input-output processing module and a switching matrix. The input-output processing module processes image data flow control, takes image data from a memory or other source address, and inputs it to a DSP inside the image processing module; Or send the data processed by the data processing module to the external memory or other destination address; the switching matrix decodes the address of the data packet, sends the data of a certain input and output processing module to any DSP according to the destination address, and can also send the data processed by a certain DSP to any input and output processing module, that is, an end-to-end fully interconnected structure; The program segment storage module is used to store the program segments of each DSP core; it has the function of flexibly configuring the size of each DSP core; The data segment storage module is used to store the data segments of each DSP core, and has the function of flexible configuration of the size of each DSP core; The lookup table storage module is used to store the lookup table of each DSP core; The row data storage module is used to store the first N-1 rows of image data required for window calculation.

2. The universal configurable image pipeline processing array architecture according to claim 1, characterized in that: Each module adopts a different interconnection mode according to different access requirements; the interconnection mode in the pixel processing module can be configured by software, and each computing node in the pipeline adopts a customized DSP.

3. The universal configurable image pipeline processing array architecture according to claim 1, characterized in that: The switching matrix is ​​divided into a host and a slave, the host is a sending end, and the slave is a receiving end. Each host is connected to all slaves, and the address of the sent data packet specifies the destination slave number; the slave receiving interface is divided into a read channel and a write channel, and the read channel and the write channel use a selector respectively, and select the specified host from the host interface connected to it according to the address configured by the software, and receive the data packet.

4. The universal configurable image pipeline processing array architecture according to claim 3, characterized in that: The input-output processing module port and the DSP or selector port can both serve as a host or a slave, and the switching matrix is ​​bidirectional, the number of DSPs can be configured by hardware, and they are interconnected in a configurable chain form, the output of each DSP can be used as the input of the next-level DSP and connected to the input selector of the next-level DSP, and can also be sent to the switching matrix as a host and output to a certain input-output processing module; at the same time, the input of each DSP can be selected from the output of the previous-level DSP or the switching matrix through the selector, that is, a certain input-output processing module, and the selector can be configured through software.

5. The universal configurable image pipeline processing array architecture according to claim 4, characterized in that: A single chain structure based on the pipeline formed by the DSP in the pixel processing module can form an n-level image processing pipeline, and each level of the pipeline can perform different image algorithm calculation operations. The architecture can also be configured through software, and the input end selector of one or more DSPs is configured to select the switching matrix, that is, the output end of a certain input-output processing module, and the output end of one or more DSPs is output to the switching matrix, that is, the input end of a certain input-output processing module. In this way, the single chain of the n-level DSP pipeline can be divided into multiple chains of different lengths or the same length.

6. The universal configurable image pipeline processing array architecture according to claim 5, characterized in that: The lookup table storage module is composed of a switching matrix and a storage unit, the storage unit is divided into different blocks, and the switching matrix has the same architecture as the switching matrix of the pixel processing module.

7. The universal configurable image pipeline processing array architecture according to claim 6, characterized in that: The row data storage module, program segment storage module and data segment storage module architecture use a semi-switching matrix as the data exchange bus architecture of the two modules; the host end of the semi-switching matrix parses the address through the decoding module and sends a request to the corresponding slave; the slave end uses a selector to select the corresponding host for communication through software configuration address.

8. The universal configurable image pipeline processing array architecture according to claim 7, characterized in that: The number of pixel processing modules required and the number of DSPs in each pixel processing module can be configured through hardware; when the number of hardware is fixed, some DSPs can be selected and unnecessary DSPs can be turned off through software configuration.

Images