Image processing device and image processing method

Abstract

The embodiment of the present application discloses an image processing device and an image processing method, wherein the image processing device of the embodiment of the present application includes a data reorganization circuit, a memory and an intelligent processor, which is used to filter image data based on a set of filter data. The data reorganization circuit performs data reorganization on a plurality of image blocks in the image data to generate a first reorganized data, and performs data reorganization on the set of filter data to generate a second reorganized data. The memory stores the first reorganized data and the second reorganized data. The intelligent processor performs a depth convolution operation based on the first reorganized data and the second reorganized data to generate a feature map data. Wherein, two adjacent image blocks in the plurality of image blocks have partially identical data, and the number of a row of the second reorganized data is related to the number of channels of the depth convolution operation. The present application can improve the processing speed of image processing.

Description

Technical Field

[0001] The present application relates to an image processing device, and more particularly to an image processing device and an image processing method that utilize reorganized data to accelerate image processing. Background Art

[0002] In the prior art, image filtering is achieved by using a processor (or image signal processor) to perform a large number of multiplication and addition operations. Because these processors can only complete one multiplication or addition operation per cycle, if the input image data size or the filter mask is too large, the processor will require a considerable number of cycles to complete the image filtering, reducing the overall image processing efficiency. Summary of the Invention

[0003] The present application proposes an image processing device and an image processing method, which utilize reorganized data to accelerate image processing.

[0004] In some embodiments, an image processing device includes a data reorganization circuit, a memory, and an intelligent processor for filtering image data based on a set of filtering data. The data reorganization circuit reorganizes a plurality of image blocks in the image data to generate first reorganized data, and reorganizes the set of filtering data to generate second reorganized data. The memory stores the first reorganized data and the second reorganized data. The intelligent processor performs a depthwise convolution operation based on the first reorganized data and the second reorganized data to generate feature map data. Two adjacent image blocks in the plurality of image blocks have some identical data, and the number of rows of the second reorganized data is related to the number of channels in the depthwise convolution operation.

[0005] In some embodiments, an image processing method for filtering image data based on a set of filtering data includes the following operations: reorganizing a plurality of image blocks in the image data to generate first reorganized data; reorganizing the set of filtering data to generate second reorganized data; utilizing an intelligent processor to perform a depthwise convolution operation based on the first reorganized data and the second reorganized data to generate feature map data; and reorganizing the feature map data to generate output image data. Two adjacent image blocks in the plurality of image blocks have some identical data, and the number of rows in the second reorganized data is related to the number of channels in the depthwise convolution operation.

[0006] In some embodiments of the present application, the image processing devices and methods can utilize data recombining to generate an image data format suitable for parallel processing. This allows the use of an intelligent processor to increase the speed of image processing (e.g., image filtering) and thus enhance the overall performance of the image processing system.

[0007] The features, implementation and effects of the present application are described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0009] Figure 1 A schematic diagram of an image processing system according to some embodiments of the present application;

[0010] Figure 2 A flowchart of an image processing method according to some embodiments of the present application;

[0011] Figure 3A According to some embodiments of the present application Figure 1 Schematic diagram of input image data, filter mask and output image data in ;

[0012] Figure 3B According to some embodiments of the present application, Figure 1 Schematic diagram of combining the input image data in into recombined data;

[0013] Figure 3C According to some embodiments of the present application Figure 1 Schematic diagram of reconstructed data generated by filtering mask;

[0014] Figure 3D According to some embodiments of the present application Figure 1 A schematic diagram of generating output feature map data based on a plurality of reorganized data D21; and

[0015] Figure 3E According to some embodiments of the present application, Figure 1 The output feature map data is arranged as a schematic diagram of output image data. DETAILED DESCRIPTION

[0016] All terms used herein have their ordinary meanings. The definitions of the above terms in commonly used dictionaries, including any examples of the use of any term discussed herein, are intended for illustrative purposes only and should not limit the scope and meaning of this application. Similarly, this application is not limited to the various embodiments described herein.

[0017] As used herein, "coupled" or "connected" may refer to direct physical or electrical contact between two or more components, or indirect physical or electrical contact between two or more components, or to the mutual operation or action of two or more components. As used herein, the term "circuit" may refer to a device comprised of at least one transistor and / or at least one active or passive component connected in a specific manner to process signals.

[0018] Figure 1 The image processing system 100 is a schematic diagram drawn according to some embodiments of the present application. The image processing system 100 includes a processor 110, a memory 120, an image signal processor 130, and an image processing device 140. In some embodiments, the image signal processor 130 and the intelligent processor (Intelligence Processor) 141 in the image processing device 140 perform different image processing or support different image processing methods. For example, the image signal processor 130 can perform image processing such as color processing, geometric transformation, and image enhancement, while the intelligent processor 141 can perform image processing such as (but not limited to) image noise reduction and Gaussian filtering. On the other hand, the image signal processor 130 can perform image processing in a sequential manner, while the intelligent processor 141 can process image data in a parallel manner. For example, the image signal processor 130 processes image data pixel by pixel or row by row, performing one operation on one piece of data per cycle, while the intelligent processor 141 performs filtering processing on the image data by performing parallel operations on multiple pieces of data simultaneously per cycle. In some embodiments, the image processing device 140 can function as an accelerator for the image signal processor 130, thereby accelerating the efficiency of filtering. In some embodiments, the image processing device 140 and the image signal processor 130 can be integrated into a single circuit system. In some embodiments, the image signal processor 130 performs image processing on input image data DI to generate image data DM, and the image processing device 140 then performs filtering on the image data DM.

[0019] The processor 110 may be used to control the operation of the image signal processor 130 and / or the image processing device 140, and provide the image signal processor 130 and / or the image processing device 140 with data and / or parameters (e.g., the plurality of width and height information described below) required for image processing. The memory 120 may be used to store input image data DI, image data DM, and output image data DO for use by the image processor 130 and / or the image processing device 140 in image processing. In some embodiments, the memory 120 may be a dynamic random access memory.

[0020] Image processing device 140 includes an intelligent processor 141, a data reconstruction circuit 142, and a memory 143. Intelligent processor 141 may be implemented by an artificial intelligence circuit or a neural network circuit to perform image filtering on image data DM to generate feature map data DF. For example, intelligent processor 141 may perform a depthwise convolution operation on image data DM and a filter mask FM to filter image data DM and generate feature map data DF. In some embodiments, filter mask FM is a matrix mask composed of a set of filter data, each of which includes filter coefficients / parameters corresponding to each position in filter mask FM.

[0021] In some embodiments, the data reassembly circuit 142 may be implemented by (but not limited to) a direct memory access (DMA) controller, and the memory 143 may be implemented by static random access memory (SRAM). The data reassembly circuit 142 may reassemble the data of multiple image blocks in the image data DM into reassembled data D21 and reassemble the data of the filter mask FM into reassembled data D22. In a depthwise convolution operation, each convolution kernel corresponds to a channel, and each channel performs a convolution operation independently. Therefore, the data reassembly circuit 142 can convert the image data DM and the filter mask FM into a data format suitable for parallel processing. For example, in this example, the data reassembly circuit 142 converts the image data DM and the filter mask FM into the data format required for the depthwise convolution operation. In this way, the intelligent processor 141 can utilize multiple channels to process the reassembled data D21 and the reassembled data D22 in parallel, more efficiently performing the depthwise convolution operation, thereby implementing the filtering process by the intelligent processor 141.

[0022] Furthermore, the data reconstruction circuit 142 further arranges the feature map data DF into output image data DO (i.e., image data after filtering) and stores the output image data DO in the memory 120. The memory 143 can be used to store the reconstructed data D21, reconstructed data D22, and feature map data DF generated during the aforementioned data combination process. The operation of the data reconstruction circuit 142 will be described below.

[0023] Figure 2 FIG. 2 is a flow chart of an image processing method 200 according to some embodiments of the present application. In some embodiments, the image processing method 200 may be Figure 1 For ease of description, the related operations of the image processing system 100 will be combined with Figure 2 Explain together.

[0024] In step S201, the number of channels supported by the depthwise convolution operation performed by the intelligent processor is determined. For example, through pre-configuration, the processor 110 may obtain the relevant specifications, the model being run, and / or relevant parameters of the intelligent processor 141, thereby determining or setting the number of channels for the depthwise convolution operation (hereinafter referred to as the number of channels C).

[0025] In step S202, the size of the output image data (eg, output image data DO) after image filtering is determined. In step S203, the size of a unit of data is confirmed.

[0026] For explanation of step S202 and step S203, please refer to Figure 3A . Figure 3A According to some embodiments of the present application Figure 1 Schematic diagram of image data DM, filter mask FM, and output image data DO in FIG. In this embodiment, the image data DM has a width of W1 and a height of H1. The filter mask FM is a mask having a width of f_w and a height of f_h. During the filtering operation, the filter mask FM is sequentially moved on the image data DM to perform multiplication and addition operations. Assuming that the filter mask FM is moved by 1 pixel each time, the processor 110 can determine the width W2 and height H2 of the output image data DO using the following formula:

[0027] W2=W1-f_w+1

[0028] H2=H1-f_h+1

[0029] For example, the width W1 and the height H1 are both 1000 (in pixels), and the width f_w and the height f_h are both 9. Based on the above formula, the width W2 and the height H2 can be deduced to be 992.

[0030] Next, the processor 110 can simulate various different ways of cutting the output image data DO to find the appropriate width tile_w2 and height tile_h2 of the unit data. In some embodiments, the width tile_w2 and the height tile_h2 meet two conditions. The first condition is that the total amount of data generated by the data reassembly circuit 142 does not exceed the capacity of the memory 143, which can be expressed as follows:

[0031] tile_w1×tile_h1×C+f_h×f_w×C+tile_w2×tile_h2×C <MS

[0032] Wherein C is the aforementioned number of channels, tile_w1 and tile_h1 are the width and height of the image block data respectively, and MS is the capacity of the memory 143 .

[0033] The second condition is to ensure that the intelligent processor 141 can process the reorganized data D21 in the most efficient manner. This condition can be expressed as minimizing the result of the following equation:

[0034] tile_w1×tile_h1×C×n

[0035] Where n is the number of operations required by the intelligent processor 141 when performing image filtering. In some embodiments, the processor 110 may determine the possible value ranges of width tile_w2 and height tile_h2 using the first condition, and then use integer programming to calculate the possible value ranges of width tile_w1, height tile_h1, and the number of times n. In this way, the processor 110 may use the possible value ranges of these parameters and the second condition to perform a trial and error method (or other suitable algorithm) to find suitable width tile_w1, height tile_h1, width tile_w2, height tile_h2, and the number of times n.

[0036] In some embodiments, the aforementioned operations of calculating the width tile_w1, the height tile_h1, and the number n can be expressed as the following formula, where the function ceil() is a ceiling function:

[0037] tile_w1=tile_w2-1+f_w

[0038] tile_h1=tile_h2-1+f_h

[0039]

[0040] exist Figure 3A In the example, the width tile_w1 and the height tile_h1 are both 6, and the width tile_w2 and the height tile_h2 are both 4. The values ​​of the above parameters are for example only, and the present application is not limited to the above values. In some embodiments, the above parameters can be stored in a register in advance, and the processor 110 can read the register to obtain these parameters.

[0041] Continue to refer to Figure 2 In step S204, the image data is divided into a plurality of image block data, wherein two adjacent image block data in the plurality of image block data have some identical data. In step S205, the plurality of image block data are combined into reconstructed data (eg, reconstructed data D21).

[0042] For explanation of step S204 and step S205, please refer to Figure 3B . Figure 3B According to some embodiments of the present application, Figure 1Schematic diagram of combining the image data DM in the image data into reconstructed data D21. As previously described, steps S201 to S203 provide information on width tile_w1 and height tile_h1. Thus, processor 110 can sequentially split the image data DM into a plurality of image block data based on width tile_w1 and height tile_h1, where each image block data has a width tile_w1 and a height tile_h1, respectively.

[0043] For ease of understanding, Figure 3B Only four image block data D1 to D4 are shown. Figure 3B As shown, image data DM includes K pixel values ​​00, 01, ..., 99, ...K. Image block data D1 includes a plurality of pixel values ​​00-05, 10-15, 20-25, 30-35, 40-45, and 50-55. Image block data D2 includes a plurality of pixel values ​​04-09, 14-19, 24-29, 34-39, 44-49, and 54-59. Image block data D3 includes a plurality of pixel values ​​40-45, 50-55, 60-65, 70-75, 80-85, and 90-95. Image block data D4 includes a plurality of pixel values ​​44-49, 54-59, 64-69, 74-79, 84-89, and 64-99. Because image filtering is performed based on multiple neighboring pixel values ​​within a certain area (e.g., a filter mask FM), to meet the requirements of image filtering, two adjacent image blocks D1-D4 overlap, thereby sharing some of the same data. In this embodiment, two adjacent image blocks D1-D4 overlap vertically by (f_h-1) columns of data and horizontally by (f_w-1) rows of data. For example, the rightmost two rows of data in image block D1 (i.e., the pixel values ​​04-05, 14-15, 24-25, 34-35, 44-45, and 54-55) are identical to the leftmost two rows of data in image block D2. The bottom two rows of data in image block D1 (i.e., the pixel values ​​40-45 and 50-55) are identical to the top two rows of data in image block D3.

[0044] Next, the data reassembly circuit 142 may reassemble the plurality of image block data into reassembled data D21. For example, the plurality of image block data D1 to D4 are sequentially arranged into a plurality of row data DC1 to DC4 of the reassembled data D21. In other words, the plurality of pixel values ​​in the row data DC1 are the same as the plurality of pixel values ​​in the image block data D1, and these plurality of pixel values ​​are re-arranged into a single row of data. Similarly, the data reassembly circuit 142 may reassemble the plurality of image block data into reassembled data D21 having a plurality of rows and a plurality of columns, and store the reassembled data D21 in the memory 143. In some embodiments, the reassembled data D21 is as follows: Figure 3B The two-dimensional arrangement of the reorganized data D21 is stored in memory 143. In some embodiments, the number of rows in the reorganized data D21 is determined based on the aforementioned number of channels C, and the number of columns in the reorganized data D21 is the product of the width tile_w1 and the height tile_h1 (i.e., tile_w1×tile_h1). For example, the number of rows in the reorganized data D21 is equal to the number of channels C, allowing the intelligent processor 141 to fully utilize all channels of the depthwise convolution operation when performing a round of depthwise convolution operation.

[0045] Continue to refer to Figure 2 In step S206 , the filter mask data is rearranged into a row of data, and the row of data is reused to form reorganized data (eg, reorganized data D22 ).

[0046] To explain step S206, please refer to Figure 3C . Figure 3C According to some embodiments of the present application Figure 1 Schematic diagram of generating reconstructed data D22 by using a filter mask FM. In this embodiment, the filter mask FM is a mask having 9 element values ​​1 to 9. The data reconstructing circuit 142 can rearrange the multiple element values ​​1 to 9 in the filter mask FM into a row of data FC. Then, the data reconstructing circuit 142 can repeatedly copy the row data FC to generate multiple row data FC, and combine the multiple row data FC into reconstructed data D22, and store the reconstructed data D22 in the memory 143. In some embodiments, the reconstructed data D22 is as follows Figure 3C The two-dimensional arrangement is stored in memory 143. In some embodiments, the number of rows in the reorganized data D22 corresponds to the number of rows in the reorganized data D21. The number of rows in the reorganized data D22 and the number of rows in the reorganized data D21 can both be determined based on the aforementioned number of channels C. For example, the number of rows in the reorganized data D22 is the same as the number of channels C.

[0047] Continue to refer to Figure 2 In step S207, the intelligent processor performs a deep convolution operation based on the plurality of reconstructed data to generate feature map data.

[0048] To explain step S207, please refer to Figure 3D . Figure 3D According to some embodiments of the present application Figure 1 Schematic diagram of generating feature map data DF from reorganized data D21 and reorganized data D22. The intelligent processor 141 can process the reorganized data D21 and reorganized data D22 in parallel to perform a depthwise convolution operation. For example, corresponding to each channel of the depthwise convolution operation, the intelligent processor 141 can read a corresponding row of data in the reorganized data D21 and a corresponding row of data in the reorganized data D22 to perform a convolution operation, thereby generating a corresponding row of data in the feature map data DF. For example, corresponding to the first channel of the depthwise convolution operation, the intelligent processor 141 can perform a convolution operation on the first row of data DC1 in the reorganized data D21 and the first row of data FC in the reorganized data D22 to generate row data MC1. Corresponding to the second channel of the depthwise convolution operation, the intelligent processor 141 can perform a convolution operation on the second row of data DC2 in the reorganized data D21 and the second row of data FC in the reorganized data D22 to generate row data MC2. Similarly, it should be understood that the intelligent processor 141 can perform multiple convolution operations in the same cycle. In this way, the intelligent processor 141 can perform image filtering more efficiently to generate feature map data DF, and store the feature map data DF in the memory 143. Figure 3D As shown, the number of columns of the feature map data DF is the product of the width tile_w2 and the height tile_h2 (ie, tile_w2×tile_h2), and the number of rows of the feature map data DF is the same as the number of channels C.

[0049] Continue to refer to Figure 2 In step S208, the feature map data is arranged as output image data. To illustrate step S208, please refer to Figure 3E . Figure 3E According to some embodiments of the present application, Figure 1 FIG1 is a schematic diagram illustrating how feature map data DF is arranged into output image data DO. Data reassembly circuit 142 reassembles multiple rows of data (e.g., row data MC1-MC4) in the feature map data into multiple unit data (e.g., unit data UD1-UD4) based on width tile_w2 and height tile_h2. The reassembly circuit 142 then combines these multiple unit data into output image data DO, which is then stored in memory 120. In this manner, image signal processor 130 can continue to process or use output image data DO.

[0050] Specifically, row data MC1 includes a plurality of pixel values ​​00′-03′, 10′-13′, 20′-23′, and 30′-33′. The data reassembly circuit 142 may arrange the plurality of pixel values ​​00′-03′, 10′-13′, 20′-23′, and 30′-33′ according to an image format having a width tile_w2 and a height tile_h2 to generate unit data UD1. Row data MC2 includes a plurality of pixel values ​​04′-07′, 14′-17′, 24′-27′, and 34′-37′. The data reassembly circuit 142 may arrange the plurality of pixel values ​​04′-07′, 14′-17′, 24′-27′, and 34′-37′ according to an image format having a width tile_w2 and a height tile_h2 to generate unit data UD2. By analogy, we can understand the corresponding relationship between the multiple pixel values ​​40'-43', 50'-53', 60'-63' and 70'-73' in the row data MC3 and the unit data UD3, and the corresponding relationship between the multiple pixel values ​​44'-47', 54'-57', 64'-67' and 74'-77' in the row data MC4 and the unit data UD4. Figure 3E It can be seen that, in some embodiments, the data reorganization circuit 142 reorganizes the feature map data DF in a manner corresponding to the data reorganization circuit 142 reorganizes the plurality of image block data D1 - D4 .

[0051] Continue to refer to Figure 2 In step S209, steps S205, S207, and S208 are repeatedly executed to complete the image filtering. For example, the data reassembly circuit 142 and the intelligent processor 141 may repeatedly execute steps S205, S207, and S208 (e.g., n times) until the complete image data DM is decomposed to form the complete output image data DO.

[0052] The operations of the image processing method 200 are merely examples and are not necessarily performed in the order shown. Without departing from the operational manner and scope of the various embodiments of the present application, the various operations of the image processing method 200 may be appropriately added, replaced, omitted, or performed in a different order (e.g., simultaneously or partially simultaneously).

[0053] In summary, the image processing devices and methods of some embodiments of the present application utilize data recombining to generate an image data format suitable for parallel processing. Furthermore, the image processing system includes both an image signal processor and an intelligent processor. Sequential image processing can be performed by the image signal processor, while parallel image processing can be performed by the intelligent processor. This allows the intelligent processor to increase the speed of image processing (e.g., image filtering) and thus enhance the overall performance of the image processing system.

[0054] Although the embodiments of the present application are described above, these embodiments are not intended to limit the present application. A person having ordinary knowledge in the technical field may make changes to the technical features of the present application based on the explicit or implicit content of the present application. All such changes may fall within the scope of the patent protection sought by the present application. In other words, the scope of patent protection of the present application shall be subject to the scope of the patent application defined in this specification.

[0055] Explanation of symbols:

[0056] 00~99,00'~77': pixel value

[0057] 1 to 9: element value

[0058] 100: Image processing system

[0059] 110: Processor

[0060] 120: Memory

[0061] 130: Image Signal Processor

[0062] 140: Image processing device

[0063] 141: Intelligent Processor

[0064] 142: Data Reconstruction Circuit

[0065] 143: Memory

[0066] 200: Image Processing Methods

[0067] C: Number of channels

[0068] D1~D4: Image block data

[0069] D21, D22: Reorganized data

[0070] DC1~DC4,FC,MC1~MC4: row data

[0071] DF: Feature map data

[0072] DI: input image data

[0073] DO: Output image data

[0074] DM: Image data

[0075] FC: Row data

[0076] FM: Filter Mask

[0077] f_h,H1,H2,tile_h1,tile_h2: height

[0078] f_w,W1,W2,tile_w1,tile_w2: width

[0079] S201~S209: Steps

[0080] UD1~UD4: unit data

Claims

1. An image processing device, characterized in that The method is used for performing filtering processing on image data according to a set of filtering data, comprising: a data reorganization circuit for performing data reorganization on a plurality of image blocks in the image data to generate first reorganized data, and for performing data reorganization on the set of filtered data to generate second reorganized data; wherein the data reorganization circuit rearranges the set of filtered data into a row of data and reuses the row of data to generate the second reorganized data; and the data reorganization circuit arranges each of the plurality of image blocks into a row of data in the first reorganized data; a memory for storing the first reorganized data and the second reorganized data; and an intelligent processor that performs a depthwise convolution operation based on the first reorganized data and the second reorganized data to generate feature map data; Two adjacent image blocks in the plurality of image blocks have partially identical data, and the number of a row of the second reorganized data is related to the number of channels of the depthwise convolution operation.

2. The image processing device according to claim 1, wherein The number of rows of the second reorganized data is equal to the number of channels of the depthwise convolution operation.

3. The image processing device according to claim 1, wherein Also includes: an image signal processor, performing image processing on input image data to generate the image data; The image signal processor performs image processing on the input image data in a sequential manner, and the intelligent processor performs the depthwise convolution operation in a parallel manner.

4. The image processing device according to claim 1, wherein The data reorganization circuit reorganizes the feature map data to generate output image data, wherein the reorganization method of the feature map data performed by the data reorganization circuit corresponds to the reorganization method of the plurality of image blocks performed by the data reorganization circuit.

5. The image processing device according to claim 1, wherein The sizes of the plurality of image blocks are determined according to the space size of the memory and the number of channels.

6. An image processing method, characterized in that: The method is used for performing filtering processing on image data according to a set of filtering data, comprising: performing data reorganization on a plurality of image blocks in the image data to generate first reorganized data; performing data reorganization on the set of filtered data to generate second reorganized data; wherein, in the data reorganization on the set of filtered data, the set of filtered data is rearranged into a row of data, and the row of data is reused to generate the second reorganized data; wherein, performing data reorganization on a plurality of image blocks in the image data to generate first reorganized data includes arranging each of the plurality of image blocks into a row of data in the first reorganized data; Utilizing an intelligent processor, performing a depthwise convolution operation based on the first reorganized data and the second reorganized data to generate feature map data; and Recombining the feature map data to generate output image data; Two adjacent image blocks in the plurality of image blocks have partially identical data, and the number of a row of the second reorganized data is related to the number of channels of the depthwise convolution operation.

7. The image processing method according to claim 6, wherein: The number of rows of the second reorganized data is equal to the number of channels of the depthwise convolution operation.

8. The image processing method according to claim 6, wherein: Also includes: Using an image signal processor, performing image processing on input image data to generate the image data; The image signal processor performs image processing on the input image data in a sequential manner, and the intelligent processor performs the depthwise convolution operation in a parallel manner.

9. The image processing method according to claim 6, wherein: Also includes: storing the first reorganized data and the second reorganized data in a memory; The sizes of the plurality of image blocks are determined according to the space size of the memory and the number of channels.

Images