A general device and method for rearranging image pixels

Abstract

The present invention discloses a general device and method for rearranging image pixels. The device includes a general control unit, a data storage unit, a data reading unit, a pixel rearrangement unit and a result collection unit, wherein the general control unit enables each internal unit to work in coordination by controlling the working state of each unit and configuring data; the data storage unit is responsible for storing the original image data and storing the rearranged image data; the pixel rearrangement unit (PixelShuffle unit) receives the data from the data reading unit, performs pixel rearrangement operation, and transmits the rearranged data to the result collection unit; the result collection unit receives the rearranged data from the pixel rearrangement unit, transmits the data to the data storage unit, and finally obtains a spliced ​​high-resolution image in the data storage unit. The present invention realizes reconstruction from low resolution to high resolution by pixel rearrangement, can process images, and has certain versatility.

Description

Technical Field

[0001] The invention relates to a method for obtaining a high-resolution image, and in particular to a general device and method for rearranging image pixels, belonging to the field of digital image processing. Background Art

[0002] Pixel shuffle is the core concept in the effective sub-pixel convolutional neural network ESPCN (Efficient Sub-Pixel Convolutional Neural Network). The input of the network is the original low-resolution image, which is passed through three convolutional layers to obtain the feature image, and then rearranged to obtain a high-resolution image. In the ESPCN network, the interpolation function of the image size enlargement process is implicitly included in the previous convolutional layer and can be automatically learned. Since the convolution operation is performed on the size of the low-resolution image, the efficiency will be higher. Some other image processing methods, such as the SRCNN (Super-Resolution Convolutional Neural Network) method, need to upsample and interpolate the low-resolution image to the same size as the high-resolution image, and then input it into the network, which means that the convolution operation must be performed at a higher resolution, thereby increasing the computational complexity.

[0003] To this end, the convolutional neural network structure of ESPCN with pixel rearrangement function is used to extract feature maps in the low-resolution LR (Low Resolution) space. After an effective sub-pixel convolution layer, it learns a set of scale expansion filters to enlarge the final low-resolution LR (Low Resolution) feature map to the high-resolution HR (High Resolution) output. Under normal circumstances, the convolution operation will reduce the height and width of the feature image. However, when stride (step length) = r1 <1, the height and width of the feature image after convolution can be increased, that is, the resolution is increased. This operation is called sub-pixel convolution, also known as pixel rearrangement PixelShuffle.

[0004] In image super-resolution and image enhancement algorithms, the process of upsampling and downsampling feature maps is required to solve some artificial traces and chessboard effects of interpolation and deconvolution. High-resolution images mean that the images have more detailed information and more delicate picture quality. These details have important application value in high-definition television, medical imaging, remote sensing satellite imaging and other fields. Summary of the invention

[0005] In order to obtain a faster operation speed, the present invention proposes a general device and method for rearranging image pixels in view of the processing mode of feature images in a neural network.

[0006] The technical solution of the device of the present invention is as follows:

[0007] A general device for rearranging image pixels, the device comprising:

[0008] The main control unit is used to control the working status and configuration data of each unit;

[0009] A data storage unit, used to store original image data in the order of image blocks, and to store rearranged image data;

[0010] A data reading unit, used for reading data from the data storage unit and transmitting the data to the pixel rearrangement unit;

[0011] A pixel rearrangement unit, used to receive data from the data reading unit, perform pixel rearrangement operation, rearrange the pixel position sequence of the image block, and transmit the rearranged data to the result collection unit;

[0012] The result collection unit is used to receive the data rearranged by the pixel rearrangement unit and transmit the data to the data storage unit.

[0013] Furthermore, the general control unit includes a status register and a configuration register, the status register is used to control the working status of each unit; the configuration register is used to cache configuration information, including the number of blocks of the input image, the starting storage address of the input image, the size of each block of the image, the starting address and reading length of the pixel rearrangement unit reading data from the data reading unit, the address of the pixel rearrangement unit writing data to the result collection unit, and the storage address of the output image.

[0014] Furthermore, the data storage unit includes a DDR memory; when the general control unit is in a state where the data storage unit transfers data to the data reading unit, the data reading unit issues a write request, and the data storage unit transfers the image data to the data reading unit; when the general control unit is in a state where the rearranged data is transferred to the data storage unit, the data storage unit issues a write request, and the result collection unit transfers the data to the data storage unit.

[0015] Furthermore, the data reading unit includes a RAM memory; the RAM memory stores data in the order of row-channel-column, that is, a row of each image block is stored first, and then the next channel of a row is stored. After all channels of a row are stored, the next row is stored until the data of one image block is stored.

[0016] Furthermore, the pixel rearrangement unit includes: a row register for storing data output by the data reading unit; a channel register for counting the depth of data placed in the row register; and a result register for storing the result after the row register is rearranged.

[0017] Furthermore, the result collection unit includes a RAM memory; when the general control unit is in a state where the data storage unit receives data from the pixel rearrangement unit, the result collection unit writes the data output by the pixel rearrangement unit into a corresponding address according to the address data in the configuration information; when the general control unit is in a state where the rearranged data is moved to the data storage unit, the data is moved from the result collection unit to the corresponding address of the data storage unit according to the address information of the output image.

[0018] The present invention also provides a general method for rearranging image pixels, the specific steps of which include:

[0019] (1) The original image size is r rows by c columns, the number of channels is h by h, and the size of each pixel is 8 bits;

[0020] (2) The original image is divided into m*n blocks, where the row direction is divided into m blocks and the column direction is divided into n blocks; each block is assumed to have x rows and y columns, and the number of channels is h; the number of rows and columns in each block is not necessarily equal;

[0021] (3) In each block, the image of the h channels of a pixel is converted into an image block of a corresponding row and h columns; then the image of the h channels of the next pixel in a row is converted into an image block of a corresponding row and h columns, and is spliced ​​with the image block of a row and h columns converted from the previous pixel, until a row of h channel data is spliced, that is, the h channels of a row of y pixels are converted into an image block of a corresponding row of y*h;

[0022] (4) converting the pixel points of all rows of h channels into corresponding image blocks according to the method of step (3);

[0023] (5) The converted images are stitched so that the original image size is converted into a high-resolution image with r*h rows, c*h columns and 1 channel, thus obtaining the upsampling result.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention realizes reconstruction from low resolution to high resolution by rearranging pixels, can process images, and has certain versatility.

[0026] (2) The processing speed is higher, the PSNR of the training results is higher, and the image performance is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is the overall architecture diagram of the device of the present invention;

[0028] Figure 2 It is the overall control unit that controls the working status of each unit;

[0029] Figure 3 It is the processing process of the pixel rearrangement unit;

[0030] Figure 4 1 is a schematic diagram of the pixel rearrangement process of the channel image in the embodiment (because the data of the embodiment is too large, only a small image is drawn for illustration); the left picture is: a 7*7, 9-channel image before rearrangement; the right picture is: an image with a size of 21*21 after rearrangement;

[0031] Figure 5 It is a change in the pixel rearrangement of the register group in the embodiment. DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings:

[0033] Figure 1 It is a structural schematic diagram of a general device for rearranging image pixels of the present invention, and the device specifically includes: a general control unit, a data storage unit, a data reading unit, a pixel rearrangement unit, and a result collection unit. The general control unit enables each internal unit to work in coordination by controlling the working state of each unit and configuring data; the data storage unit is responsible for storing the original image data and the rearranged image data; the pixel rearrangement unit (PixelShuffle unit) receives the data from the data reading unit, performs pixel rearrangement operation, and transmits the rearranged data to the result collection unit; the result collection unit receives the rearranged data of the pixel rearrangement unit and transmits the data to the data storage unit.

[0034] The general control unit includes a status register and a configuration register. The status register is used to control the working state of each unit. The image data is transferred from the data storage unit to the data reading unit. After the transfer, the pixel rearrangement unit takes out the data from the data reading unit and then rearranges the input pixel data. The result collection unit receives the image data rearranged by the pixel rearrangement unit and then transfers the rearranged data to the data storage unit. The configuration register is used to cache configuration information, including the number of input image blocks, the starting storage address of the input image, the size of each image block, the first address of the data read by the data rearrangement unit to the data reading unit and its reading length (reading h addresses), the address of the data written by the data rearrangement unit to the result collection unit, and the storage address of the output image.

[0035] The pixel rearrangement unit includes: (1) a row register with a depth of h and a bit width of data of one row of an image block (8*y), which is used to store the data output by the data reading unit. When the status register of the general control unit is in the working state of the pixel rearrangement unit, the pixel rearrangement unit sends a read request to the data reading unit, and reads the data to be rearranged according to the address data and address length (reading h addresses) in the configuration information; (2) a channel register, which is used to count the depth of the data placed in the row register, and the count value is 0-h-1; (3) a result register, which is used to store the result after the row register is rearranged, and the bit width is y*8*h.

[0036] This embodiment also provides a general method for rearranging image pixels, comprising the following steps: (1) the original image size is r (row) * c (column), the number of channels is h * h, and the size of each pixel is 8 bits; (2) the original image is divided into m * n blocks, where the row direction is divided into m blocks and the column direction is divided into n blocks. It is assumed that each block has x rows and y columns, and the number of channels is h. The number of rows and columns in each block is not necessarily equal; (3) in each block, the image of the h channels of a pixel point is converted into an image block of a corresponding row and h columns. Then the image of the h channels of the next pixel point in a row is converted into an image block of a corresponding row and h columns, and spliced ​​with the image block of a row and h columns converted from the previous pixel point, until a row of h channel data is spliced. Convert a row of y pixels with h channels into a corresponding row of y*h image blocks; (4) Convert all rows of h channels of pixels into corresponding image blocks according to the method in step (3); (5) Splice the converted images; (6) The original image size is r (row)*c (column), and the number of channels is h*h. Convert it into a high-resolution image with r*h rows, c*h columns, and 1 channel, and get the upsampling result.

[0037] Assuming that the method of the present invention is used to process an image with a size of 68 (rows) * 120 (columns) and 64 channels, and each pixel is 8 bits, the reconstruction from a low-resolution image to a high-resolution image will be carried out in the following steps:

[0038] For an image with a size of 68 (rows) * 120 (columns) and 64 channels, the 8*8 channels of a pixel are converted into corresponding image blocks of 8*8 in sequence. Each pixel is converted into a corresponding image block according to the same rule. The image blocks generated by each pixel are spliced, and the feature map of the 8*8 channels is converted into a new 68*8, 120*8 upsampling result to obtain a high-resolution image. The low-resolution image with a size of 68 (rows) * 120 (columns) and 64 channels is converted into a high-resolution image with a size of 68*8 rows and 120*8 columns.

[0039] The image is divided into 16*3 blocks, where the row direction is divided into 16 blocks and the column direction is divided into 3 blocks. In the row direction, each of the first 15 blocks includes 4 rows, and the 16th block includes 8 rows. In the column direction, each block includes 40 columns. The 8 channels of each row and 40 columns are rearranged into 320 columns and 1 channel. The number of rows in the final image is 68*8=544, and the number of columns is 40*8=320.

[0040] The above process can be completed using the following processing units:

[0041] 1. General control unit: through the control of the working status of each unit and the configuration of data, the internal units can work together.

[0042] 2. Data storage unit: responsible for storing the original image data and the rearranged image data. This embodiment uses DDR memory. The size of the stored original image is 68 (rows) * 120 (columns), 64 channels, and each pixel size is 8 bits. The size of the rearranged image stored is 68*8, 120*8, 1 channel, which can be regarded as the splicing result output by the result collection module. When the status register of the general control unit is in the state of the data storage unit moving data to the data reading unit, the data reading unit issues a write request, and the data storage unit moves the image data to the data reading unit; when the status register of the general control unit is in the state of the rearranged data moving to the data storage unit, the data storage unit issues a write request, and the result collection unit moves the data to the data storage unit.

[0043] 3. Data reading unit: reads the data of the data storage unit and transmits the data to the pixel rearrangement unit. This embodiment uses RAM memory. Since the bus width of the DDR memory is inconsistent with the input data width of the pixel rearrangement unit, RAM is required to achieve bit width matching of data transmission. At the same time, since the amount of original image data is too large, it is necessary to use an image block strategy. According to the starting address of the input image and the size of the image in the configuration information of the total control unit, an image block data is stored in the RAM. It is stored in the order of row-channel-column, that is, a row of each image block is stored first, and then the next channel of a row is stored. After all channels of a row are stored, the next row is stored until the image block data storage is completed. Since the original image is divided into 16*3 blocks, the largest block is 8 rows and 40 columns, so the RAM storage depth is 8*40*64 and the storage bit width is 40*8 bits.

[0044] 4. Pixel Shuffle Unit: Receives data from the data reading unit, performs pixel rearrangement operations, and transmits the rearranged data to the result collection unit. The row register, with a depth of 8 and a bit width of one row of data in an image block (8*40 bits), is used to store the data output by the data reading unit. When the status register of the general control unit is in the pixel rearrangement unit working state, the pixel rearrangement unit sends a read request to the data reading unit, and reads the data to be rearranged according to the address data and address length (reading 8 addresses) in the configuration information; the channel register is used to count the depth of the data placed in the row register, with a count value of 0-7; the result register is used to store the result after the row register is rearranged, with a bit width of 40*8*8.

[0045] 5. Result collection unit: receives the data rearranged by the pixel rearrangement unit and transmits the data to the data storage unit. This embodiment uses a RAM memory. When the status register of the general control unit is in a state where the data storage unit receives the data of the pixel rearrangement unit, the result collection unit writes the data output by the pixel rearrangement unit into the corresponding address according to the address data in the configuration information. When the status register of the general control unit is in a state where the rearranged data is transferred to the data storage unit, the data is transferred from the result collection unit to the corresponding address of the data storage unit according to the address information of the output image.

[0046] When the status register of the general control unit is in the working state of the pixel rearrangement unit, the pixel rearrangement unit sends a read request to the data reading unit, and reads the data to be rearranged according to the address data and address length (reading 8 addresses) in the configuration information. The data (I_data) taken out from the data reading unit in each clock cycle has a bit width of 8*40 bits, which is the data of 1 row and 40 columns of pixels. Each clock cycle corresponds to a different channel register (ch_count) value, and the taken out data (I_data) is placed in the depth corresponding to the row register (row_reg). After taking out the 8-channel data and storing it in the row register (row_reg), the data of different depths and the same bit width in the row register are taken out in sequence and placed in the result register (shuffle_reg), completing the data rearrangement of 1 row, 40 columns and 8 channels. For example, in the row register, the bit width is 0 to 7, and the data of all depths is stored in the result register 0 to 63 bit width; the bit width is 8 to 15, and the data of all depths is stored in the result register 64 to 127 bit width, and the other data are arranged in sequence.

[0047] When the status register of the general control unit is in the working state of the pixel rearrangement unit next time, the pixel rearrangement unit sends a read request to the data reading unit again, according to the address data and address length in the new configuration information (read 8 addresses, which are the next 8 addresses of the last pixel rearrangement unit operation). The above method is continuously iterated. Rearrange the other channels of a row in turn, and after the rearrangement of all channel data of a row is completed, the next row of data is replaced for rearrangement. The converted images are spliced ​​to obtain high-resolution images with sizes of 68*8, 120*8, and 1 channel.

Claims

1. A general method for rearranging image pixels, characterized in that: The device for implementing the method includes: a general control unit for controlling the working state and configuration data of each unit; a data storage unit for storing original image data in the order of image blocks and storing rearranged image data; a data reading unit for reading data from the data storage unit and transmitting the data to a pixel rearrangement unit; a pixel rearrangement unit for receiving data from the data reading unit, performing pixel rearrangement calculation, rearranging the pixel position order of the image block, and transmitting the rearranged data to a result collection unit; a result collection unit for receiving the data rearranged by the pixel rearrangement unit and transmitting the data to the data storage unit; the specific steps of the general method include: (1) The original image size is r rows by c columns, the number of channels is h by h, and the size of each pixel is 8 bits; (2) The original image is divided into m*n blocks, where the row direction is divided into m blocks and the column direction is divided into n blocks; each block is assumed to have x rows and y columns, and the number of channels is h; the number of rows and columns in each block is not necessarily equal; (3) In each block, the image of the h channels of a pixel is converted into an image block of a corresponding row and h columns; then the image of the h channels of the next pixel in a row is converted into an image block of a corresponding row and h columns, and is spliced ​​with the image block of a row and h columns converted from the previous pixel, until a row of h channel data is spliced, that is, the h channels of a row of y pixels are converted into an image block of a corresponding row of y*h; (4) converting the pixel points of all rows of h channels into corresponding image blocks according to the method of step (3); (5) The converted images are stitched so that the original image size is converted into a high-resolution image with r*h rows, c*h columns and 1 channel, thus obtaining the upsampling result.

2. A general method for rearranging image pixels according to claim 1, characterized in that: The overall control unit includes a status register and a configuration register. The status register is used to control the working state of each unit; the configuration register is used to cache configuration information, including the number of blocks of the input image, the starting storage address of the input image, the size of each block of the image, the starting address and reading length of the data read by the pixel rearrangement unit from the data reading unit, the address at which the pixel rearrangement unit writes data to the result collection unit, and the storage address of the output image.

3. A general method for image pixel rearrangement according to claim 1, characterized in that: The data storage unit includes a DDR memory; when the general control unit is in a state where the data storage unit transfers data to the data reading unit, the data reading unit issues a write request, and the data storage unit transfers the image data to the data reading unit; when the general control unit is in a state where the rearranged data is transferred to the data storage unit, the data storage unit issues a write request, and the result collection unit transfers the data to the data storage unit.

4. A general method for image pixel rearrangement according to claim 1, characterized in that: The data reading unit includes a RAM memory; the RAM memory stores data in the order of row-channel-column, that is, a row of each image block is stored first, and then the next channel of a row is stored. After all channels of a row are stored, the next row is stored until the data of an image block is stored.

5. A general method for rearranging image pixels according to claim 1, characterized in that: The pixel rearrangement unit comprises: A row register, used for storing data output by the data reading unit; Channel registers, which count how deep the data is placed into the row registers; The result register is used to store the result after the row registers are rearranged.

6. A general method for image pixel rearrangement according to claim 1, characterized in that: The result collection unit includes a RAM memory; when the general control unit is in a state where the data storage unit receives data from the pixel rearrangement unit, the result collection unit writes the data output by the pixel rearrangement unit into the corresponding address according to the address data in the configuration information; when the general control unit is in a state where the rearranged data is moved to the data storage unit, the data is moved from the result collection unit to the corresponding address of the data storage unit according to the address information of the output image.

Images