Data processing system based on FPGA affine inverse transform

Abstract

The present application relates to a kind of data processing systems based on FPGA affine inverse transform, belong to image processing technical field, solve the problem of insufficient BRAM space when loading original image data in prior art.The data processing system includes: first external memory, for storing multiple fill-in subgraph blocks;Second external memory is used to store original image;Matte control module is used to determine matte subgraph block in original image according to fill-in subgraph block and affine inverse matrix, and determine first preloading subgraph block according to matte subgraph block;Preloading control module is used to determine whether the size of first preloading subgraph block is greater than the size of BRAM space, and obtain second preloading subgraph block;BRAM space is used to store second preloading subgraph block;Fill-in control module is used to determine the pixel value of each pixel point in fill-in subgraph block.Original image data is saved to BRAM space normally, and data execution is guaranteed to be successful.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a data processing system based on FPGA inverse affine transform. Background Technology

[0002] With the development of technology, people's demand for high-definition video is becoming increasingly strong. Upscaling low-resolution video to high-definition video has become a major challenge. With the increasingly widespread application of FPGA (Field Programmable Gate Array), implementing high-definition video based on FPGA is gradually becoming the mainstream.

[0003] Affine transformation algorithms can perform functions such as image rotation, translation, and scaling, and are widely used in many video and image processing fields.

[0004] However, in the existing technology, dividing the image into blocks reduces the requirement for Block Random Access Memory (BRAM) space. When faced with the need to load large amounts of original image data due to image downsizing, the BRAM space is insufficient to hold the required original image data, resulting in data execution failure. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a data processing system based on FPGA inverse affine transform to solve the problem of insufficient BRAM space when loading raw image data in the prior art.

[0006] This invention provides a data processing system based on FPGA inverse affine transform, the data processing system comprising:

[0007] The first external memory is used to store multiple filled sub-blocks, which are sub-blocks divided according to the result image;

[0008] The second external memory is used to store the original image;

[0009] The image matting control module is used to determine the matting sub-block in the original image based on the filling sub-block and the inverse affine transformation matrix, and to determine the first preloaded sub-block based on the matting sub-block;

[0010] The preloading control module is used to determine whether the size of the first preloaded sub-block is greater than the size of the BRAM space; if the determination is yes, the first preloaded sub-block is reduced according to the size of the first preloaded sub-block and the size of the BRAM space to obtain the second preloaded sub-block; if the determination is no, the first preloaded sub-block is used as the second preloaded sub-block.

[0011] BRAM space is used to store the second preloaded sub-block;

[0012] The filling control module is used to sequentially traverse each pixel in the filling sub-block, determine the pixel value of each pixel according to the inverse affine transformation matrix and the second preloaded sub-block, and send the pixel value to the corresponding pixel in the first external memory.

[0013] Based on further improvements to the above system, the data processing system also includes:

[0014] A coordinate generator is used to generate the coordinates of any pixel in the original image and the result image, and to generate the coordinates of any pixel in the second preloaded sub-plot.

[0015] Based on further improvements to the above system, the data processing system also includes:

[0016] A buffer register is used to cache the pixel values ​​in the first preloaded sub-block sent by the second external memory, and to transmit the corresponding pixel values ​​to the preload control module according to the control instructions of the preload control module. The preload control module obtains the pixel values ​​of each point in the second preloaded sub-block based on the obtained pixel values ​​of the first preloaded sub-block.

[0017] Based on a further improvement to the above system, the step of determining the cutout sub-pattern in the original image according to the filled sub-pattern and the inverse affine transformation matrix includes:

[0018] Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of the four mapping vertices corresponding to the mapping sub-block.

[0019] Based on the inverse affine transformation, the matting coordinates of the matted sub-plot are determined according to the coordinates of the four matting vertices and the inverse affine transformation matrix. The matting coordinates include the coordinates of the four matting vertices.

[0020] The cutout sub-block is determined based on the coordinates of the four cutout vertices.

[0021] Based on further improvements to the above system, the step of determining the first preloaded sub-block according to the cutout sub-block includes:

[0022] Based on the coordinates of the four cutout vertices of the cutout sub-block, the maximum and minimum values ​​of the cutout sub-block on the two coordinate axes are determined;

[0023] Based on the maximum and minimum values ​​on the two coordinate axes, determine the coordinates of the four vertices of the first preloaded sub-plot:

[0024]

[0025] Where A, B, C, and D represent the coordinates of the four vertices of the first preloaded sub-plot, Xmax represents the maximum value of the cutout sub-plot on the X-axis, Xmin represents the minimum value of the cutout sub-plot on the X-axis, Ymax represents the maximum value of the cutout sub-plot on the Y-axis, and Ymin represents the minimum value of the cutout sub-plot on the Y-axis.

[0026] Based on further improvements to the above system, determining whether the size of the first preloaded sub-plot is greater than the size of the BRAM space includes:

[0027] Determine the length and width of the first preloaded sub-block and determine the length and width of the BRAM space;

[0028] Calculate the length ratio of the first preloaded sub-plot to the length of the BRAM space, and calculate the width ratio of the first preloaded sub-plot to the width of the BRAM space;

[0029] If either the length ratio or the width ratio is greater than 1, it is judged as yes; otherwise, it is judged as no.

[0030] Based on a further improvement to the above system, the step of reducing the first preloaded sub-block according to the size of the first preloaded sub-block and the BRAM space to obtain a second preloaded sub-block includes:

[0031] Determine whether the length ratio is greater than the width ratio; if it is, determine the pre-shrunk ratio parameter based on the length ratio; otherwise, determine the pre-shrunk ratio parameter based on the width ratio; the pre-shrunk ratio parameter is N / M, where N and M are both integers in the range [1, 6], and M > N;

[0032] The first preloaded sub-block is reduced according to the pre-scaling ratio parameter to obtain the second preloaded sub-block.

[0033] Based on a further improvement to the above system, the step of reducing the first preloaded sub-block according to the pre-scaling ratio parameter to obtain the second preloaded sub-block includes:

[0034] N-1 pixels are inserted between two adjacent pixels in each row of the first preloaded sub-block. Then, N-1 pixels are inserted between two adjacent pixels in each column of the first preloaded sub-block, thereby expanding the first preloaded sub-block by N times to obtain an intermediate sub-block.

[0035] In the intermediate sub-block, a row is selected every M-1 rows as a row to be sampled. In each row to be sampled, a pixel is selected every M-1 points as a sampling point, thereby reducing the intermediate sub-block by a factor of M to obtain the second preloaded sub-block.

[0036] Based on further improvements to the above system, when inserting pixels into the first preloaded sub-block, the expression for the pixel value of the interpolated pixel is:

[0037] Dix_j=DixJ-j[(DixJ-DixK) / N];

[0038] Where DixJ and DixK represent the pixel values ​​of two adjacent pixels J and K in the first preloaded sub-block, respectively, and Dix_j represents the pixel value of the j-th pixel among the N-1 pixels inserted sequentially from pixel J to pixel K, where 1≤j≤(N-1).

[0039] Based on a further improvement to the above system, the step of determining the pixel value of each pixel point according to the inverse affine transformation matrix and the second preloaded sub-block includes:

[0040] By combining the inverse affine transformation matrix and the pre-scaling ratio parameter, the first coordinates of each pixel in the BRAM space are determined.

[0041] Let P(x, y) represent the first coordinate;

[0042] When both x and y are integers, the pixel value of point P is used as the pixel value of each corresponding pixel.

[0043] When x and y are not integers, four second coordinates around point P are determined based on point P, and the pixel values ​​of the four second coordinates are determined by addressing the BRAM space; based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated through the pixel values ​​of the four second coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel point;

[0044] When only one of x and y is an integer, two third coordinates around point P are determined based on point P, and the pixel values ​​of the two third coordinates are determined by addressing the BRAM space. Based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated using the pixel values ​​of the two third coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel.

[0045] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0046] 1. By determining whether the size of the original image data to be loaded (the first preloaded sub-plot) is greater than the size of the BRAM space; if the determination is yes, the original image data is reduced in size according to the original image data and the size of the BRAM space, and the reduced original image data is saved to the BRAM space, thus realizing the normal saving of the original image data to the BRAM space and ensuring the successful execution of data.

[0047] 2. By using the mapping coordinates of the mapping sub-blocks, the original image data to be loaded is determined based on the inverse affine transformation, which improves the accuracy of calculating the original image data to be loaded.

[0048] 3. Based on the length ratio and width ratio of the original image data to the BRAM space required for loading, the pre-reduction ratio parameter was further determined, which ensured the reduction range and reduced the loss of the original image data when reducing the original image data.

[0049] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0050] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0051] Figure 1 This is a schematic diagram of the structure of a data processing system based on FPGA inverse affine transform provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of the structure of the resulting image divided into filling sub-blocks according to an embodiment of the present invention;

[0053] Figure 3 This is one of the structural schematic diagrams of the cutout sub-block corresponding to the filling sub-block provided in the embodiments of the present invention;

[0054] Figure 4 This is the second schematic diagram of the structure of the cutout sub-block corresponding to the filling sub-block provided in the embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram illustrating the interpolation and enlargement of a first preloaded sub-block according to an embodiment of the present invention;

[0056] Figure 6 This is a schematic diagram illustrating the segmentation and scaling of intermediate sub-plots using interval sampling, as provided in an embodiment of the present invention.

[0057] Figure 7 One of the structural schematic diagrams of the first coordinate provided in the embodiment of the present invention;

[0058] Figure 8 This is a second schematic diagram of the structure of the first coordinate provided in an embodiment of the present invention. Detailed Implementation

[0059] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0060] A specific embodiment of the present invention discloses a data processing system based on FPGA inverse affine transform, such as... Figure 1 As shown, the data processing system includes:

[0061] The first external memory is used to store multiple filled sub-blocks, which are sub-blocks divided according to the result image;

[0062] The second external memory is used to store the original image;

[0063] The image matting control module is used to determine the matting sub-block in the original image based on the filling sub-block and the inverse affine transformation matrix, and to determine the first preloaded sub-block based on the matting sub-block;

[0064] The preloading control module is used to determine whether the size of the first preloaded sub-block is greater than the size of the BRAM space; if the determination is yes, the first preloaded sub-block is reduced according to the size of the first preloaded sub-block and the size of the BRAM space to obtain the second preloaded sub-block; if the determination is no, the first preloaded sub-block is used as the second preloaded sub-block.

[0065] BRAM space is used to store the second preloaded sub-block;

[0066] The filling control module is used to sequentially traverse each pixel in the filling sub-block, determine the pixel value of each pixel according to the inverse affine transformation matrix and the second preloaded sub-block, and send the pixel value to the corresponding pixel in the first external memory.

[0067] Specifically, when using an FPGA for video image processing, the original image pixel data needs to be stored from the externally configured DDR (Double Data Rate) memory to the on-chip BRAM space of the FPGA. Then, the original image pixel data is mapped to the result image according to the affine transformation matrix, and the result image is also saved to the DDR.

[0068] It is worth noting that the first external memory is used to store the pixel value data of the resulting image, and the second external memory is used to store the pixel value data of the original image. The first and second external memory can be the same DDR configured externally to the FPGA, or they can be different DDRs. After the filling control module determines the pixel value of each pixel, it needs to send the pixel value of each pixel to the first external memory for storage.

[0069] Understandably, the second external memory is connected to the matting control module via the AXI4 bus, and the first external memory is connected to the filling control module via the AXI4 bus. AXI (Advanced eXtensible Interface) is a bus protocol, the most important part of the AMBA (Advanced Microcontroller Bus Architecture) 3.0 protocol proposed by ARM, and is an on-chip bus designed for high performance, high bandwidth, and low latency.

[0070] Specifically, in the first external memory, the resulting image is divided into multiple sub-blocks as filling sub-blocks. These filling sub-blocks are further divided into odd-column blocks, even-column blocks, odd-row blocks, and even-row blocks according to their row and column location. Once the size of each filling sub-block is designed, it will not be changed. For example, as shown... Figure 2 As shown, the resulting image has a size of 1280*1024 pixels, with a length of 1280 pixels and a width of 1024 pixels. The size of the filled sub-plots is 128*128 pixels, with a length of 128 pixels and a width of 128 pixels. The resulting image is divided into 80 filled sub-plots, each with a length of 10 and a width of 8. These multiple filled sub-plots together form the final image.

[0071] Specifically, the second external memory is used to store multiple cutout sub-plots, which together form the original image.

[0072] Preferably, such as Figure 1 As shown, the data processing system also includes:

[0073] A coordinate generator is used to generate the coordinates of any pixel in the original image and the result image, and to generate the coordinates of any pixel in the second preloaded sub-plot.

[0074] Understandably, the coordinate generator uses a two-dimensional coordinate system to generate and locate the coordinates of all pixels in the resulting image and the original image.

[0075] The inverse matrix of an affine transformation in a two-dimensional coordinate system includes:

[0076] 1. The 3x3 control matrix for two-dimensional translation is:

[0077] In this matrix, tx and ty represent the pixel distances of translation, corresponding to the directions of the arrows on the X and Y coordinate axes. For the inverse transform, its forward translation is exactly opposite to the direction of the arrows on the coordinate axes, and therefore is represented by a negative sign.

[0078] 2. The 3x3 control matrix for two-dimensional rotation is:

[0079] In this matrix, θ represents the rotation angle, and the sign of the sine parameter of this matrix is ​​exactly the opposite of that in the forward transformation.

[0080] 3. The 3x3 control matrix for two-dimensional scaling is:

[0081] In this matrix, Sx and Sy are magnification factors. Since the magnification factor of the forward transformation is the reduction factor of the inverse transformation, the magnification factors here exist in reciprocal form.

[0082] Based on the inverse affine transformation matrix, the image coordinates of the resulting image and the original image are rearranged. It's worth noting that the pixel coordinates of the resulting image are mapped to the coordinates of the original image, and then the pixel values ​​from the original image coordinates are transferred to the pixel coordinates of the resulting image. The matrix operation expression is as follows:

[0083]

[0084] Where (u, v) are the coordinates of any pixel in the resulting image, and (x, y) are the corresponding coordinates in the original image obtained from the inverse affine transformation matrix. It is the inverse matrix of the affine transformation.

[0085] It is worth noting that image transformation can be achieved by changing the parameters in the control matrix. Of course, multiple deformation parameters can also be mixed to achieve the purpose of multiple deformations occurring simultaneously in a single affine transformation.

[0086] Specifically, in the image matting control module, the matting sub-blocks are determined in the original image based on the filling sub-blocks and the inverse affine transformation matrix. The matting sub-blocks are sub-blocks in the original image, and the matting sub-blocks are the corresponding sub-blocks of the filling sub-blocks in the original image. Simultaneously, the first preloaded sub-block is calculated in the image matting control module based on the matting sub-blocks.

[0087] Preferably, determining the cutout sub-pattern in the original image based on the cutout sub-pattern and the inverse affine transformation matrix includes:

[0088] Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of the four mapping vertices corresponding to the mapping sub-block.

[0089] Based on the inverse affine transformation, the matting coordinates of the matted sub-plot are determined according to the coordinates of the four matting vertices and the inverse affine transformation matrix. The matting coordinates include the coordinates of the four matting vertices.

[0090] The cutout sub-block is determined based on the coordinates of the four cutout vertices.

[0091] Specifically, such as Figure 3 As shown, the filling sub-block is any sub-block in the result image. Combining the two-dimensional coordinate system of the result image, the coordinates of the four corresponding filling vertices Q1, Q2, Q3 and Q4 can be determined for this filling sub-block.

[0092] Based on the inverse affine transformation, according to the coordinates of the four filling vertices Q1, Q2, Q3, and Q4, the corresponding coordinates of the four matting vertices in the original image are determined by calculating the inverse affine transformation matrix, namely Q1', Q2', Q3', and Q4'. Figure 3 As shown, the cutout sub-block is the sub-block in the original image that corresponds to the filled sub-block.

[0093] Specifically, in the matting control module, the first preloaded sub-block is determined based on the matting sub-blocks. It's worth noting that the DDR memory configured externally to the FPGA is read in burst lengths; directly extracting the pixel data corresponding to the matting sub-blocks from the DDR would result in significant timing waste. Therefore, the first preloaded sub-block needs to be determined based on the matting sub-blocks of the original image. This first preloaded sub-block contains the pixel data that needs to be stored from the DDR into the BRAM space.

[0094] Preferably, determining the first preloaded sub-block based on the cutout sub-block includes:

[0095] Based on the coordinates of the four cutout vertices of the cutout sub-block, the maximum and minimum values ​​of the cutout sub-block on the two coordinate axes are determined;

[0096] Based on the maximum and minimum values ​​on the two coordinate axes, determine the coordinates of the four vertices of the first preloaded sub-plot:

[0097]

[0098] Where A, B, C, and D represent the coordinates of the four vertices of the first preloaded sub-plot, Xmax represents the maximum value of the cutout sub-plot on the X-axis, Xmin represents the minimum value of the cutout sub-plot on the X-axis, Ymax represents the maximum value of the cutout sub-plot on the Y-axis, and Ymin represents the minimum value of the cutout sub-plot on the Y-axis.

[0099] Specifically, such as Figure 3 As shown, the coordinates of the four cutout vertices Q1', Q2', Q3', and Q4' have a maximum value of Xmax and a minimum value of Xmin on the X-axis; and a maximum value of Ymax and a minimum value of Ymin on the Y-axis. The coordinates of the four vertices of the first preloaded sub-tile are... It is worth noting that when reading the pixel value from the DDR, the coordinates of each pixel are integers.

[0100] It is understandable that the rectangle enclosed by the first preloaded sub-tile includes the cutout region formed by the cutout sub-tile, and the coordinates of the four vertices of the first preloaded sub-tile are all integers.

[0101] Specifically, in the preloading control module, it is determined whether the size of the first preloading sub-block is greater than the size of the BRAM space; if it is determined to be yes, the first preloading sub-block is shrunk according to the size of the first preloading sub-block and the BRAM space to obtain the second preloading sub-block; if it is determined to be no, the first preloading sub-block is used as the second preloading sub-block.

[0102] Specifically, the size of the BRAM space is generally designed to be 1.5-2 times the size of the filled sub-block. For example, if the size of the filled sub-block is 128*128, the size of the BRAM space is designed to be 256*256. Once the size of the BRAM space is determined, it will not be changed.

[0103] It is worth noting that when dealing with a scaled-down original image, loading pixel data from the original image based on the fill sub-blocks of the resulting image may require loading a large region of the original image. The pre-defined BRAM space may not be sufficient to hold the required pixel data from the original image, leading to data execution failure. Therefore, this invention compares the size of the first pre-loaded sub-block with the size of the BRAM space before storing it in the BRAM.

[0104] like Figure 3 As shown, when the size of the first preloaded sub-block is larger than the size of the BRAM space, the first preloaded sub-block is shrunk according to the size of the first preloaded sub-block and the size of the BRAM space to obtain the second preloaded sub-block, so that the BRAM space is larger than the second preloaded sub-block, so that the second preloaded sub-block can be placed normally in the BRAM space and the data can be executed successfully.

[0105] like Figure 4 As shown, when the size of the first preloaded sub-block is less than or equal to the size of the BRAM space, the first preloaded sub-block is directly used as the second preloaded sub-block.

[0106] Preferably, determining whether the size of the first preloaded sub-plot is greater than the size of the BRAM space includes:

[0107] Determine the length and width of the first preloaded sub-block and determine the length and width of the BRAM space;

[0108] Calculate the length ratio of the first preloaded sub-plot to the length of the BRAM space, and calculate the width ratio of the first preloaded sub-plot to the width of the BRAM space;

[0109] If either the length ratio or the width ratio is greater than 1, it is judged as yes; otherwise, it is judged as no.

[0110] Specifically, the size of the BRAM space is fixed, such as L*W; L is the length, usually 256 pixels, and W is the width, usually 256 pixels. The size of the first preloaded sub-tile is determined based on the coordinates of its four vertices A, B, C, and D. The length of the first preloaded sub-tile is... 1 pixel, width is 1 pixel.

[0111] Specifically, the length ratio of the first preloaded sub-plot to the length of the BRAM space is 1. The width ratio of the first preloaded sub-tile to the width of the BRAM space is:

[0112] Specifically, compare length ratios Compare the width ratio with the size of 1. The value of the length ratio and the width ratio is determined by the following condition: if either the length ratio or the width ratio is greater than 1, it is considered yes; otherwise, it is considered no.

[0113] Specifically, in the preloading control module, if the determination is yes, the first preloading sub-block is reduced in size according to the size of the first preloading sub-block and the BRAM space to obtain the second preloading sub-block.

[0114] Preferably, the step of reducing the size of the first preloaded sub-block according to the dimensions of the first preloaded sub-block and the BRAM space to obtain the second preloaded sub-block includes:

[0115] Determine whether the length ratio is greater than the width ratio; if yes, determine the pre-shrunk ratio parameter based on the length ratio; otherwise, determine the pre-shrunk ratio parameter based on the width ratio.

[0116] The first preloaded sub-block is reduced according to the pre-scaling ratio parameter to obtain the second preloaded sub-block.

[0117] Specifically, determining the length ratio Ratio of width The size. If the length ratio Greater than the width ratio Then it is determined to be yes, based on the length ratio. Determine the pre-shrinkage ratio parameter; if the length ratio Less than or equal to the width ratio The result is then determined as no, based on the width ratio. Determine the pre-shrinkage ratio parameter.

[0118] Preferably, the pre-shrinkage ratio parameter is N / M, where N and M are both integers in the range [1, 6], and M > N.

[0119] It is worth noting that when the length ratio Greater than the width ratio When the length ratio is 1, the reciprocal of the length ratio is taken, and then the reciprocal is simplified to a fraction. The fraction is then made as close as possible to the reciprocal without being greater than it. This fraction is the pre-scaling ratio parameter N / M, where N and M are both integers in the range [1, 6], and M > N. Similarly, when the length ratio is 1, the reciprocal is calculated to be 1, and ... Less than or equal to the width ratio When the width ratio is taken as a reciprocal, the reciprocal value is simplified into a fraction and the score is made as close as possible to the reciprocal value without being greater than it. The score is the pre-scaling ratio parameter N / M, where N and M are both integers in the range [1, 6] and M > N.

[0120] For example, if the calculated length ratio is 1.6 and the width ratio is 1.2, the pre-scaling ratio parameter can be determined to be 3 / 5 based on the length ratio. Specifically, first take the reciprocal of the length ratio as 1 / 1.6 = 0.625. Then select integers N and M such that N / M is no greater than 0.625 and as close to 0.625 as possible. Therefore, N is selected as 3 and M as 5, which simplifies the reciprocal value to the fraction 3 / 5, resulting in a pre-scaling ratio parameter of 3 / 5.

[0121] Preferably, the step of reducing the first preloaded sub-block according to the pre-scaling ratio parameter to obtain the second preloaded sub-block includes:

[0122] N-1 pixels are inserted between two adjacent pixels in each row of the first preloaded sub-block. Then, N-1 pixels are inserted between two adjacent pixels in each column of the first preloaded sub-block, thereby expanding the first preloaded sub-block by N times to obtain an intermediate sub-block.

[0123] In the intermediate sub-block, a row is selected every M-1 rows as a row to be sampled. In each row to be sampled, a pixel is selected every M-1 points as a sampling point, thereby reducing the intermediate sub-block by a factor of M to obtain the second preloaded sub-block.

[0124] Specifically, such as Figure 5 As shown, for example: when N is 2, one pixel is inserted between two adjacent pixels in each row of the first preloaded sub-block. Then, one pixel is inserted between two adjacent pixels in each column of the first preloaded sub-block, thereby enlarging the first preloaded sub-block by a factor of 2 to obtain an intermediate sub-block. When N is 3, two pixels are inserted between two adjacent pixels in each row of the first preloaded sub-block. Then, two pixels are inserted between two adjacent pixels in each column of the first preloaded sub-block, thereby enlarging the first preloaded sub-block by a factor of 3 to obtain an intermediate sub-block. The original pixels represent the pixels in the first preloaded sub-block, and the interpolated pixels represent the inserted pixels.

[0125] Preferably, when inserting pixels into the first preloaded sub-block, the expression for the pixel value of the interpolated pixel is:

[0126] Dix_j=DixJ-j[(DixJ-DixK) / N];

[0127] Where DixJ and DixK represent the pixel values ​​of two adjacent pixels J and K in the first preloaded sub-block, respectively, and Dix_j represents the pixel value of the j-th pixel among the N-1 pixels inserted sequentially from pixel J to pixel K, where 1≤j≤(N-1).

[0128] Specifically, such as Figure 5 As shown, for example: when N is 2, in the first row of the first preloaded sub-block, one pixel is inserted between two adjacent original pixels. From left to right, the pixel values ​​of the two adjacent original pixels are 20 and 30 respectively. Then the pixel value of the inserted pixel is 20-1*[(20-30) / 2], that is, the pixel value of the inserted pixel is 25. When N is 3, in the first row, two pixels are inserted between two adjacent original pixels. From left to right, the pixel values ​​of the two adjacent original pixels are 20 and 30 respectively. Then the pixel values ​​of the inserted pixels are 20-1*[(20-30) / 3] and 20-2*[(20-30) / 3], that is, the pixel values ​​of the two inserted pixels are 23 and 26.

[0129] Specifically, such as Figure 5 As shown, for example: when N is 2, in the first column of the first preloaded sub-block, one pixel is inserted between two adjacent original pixels. From top to bottom, the pixel values ​​of the two adjacent original pixels are 30 and 40 respectively. Then the pixel value of the inserted pixel is 30-1*[(30-40) / 2], that is, the pixel value of the inserted pixel is 35. When N is 3, in the first column of the first preloaded sub-block, two pixels are inserted between two adjacent original pixels. From top to bottom, the pixel values ​​of the two adjacent original pixels are 30 and 40 respectively. Then the pixel values ​​of the inserted pixels are 30-1*[(30-40) / 3] and 30-2*[(30-40) / 3], that is, the pixel values ​​of the two inserted pixels are 33 and 36.

[0130] Specifically, such as Figure 6 As shown, for example: when M is 2, in the intermediate sub-block, a row is selected every other row as a row to be sampled. In each row to be sampled, a pixel is selected every other point as a sampling point, thereby reducing the intermediate sub-block by a factor of 2 to obtain the second preloaded sub-block; when M is 3, in the intermediate sub-block, a row is selected every other row as a row to be sampled. In each row to be sampled, a pixel is selected every other point as a sampling point, thereby reducing the intermediate sub-block by a factor of 3 to obtain the second preloaded sub-block. Pixels represent the pixels of the sampled points, and discarded pixels represent the pixels that were not sampled.

[0131] Compared with existing technologies, the data processing system based on FPGA inverse affine transformation provided in this invention solves the technical problem that affine reduction transformation cannot be performed due to the original image being too large and the BRAM space being too small during the affine transformation process. This is achieved by determining whether the size of the original image data to be loaded is larger than the size of the BRAM space. If so, the original image data is reduced in size according to the original image data and the BRAM space, and the reduced original image data is saved to the BRAM space. Furthermore, by using the filling coordinates of the filling sub-blocks, the original image data to be loaded is determined based on the inverse affine transformation, improving the accuracy of calculating the original image data to be loaded. Finally, by determining the length ratio and width ratio of the original image data to be loaded to the BRAM space, a pre-reduction ratio parameter is further determined, ensuring the reduction range and reducing the loss of original image data during the reduction process.

[0132] The BRAM space is used to store the second preloaded sub-block.

[0133] Specifically, the coordinate generator is used to locate the coordinates of any pixel in the second preloaded sub-tile.

[0134] Preferably, such as Figure 1 As shown, the data processing system also includes:

[0135] A buffer register is used to cache the pixel values ​​in the first preloaded sub-block sent by the second external memory, and to transmit the corresponding pixel values ​​to the preload control module according to the control instructions of the preload control module. The preload control module obtains the pixel values ​​of each point in the second preloaded sub-block based on the obtained pixel values ​​of the first preloaded sub-block.

[0136] That is, when the size of the first preloaded sub-block is larger than the size of the BRAM space, the first preloaded sub-block is interpolated and sampled at intervals to obtain the second preloaded sub-block; when the size of the first preloaded sub-block is not larger than the size of the BRAM space, each pixel of the first preloaded sub-block is directly used as each pixel of the second preloaded sub-block, and the pixel values ​​of each pixel in the second preloaded sub-block are stored in the BRAM space.

[0137] In the filling control module, each pixel in the filling sub-block is traversed sequentially, and the pixel value of each pixel is determined according to the inverse affine transformation matrix and the second preloaded sub-block. The pixel value of each pixel is then filled.

[0138] Preferably, determining the pixel value of each pixel based on the inverse affine transformation matrix and the second preloaded sub-block includes:

[0139] By combining the inverse affine transformation matrix and the pre-scaling ratio parameter, the first coordinates of each pixel in the BRAM space are determined.

[0140] The pixel value of each pixel is determined based on the first coordinate.

[0141] Specifically, by combining the inverse affine transformation matrix and the pre-scaling parameter, the first coordinates corresponding to each pixel in the BRAM space are determined, including:

[0142] The coordinates of the corresponding pixel in the original image are obtained by using the coordinates of each pixel and the inverse affine transformation matrix. The first coordinate is obtained by subtracting the first coordinate of the first preloaded sub-plot from the coordinates of the pixel in the original image and then multiplying it by the pre-scaling parameter.

[0143] Specifically, for any pixel in the filling sub-block with coordinates (u, v), the corresponding coordinates of the original image are calculated as (u', v') using the inverse of the radiometric transformation matrix. It's worth noting that (u, v) and (u', v') are both coordinates within the same coordinate system. Understandably, the coordinate system in BRAM space differs from the aforementioned coordinate system. Therefore, the corresponding coordinates (u'v') need to be subtracted from the first coordinate of the first preloaded sub-block corresponding to the filling sub-block to convert the corresponding coordinate point (u', v') into a coordinate point (u”, v”) in the BRAM space coordinate system. Based on the pre-scaling factor parameter N / M, the first coordinates of the pixel point (u, v) in BRAM space are obtained as (u”*N / M, v”*N / M).

[0144] Preferably, determining the pixel value of each pixel point based on the first coordinate includes:

[0145] Let P(x, y) represent the first coordinate;

[0146] When both x and y are integers, the pixel value of point P is used as the pixel value of each corresponding pixel.

[0147] When x and y are not integers, four second coordinates around point P are determined based on point P, and the pixel values ​​of the four second coordinates are determined by addressing the BRAM space; based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated through the pixel values ​​of the four second coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel point;

[0148] When only one of x and y is an integer, two third coordinates around point P are determined based on point P, and the pixel values ​​of the two third coordinates are determined by addressing the BRAM space. Based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated using the pixel values ​​of the two third coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel.

[0149] Specifically, when both x and y are integers, it means that the first coordinate P(x, y) is a pixel. In this case, the pixel value corresponding to point P in the BRAM space is directly taken out as the pixel value of the pixel in the filling sub-block.

[0150] When only one of x and y is an integer, for example, such as Figure 7 As shown, y is an integer and x is not an integer, indicating that the first coordinate P(x, y) lies between the integer coordinates of two adjacent pixels. Based on point P, two third coordinates P1 and P2 are determined. The pixel values ​​corresponding to the third coordinates P1 and P2 in the BRAM space are extracted as DixP1 and DixP2. Therefore, the pixel value corresponding to the first coordinate P is DixP = DixP1*k1 + Dix*k2, where... Use the pixel value corresponding to point P as the pixel value of the pixel in the filled sub-block.

[0151] When both x and y are not integers, such as Figure 8 As shown, the first coordinate P(x, y) lies between four adjacent pixels. Based on point P, four second coordinates P1, P2, P3, and P4 are determined. The pixel values ​​of these second coordinates P1, P2, P3, and P4 in the BRAM space are extracted as DixP1, DixP2, DixP3, and DixP4. Therefore, the pixel value corresponding to the first coordinate P is DixP = (DixP1*k1 + DixP2*k3 + DixP3*k1 + DixP4*k3 + DixP1*k4 + DixP3*k2 + DixP2*k4 + DixP4*k2) / 4, where... Use the pixel value corresponding to point P as the pixel value of the pixel in the filled sub-block.

[0152] After obtaining the pixel values ​​of each filled sub-block in sequence using the above method, all filled sub-blocks form the result image.

[0153] Compared with the prior art, the data processing system based on FPGA inverse affine transformation provided in this embodiment of the invention determines the first coordinate in BRAM space by any pixel point in the filling sub-block, determines the pixel value corresponding to the first coordinate by different cases of the first coordinate, and uses the pixel value of the first coordinate as the pixel value of the corresponding pixel point in the filling sub-block.

[0154] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0155] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing system based on FPGA inverse affine transform, characterized in that, The data processing system includes: The first external memory is used to store multiple filling sub-blocks. The filling sub-blocks are sub-blocks divided according to the result image. Once the size of each filling sub-block is designed, it will not be changed. The second external memory is used to store the original image; The image matting control module is used to determine the matting sub-block in the original image based on the filling sub-block and the inverse affine transformation matrix, and to determine the first preloaded sub-block based on the matting sub-block; The preloading control module is used to determine whether the size of the first preloaded sub-block is greater than the size of the BRAM space, including: Determine the length and width of the first preloaded sub-block and determine the length and width of the BRAM space; Calculate the length ratio of the first preloaded sub-plot to the length of the BRAM space, and calculate the width ratio of the first preloaded sub-plot to the width of the BRAM space; If either the length ratio or the width ratio is greater than 1, it is judged as yes; otherwise, it is judged as no. If the determination is yes, the first preloaded sub-block is reduced in size according to the first preloaded sub-block and the BRAM space to obtain a second preloaded sub-block, including: Determine whether the length ratio is greater than the width ratio; if it is, determine the pre-shrunk ratio parameter based on the length ratio; otherwise, determine the pre-shrunk ratio parameter based on the width ratio; the pre-shrunk ratio parameter is N / M, where N and M are both integers in the range [1, 6], and M > N; The first preloaded sub-block is reduced according to the pre-scaling ratio parameter to obtain the second preloaded sub-block; If the determination is negative, the first preloaded sub-block is used as the second preloaded sub-block; BRAM space is used to store the second preloaded sub-block; The filling control module is used to sequentially traverse each pixel in the filling sub-block, determine the pixel value of each pixel according to the inverse affine transformation matrix and the second preloaded sub-block, and send the pixel value to the corresponding pixel in the first external memory.

2. The data processing system according to claim 1, characterized in that, The data processing system also includes: A coordinate generator is used to generate the coordinates of any pixel in the original image and the result image, and to generate the coordinates of any pixel in the second preloaded sub-plot.

3. The data processing system according to claim 2, characterized in that, The data processing system also includes: A buffer register is used to cache the pixel values ​​in the first preloaded sub-block sent by the second external memory, and to transmit the corresponding pixel values ​​to the preload control module according to the control instructions of the preload control module. The preload control module obtains the pixel values ​​of each point in the second preloaded sub-block based on the obtained pixel values ​​of the first preloaded sub-block.

4. The data processing system according to claim 3, characterized in that, The step of determining the cutout sub-pattern in the original image based on the cutout sub-pattern and the inverse affine transformation matrix includes: Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of the four mapping vertices corresponding to the mapping sub-block. Based on the inverse affine transformation, the matting coordinates of the matted sub-plot are determined according to the coordinates of the four matting vertices and the inverse affine transformation matrix. The matting coordinates include the coordinates of the four matting vertices. The cutout sub-block is determined based on the coordinates of the four cutout vertices.

5. The data processing system according to claim 4, characterized in that, The step of determining the first preloaded sub-block based on the cutout sub-block includes: Based on the coordinates of the four cutout vertices of the cutout sub-block, the maximum and minimum values ​​of the cutout sub-block on the two coordinate axes are determined; Based on the maximum and minimum values ​​on the two coordinate axes, determine the coordinates of the four vertices of the first preloaded sub-plot: A( Xmin , Ymin );B( Xmin , Ymax );C ( Xmax , Ymin );D( Xmax , Ymax ); Where A, B, C, and D represent the coordinates of the four vertices of the first preloaded sub-plot, Xmax represents the maximum value of the cutout sub-plot on the X-axis, Xmin represents the minimum value of the cutout sub-plot on the X-axis, Ymax represents the maximum value of the cutout sub-plot on the Y-axis, and Ymin represents the minimum value of the cutout sub-plot on the Y-axis.

6. The data processing system according to claim 1, characterized in that, The step of reducing the first preloaded sub-block according to the pre-scaling ratio parameter to obtain the second preloaded sub-block includes: N-1 pixels are inserted between two adjacent pixels in each row of the first preloaded sub-block. Then, N-1 pixels are inserted between two adjacent pixels in each column of the first preloaded sub-block, thereby expanding the first preloaded sub-block by N times to obtain an intermediate sub-block. In the intermediate sub-block, a row is selected every M-1 rows as a row to be sampled. In each row to be sampled, a pixel is selected every M-1 points as a sampling point, thereby reducing the intermediate sub-block by a factor of M to obtain the second preloaded sub-block.

7. The data processing system according to claim 6, characterized in that, When inserting a pixel in the first preloaded sub-block, the expression for the pixel value of the interpolated pixel is: Dix_j=DixJ-j[(DixJ-DixK) / N]; Where DixJ and DixK represent the pixel values ​​of two adjacent pixels J and K in the first preloaded sub-block, respectively, and Dix_j represents the pixel value of the j-th pixel among the N-1 pixels inserted sequentially from pixel J to pixel K, where 1≤j≤(N-1).

8. The data processing system according to claim 1, characterized in that, The step of determining the pixel value of each pixel based on the inverse affine transformation matrix and the second preloaded sub-block includes: By combining the inverse affine transformation matrix and the pre-scaling ratio parameter, the first coordinates of each pixel in the BRAM space are determined. Let P(x, y) represent the first coordinate; When both x and y are integers, the pixel value of point P is used as the pixel value of each corresponding pixel. When x and y are not integers, four second coordinates around point P are determined based on point P, and the pixel values ​​of the four second coordinates are determined by addressing the BRAM space; based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated through the pixel values ​​of the four second coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel point; When only one of x and y is an integer, two third coordinates around point P are determined based on point P, and the pixel values ​​of the two third coordinates are determined by addressing the BRAM space. Based on the bilinear interpolation algorithm, the pixel value corresponding to the first coordinate is calculated using the pixel values ​​of the two third coordinates, and the pixel value corresponding to the first coordinate is used as the pixel value of each corresponding pixel.

Images