Bayer image decoding method based on FPGA affine inverse transformation

Abstract

The present invention relates to a Bayer image decoding method based on FPGA affine inverse transformation, which belongs to the field of image processing technology and solves the problem of excessive use of affine transformation resources for color images in the prior art. The decoding method includes: determining a cutout sub-block based on a filling sub-block and an affine transformation inverse matrix; determining a first preloaded sub-block based on the cutout sub-block and a BRAM space; and storing the preloaded sub-block in the BRAM space; the BRAM space includes 8 dual-port BRAMs; sequentially traversing each pixel point in the filling sub-block, and determining the first coordinate corresponding to the coordinate of each pixel point in the BRAM space based on the affine transformation inverse matrix and the first preloaded sub-block; decoding the Bayer value corresponding to the first coordinate based on the Bayer image decoding, obtaining the RGB pixel value corresponding to the first coordinate, and filling the RGB pixel value corresponding to the first coordinate as the RGB pixel value of each pixel point. The affine transformation and Bayer image decoding are achieved simultaneously, saving resources.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a Bayer image decoding method based on FPGA affine inverse transformation. Background Art

[0002] With the advancement of technology, the demand for high-definition video is growing. Scaling low-resolution videos to high-definition has become a major challenge. With the increasing use of FPGAs (Field Programmable Gate Arrays), FPGA-based high-definition video processing is becoming mainstream. Affine transformation algorithms can perform functions such as derotation, translation, and mirroring of images, and are widely used in many video and image processing fields.

[0003] However, in the prior art, when performing affine transformation on a Bayer image using FPGA processing, it is necessary to first decode the Bayer image into an RGB color image, and then perform affine transformation on the RGB color image separately. The resources used are three times that of processing a grayscale image, resulting in excessive resource usage.

[0004] Therefore, a new affine transform decoding method for Bayer images is urgently needed. Summary of the Invention

[0005] In view of the above analysis, an embodiment of the present invention aims to provide a Bayer image decoding method based on FPGA affine inverse transformation to solve the problem of excessive use of resources for affine transformation of color images in the prior art.

[0006] An embodiment of the present invention provides a Bayer image decoding method based on FPGA affine inverse transformation, the decoding method comprising:

[0007] Determine a cutout sub-block according to the filling sub-block and the inverse affine transformation matrix, wherein the filling sub-block is a sub-block divided according to the result image, and the cutout sub-block is a sub-block of the original image;

[0008] Determine a first preloaded sub-image block according to the cutout sub-image block and the BRAM space; store the first preloaded sub-image block in the BRAM space; the BRAM space includes 8 dual-port BRAMs;

[0009] Traversing each pixel point in the mapping sub-block in sequence, and determining the first coordinate corresponding to the coordinate of each pixel point in the BRAM space according to the inverse affine transformation matrix and the first preloaded sub-block;

[0010] Obtain the bayer value corresponding to the first coordinate from the BRAM space, decode the bayer value corresponding to the first coordinate based on the bayer image decoding, obtain the RGB pixel value corresponding to the first coordinate, and fill the RGB pixel value corresponding to the first coordinate as the RGB pixel value of each pixel point.

[0011] Based on a further improvement of the above method, storing the first preloaded sub-image block in the BRAM space includes:

[0012] Divide the bayer values ​​stored in the first preloaded sub-block into a plurality of preset matrices in order from left to right and from top to bottom, where the length and width of the preset matrices are both 4 bayer values;

[0013] Each of the preset matrices is divided into 8 sub-matrices, each of the sub-matrices includes 2 bayer values, and each of the 8 sub-matrices is stored in a dual-port BRAM.

[0014] Based on a further improvement of the above method, obtaining the Bayer value corresponding to the first coordinate from the BRAM space, decoding the Bayer value corresponding to the first coordinate based on Bayer image decoding, and obtaining the RGB pixel value corresponding to the first coordinate includes:

[0015] Let P(x, y) denote the first coordinate;

[0016] When neither x nor y is an integer, determine four second coordinates around point P based on point P, decode the Bayer values ​​of the four second coordinates to obtain RGB pixel values ​​of the four second coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the four second coordinates based on a bilinear interpolation algorithm;

[0017] When only one of x and y is an integer, determine two third coordinates around point P based on point P, decode the Bayer values ​​of the two third coordinates to obtain the RGB pixel values ​​of the two third coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the two third coordinates based on a bilinear interpolation algorithm;

[0018] When x and y are both integers, the Bayer value of the coordinate of point P is decoded to obtain the RGB pixel value corresponding to the first coordinate.

[0019] Based on a further improvement of the above method, decoding the Bayer values ​​of the four second coordinates to obtain the RGB pixel values ​​of the four second coordinates includes:

[0020] Four 3*3 matrices centered on the four second coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding second coordinates are decoded to determine the RGB pixel values ​​of the second coordinates.

[0021] Based on a further improvement of the above method, decoding the bayer values ​​of the two third coordinates to obtain the RGB pixel values ​​of the two third coordinates includes:

[0022] Two 3*3 matrices centered on the two third coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding third coordinates are decoded to determine the RGB pixel values ​​of the third coordinates.

[0023] Based on a further improvement of the above method, decoding the Bayer value of the coordinate of point P to obtain the RGB pixel value corresponding to the first coordinate includes:

[0024] A 3*3 matrix centered on the first coordinate is obtained from the BRAM space, and based on the bayer values ​​included in the 3*3 matrix, the bayer values ​​of the first coordinate are decoded to determine the RGB pixel values ​​of the first coordinate.

[0025] Based on a further improvement of the above method, decoding the Bayer value of the coordinate to determine the RGB pixel value of the coordinate includes:

[0026] Assume the coordinates are C(M, N);

[0027] When M and N are both odd numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points above and below point C; the G pixel value of point C is the Bayer value of point C; the B pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C;

[0028] When M is an odd number and N is an even number, the R pixel value of point C is the Bayer value of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left and right of point C; the B pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right and upper right directions of point C.

[0029] Based on the further improvement of the above method, when M and N are both even numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C; the G pixel value of point C is the Bayer value of point C; and the B pixel value of point C is the average of the Bayer values ​​of the two points above and below point C.

[0030] When M is an even number and N is an odd number, the R pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right, and upper right directions of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left, and right of point C; the B pixel value of point C is the Bayer value of point C.

[0031] Based on a further improvement of the above method, the step of determining the cutout sub-block according to the filling sub-block and the inverse affine transformation matrix includes:

[0032] Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of four mapping vertices corresponding to the mapping sub-block;

[0033] Based on the inverse affine transformation, determining the cutout coordinates of the cutout sub-block according to the four filling vertex coordinates and the inverse affine transformation matrix, wherein the cutout coordinates include the coordinates of the four cutout vertices;

[0034] The cutout sub-block is determined according to the coordinates of the four cutout vertices.

[0035] Based on a further improvement of the above method, determining the first preloaded sub-image block according to the cutout sub-image block and the BRAM space includes:

[0036] Determining the maximum value and the minimum value of the cutout sub-block on two coordinate axes according to the coordinates of the four cutout vertices of the cutout sub-block;

[0037] Determine the coordinates of the four vertices of the second preloaded sub-tile based on the maximum and minimum values ​​on the two coordinate axes:

[0038]

[0039] Wherein, A, B, C and D represent the coordinates of the four vertices of the second preloaded sub-block, Xmax represents the maximum value of the cutout sub-block on the X-axis, Xmin represents the minimum value of the cutout sub-block on the X-axis, Ymax represents the maximum value of the cutout sub-block on the Y-axis, and Ymin represents the minimum value of the cutout sub-block on the Y-axis;

[0040] A first preloaded sub-tile is determined based on the four vertex coordinates of the second preloaded sub-tile and the BRAM space, so that the bayer value included in the first preloaded sub-tile contains the bayer value included in the second preloaded sub-tile, and the size of the first preloaded sub-tile is less than or equal to the size of the BRAM space.

[0041] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0042] 1. By storing the loaded original image data (the first preloaded sub-block) into the BRAM space including 8 dual-port BRAMs, the Bayer image is decoded while implementing the affine transformation, saving resources.

[0043] 2. By filling the mapping coordinates of the sub-block, the original image data that needs to be loaded is determined based on the affine inverse transformation, which improves the accuracy of calculating the original image data that needs to be loaded.

[0044] In the present invention, the above-mentioned technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of the present invention will be described in the following description, and some advantages will become apparent from the description or be learned through practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0046] Figure 1 A schematic diagram of a flow chart of a Bayer image decoding method based on FPGA affine inverse transform provided in an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of a structure in which a result image provided by an embodiment of the present invention is divided into mapping sub-blocks;

[0048] Figure 3 A schematic structural diagram of a cutout sub-block, a first preloaded sub-block, and a second preloaded sub-block corresponding to a fill sub-block provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the structure of storing an 8*8 Bayer image into BRAM space provided by an embodiment of the present invention;

[0050] Figure 5 One of the structural schematic diagrams of the first coordinate provided in an embodiment of the present invention;

[0051] Figure 6 The second structural diagram of the first coordinate provided by an embodiment of the present invention;

[0052] Figure 7 One of the structural diagrams of the first preloaded sub-tile storage provided by an embodiment of the present invention;

[0053] Figure 8 A second structural diagram of a first preloaded sub-tile storage provided by an embodiment of the present invention;

[0054] Figure 9 One of the schematic diagrams of the structure of decoding the Bayer value of a coordinate to determine the RGB pixel value of the coordinate provided by an embodiment of the present invention;

[0055] Figure 10 A second structural diagram of decoding the Bayer value of a coordinate to determine the RGB pixel value of the coordinate provided by an embodiment of the present invention;

[0056] Figure 11 A third structural diagram of decoding the Bayer value of a coordinate to determine the RGB pixel value of the coordinate provided by an embodiment of the present invention;

[0057] Figure 12 One of the structural schematic diagrams of decoding the Bayer value of a coordinate to determine the RGB pixel value of the coordinate provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, and are not used to limit the scope of the present invention.

[0059] A specific embodiment of the present invention discloses a Bayer image decoding method based on FPGA affine inverse transformation, such as Figure 1 shown.

[0060] Step S1: determining a cutout sub-block according to a filling sub-block and an inverse affine transformation matrix, wherein the filling sub-block is a sub-block divided according to the result image, and the cutout sub-block is a sub-block of the original image;

[0061] Step S2: determining a first preloaded sub-image block according to the cutout sub-image block and the BRAM space; storing the first preloaded sub-image block in the BRAM space; the BRAM space includes 8 dual-port BRAMs;

[0062] Step S3: traverse each pixel point in the mapping sub-block in sequence, and determine the first coordinate corresponding to the coordinate of each pixel point in the BRAM space according to the inverse affine transformation matrix and the first preloaded sub-block;

[0063] Step S4: Obtain the Bayer value corresponding to the first coordinate from the BRAM space, decode the Bayer value corresponding to the first coordinate based on Bayer image decoding, obtain the RGB pixel value corresponding to the first coordinate, and fill the RGB pixel value corresponding to the first coordinate as the RGB pixel value of each pixel point.

[0064] Specifically, when using FPGA for video image processing, the original image data needs to be stored from the DDR (Double Data Rate) memory configured outside the FPGA to the BRAM space within the FPGA chip, and then the original image pixel data is mapped to the result image according to the affine transformation matrix, and the result image is also saved in the DDR.

[0065] Specifically, in an embodiment of the present invention, the FPGA is directly connected to the image acquisition module externally, such as the CMOS camera 0V7670, which supports video output with a resolution of 640*480@60Hz. The Bayer image data output by the CMOS camera is used as the original image data, and the Bayer image data is stored in the BRAM space in sequence. The affine transformation and decoding of the Bayer image are implemented based on the FPGA, and the decoded RGB pixel value data is stored in the memory of the result image.

[0066] It's worth noting that in the original image data, each pixel coordinate corresponds to a Bayer value, while in the resulting image data, each pixel coordinate corresponds to three pixel values: R, G, and B. It's understandable that storing the RGB pixel value data corresponding to a coordinate point in the resulting image takes up three times the memory of storing the Bayer value corresponding to a coordinate point in the original image.

[0067] Specifically, if Figure 1 As shown, in step S1, the result image is divided into a plurality of sub-blocks as filling sub-blocks. After the size of each filling sub-block is designed, it will not be changed. Figure 2 As shown, the resulting image size is 1280*1024, with a length of 1280 pixels and a width of 1024 pixels. The fill sub-block size is 128*128, with a length of 128 pixels and a width of 128 pixels. The resulting image is divided into 10 fill sub-blocks of length and 8 fill sub-blocks of width, for a total of 80 fill sub-blocks. It can be understood that in the resulting image, each pixel can store an RGB pixel value.

[0068] It is understandable that a two-dimensional coordinate system is used to locate all pixels of the result image and the original image.

[0069] For the affine transformation in the two-dimensional coordinate system, the inverse matrix includes:

[0070] 1. The 3*3 control matrix of two-dimensional translation is:

[0071] In this matrix, tx and ty are the pixel distances of the translation, which correspond to the directions of the arrows on the X and Y coordinate axes. For the inverse transformation, its positive translation is exactly opposite to the direction of the arrows on the coordinate axes, so it is represented by a negative sign.

[0072] 2. The 3*3 control matrix of two-dimensional rotation is:

[0073] In this matrix, θ represents the rotation angle, and the sign of the sine parameter of the matrix in the inverse transformation is exactly opposite to that in the forward transformation.

[0074] 3. The 3*3 control matrix for two-dimensional scaling is:

[0075] In this matrix, Sx and Sy are magnification factors. Since the magnification factor of the forward transformation is the reduction factor of the inverse transformation, the magnification factor here exists in the inverse form.

[0076] According to the inverse affine transformation matrix, the image coordinates of the result image and the original image are rearranged. It is worth noting that the pixel coordinates of the result image are mapped to the coordinates of the original image, and then the pixel values ​​on the coordinates of the original image are moved to the pixel coordinates of the result image. The matrix operation expression is:

[0077]

[0078] Among them, the coordinates (u, v) are the coordinates of any pixel point in the result image, and the coordinates (x, y) are the corresponding coordinates in the original image obtained according to the inverse matrix of the affine transformation. is the inverse affine transformation matrix.

[0079] It is worth noting that the image transformation is achieved by changing the parameters in the control matrix. Of course, multiple deformation parameters can also be mixed to achieve the purpose of multiple deformations simultaneously in one affine transformation process.

[0080] Specifically, if Figure 1 As shown, in step S1, a cutout sub-block is calculated based on the filling sub-block and the inverse affine transformation matrix. The cutout sub-block is a sub-block in the original image, and the cutout sub-block is a sub-block corresponding to the filling sub-block in the original image.

[0081] Preferably, determining the cutout sub-block according to the filling sub-block and the inverse affine transformation matrix includes:

[0082] Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of four mapping vertices corresponding to the mapping sub-block;

[0083] Based on the inverse affine transformation, determining the cutout coordinates of the cutout sub-block according to the four filling vertex coordinates and the inverse affine transformation matrix, wherein the cutout coordinates include the coordinates of the four cutout vertices;

[0084] The cutout sub-block is determined according to the coordinates of the four cutout vertices.

[0085] Specifically, if Figure 3 As shown, the mapping sub-block is any sub-block in the result image. Combined with the two-dimensional coordinate system of the result image, the corresponding four mapping vertex coordinates Q1, Q2, Q3 and Q4 can be determined for the mapping sub-block.

[0086] Based on the inverse affine transformation, according to the four filling vertex coordinates Q1, Q2, Q3 and Q4, the corresponding four cutout vertex coordinates Q1', Q2', Q3' and Q4' in the original image are determined by the inverse affine transformation matrix, as shown in Figure 5. Figure 3 As shown in FIG, the cutout sub-block is the sub-block corresponding to the filling sub-block in the original image.

[0087] Specifically, if Figure 1 As shown, in step S2, the first preloaded sub-block is determined based on the cutout sub-block and the BRAM space. It is worth noting that the original image data memory configured externally by the FPGA is read in burst length. If the pixel data corresponding to the cutout sub-block is to be directly extracted from it, a large amount of timing waste will be generated. Therefore, it is necessary to determine the first preloaded sub-block based on the cutout sub-block of the original image. The first preloaded sub-block is the bayer image data that needs to be stored from the memory configured externally by the FPGA to the BRAM space.

[0088] Preferably, determining the first preloaded sub-image block according to the cutout sub-image block and the BRAM space includes:

[0089] Determining the maximum value and the minimum value of the cutout sub-block on two coordinate axes according to the coordinates of the four cutout vertices of the cutout sub-block;

[0090] Determine the coordinates of the four vertices of the second preloaded sub-tile based on the maximum and minimum values ​​on the two coordinate axes:

[0091]

[0092] Wherein, A, B, C and D represent the coordinates of the four vertices of the second preloaded sub-block, Xmax represents the maximum value of the cutout sub-block on the X-axis, Xmin represents the minimum value of the cutout sub-block on the X-axis, Ymax represents the maximum value of the cutout sub-block on the Y-axis, and Ymin represents the minimum value of the cutout sub-block on the Y-axis;

[0093] A first preloaded sub-tile is determined based on the four vertex coordinates of the second preloaded sub-tile and the BRAM space, so that the bayer value included in the first preloaded sub-tile contains the bayer value included in the second preloaded sub-tile, and the size of the first preloaded sub-tile is less than or equal to the size of the BRAM space.

[0094] Specifically, if Figure 3 As shown, the four vertex coordinates Q1', Q2', Q3' and Q4' on the X axis have a maximum value of Xmax and a minimum value of Xmin; the maximum value on the Y axis has a maximum value of Ymax and a minimum value of Ymin. The four vertex coordinates of the second preloaded sub-block are It is worth noting that when reading the Bayer value of a pixel from the memory configured outside the FPGA, the coordinates of each pixel are integers.

[0095] It can be understood that the rectangle enclosed by the second preloaded sub-image block includes the cutout area formed by the cutout sub-image block, and the coordinates of the four vertices of the second preloaded sub-image block are all integers.

[0096] According to the Bayer image decoding method, the Bayer values ​​of the surrounding pixel coordinates are needed to decode the Bayer value of a certain pixel coordinate point. Therefore, when reading the second preloaded sub-block from the original image data, the Bayer value data of some more pixels can be read. That is, based on the second preloaded sub-block, more data of the same rows or columns are read from the four directions of up, down, left and right to obtain the first preloaded sub-block, and the size of the first preloaded sub-block is made smaller than the total size of the BRAM space. For example, based on the second preloaded sub-block, 3 more rows are read from the top and bottom, and 3 more columns are read from the left and right to obtain the first preloaded sub-block.

[0097] It can be understood that the size of the first preloaded sub-tile is between the size of the second preloaded sub-tile and the size of the BRAM space.

[0098] Specifically, if Figure 1 As shown, in step S2, the first preloaded sub-image block is stored in the BRAM space. It can be understood that the size of the first preloaded sub-image block is smaller than the BRAM space.

[0099] Preferably, storing the first preloaded sub-image block in the BRAM space includes:

[0100] Divide the bayer values ​​stored in the first preloaded sub-block into a plurality of preset matrices in order from left to right and from top to bottom, where the length and width of the preset matrices are both 4 bayer values;

[0101] Each of the preset matrices is divided into 8 sub-matrices, each of the sub-matrices includes 2 bayer values, and each of the 8 sub-matrices is stored in a dual-port BRAM.

[0102] Specifically, if Figure 4 As shown in the figure, an 8*8 Bayer image is used as the first preloaded sub-tile and is saved to the BRAM space. The 8*8 Bayer image is divided into four preset matrices from left to right and from top to bottom, with the length and width of each matrix being 4 Bayer values. It is worth noting that each preset matrix is ​​divided into 8 sub-matrices, each of which contains 2 Bayer values.

[0103] like Figure 4 As shown in the figure, 1, 2, 3, 4, 5, 6, 7 and 8 represent the numbers of the 8 dual-port BRAMs in the BRAM space. Each preset matrix is ​​divided into 8 sub-matrices, and the bayer values ​​in each sub-matrix are stored in a dual-port BRAM; the 8 sub-matrices are respectively stored in the 8 dual-port BRAMs in the BRAM space.

[0104] It is worth noting that the bayer image data in the first preloaded sub-block is Figure 4 When the Bayer image is retrieved, for any connected 4*4 matrix, the FPGA can read the 16 stored Bayer values ​​from the 8 dual-port BRAMs included in the BRAM space at one time.

[0105] Specifically, if Figure 1 As shown, in step S3, each pixel point in the mapping sub-block is traversed in turn, and the first coordinate corresponding to the coordinate of each pixel point in the BRAM space is determined according to the inverse affine transformation matrix and the first preloaded sub-block.

[0106] Preferably, determining the first coordinate corresponding to the coordinate of each pixel point in the BRAM space according to the inverse affine transformation matrix and the first preloaded sub-block includes:

[0107] The coordinates of the corresponding pixel in the original image are obtained according to the coordinates of each pixel and the inverse affine transformation matrix, and the first coordinates are obtained by subtracting the first coordinates of the first preloaded sub-block from the coordinates of the pixel in the original image.

[0108] Specifically, for any pixel point in the mapping sub-block with coordinates (u, v), the coordinates of the corresponding original image are calculated based on the inverse matrix of the radial transformation, which is (u', v'). It is worth noting that (u, v) and (u', v') are both coordinate points in the same coordinate system. It is understandable that the coordinate system in the BRAM space is different from the above coordinate system. The corresponding coordinate (u'v') needs to be subtracted from the first coordinate of the first preloaded sub-block corresponding to the mapping sub-block, so that the corresponding coordinate point (u', v') is converted into the coordinate point (u", v") in the BRAM space coordinate system.

[0109] Specifically, if Figure 1 As shown, in step S4, the bayer value corresponding to the first coordinate is obtained from the BRAM space, the bayer value corresponding to the first coordinate is decoded based on the bayer image decoding, the RGB pixel value corresponding to the first coordinate is obtained, and the RGB pixel value corresponding to the first coordinate is used as the RGB pixel value of each pixel point for filling.

[0110] After obtaining the first coordinate, the Bayer value corresponding to the first coordinate can be decoded based on the Bayer image decoding to obtain the RGB pixel value corresponding to the first coordinate, and the RGB pixel value corresponding to the first coordinate is stored in the corresponding pixel point in the result image. It is worth noting that the memory address of the pixel point in the result image can store a single RGB pixel value.

[0111] Preferably, obtaining the Bayer value corresponding to the first coordinate from the BRAM space, decoding the Bayer value corresponding to the first coordinate based on Bayer image decoding, and obtaining the RGB pixel value corresponding to the first coordinate includes:

[0112] Let P(x, y) denote the first coordinate;

[0113] When neither x nor y is an integer, determine four second coordinates around point P based on point P, decode the Bayer values ​​of the four second coordinates to obtain RGB pixel values ​​of the four second coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the four second coordinates based on a bilinear interpolation algorithm;

[0114] When only one of x and y is an integer, determine two third coordinates around point P based on point P, decode the Bayer values ​​of the two third coordinates to obtain the RGB pixel values ​​of the two third coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the two third coordinates based on a bilinear interpolation algorithm;

[0115] When x and y are both integers, the Bayer value of the coordinate of point P is decoded to obtain the RGB pixel value corresponding to the first coordinate.

[0116] Specifically, when x and y are both integers, it means that the first coordinate P(x, y) is a pixel point. At this time, the corresponding bayer value of the P point coordinate in the BRAM space is directly decoded to obtain the RGB pixel value of the P point coordinate, and the RGB pixel value is used as the RGB pixel value of the pixel point in the mapping sub-block.

[0117] When only one of x and y is an integer, for example, Figure 5 As shown, y is an integer and x is not an integer, indicating that the first coordinate P(x, y) is between the integer coordinates of two adjacent pixel points. At this time, two third coordinates P1' and P2' are determined based on point P. The Bayer values ​​corresponding to the two third coordinates P1' and P2' are decoded to obtain the RGB pixel values ​​of the two third coordinates P1' and P2'. The RGB pixel values ​​corresponding to the third coordinates P1' and P2' in the BRAM space are taken out as DixP1'(RGB) and DixP2'(RGB). The RGB pixel value corresponding to the first coordinate point P is DixP(RGB)=DixP1'(RGB)*k1+DixP2'(RGB)*k2, where The RGB pixel value corresponding to point P is used as the RGB pixel value of the pixel in the mapping sub-block.

[0118] When both x and y are not integers, such as Figure 6 As shown, it means that the first coordinate P(x, y) is between four adjacent pixel points. At this time, the four second coordinates P1, P2, P3 and P4 are determined according to point P, and the bayer values ​​corresponding to the four second coordinates P1, P2, P3 and P4 are decoded to obtain the RGB pixel values ​​of the four second coordinates. The pixel values ​​of the second coordinates P1, P2, P3 and P4 in the BRAM space are taken out as DixP1(RGB), DixP2(RGB), DixP3(RGB) and DixP4(RGB). Then the pixel value corresponding to the first coordinate point P is DixP(RGB)=(DixP1(RGB)*k1+DixP2(RGB)*k3+DixP3(RGB)*k1+DixP4(RGB)*k3+DixP1(RGB)*k4+DixP3(RGB)*k2+DixP2(RGB)*k4+DixP4(RGB)*k2) / 4, where The RGB pixel value corresponding to point P is used as the RGB pixel value of the pixel in the mapping sub-block. To round up, To round down.

[0119] After obtaining the RGB pixel values ​​in each mapping sub-block in sequence according to the above method, all the mapping sub-blocks form the result image.

[0120] Preferably, decoding the Bayer values ​​of the four second coordinates to obtain RGB pixel values ​​of the four second coordinates includes:

[0121] Four 3*3 matrices centered on the four second coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding second coordinates are decoded to determine the RGB pixel values ​​of the second coordinates.

[0122] Preferably, decoding the Bayer values ​​of the two third coordinates to obtain RGB pixel values ​​of the two third coordinates includes:

[0123] Two 3*3 matrices centered on the two third coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding third coordinates are decoded to determine the RGB pixel values ​​of the third coordinates.

[0124] Preferably, decoding the Bayer value of the coordinate of point P to obtain the RGB pixel value corresponding to the first coordinate includes:

[0125] A 3*3 matrix centered on the first coordinate is obtained from the BRAM space, and based on the bayer values ​​included in the 3*3 matrix, the bayer values ​​of the first coordinate are decoded to determine the RGB pixel values ​​of the first coordinate.

[0126] Specifically, if Figure 6 and Figure 7 As shown in the figure, when both x and y are not integers, P1, P2, P3 and P4 are the four second coordinates, and four 3*3 matrices are taken with P1, P2, P3 and P4 as the center respectively. For each 3*3 matrix, decoding is performed based on the Bayer image to obtain the RGB pixel values ​​corresponding to the P1, P2, P3 and P4 coordinates.

[0127] It is understandable that if Figure 8 As shown, for a 3*3 matrix, its center is the element in the 2nd row and 2nd column. That is, in Figure 8 The shaded part is a 3*3 matrix with point C as its center.

[0128] Specifically, if Figure 5 and Figure 7As shown in the figure, when only one of x and y is an integer, P1' and P2' are two third coordinates. Two 3*3 matrices are taken with P1' and P2' as the center respectively. For each 3*3 matrix, decoding is performed based on the Bayer image to obtain the RGB pixel values ​​corresponding to the P1' and P2' coordinates.

[0129] Specifically, if Figure 8 As shown in the figure, when x and y are both integers, the corresponding first coordinate is taken as point C, a 3*3 matrix is ​​taken with point C as the center, and the RGB pixel value corresponding to the C coordinate is obtained by decoding based on the Bayer image.

[0130] It is worth noting that in Figure 7 and Figure 8 The figure shows a schematic diagram of the logical structure of the first preloaded sub-block stored in the BRAM space. It can be seen from the figure that when reading any 3*3 matrix in the BRAM space, 9 bayer values ​​can be read at one time, and the RGB pixel value of the center coordinate point can be obtained by decoding based on the 9 bayer values.

[0131] Preferably, decoding the Bayer value of the coordinate to determine the RGB pixel value of the coordinate includes:

[0132] Assume the coordinates are C(M, N);

[0133] When M and N are both odd numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points above and below point C; the G pixel value of point C is the Bayer value of point C; the B pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C;

[0134] When M is an odd number and N is an even number, the R pixel value of point C is the Bayer value of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left and right of point C; the B pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right and upper right directions of point C.

[0135] When M and N are both even numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C; the G pixel value of point C is the Bayer value of point C; the B pixel value of point C is the average of the Bayer values ​​of the two points above and below point C;

[0136] When M is an even number and N is an odd number, the R pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right, and upper right directions of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left, and right of point C; the B pixel value of point C is the Bayer value of point C.

[0137] Specifically, assuming that the four second coordinates, the two third coordinates or the first coordinate in the BRAM space is C(M,N), the RGB pixel value of point C can be obtained according to the above decoding method.

[0138] It can be understood that for the global coordinates (u, v) of the result image corresponding to each pixel point in the mapping sub-block, the global coordinates (u', v') of the original image corresponding to each pixel point coordinate can be calculated according to the inverse affine transformation matrix, and (u', v') is converted into the coordinate point (u", v") in the BRAM space coordinate system. For any coordinate point in the BRAM space coordinate system, the global coordinate in the original image can be obtained by adding the first coordinate of the cutout of the first preloaded sub-block, and the parity of the global coordinate in the original image is used as the parity of the coordinate point in the BRAM space coordinate system. That is, the parity of the global coordinate corresponding to the second coordinate, the third coordinate or the first coordinate in the BRAM space in the original image is used as the parity of the second coordinate, the third coordinate or the first coordinate.

[0139] Specifically, if Figure 9 As shown in the figure, when M and N are both odd numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points above and below point C; the G pixel value of point C is the Bayer value of point C; and the B pixel value of point C is the average of the Bayer values ​​of the two points left and right of point C.

[0140] Specifically, if Figure 10 As shown in the figure, when M is an odd number and N is an even number, the R pixel value of point C is the bayer value of point C; the G pixel value of point C is the average of the bayer values ​​of the four points above, below, left and right of point C; the B pixel value of point C is the average of the bayer values ​​of the four points in the upper left, lower left, lower right and upper right directions of point C.

[0141] Specifically, such as Figure 11 As shown in the figure, when M and N are both even numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C; the G pixel value of point C is the Bayer value of point C; and the B pixel value of point C is the average of the Bayer values ​​of the two points above and below point C.

[0142] Specifically, if Figure 12 As shown in the figure, when M is an even number and N is an odd number, the R pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right, and upper right directions of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left, and right of point C; the B pixel value of point C is the Bayer value of point C.

[0143] Compared with the prior art, the Bayer image decoding method based on FPGA affine inverse transformation provided by the embodiment of the present invention realizes Bayer image decoding while implementing affine transformation by storing the loaded original image data (first preloaded sub-block) in a BRAM space including 8 dual-port BRAMs, thereby saving resources; and determines the original image data to be loaded based on the affine inverse transformation by filling the mapping coordinates of the mapping sub-block, thereby improving the accuracy of calculating the original image data to be loaded.

[0144] Those skilled in the art will appreciate that all or part of the process steps of the above-described embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, such as a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0145] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the present invention should be covered by the scope of protection of the present invention.

Claims

1. A Bayer image decoding method based on FPGA affine inverse transform, characterized in that: The decoding method comprises: Determine a cutout sub-block according to the filling sub-block and the inverse affine transformation matrix, wherein the filling sub-block is a sub-block divided according to the result image, and the cutout sub-block is a sub-block of the original image; Determine a first preloaded sub-image block according to the cutout sub-image block and the BRAM space; store the first preloaded sub-image block in the BRAM space; the BRAM space includes 8 dual-port BRAMs; Traversing each pixel point in the mapping sub-block in sequence, and determining the first coordinate corresponding to the coordinate of each pixel point in the BRAM space according to the inverse affine transformation matrix and the first preloaded sub-block; Obtain the bayer value corresponding to the first coordinate from the BRAM space, decode the bayer value corresponding to the first coordinate based on the bayer image decoding, obtain the RGB pixel value corresponding to the first coordinate, and fill the RGB pixel value corresponding to the first coordinate as the RGB pixel value of each pixel point.

2. The decoding method according to claim 1, wherein: The storing the first preloaded sub-image block into the BRAM space includes: Divide the bayer values ​​stored in the first preloaded sub-block into a plurality of preset matrices in order from left to right and from top to bottom, where the length and width of the preset matrices are both 4 bayer values; Each of the preset matrices is divided into 8 sub-matrices, each of the sub-matrices includes 2 bayer values, and each of the 8 sub-matrices is stored in a dual-port BRAM.

3. The decoding method according to claim 2, wherein: The obtaining of the Bayer value corresponding to the first coordinate from the BRAM space, decoding the Bayer value corresponding to the first coordinate based on Bayer image decoding, and obtaining the RGB pixel value corresponding to the first coordinate includes: Let P(x, y) denote the first coordinate; When neither x nor y is an integer, determine four second coordinates around point P based on point P, decode the Bayer values ​​of the four second coordinates to obtain RGB pixel values ​​of the four second coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the four second coordinates based on a bilinear interpolation algorithm; When only one of x and y is an integer, determine two third coordinates around point P based on point P, decode the Bayer values ​​of the two third coordinates to obtain the RGB pixel values ​​of the two third coordinates, and calculate the RGB pixel value corresponding to the first coordinate using the RGB pixel values ​​of the two third coordinates based on a bilinear interpolation algorithm; When x and y are both integers, the Bayer value of the coordinate of point P is decoded to obtain the RGB pixel value corresponding to the first coordinate.

4. The decoding method according to claim 3, wherein: Decoding the Bayer values ​​of the four second coordinates to obtain RGB pixel values ​​of the four second coordinates includes: Four 3*3 matrices centered on the four second coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding second coordinates are decoded to determine the RGB pixel values ​​of the second coordinates.

5. The decoding method according to claim 3, wherein: Decoding the bayer values ​​of the two third coordinates to obtain RGB pixel values ​​of the two third coordinates includes: Two 3*3 matrices centered on the two third coordinates are respectively obtained from the BRAM space, and based on the bayer values ​​included in each 3*3 matrix, the bayer values ​​of the corresponding third coordinates are decoded to determine the RGB pixel values ​​of the third coordinates.

6. The decoding method according to claim 3, wherein: Decoding the Bayer value of the coordinate of point P to obtain the RGB pixel value corresponding to the first coordinate includes: A 3*3 matrix centered on the first coordinate is obtained from the BRAM space, and based on the bayer values ​​included in the 3*3 matrix, the bayer values ​​of the first coordinate are decoded to determine the RGB pixel values ​​of the first coordinate.

7. The decoding method according to any one of claims 4 to 6, characterized in that: Decoding the Bayer value of the coordinate to determine the RGB pixel value of the coordinate includes: Assume the coordinates are C(M, N); When M and N are both odd numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points above and below point C; the G pixel value of point C is the Bayer value of point C; the B pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C; When M is an odd number and N is an even number, the R pixel value of point C is the Bayer value of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left and right of point C; the B pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right and upper right directions of point C.

8. The decoding method according to claim 7, wherein: When M and N are both even numbers, the R pixel value of point C is the average of the Bayer values ​​of the two points to the left and right of point C; the G pixel value of point C is the Bayer value of point C; the B pixel value of point C is the average of the Bayer values ​​of the two points above and below point C; When M is an even number and N is an odd number, the R pixel value of point C is the average of the Bayer values ​​of the four points in the upper left, lower left, lower right, and upper right directions of point C; the G pixel value of point C is the average of the Bayer values ​​of the four points above, below, left, and right of point C; the B pixel value of point C is the Bayer value of point C.

9. The decoding method according to claim 1, wherein: The step of determining the cutout sub-block according to the filling sub-block and the inverse affine transformation matrix includes: Determine the mapping coordinates corresponding to the mapping sub-block; the mapping coordinates include the coordinates of four mapping vertices corresponding to the mapping sub-block; Based on the inverse affine transformation, determining the cutout coordinates of the cutout sub-block according to the four filling vertex coordinates and the inverse affine transformation matrix, wherein the cutout coordinates include the coordinates of the four cutout vertices; The cutout sub-block is determined according to the coordinates of the four cutout vertices.

10. The decoding method according to claim 9, characterized in that: The determining a first preloaded sub-image block according to the cutout sub-image block and the BRAM space includes: Determining the maximum value and the minimum value of the cutout sub-block on two coordinate axes according to the coordinates of the four cutout vertices of the cutout sub-block; The coordinates of the four vertices of the second preloaded sub-tile are determined based on the maximum and minimum values ​​on the two coordinate axes: A B C D Wherein, A, B, C and D represent the coordinates of the four vertices of the second preloaded sub-block, Xmax represents the maximum value of the cutout sub-block on the X-axis, Xmin represents the minimum value of the cutout sub-block on the X-axis, Ymax represents the maximum value of the cutout sub-block on the Y-axis, and Ymin represents the minimum value of the cutout sub-block on the Y-axis; A first preloaded sub-tile is determined based on the four vertex coordinates of the second preloaded sub-tile and the BRAM space, so that the bayer value included in the first preloaded sub-tile contains the bayer value included in the second preloaded sub-tile, and the size of the first preloaded sub-tile is less than or equal to the size of the BRAM space.

Images