Camera image distortion correction method based on FPGA

Abstract

The invention discloses a camera image distortion correction method based on an FPGA, and the method comprises the steps: 1, carrying out the frame caching of a received original distortion image, and reading a preset quadratic fitting curve coefficient; step 2, according to a preset quadratic fitting curve coefficient and a preset FPGA local line cache capacity, determining a block region through a coordinate calculation mode; step 3, according to the block regions, determining a backward reading region in the original distorted image line by line; 4, the original distortion image is read back according to the read-back area, the original distortion image is recorded as a read-back image, pixel values of pixel points around correction coordinates in the read-back image are read, and the correction coordinates are determined by a preset quadratic fitting curve coefficient; and step 5, determining the pixel value of the to-be-corrected point according to the pixel values of the pixel points around the correction coordinates in the read-back image, and generating a corrected image according to the pixel value of the to-be-corrected point. The problem of the demand for a large number of local storage resources of the FPGA is solved by processing backward reading of the distorted image in a block mode.

Description

[0001] This application is a divisional application for the invention patent with an application date of August 21, 2023, application number 2023110518355, and invention name "A camera image distortion correction method based on FPGA". Technical Field

[0002] The present invention relates to the field of image processing, and in particular to a camera image distortion correction method based on FPGA. Background Art

[0003] Cameras are important sensors in the field of computer vision, but the images captured by cameras will have more or less image distortion, such as barrel distortion and pincushion distortion. The direct cause of this type of image distortion is that the magnification of the edge and center of the lens is different, which is an inherent characteristic of optical lenses. This distortion causes only the image passing through the center area of ​​the image captured by the camera to remain straight, and the image of other parts will become curved, and the farther away from the center of the image, the greater the degree of deformation. Therefore, before performing image detection / recognition, it is necessary to eliminate the adverse effects of image distortion.

[0004] Most of the existing image distortion correction technology solutions adopt two implementation methods: backward mapping or forward mapping. The backward mapping is to map the integer coordinates of the corrected image to the distorted image, and interpolate the pixels around the mapping point to obtain the corresponding new pixels. In the implementation process of backward mapping, a certain size of image data needs to be cached. The size of the cached image is proportional to the degree of image distortion and image resolution. As the camera resolution becomes higher and the image distortion becomes greater, this will lead to an increasing demand for hardware storage resources in the camera.

[0005] Taking FPGA chips as an example, since general devices also require certain storage resources when designing / processing other logic, it is impossible to use most of the storage resources for image caching; even if other logics occupy relatively few storage resources, using a large amount of storage resources in the FPGA chip for image caching will also cause resource waste and increase the difficulty of layout and routing. Therefore, most design solutions will use external SDRAM (synchronous dynamic random access memory) or DDR (double data rate synchronous dynamic random access memory) for image caching, but since the external storage method requires real-time access when performing image distortion correction, this introduces the delay problem caused by frequent bus access and DDR bandwidth occupation, which in turn causes the FPGA chip processing frame rate to drop and the SOC processing speed that shares DDR with it to drop.

[0006] Therefore, technicians in this field urgently need to solve the problem of how to solve the problem of backward mapping occupying a large amount of storage resources of the FPGA chip when implementing image distortion correction using FPGA hardware, especially for images with large distortion and high resolution, and the problem of latency and low bandwidth utilization when using external memory such as DDR or SDRAM for image caching. Summary of the invention

[0007] The technical problem to be solved by this application is: how to solve the problem that the backward mapping method occupies a large amount of storage resources of the FPGA chip when implementing image distortion correction by FPGA hardware, especially for large distortion and high resolution images, and the problem of delay and low bandwidth utilization when using external memory such as DDR or SDRAM for image caching.

[0008] In order to solve the above technical problems, the present application proposes a camera image distortion correction method based on FPGA, the correction method comprising:

[0009] Step 1: Frame cache the received original distorted image, read the preset quadratic fitting curve coefficients, and read the preset quadratic fitting coefficients corresponding to the correction coordinates from the coefficient cache area in DDR to the local FPGA;

[0010] Step 2: Determine the block area by coordinate calculation according to the preset quadratic fitting curve coefficient and the preset FPGA local row cache capacity;

[0011] Any set of preset quadratic fitting curve coefficients corresponds to a row of pixels in the correction image. Step 2 specifically includes:

[0012] Step 2.1, construct the corresponding correction coordinate calculation equation according to any set of preset quadratic fitting curve coefficients;

[0013] Step 2.2, according to the correction coordinate calculation equation and the preset FPGA local row cache capacity, by means of coordinate calculation, the first starting point and the maximum block position of the first block area corresponding to each group of preset quadratic fitting curve coefficients are determined in turn;

[0014] Step 2.3, selecting the minimum value of the column coordinates in each maximum block position, and taking the corresponding maximum block position as the first end point of the first block area, and determining the first block area according to the first starting point and the first end point;

[0015] Step 2.4, determining a second starting point of the second block area according to the first end point and the corresponding correction coordinate calculation equation, repeating step 2.2, and determining each block area of ​​the original distorted image in turn;

[0016] Wherein, the first starting point is determined by the constant term coefficient in the preset quadratic fitting curve coefficient;

[0017] Step 3: Determine the read-back area in the original distorted image row by row according to the block area;

[0018] Step 4: according to the readback area, read back the original distorted image from the DDR and cache it locally, read back the original distorted image, record it as the readback image, and read the pixel values ​​of the pixel points around the correction coordinates in the readback image; wherein the correction coordinates are determined by the preset quadratic fitting curve coefficients;

[0019] Step 5: According to the pixel values ​​of the pixels around the correction coordinates in the read-back image, the pixel values ​​of the four pixels to be read corresponding to the correction coordinates read from the read-back image are interpolated to determine the pixel values ​​of the points to be corrected, and a corrected image is generated based on the pixel values ​​of the points to be corrected.

[0020] In any of the above technical solutions, step 2.2 further includes: rounding up the first starting point;

[0021] According to the preset FPGA local row cache capacity, the row coordinates of the first starting point after rounding are moved up or down; according to the moved row coordinates, the corresponding column coordinates are calculated using the corresponding correction coordinate calculation equation; the calculated column coordinates are rounded up, and the maximum block position is determined based on the moved row coordinates and the rounded column coordinates.

[0022] In any of the above technical solutions, according to the preset FPGA local row cache capacity, the row coordinates of the rounded first starting point are moved upward or downward, specifically including:

[0023] According to the correction coordinate calculation equation, a preset number of correction coordinates after the first starting point are obtained; according to the row coordinate change trend of the preset number of correction coordinates, the row coordinate moving direction is determined; according to the preset FPGA local row cache capacity and the row coordinate moving direction, the rounded row coordinate of the first starting point is moved.

[0024] In any of the above technical solutions, step 2.4, determining the second starting point of the second block area according to the first end point and the corresponding correction coordinate calculation equation, specifically includes:

[0025] By using the coordinate calculation method, according to the correction coordinate calculation equation, it is determined whether the column coordinates of the first end point correspond to the column coordinates of the correction coordinates. If so, the pixel point corresponding to the correction coordinate is used as the second starting point of the second block area. If not, the pixel point corresponding to the last correction coordinate contained in the first block area is used as the second starting point of the second block area.

[0026] In any of the above technical solutions, step 2.2 also includes: counting the number of corrected coordinates contained between the first starting point and the maximum block position in the first block area; determining whether the number is less than or equal to the preset local coordinate cache capacity, if so, executing step 2.3; if not, according to the preset local coordinate cache capacity, selecting the last corrected coordinate to update the maximum block position, and executing step 2.3.

[0027] In any of the above technical solutions, the preset quadratic fitting curve coefficients include an x-axis fitting coefficient and a y-axis fitting coefficient, the x-axis fitting coefficient is determined by fitting the horizontal axis coordinate of the corresponding correction coordinate in the original distorted image, and the y-axis fitting coefficient is determined by fitting the vertical axis coordinate of the corresponding correction coordinate in the original distorted image.

[0028] The calculation formula of the correction coordinate calculation equation is:

[0029]

[0030] Wherein, Fx(z) is the calculated row coordinate value of the zth point to be corrected corresponding to the corrected coordinates in the original distorted image, ax, bx, cx are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the x-axis fitting coefficient, respectively; Fy(z) is the column coordinate of the zth point to be corrected corresponding to the corrected coordinates in the original distorted image, ay, by, cy are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the y-axis fitting coefficient, respectively; z is the number of the point to be corrected, z=0,1,2,...,n, and n is the set number of points to be corrected.

[0031] In any of the above technical solutions, in step 4, the pixel values ​​of the pixel points around the correction coordinates in the read-back image are read, specifically including: rounding down the correction coordinates corresponding to the current point to be corrected in the read-back image; according to the rounded correction coordinates, determining the pixel point in the upper left corner as the first pixel point to be read, and taking the first pixel point to be read as the starting point, selecting the remaining three coordinate points adjacent to the first pixel point to be read in a clockwise direction as the second to fourth pixel points to be read; and reading the pixel values ​​of the four pixel points to be read from the read-back image.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present application provides a method for correcting camera image distortion based on FPGA, which introduces a method for reading back the distorted image from the hardware storage to the local and caching it in a block manner. According to the preset quadratic fitting curve coefficient and the preset FPGA local row cache capacity, the area where the distorted image needs to be read back is determined by coordinate calculation. According to the read-back area, the pixel values ​​of the pixel points around the correction coordinates in the read-back image are read, and the pixel values ​​of the points to be corrected are determined, thereby generating a corrected image. By reading back the distorted image from the hardware storage to the FPGA local and caching it in a block manner, only a few rows are cached for each image block, which solves the problem of the need for a large amount of FPGA local storage resources. Even under the requirements of large distortion and high resolution, the occupation of FPGA local storage resources for realizing the distortion correction function is reduced as much as possible, thereby improving the utilization rate of FPGA resources.

[0034] 2. The present application provides an FPGA-based camera image distortion correction method, which uses hardware storage to cache the entire frame image, reads back part of the image to the FPGA local storage and uses it for correction calculations, thereby avoiding the delay problem caused by directly operating the memory itself. The entire process uses a pipeline processing method, which greatly improves the processing speed, and the processing frame rate of the distortion correction is greatly improved.

[0035] 3. The camera image distortion correction method based on FPGA provided in the present application is compared with the traditional scheme that directly operates DDR and needs to access DDR once or multiple times for each point processed. The amount of data read back is large, and frequent bus operations occupy more DDR bandwidth, resulting in a decrease in the processing speed of the operating system. In the present application, the readback of the distorted image is processed in a block form, and the amount of image data read back in each block is at most the number of locally cached rows. In general, only the amount of data of the row difference between the current row and the previous row in the current block needs to be read back, which greatly reduces the occupation of DDR bandwidth for realizing the distortion correction function and ensures the efficiency of DDR use.

[0036] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application so that it can be implemented in accordance with the contents of the specification, and to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following is a detailed description of the preferred embodiments of the present application in conjunction with the accompanying drawings as follows.

[0037] Based on the detailed description of the specific embodiments of the present application in combination with the accompanying drawings below, those skilled in the art will become more aware of the above and other objects, advantages and features of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale.

[0039] Figure 1 This is a flow chart of the camera image distortion correction method based on FPGA in this application;

[0040] Figure 2 This is a schematic diagram of the correction coordinates and quadratic fitting coefficients in the embodiment of the present application;

[0041] Figure 3 A schematic diagram of a process of dividing a distorted image into blocks in an embodiment of the present application;

[0042] Figure 4 Another schematic diagram of the process of dividing a distorted image into blocks in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of a process for determining a readback position in an embodiment of the present application;

[0044] Figure 6 Schematic diagram of block division and caching of distorted images in an embodiment of the present application;

[0045] Figure 7 This is a schematic diagram of the bilinear interpolation principle in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, rather than all of the embodiments. In the following description, specific details such as specific configuration and components are provided only to help fully understand the embodiments of the present application. Therefore, it should be clear to those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. In addition, for clarity and brevity, the description of known functions and structures is omitted in the embodiment.

[0047] It should be understood that the references to "one embodiment" or "this embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "one embodiment" or "this embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0048] In addition, the present application may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplicity and clarity, and does not in itself indicate the relationship between the various embodiments and / or settings discussed.

[0049] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" in this article describes another type of association object relationship, indicating that there can be two relationships. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship.

[0050] The term "at least one" in this article is merely a description of the association relationship of associated objects, indicating that there may be three relationships. For example, at least one of A and B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0051] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusions.

[0052] Example 1

[0053] The existing image distortion correction method can achieve distortion correction by backward mapping. Backward mapping is to map the integer coordinates of the corrected image to the distorted image, and interpolate the pixels around the mapping point to obtain the corresponding new pixels. In the implementation of backward mapping, it is necessary to cache image data of a certain size. The size of the cached image is proportional to the degree of image distortion and image resolution. As the camera resolution becomes higher and the image distortion becomes greater, this will lead to an increasing demand for hardware storage resources in the camera. Therefore, for distorted images with large distortion and high resolution, the backward mapping method occupies a large amount of FPGA storage resources, and when using external hardware memory for image caching, it will also bring a large delay and low bandwidth utilization.

[0054] To this end, this embodiment provides a camera image distortion correction method based on FPGA. In this correction method, an idea of ​​reading back the distorted image from the hardware storage to the local and caching it in a block manner is introduced, which solves the problem of large occupation of FPGA local storage resources during image distortion correction. Even in the case of large distortion and high resolution, the occupation of FPGA local storage resources by distortion correction can be effectively reduced, thereby improving the utilization rate of FPGA resources.

[0055] Combined with Figure 1 As shown, the correction method in this embodiment includes: step 1, frame caching the received original distorted image, and reading the preset quadratic fitting curve coefficients; step 2, determining the block area by coordinate calculation according to the preset quadratic fitting curve coefficients and the preset FPGA local row cache capacity; step 3, determining the readback area in the original distorted image row by row according to the block area; step 4, reading back the original distorted image according to the readback area, recording it as the readback image, and reading the pixel values ​​of the pixel points around the correction coordinates in the readback image, wherein the correction coordinates are determined by the preset quadratic fitting curve coefficients; step 5, determining the pixel value of the point to be corrected according to the pixel values ​​of the pixel points around the correction coordinates in the readback image, and generating a corrected image according to the pixel value of the point to be corrected.

[0056] Those skilled in the art can understand that, for a camera, the coordinate correspondence between each pixel in the rectified image and the original distorted image before and after correction is obtained by calculating the camera internal distortion parameters obtained by camera calibration based on the pinhole camera mathematical model and the camera distortion mathematical model. Therefore, after the above parameters of the camera are determined, the coordinates of each pixel in the rectified image mapped to the distortion map are fixed.

[0057] In this embodiment, the image obtained by taking a camera is recorded as the original distorted image, the image generated after distortion correction is recorded as the corrected image, the pixel points in the corrected image are recorded as points to be corrected, the corresponding coordinates are recorded as coordinates of the points to be corrected, and the coordinates of the points to be corrected mapped to the coordinates in the original distorted image are recorded as corrected coordinates.

[0058] Considering that if the correction coordinates are used directly during the distortion correction process, the memory size required to store the correction coordinates in the DDR and the bandwidth required to read them back from the DDR will become very large as the image resolution increases. Therefore, this embodiment fits two quadratic curves based on the x-axis coordinates and y-axis coordinates of a number of correction coordinates, which are recorded as preset quadratic fitting curves. Only the coefficients of the preset quadratic fitting curve (preset quadratic fitting curve coefficients) are stored in the DDR. When performing distortion correction, the preset quadratic fitting curve coefficients are read from the DDR to the local FPGA, and then the correction coordinates can be obtained by analysis through the inverse fitting operation.

[0059] like Figure 2 As shown, those skilled in the art can understand that the position coordinates of each point are 8 bytes, including 4 bytes for horizontal and vertical coordinates, each of which contains a 2-byte integer part and a 2-byte decimal part; each quadratic fitting curve requires 3 coefficients, each of which is 4 bytes. This embodiment can be set to use 2 preset quadratic fitting curves that fit the horizontal and vertical coordinates with every 64 correction point coordinates. Calculated in units of 64 correction point coordinates, directly using the coordinates requires 512 bytes, while using the quadratic fitting solution only requires 24 bytes. In this case, the scheme for reading the preset quadratic fitting curve coefficients proposed in this embodiment reduces about 95% of resources in DDR memory and bandwidth usage compared to the traditional scheme of directly using the correction coordinates.

[0060] It should be noted that any set of preset quadratic fitting curve coefficients corresponds to a row of pixel point coordinates in the corrected image.

[0061] Specifically, when the resolution of the corrected image is 1920*1080, each row of pixels can be divided into 30 segments from left to right with 64 pixels as a group. Each segment of pixels can be fitted to obtain a quadratic fitting curve, corresponding to a quadratic fitting curve coefficient, and each group of pixels (points to be corrected) are marked as 0-63 in sequence.

[0062] Therefore, every 30 segments of quadratic fitting curve coefficients can form a set of preset quadratic fitting curve coefficients, corresponding to a row of pixels in the corrected image. Therefore, a total of 30*1080 segments and 1080 sets of preset quadratic fitting curve coefficients can be obtained.

[0063] After obtaining the preset quadratic fitting curve coefficients by reading back, the numbers of the points to be corrected are sequentially substituted, and the corresponding correction coordinates in the original distorted image can be obtained by coordinate calculation. After that, the first correction coordinate corresponding to each set of preset quadratic fitting curve coefficients is used as the starting point of the block area according to the preset FPGA local row cache capacity, for example, 8 rows. After moving 8 rows, the last correction coordinate of the corresponding number of rows is used as the end point of the current block area, and the correction coordinate corresponding to this end point is used as the starting point of the next block area. The above process is repeated until the block areas corresponding to all sets of preset quadratic fitting curve coefficients are determined.

[0064] After that, when correcting each frame of the original distorted image, the block area of ​​the image will remain unchanged. Based on the determined block area, the readback area of ​​the corrected image corresponding to the original distorted image is determined row by row. First, the column coordinates of the correction coordinates in the original distorted image are compared with the block area to determine the block in which it is located, and the column starting position and number of columns of the image to be read back are obtained; then, the minimum row coordinate of all the correction coordinates in the current row in each block area in the original distorted image is found as the row starting position of the readback area; finally, the number of rows of the preset FPGA local row cache capacity is determined downward or upward, that is, 8 rows are moved downward or upward, thereby determining the area of ​​the original distorted image that needs to be read back, that is, the readback area.

[0065] When correcting the next row, repeat the above process. When the readback area obtained at this time is compared with the readback area of ​​the previous corrected row, only a few rows have changed or no change has occurred. It is only necessary to update the number of the changed image rows to meet the correction processing requirements of the current row to be corrected. For example, when the current readback area is compared with the readback area of ​​the previous corrected row, only 2 rows have changed. At this time, only the changed 2 rows need to be read.

[0066] Therefore, after blocking, the image data that needs to be read back for each correction row is at most the locally set row cache capacity, and the amount of image data that needs to be read back after the entire frame correction is completed does not exceed the size of one frame of image. Compared with the amount of data that requires reading back at least 4 frames of images in the traditional DDR solution, the DDR bandwidth occupancy is reduced by more than 75%.

[0067] Furthermore, according to the read-back area determined as above, the original distorted image can be read back, and then pixel value operations (such as interpolation operations) can be performed according to the pixel values ​​of the pixel points around the correction coordinates to calculate the corresponding pixel values ​​to be corrected, generate the final corrected image to be output, and complete the image distortion correction.

[0068] In some embodiments, a method for determining a block area in the above-mentioned embodiment 1 is shown, and the step 2 specifically includes: step 201, constructing a corresponding correction coordinate calculation equation according to any set of preset quadratic fitting curve coefficients; step 202, according to the correction coordinate calculation equation, calculating each correction coordinate in the original distorted image in turn by coordinate calculation; step 203, according to the preset FPGA local row cache capacity, in accordance with the order of the correction coordinates, sequentially determining the first starting point and the maximum block position of the first block area corresponding to each set of preset quadratic fitting curve coefficients; step 204, selecting each maximum block position The minimum value of the column coordinates in the image, and the corresponding maximum block position is used as the first end point of the first block area, and the first block area is determined according to the first starting point and the first end point; step 205, according to the first end point and the correction coordinates, the second starting point of the second block area is determined, and step 203 is repeated to determine the respective block areas of the original distorted image in turn; wherein the first starting point is the first correction coordinate corresponding to the preset quadratic fitting curve coefficient, and the second starting point is the first correction coordinate on the right side of the first end point in the correction coordinates, or the second starting point is the correction coordinate with the same column coordinates as the first end point in the correction coordinates.

[0069] Specifically, by reading back the preset quadratic fitting curve coefficients, the corresponding curve and the corresponding correction coordinate calculation equation can be fitted, and then all the correction coordinates for pixel correction in the original distorted image can be calculated by coordinate calculation.

[0070] First, the first correction coordinate corresponding to each set of preset quadratic fitting curve coefficients is used as the first starting point. According to the preset FPGA local row cache capacity (such as 8 rows), in the order of the correction coordinates, when the difference between the selected correction coordinate and the row coordinate of the first starting point reaches 8, the correction coordinate is the maximum block position corresponding to the first starting point.

[0071] Then, in order to make the limited read-back image meet the requirements of the maximum distortion correction, the original distorted image is divided into several regions (blocks) with the same number of columns in the horizontal direction, and the corresponding regional positions of each row in the original distorted image are unified to reduce the number of parameters of the regional block positions, and the minimum value of the column coordinates in each maximum block position corresponding to each set of preset quadratic fitting curve coefficients is selected, and the corresponding coordinate point is used as the first end point. The first block area can be determined according to the first starting point and the first end point.

[0072] Finally, after the first block area is determined, the last corrected coordinate contained in the area may be located on the boundary of the area (the corrected coordinate is the same as the minimum value of the column coordinate in the maximum block position), or may be located on the right side of the area boundary. Therefore, the first corrected coordinate on the right side of the first end point, or the corrected coordinate that is the same as the column coordinate of the first end point is selected as the second starting point of the second block area (the next block area), and step 203 is repeatedly performed to determine the maximum block position corresponding to each second starting point. The above process is repeated to determine each block area of ​​the original distorted image in turn.

[0073] Afterwards, the read-back area in the corrected image corresponding to the original distorted image is determined line by line.

[0074] In some embodiments, step 203 specifically includes: selecting the first correction coordinate corresponding to each group of preset quadratic fitting curve coefficients, and recording it as the first starting point of the first block area; calculating the row difference between the row coordinates of each correction coordinate after the first starting point and the row coordinates of the first correction coordinate in sequence according to the order of the correction coordinates; when the row difference is greater than the preset FPGA local row cache capacity, selecting the previous correction coordinate and recording it as the maximum block position of the first block area.

[0075] In order to make the limited readback image meet the requirements of correcting the maximum distortion, the distorted image is horizontally divided into several areas with the same number of columns, and in order to reduce the number of parameters of the block position, the idea of ​​determining the maximum block position is introduced to unify the position of each row of block areas. In some embodiments, another method for determining the block area in the above embodiment 1 is also shown, and the step 2 includes: step 2.1, according to any set of preset quadratic fitting curve coefficients, constructing a corresponding correction coordinate calculation equation; step 2.2, according to the correction coordinate calculation equation and the preset FPGA local row cache capacity, by coordinate calculation , determine the first starting point and the maximum block position of the first block area corresponding to each group of correction coordinate calculation equations in turn; step 2.3, select the minimum value of the column coordinates in each maximum block position, and use the corresponding maximum block position as the first end point of the first block area, and determine the first block area according to the first starting point and the first end point; step 2.4, determine the second starting point of the second block area according to the first end point and the corresponding correction coordinate calculation equation, repeat step 2.2, and determine each block area of ​​the original distorted image in turn; wherein the first starting point is determined by the constant term coefficient in the preset quadratic fitting curve coefficient.

[0076] Specifically, Figure 3 and Figure 4As shown, taking barrel distortion as an example, the horizontal midline of the original distorted image divides the distorted image into two parts, an upper part and a lower part, and the first correction coordinate corresponding to each set of preset quadratic fitting curve coefficients is used as the first starting point (such as point A), where the first starting point includes row coordinates and column coordinates.

[0077] For the upper image, according to the preset FPGA local row cache capacity (e.g., 8 rows), the row coordinates of the first starting point are moved up by 8 rows, i.e., the coordinate value is -8, and the column coordinates remain unchanged. The intersection of the 8th row after upward movement and the curve corresponding to the quadratic fitting curve coefficient of the row can be determined, and the intersection is the maximum block position. For the lower image, it is moved down by 8 rows, and the rest of the process is the same.

[0078] It should be noted that, for pincushion distortion, the upper part of the image moves downward, and the lower part of the image moves upward.

[0079] Then, the first starting point and the maximum block position of the next set of preset quadratic fitting curve coefficients are determined until the maximum block positions of all sets of preset quadratic fitting curve coefficients are determined.

[0080] In order to ensure the consistency of the size of the block area and reduce the DDR bandwidth occupied when the image is read back, the minimum value of the column coordinates in the above-mentioned maximum block positions is selected as the column coordinates of the first end point, and the row coordinates after moving 8 rows are used as the row coordinates of the first end point. The first end point (such as point B) of the first block area corresponding to all groups of preset quadratic fitting curve coefficients is combined to obtain the first block area.

[0081] Then, according to the column coordinates of the first end point and the corresponding correction coordinate calculation equation, the intersection point (such as point C) of the first block area and the quadratic fitting curve corresponding to the preset quadratic fitting curve coefficient is calculated, and the intersection point is used as the second starting point of the second block area. The above process is repeated to determine each block area of ​​the original distorted image.

[0082] In some embodiments, considering that the correction coordinates mapped to the original distorted image are not continuous and may be decimals, the left boundary of the block area may be the same as the column coordinates of a certain correction coordinate, and the pixel point corresponding to the correction coordinate is located on the left boundary of the block area; the left boundary of the block area may also be located between the pixel points corresponding to the two correction coordinates. Therefore, in order to ensure the accuracy of the starting point of the next block area and to obtain accurate pixel values ​​in the subsequent pixel correction operation, the above step 2.4 also includes: using the coordinate calculation method, according to the correction coordinate calculation equation, judging whether the column coordinate of the first end point corresponds to the column coordinate of the correction coordinate, if so, it indicates that the pixel point corresponding to the correction coordinate is located on the left boundary of the block area, so the pixel point is used as the second starting point of the second block area; if not, it indicates that the left boundary of the block area is located between the pixel points corresponding to the two correction coordinates, at this time, the pixel point corresponding to the last correction coordinate contained in the first block area (the previous block area) is used as the second starting point of the second block area, so as to ensure that the next block area can contain all the pixel points required for the correction of the corresponding correction coordinates, so as to ensure that the subsequent pixel correction operation can obtain accurate pixel values.

[0083] In some embodiments, the preset quadratic fitting curve coefficients include an x-axis fitting coefficient and a y-axis fitting coefficient, the x-axis fitting coefficient is determined by fitting the horizontal axis coordinate of the corresponding correction coordinate in the original distorted image, and the y-axis fitting coefficient is determined by fitting the vertical axis coordinate of the corresponding correction coordinate in the original distorted image. The calculation formula of the correction coordinate calculation equation is:

[0084]

[0085] In the formula, F x (z) is the row coordinate of the zth point to be corrected corresponding to the corrected coordinate in the original distorted image, a x 、b x 、c x They are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the x-axis fitting coefficient, respectively. y (z) is the column coordinate of the zth point to be corrected corresponding to the corrected coordinates in the original distorted image, a y 、b y 、c y They are the quadratic term coefficient, linear term coefficient and constant term coefficient in the y-axis fitting coefficient respectively. z is the number of the point to be calibrated, z=0,1,2,...,n, and n is the number of the set points to be calibrated.

[0086] In some embodiments, step 2.2 also includes: rounding up the first starting point; moving the row coordinates of the rounded first starting point up or down by M rows according to the preset FPGA local row cache capacity; calculating the corresponding column coordinates according to the moved row coordinates using the corresponding correction coordinate calculation equation; rounding up the calculated column coordinates, and determining the maximum block position according to the moved row coordinates and the rounded column coordinates.

[0087] In some embodiments, the row coordinates of the first starting point after rounding are moved upward or downward by M rows according to the preset FPGA local row cache capacity, specifically including: when it is determined that the distortion type of the original distorted image is barrel distortion, it is determined whether the row coordinates of the first starting point after rounding are less than or equal to 1 / 2 of the number of rows of the original distorted image; if so, the row coordinates of the first starting point after rounding are moved upward by M rows according to the preset FPGA local row cache capacity; if not, the row coordinates of the first starting point after rounding are moved downward by M rows according to the preset FPGA local row cache capacity. When it is determined that the distortion type of the original distorted image is pincushion distortion, it is determined whether the row coordinates of the first starting point after rounding are less than or equal to 1 / 2 of the number of rows of the original distorted image; if so, the row coordinates of the first starting point after rounding are moved downward by M rows according to the preset FPGA local row cache capacity; if not, the row coordinates of the first starting point after rounding are moved upward by M rows according to the preset FPGA local row cache capacity.

[0088] In some embodiments, the row coordinates of the first starting point after rounding are moved up or down according to a preset FPGA local row cache capacity, specifically including: according to a correction coordinate calculation equation, a preset number of correction coordinates after the first starting point are obtained, wherein the preset number can be manually set; according to the row coordinate change trend of the preset number of correction coordinates, the row coordinate moving direction is determined, that is, when the row coordinates show a decreasing trend, it indicates that the corresponding quadratic fitting curve is bent upward, and the row coordinate moving direction should be upward at this time, and when the row coordinates show an increasing trend, it indicates that the corresponding quadratic fitting curve is bent downward, and the row coordinate moving direction should be downward at this time; according to the preset FPGA local row cache capacity and the row coordinate moving direction, the row coordinates of the first starting point after rounding are moved.

[0089] In some embodiments, the method for determining the maximum block position can also be: sequentially counting the values ​​of the row coordinates in the adjacent corrected coordinates after the first starting point, when the row difference reaches the preset FPGA local row cache capacity, the corresponding position is the maximum block position. If it is assumed that the local row cache capacity is M rows, the coordinates of each point on the curve in the distorted image can be calculated, a starting point is determined, and the starting point is used as the first coordinate point on the curve, and the row difference between the first coordinate point and the second coordinate point is calculated to be greater than or equal to M. If not, the row difference between the first coordinate point and the third coordinate point is calculated until the row difference between the last coordinate point on the curve and the first coordinate point is greater than or equal to M. At this time, the last coordinate point is used as the final maximum block position.

[0090] In some embodiments, the above step 2.2 also includes: counting the number of corrected coordinates contained between the first starting point and the maximum block position in the first block area; determining whether the number obtained by counting is less than or equal to the preset local coordinate cache capacity N, if so, executing step 2.3; if not, according to the preset local coordinate cache capacity N, starting from the first starting point, selecting the last corrected coordinate among the N corrected coordinates to update the maximum block position, and executing step 2.3.

[0091] In this embodiment, the distorted image is read back in blocks, and the amount of image data read back in each block is at most the number of locally cached rows. In general, only the amount of data of the row difference between the current row and the previous row in the current block needs to be read back, which greatly reduces the occupation of the hardware memory bandwidth for realizing the distortion correction function and ensures the efficient use of the memory.

[0092] In some embodiments, in step 4, the pixel values ​​of the pixel points around the correction coordinates in the read-back image are read, specifically including: rounding down the correction coordinates in the read-back image corresponding to the current point to be corrected; according to the rounded correction coordinates, determining the pixel point in the upper left corner as the first pixel point to be read, and taking the first pixel point to be read as the starting point, selecting the remaining three coordinate points adjacent to the first pixel point to be read in a clockwise direction as the second to fourth pixel points to be read; and reading the pixel values ​​of the four pixel points to be read from the read-back image.

[0093] In some embodiments, in step 5, the pixel value of the point to be corrected is determined according to the pixel values ​​of the pixel points around the correction coordinates in the read-back image, specifically including: determining the pixel value of the point to be corrected by interpolation according to the pixel values ​​of the four pixel points to be read corresponding to the correction coordinates read from the read-back image, and the interpolation principle is as follows: Figure 7 As shown. The calculation formula of the pixel value of the point to be corrected is as follows:

[0094] X=X lu *(1-a)*(1-b)+Xru *a*(1-b)+X ld *(1-a)*b+X rd *a*b

[0095] In the formula, X is the pixel value of the point to be corrected, X lu , X ru , X ld , X rd are the pixel values ​​of the four pixels to be read, and a and b are the decimal parts of the column coordinates and the row coordinates in the correction coordinates, respectively.

[0096] The FPGA-based camera image distortion correction method proposed in this embodiment only needs to cache a few lines regardless of the degree of image distortion, and the FPGA local storage resources used are relatively fixed and limited. Compared with the cache solution that caches as many lines as the distortion before segmentation, it can greatly alleviate the pressure on FPGA storage resources when the resolution is large and the degree of distortion is large.

[0097] Example 2

[0098] Based on Example 1, this example takes barrel distortion as an example to specifically describe a camera image distortion correction method based on FPGA. The correction method includes the following steps:

[0099] Step 100: Frame buffering of the received original distorted image; this step 1 is to receive the original distorted image and then write the original distorted image into a frame buffer area reserved in the DDR through a high-speed data bus interface.

[0100] In the implementation process of step 1, the FPGA includes two modules, one is the image_cache module, which is used to receive images, cache a burst transmission length, and complete axi transmission data control; the other is the axi_datamover module, which is used to handle data interaction with the axi high-speed data interface.

[0101] Step 200: Read the preset quadratic fitting curve coefficients; The main function of step 2 is to read the preset quadratic fitting coefficients corresponding to the correction coordinates from the coefficient buffer in the DDR to the local FPGA. During implementation, the FPGA includes two modules, one is the fit_coef_read module, which is used to control the alternate issuance of commands for reading the left and right image fitting coefficients in the DDR; the other is the axi_datamover module, which is used to handle data interaction with the axi high-speed data interface.

[0102] In this embodiment, the backward mapping implementation method is adopted. The correspondence between the pixel coordinates before and after correction is obtained by calculating the camera internal distortion parameters obtained by camera calibration based on the pinhole camera mathematical model and the camera distortion mathematical model. The calculation process is not described here. Therefore, after the above parameters of the camera are determined, the coordinates of each pixel in the corrected image mapped to the original distortion map are fixed.

[0103] The coordinates of the pixel points in the corrected image are mapped to the coordinates in the original distorted image as the corrected coordinates. However, if the corrected coordinates are used directly during the correction process, the memory size required to store the corrected coordinates in the DDR and the bandwidth required to read them back from the DDR will become very large as the resolution increases. Therefore, in this implementation, two quadratic curves are fitted based on the x-axis coordinates and y-axis coordinates of several points to be corrected, and only the quadratic fitting curve coefficients are stored in the DDR. After the quadratic fitting curve coefficients are read back from the DDR to the local FPGA, the corrected coordinates are obtained by the inverse fitting operation. The corrected coordinates are determined by the coefficients of the quadratic fitting curve.

[0104] In the process of quadratic fitting and inverse fitting, because of image distortion, after each row of pixels in the corrected image is mapped to the original distorted image, the line connecting the corrected coordinates is close to a curve. Therefore, the order of magnitude of the error generated by the quadratic fitting and inverse fitting process can be 10 -4 -10 -5 , which can ensure that the calculation of the pixel value (grayscale value) of the correction point has only a small loss of accuracy during the correction process, and does not affect the integer bits of the calculated pixel value.

[0105] like Figure 2 As shown in the figure, the position coordinates of each point to be corrected are 8 bytes, including 4 bytes for horizontal and vertical coordinates, each of which contains a 2-byte integer part and a 2-byte decimal part. Each quadratic fitting curve requires 3 coefficients, each of which is 4 bytes. Two quadratic curves of horizontal and vertical coordinates are fitted using every 64 points to be corrected. Calculated in units of 64 coordinates of the points to be corrected, directly using the coordinates requires 512 bytes, while using the quadratic fitting solution only requires 24 bytes.

[0106] Step 300: Calculate the coordinates of the points to be corrected in the original distorted image; specifically, cache the correction coordinate fitting coefficients read in step 2 locally, and parse each set of quadratic term coefficients, linear term coefficients, constant term coefficients, etc. corresponding to the horizontal and vertical coordinates according to the protocol; through discrete fitting inverse operations, each set of coefficients can obtain the coordinates of several points to be corrected; wherein, the coordinates of the points to be corrected include their integer coordinates and decimal coordinates in the original distorted image.

[0107] Assuming that each curve read back and parsed from DDR is obtained by fitting 64 points to be calibrated, each quadratic fitting curve can also reversely calculate the coordinates of the 64 points to be calibrated. For any group of 64 points to be calibrated, the corresponding calculation formula is as follows:

[0108] F x (z) = a x *z 2 +b x *z+c x

[0109] F y (z) = a y *z 2 +b y *z+c y

[0110] In the above formula, F x (z) is the row coordinate of the zth point to be corrected corresponding to the corrected coordinate in the original distorted image, a x 、b x 、c x They are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the x-axis fitting coefficient, respectively. y (z) is the column coordinate of the zth point to be corrected corresponding to the corrected coordinates in the original distorted image, a y 、b y 、c y are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the y-axis fitting coefficient, respectively. z is the number of the point to be corrected, z = 0, 1, 2, ..., n, and n is the number of points to be corrected. Therefore, the coordinates of the point to be corrected in the original distorted image calculated in reverse are (F x (z),F y (z)).

[0111] Step 400: Determine the readback area by coordinate calculation according to the preset quadratic fitting curve coefficient and the preset FPGA local row buffer capacity. Step 400 is to determine the position information of the original distorted image required for the current row correction that can be read back from the DDR frame memory.

[0112] This embodiment takes barrel distortion as an example, and first divides the entire original distorted image into several blocks, as shown in the attached figure. Figure 3 , 4 As shown, for an original distorted image to be subjected to barrel distortion correction, the process of dividing the blocks includes the following steps:

[0113] Step 410: Taking the coordinates of the first point to be corrected corresponding to each row of quadratic fitting curve coefficients in the original distorted image as the starting point, according to the preset FPGA local row cache capacity (such as 8 rows) and the FPGA local coordinate cache capacity (such as 100 coordinate points), determine the maximum block position of the first point to be corrected corresponding to each row of quadratic fitting curve coefficients in turn, which is the column coordinate.

[0114] In this embodiment, the horizontal midline of the original distorted image divides the original distorted image into upper and lower parts;

[0115] For the upper image, after determining the coordinates of the first point to be corrected in the original distorted image, use it as the first starting point and move it upward by 8 rows (or downward by 8 rows for pincushion distortion) according to the preset FPGA local row cache capacity, that is, the row coordinate value is -8 and the column coordinate value remains unchanged. At this time, combined with the corresponding quadratic fitting curve coefficient, the intersection of the 8th row after upward movement and the curve corresponding to the quadratic fitting curve coefficient of this row can be determined.

[0116] It should be noted that since the coordinates of the point to be corrected in the original distorted image may be decimals, before determining the starting point, the first starting point is rounded up and the rounded first starting point is moved. This ensures that the data below the point can be obtained when reading back as a basis for interpolation.

[0117] Because the coordinates of the points to be corrected in the original distorted image have been calculated in step 300, the number of points to be corrected between the intersection and the first starting point can be counted. If the number is less than or equal to the set FPGA local coordinate cache capacity, the column coordinates corresponding to the intersection are determined to be the maximum block position corresponding to the current starting point of the set of quadratic fitting curve coefficients.

[0118] If this number is greater than the set FPGA local coordinate cache capacity, then starting from the starting point, select the FPGA local coordinate cache capacity points to be corrected, and use the column coordinates of the correction coordinates corresponding to the last selected point to be corrected as the maximum block position corresponding to the current starting point of the set of quadratic fitting curve coefficients.

[0119] For the lower part of the image, move down 8 lines and repeat the above process.

[0120] Then, the above process is repeated for the next row of quadratic fitting curve coefficients until the maximum block position corresponding to the quadratic fitting curve coefficients of all rows is determined.

[0121] Step 420: Select the minimum value among the positions of each maximum block as the first vertical block boundary of each row of quadratic fitting curve coefficients.

[0122] Step 430: Count the correction coordinates of the last point to be corrected of the quadratic fitting curve coefficients of each row corresponding to the current vertical block boundary, and determine whether the coordinates are on the vertical block boundary; if so, use the coordinates as the starting position, re-execute step 410, and determine the next maximum block position of each row in turn; if not, use the next point to be corrected in the current row as the starting position, re-execute step 410, and determine the next maximum block position of each row in turn.

[0123] Steps 410-430 are executed repeatedly until the barrel-shaped original distorted image is divided into blocks. After the block division is completed, the leftmost column coordinates of the block (col1, col2, ..., col n ) and the number of columns (col_num1, col_num2, ..., col_num n-1 ) is written to the FPGA local cache, such as Figure 5 shown.

[0124] When implementing the above process, the FPGA includes three modules: one is the column_block module, which is used to store the results obtained by the above block division algorithm; one is the column_point module, which is used to preprocess and delay the coordinates obtained in step 3; and the other is the readback_positin module, which first needs to compare the column coordinates of the point to be corrected in the original distorted image with the block position, determine the block where it is located, and obtain the column starting position and number of columns that need to be read back; then it needs to find the minimum row coordinates of all the points to be corrected in each block of the current row in the original distorted image as the row starting position for reading back, so as to determine the area of ​​the original distorted image that needs to be read back.

[0125] like Figure 5 As shown, Figure 5 The middle curve ① is corrected, and the part of the original distorted image that needs to be read back in the first block is the solid frame area. Among them, the leftmost column coordinates of the block, col1, col2, etc. and the column numbers, col_num1, col_num2, etc., have been obtained through the above block division process and cached in the FPGA local RAM.

[0126] By traversing all the points of curve ① in the first block, we can get the minimum row position of this part of the curve in the original distorted image, that is, row1, which is used as the starting position of the row to be read back; combined with the number of columns col_num1, we can determine the number of columns that need to be read back at this time.

[0127] The number of rows read is determined by the FPGA local row cache capacity (eg, 8 rows), that is, row1 is shifted downward by 8 rows.

[0128] In summary, we can get the original distorted image area that needs to be read back in the current block for the current correction line. Similarly, the part of the original distorted image that needs to be read back in the second block is the dotted box area.

[0129] Step 500: read back the original distorted image from the DDR and cache it locally, wherein the read-back original distorted image is recorded as the read-back image.

[0130] The FPGA implementation of this process includes three modules: one is the image_readback module, which further judges the position obtained in step 400. If the current block needs to update the original distorted image, it issues an axi command to retrieve the corresponding part of the original distorted image from the DDR. Otherwise, there is no need to initiate a readback operation; one is the axi_datamover module, which is used to handle data interaction with the axi high-speed data interface; and the other is the line_buffer module, which is used to store the original distorted image retrieved from the DDR.

[0131] In order to minimize the DDR bandwidth usage and improve the utilization rate when reading back the original distorted image, the entire original distorted image is first divided into several blocks, such as Figure 6 Assume that the distribution curve of the row to be corrected in the distortion map is as follows: Figure 6 As shown in the middle curve ①, the blocks divided in this implementation will split the curve ① into several parts. The curve corresponding to each part only needs to retrieve the corresponding several rows of original distorted images from DDR to complete the correction processing of the current row and output the first row of corrected pixels of the corrected image.

[0132] like Figure 6 Curve ② in the middle is the next row distribution curve to be corrected after curve ①. It is also divided into several parts by the divided block. According to row 1 corresponding to the curve, the starting position of the minimum row is determined. Similarly, it moves down 8 rows to determine the original distorted image area that needs to be read back in the current block.

[0133] By comparing the original distorted image areas that need to be read back between curve ① and curve ②, the corresponding original distorted image lines that need to be retrieved from DDR only have a few lines changed or have not changed. Only the changed part of the image needs to be updated to meet the correction processing requirements of the lines to be corrected corresponding to curve ②.

[0134] The above process can be determined by comparing the values ​​of row.

[0135] By analogy, the image that needs to be read back for each correction row after block division is at most the locally set row cache capacity, and the image data amount that needs to be read back after the whole frame correction is completed does not exceed the size of one frame of image. Compared with the data amount that needs to read back at least 4 frames of images in the traditional DDR solution, this application reduces the DDR bandwidth occupation by more than 75%. It solves the problem that when directly operating DDR in the traditional solution, each point needs to be accessed once or multiple times, the amount of data read back is relatively large, and frequent bus operations will occupy more DDR bandwidth, which may cause the operating system processing speed to decrease.

[0136] In some embodiments, it is also necessary to pay attention to the following when dividing blocks:

[0137] 1) The maximum row difference between all the points to be corrected in the same block of each correction row and the original distorted image does not exceed the capacity of the FPGA local row cache, thus controlling the storage resources occupied by the original distorted image cache;

[0138] 2) If the point to be corrected in the previous block is at the boundary of the division, the right image required for the correction of the boundary point of the previous block needs to be read back when the next block image is read back to ensure the correctness of the correction;

[0139] 3) The maximum number of points to be corrected in each block does not exceed the capacity of the FPGA local coordinate cache, controlling the storage resources occupied by the coordinate cache.

[0140] Step 600: Read the pixel values ​​of the pixel points around the correction coordinates in the read-back image;

[0141] In this embodiment, step 600 needs to read out the pixel values ​​of points around the coordinates from the line_buffer of step 500 according to the position of the point to be corrected in the original distorted image obtained in step 300.

[0142] When determining the surrounding pixel points, the correction coordinates of the current point to be corrected are rounded down, and based on the rounded correction coordinates, the pixel point in its upper left corner is determined as the first pixel point to be read, and the first pixel point to be read is taken as the starting point, and the remaining three coordinate points adjacent to the first pixel point to be read are selected in a clockwise direction as the second to fourth pixel points to be read; then the pixel values ​​of the four pixel points to be read are read out from the read-back image of step 500.

[0143] Step 700: Determine the pixel value of the point to be corrected according to the pixel values ​​of the pixels around the correction coordinates in the read-back image, and generate a corrected image according to the pixel values ​​of the point to be corrected. Specifically, it includes: determining the pixel value of the point to be corrected by interpolation according to the pixel values ​​of the four pixels to be read corresponding to the correction coordinates read from the read-back image. The interpolation principle is as follows: Figure 7As shown. The calculation formula of the pixel value of the point to be corrected is as follows:

[0144] X=X lu *(1-a)*(1-b)+X ru *a*(1-b)+X ld *(1-a)*b+X rd *a*b

[0145] In the formula, X is the pixel value of the point to be corrected, X lu , X ru , X ld , X rd are the pixel values ​​of the four pixels to be read, and a and b are the decimal parts of the column coordinates and the row coordinates in the correction coordinates, respectively.

[0146] In this embodiment, in order to occupy as little local storage resources of the FPGA as possible when reading back the original distorted image in the local cache of the FPGA, a block cache scheme is proposed, which first divides the entire original distorted image into several blocks, and the original distorted image retrieved from the DDR corresponds to the corresponding block and is stored according to the relative position of the row. No matter how large the image distortion is, only a few rows need to be cached, which greatly reduces the occupation of the DDR bandwidth for implementing the distortion correction function, ensures the efficiency of DDR use, and is compared with the cache scheme that requires as many rows as the distortion before the block is divided. In the case of large resolution and large distortion, it can greatly alleviate the pressure on FPGA storage resources.

[0147] The above is only the preferred embodiment of the present application, which does not limit the protection scope of the present application. For those skilled in the art, the present application may have various changes and modifications. Within the spirit and principle of the present application, the changes, modifications, replacements, integrations and parameter changes of these embodiments by conventional substitutions or capable of achieving the same functions without departing from the principles and spirit of the present application shall fall within the protection scope of the present application.

Claims

1. A camera image distortion correction method based on FPGA, characterized in that: The correction method comprises: Step 1: Frame cache the received original distorted image, read the preset quadratic fitting curve coefficients, and read the preset quadratic fitting coefficients corresponding to the correction coordinates from the coefficient cache area in DDR to the local FPGA; Step 2: Determine the block area by coordinate calculation according to the preset quadratic fitting curve coefficient and the preset FPGA local row cache capacity; Any set of the preset quadratic fitting curve coefficients corresponds to a row of pixels in the corrected image, and the step 2 specifically includes: Step 2.1, constructing a corresponding correction coordinate calculation equation according to any set of preset quadratic fitting curve coefficients; Step 2.2, according to the correction coordinate calculation equation and the preset FPGA local row cache capacity, by means of coordinate calculation, the first starting point and the maximum block position of the first block area corresponding to each group of the preset quadratic fitting curve coefficients are determined in turn; Step 2.3, selecting the minimum value of the column coordinates in each of the maximum block positions, and taking the corresponding maximum block position as the first end point of the first block area, and determining the first block area according to the first starting point and the first end point; Step 2.4, determining a second starting point of a second block area according to the first end point and the corresponding correction coordinate calculation equation, and repeating step 2.2 to sequentially determine each block area of ​​the original distorted image; Wherein, the first starting point is determined by the constant term coefficient in the preset quadratic fitting curve coefficient; Step 3: determining the read-back area in the original distorted image row by row according to the block area; Step 4: according to the readback area, read back the original distorted image from the DDR and cache it locally, read back the original distorted image and record it as the readback image, and read the pixel values ​​of the pixel points around the correction coordinates in the readback image; wherein the correction coordinates are determined by the preset quadratic fitting curve coefficients; Step 5: According to the pixel values ​​of the pixel points around the correction coordinates in the read-back image, the pixel values ​​of the four pixel points to be read corresponding to the correction coordinates read from the read-back image are interpolated to determine the pixel values ​​of the points to be corrected, and a corrected image is generated based on the pixel values ​​of the points to be corrected.

2. The camera image distortion correction method based on FPGA according to claim 1, characterized in that: The step 2.2 further includes: rounding up the first starting point; According to the preset FPGA local row cache capacity, the rounded row coordinates of the first starting point are moved upward or downward; according to the moved row coordinates, the corresponding column coordinates are calculated using the corresponding correction coordinate calculation equation; the calculated column coordinates are rounded upward, and the maximum block position is determined according to the moved row coordinates and the rounded column coordinates.

3. The camera image distortion correction method based on FPGA according to claim 2, characterized in that: The step of moving the rounded row coordinate of the first starting point upward or downward according to the preset FPGA local row cache capacity specifically includes: According to the correction coordinate calculation equation, a preset number of correction coordinates after the first starting point are obtained; according to the row coordinate change trend of the preset number of correction coordinates, the row coordinate moving direction is determined; according to the preset FPGA local row cache capacity and the row coordinate moving direction, the rounded row coordinate of the first starting point is moved.

4. The camera image distortion correction method based on FPGA according to claim 1, characterized in that: The step 2.4, determining the second starting point of the second block area according to the first end point and the corresponding correction coordinate calculation equation, specifically includes: By means of coordinate calculation, according to the correction coordinate calculation equation, it is determined whether the column coordinates of the first end point correspond to the column coordinates of the correction coordinates. If so, the pixel point corresponding to the correction coordinates is used as the second starting point of the second block area. If not, the pixel point corresponding to the last correction coordinate contained in the first block area is used as the second starting point of the second block area.

5. The camera image distortion correction method based on FPGA according to claim 2, characterized in that: The step 2.2 also includes: counting the number of corrected coordinates contained between the first starting point and the maximum block position in the first block area; determining whether the number is less than or equal to a preset local coordinate cache capacity, if so, executing step 2.3; if not, selecting the last corrected coordinate to update the maximum block position according to the preset local coordinate cache capacity, and executing step 2.

3.

6. The camera image distortion correction method based on FPGA according to any one of claims 1 to 5, characterized in that: The preset quadratic fitting curve coefficients include an x-axis fitting coefficient and a y-axis fitting coefficient. The x-axis fitting coefficient is determined by fitting the horizontal axis coordinate of the corresponding correction coordinate in the original distorted image. The y-axis fitting coefficient is determined by fitting the vertical axis coordinate of the corresponding correction coordinate in the original distorted image. The calculation formula of the correction coordinate calculation equation is: In the formula, F x (z) is the calculated row coordinate value of the zth point to be corrected corresponding to the corrected coordinate in the original distorted image, a x , b x 、c x are the quadratic term coefficient, linear term coefficient, and constant term coefficient in the x-axis fitting coefficient, respectively. Fy(z) is the column coordinate of the z-th point to be corrected corresponding to the correction coordinate in the original distorted image. a y , b y 、c y are respectively the quadratic term coefficient, the linear term coefficient and the constant term coefficient in the y-axis fitting coefficient, z is the number of the point to be calibrated, z=0,1,2,...,n, and n is the set number of the points to be calibrated.

7. The camera image distortion correction method based on FPGA according to any one of claims 1 to 5, characterized in that: In the step 4, the pixel values ​​of the pixel points around the correction coordinates in the read-back image are read, specifically including: rounding down the correction coordinates corresponding to the current point to be corrected in the read-back image; determining the pixel point in the upper left corner as the first pixel point to be read according to the rounded correction coordinates, and taking the first pixel point to be read as the starting point, selecting the remaining three coordinate points adjacent to the first pixel point to be read in a clockwise direction as the second to fourth pixel points to be read; and reading the pixel values ​​of the four pixel points to be read from the read-back image.

Images