Lens distortion correction circuit and correction method
Through the linear fitting method of the lens distortion correction circuit, the combination of comparator, register, selector, multiplier and adder is used to solve the problems of large resource consumption and long calculation time in traditional lens distortion correction, and achieve hardware resource conservation and calculation speed improvement.
Patent Information
- Application Number
- CN202311849746.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-28
- Publication Date
- 2025-07-01
AI Technical Summary
Traditional lens distortion correction hardware circuit resources are high and the calculation takes a long time, especially in the open square operation process, requiring a large amount of hardware resources and time.
The combined circuit of comparator, register, selector, multiplier and adder is used to calculate the distortion coefficient through a linear fitting method, simplifying the square operation, using only one multiplier and up to two adders to achieve lens distortion correction.
Significantly reduce hardware resource consumption and accelerate calculation speed, only one multiplier and up to two adders are required, the timing is fast, the clock cycle is reduced, and the error is within an acceptable range.
Smart Images
Figure CN120238750A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of semiconductor manufacturing, and particularly to a lens distortion correction circuit and a correction method. Background Art
[0002] Lens distortion refers to the deformation of physical imaging caused by the magnification difference of the lens lens system, manufacturing precision, and assembly process deviation, which is mainly divided into radial distortion and tangential distortion. Generally speaking, lens distortion is inevitable. In general, there are two solutions. One is to improve lens distortion by optimizing the lens group structure and selecting good lenses. The other is to correct the distortion through a hardware circuit. The defect of the traditional hardware circuit is that it occupies too many resources and takes a long time. The areas where the traditional hardware circuit has large computational overhead resources are the multiplier and the correction circuit. The square root circuit often uses approximation methods in hardware design, such as Newton iteration method or dichotomy, to convert the square root operation into a series of iterative calculations. For example, when calculating the radius r from the center given the pixel coordinates (x, y), the square root operation has too many pipeline stages, and each stage requires a certain amount of time, so the overall delay increases and the time consumption is too long. At the same time, enough pipeline registers are needed to store the data of each stage, consuming a large amount of storage resources, large area, and the execution time of the first computing unit increases due to the entire hardware design, resulting in an increase in the overall delay, so the time consumption is long. Summary of the Invention
[0003] The purpose of the present invention is to provide a lens distortion correction circuit and a correction method to improve the defects of large consumption of hardware resources and long computational time.
[0004] To achieve the above purpose, the present invention provides a lens distortion correction circuit, including: a comparator, a register, a selector, a multiplier, and a first adder:
[0005] The input interface of the comparator is used to input the first-direction distance and the second-direction distance between the pixel points of the distorted image and the pixel center; the larger value output by the output interface of the comparator is stored in the first register, and the smaller value output by the output interface of the comparator is stored in the second register;
[0006] The first input interface of the selector is used to input the data stored in the second register, the second input interface of the selector is used to input the data stored in the first register or the processed data, the processed data is the data obtained by adding the data stored in the first register, and the output interface of the selector is used to output the distortion coefficient determined according to the comparison result;
[0007] The first input interface of the multiplier inputs the distortion coefficient, and the second input interface of the multiplier inputs the smaller value output by the output interface of the comparator; the multiplier outputs a product result.
[0008] The input interface of the first adder is used to input the data in the first register and the product result, and the output interface of the first adder is used to output the distance between the pixel point and the optical axis center.
[0009] In a possible implementation manner, when the second input interface of the selector is used to input the processed data, the input data of the second input interface of the selector is the output data of the second adder, and the input of the second adder is the data stored in the second register.
[0010] In another possible implementation manner, the value of the distortion coefficient is related to the ratio between the first direction distance and the second direction distance.
[0011] In other possible implementation manners, the distance r between the pixel point and the optical axis center satisfies the following formula:
[0012] r = X + sY
[0013] Wherein, X is the larger value output by the output interface of the comparator, and Y is the smaller value output by the output interface of the comparator.
[0014] In yet another possible implementation manner, when the selector is an N-to-1 selector, N is 2 n , n is a positive integer greater than or equal to 1, and the selector is used to: when the data in the second register is less than i / N of the data in the first register, output the distortion coefficient corresponding to i, and the value of i ranges from 1 to N.
[0015] In yet another possible implementation manner, N is 4, 8, or 16.
[0016] In a second aspect, an embodiment of the present invention further provides a lens distortion correction method, including:
[0017] Input the first-direction distance and the second-direction distance between the pixel points of the distorted image and the pixel center into the comparator; save the larger value output from the output interface of the comparator in the first register, and save the smaller value output from the output interface of the comparator in the second register; input the data saved in the second register into the first input interface of the selector, and input the data saved in the first register or the processed data into the second input interface of the selector, where the processed data is the data obtained by performing a shift or addition operation on the data saved in the first register; obtain the distortion coefficient determined according to the comparison result from the output interface of the selector; input the distortion coefficient into the first input interface of the multiplier, and input the smaller value output from the output interface of the comparator into the second input interface of the multiplier; obtain the output product result from the multiplier; input the data in the first register and the product result into the input interface of the first adder; obtain the distance between the pixel point and the optical axis center output from the output interface of the first adder.
[0018] In a possible embodiment, when inputting the processed data into the second input interface of the selector, the data input into the second input interface of the selector is the output data of the second adder, and the input of the second adder is the data saved in the second register.
[0019] The lens distortion correction circuit, correction method, device and medium provided by the present invention have the beneficial effects that: the method of the present invention is simple to implement, and in terms of hardware resource consumption, it consumes very little. The present invention only uses one multiplier and at most two adders, which has obvious advantages compared with the original method that requires two multipliers, five adders and several selectors, etc.; in terms of calculation speed, the timing of the present invention is fast, and it only needs to consume one to two clock cycles of the multiplier itself. The traditional method requires 15-stage pipelining implementation, and the present invention also has obvious advantages in comparison. Description of the Drawings
[0020] Figure 1 It is a schematic diagram of a traditional hardware correction circuit provided by the prior art;
[0021] Figure 2 It is a diagram of area division provided by the present invention;
[0022] Figure 3 It is a schematic diagram of a linear function relationship provided by the present invention;
[0023] Figure 4 It is a schematic diagram of the correction circuit provided by the present invention;
[0024] Figure 5 It is a schematic diagram of two function polar coordinate systems provided by the present invention;
[0025] Figure 6 Schematic diagram of error statistical comparison effects for different equal divisions provided by the present invention;
[0026] Figure 7 Schematic flow diagram of a lens distortion correction method provided by the present invention. Detailed implementation manners
[0027] It should be noted that conventional distortion correction mainly includes radial distortion and tangential distortion. The present invention only considers the optimization of radial distortion. Radial distortion is the distortion along the radius direction of the lens. The distortion at the optical axis center is 0, and along the radius direction of the lens, the distortion increases. Radial distortion is divided into two types: barrel distortion and pincushion distortion. Barrel distortion makes the objects in the central part of the image look smaller than the actual ones, while pincushion distortion makes the objects in the central part of the image look larger than the actual ones. These distortions will affect the geometric shape and size of the image, resulting in image distortion.
[0028] Traditionally, the method of using a polynomial distortion model is mostly adopted for distortion correction. It is mainly described by the first few terms of the Taylor series expansion around the principal point, generally the first two terms. The adjustment formula is as follows, where k1 and k2 are the radial deformation coefficients, the original position projected onto the imager is (x0, y0), (x, y) is the ideal position, or the position after correction, and r is the distance between (x0, y0) and the optical axis center.
[0029] x0 = x(1 + k1r 2 + k2r 4 + k3r 6 )
[0030] y0 = y(1 + k1r 2 + k2r 4 + k3r 6 )
[0031] Traditional hardware circuit implementation is as Figure 1 shown. First, the center point of the image is selected, and the coordinates after distortion input are (x0, y0). The subtractor calculates the distances, dist_x and dist_y, between the distorted pixel coordinates and the center pixel point at this time. Then, two multipliers are used to calculate the squares of the distances respectively, and then added by the adder to obtain dist_x 2 + dist_y 2 , that is, r 2 . Finally, a special correction circuit is used to implement the square root operation of r 2 to obtain r. Finally, with the known model parameters of k1 and k2, the normal pixel coordinates are obtained using the above model. Figure 1The areas with large hardware circuit overhead shown are the multiplier and the correction circuit. The square root circuit often uses approximation methods in hardware, such as Newton's iteration method or the bisection method, to convert the square root operation into a series of iterative calculations. Set an iteration count, and then within this count range, approximate step by step until they are close or equal. It can be abstractly expressed for the algorithm of the square root circuit using the area division diagram as Figure 2 shown. Take the square root of 10000 as an example. Input 10000 (the original value), select 15 for the iteration count (denoted as shift), and output its square root value sqrt.
[0032] The hardware implementation of the square root circuit in the prior art is as follows: One register stores the original value 10000, one register stores the estimated original value (denoted as sqrt2), one register stores the square root value (denoted as sqrt), and one register stores the iteration count (denoted as shift). The four registers have a relatively large bit width, 4 in each stage of the pipeline, and 60 registers are required for 15 stages of the pipeline, occupying a large area. First, initialize and assume the value of sqrt is 0, the iteration count shift is 15, and n is the input original value. Then enter the next stage of the pipeline until the iteration count decreases to 0 and the pipeline execution is completed.
[0033] In each stage of the pipeline, the combinational logic is divided into three steps:
[0034] The first step, addition and shift logic. Shift the value of sqrt one bit to the left and add the result of shifting 1 one bit to the left by shift bits and then shift it by shift bits again. This value is assigned to sqrt2 at this stage. This step is actually Figure 2 the sum of several areas.
[0035] The second step, judgment and addition logic. Judge if the obtained sqrt2 is less than or equal to the original value, then add the value of sqrt shifted one bit to the left by shift bits. This is Figure 2 the side length of the square, that is, (1) side length + (2) side length + …, that is, the sum of several side lengths, and then subtract sqrt_2 from n, that is Figure 2 the remaining area, that is, (3) + (4) + …; Judge if sqrt2 is greater than or equal to the original value, then the values of sqrt and n remain unchanged and are passed to the next stage of the pipeline register.
[0036] The third step, judgment logic. Judge if shift is 0. If it is 0, it represents the end of the entire pipeline and the final result can be output. If not, subtract 1 from shift and assign it to shift at this stage. Thus, the hardware implementation of the square root requires 4 adders and 2 comparators. The entire operation requires adding two multipliers and one adder on the basis of the correction circuit. After the pipeline ends, the value of sqrt is obtained, that is Figure 2The (1) side length + (2) side length + …, that is, the sum of the side lengths of all iteration times. A simplified hardware implementation diagram is as Figure 1 shown.
[0037] It can be seen that: in the traditional technology, 15 iterations require 15 - stage pipelining to implement. Each pipelining requires registers to save all intermediate variable calculation values. In terms of computer resource overhead: there are many pipeline registers, and the entire hardware circuit also requires 2 multipliers, 5 adders, and several selectors, etc., occupying a large area. In terms of time: the pipeline is long, and the entire hardware design causes the execution time of the first calculation unit to increase (it takes 15 clocks to process well), and the overall delay increases. In summary, when finding the distance from the center radius r given the pixel coordinates (x, y), the square - root operation has too many pipeline stages (generally 16 stages), and each stage requires a certain amount of time, so the overall delay increases and it takes too much time. At the same time, enough pipeline registers are needed to store data at each stage, consuming a large amount of storage resources and a large area. So, it is considered to use a linear fitting method to replace the complex transcendental function. As Figure 3 shown, triangle ABD is an isosceles right - triangle, so A(x, y), B(x, 0), C(r, 0), D(x + y, 0); therefore, s = BC / BD = (r - x) / y, and the relationship between s, x, and y satisfies the following formula:
[0038]
[0039] where k and b are known numbers.
[0040] It can be seen that s and are nearly in a linear - function relationship, so the distortion coefficient s can be calculated using pixel coordinates.
[0041] Based on the above principle, the lens distortion correction circuit provided by the present invention is as Figure 5 shown, including: a comparator, a first register, a second register, a selector, a multiplier, a first adder, and a second adder:
[0042] The input interface of the comparator is used to input the first - direction distance dist_x and the second - direction distance dist_y between the pixel points of the distorted image and the pixel center; the larger value output by the output interface of the comparator is saved in the first register, and the smaller value output by the output interface of the comparator is saved in the second register.
[0043] The first input interface of the selector is used to input the data stored in the second register. The second input interface of the selector is used to input the data stored in the first register or the processed data. The processed data is the data obtained by adding the data stored in the first register. The output interface of the selector is used to output the distortion coefficient s determined according to the comparison result.
[0044] The first input interface of the multiplier inputs the distortion coefficient s, and the second input interface of the multiplier inputs the smaller value output by the output interface of the comparator; the multiplier outputs the product result.
[0045] The input interface of the first adder is used to input the data in the first register and the product result, and the output interface of the first adder is used to output the distance between the pixel point and the optical axis center.
[0046] When the second input interface of the selector is used to input the processed data, the input data of the second input interface of the selector is the output data of the second adder. The processed data is the data obtained by shifting or adding the data stored in the first register. The input of the second adder is the data stored in the second register.
[0047] It can be seen that Figure 4 In terms of the shown hardware resource consumption, only 1 multiplier and at most 2 adders are used to implement the operation process of the entire calibration circuit. In terms of time, generally only the multiplier needs to consume 1 to 2 clock cycles, and the remaining adders and selectors are combinational logics that do not consume clock cycles.
[0048] The specific hardware implementation steps of the present invention are as follows: First, the input interface data is the distances dist_x and dist_y of the current pixel point (x, y) from the pixel center. Then, dist_x and dist_y are input into the comparator. Among them, the larger value (denoted as X) in the comparison result is stored in the first register, and the smaller value (denoted as Y) in the comparison result is stored in the second register. After that, the coefficient s is determined. The data in the first register and the second register are sent to the selector for comparison. This selector can be an N-to-1 selector, where N is 2 n , n is a positive integer greater than or equal to 1. The selector is used to: when the data in the second register is less than i / N of the data in the first register, output the distortion coefficient corresponding to i, where the value of i ranges from 1 to N. In practical applications, 4-to-1, 8-to-1, 16-to-1 or others can be selected according to the required image correction accuracy. Combined Figure 5 For example, line A represents the result calculated by the traditional square root operation The function relationship curve diagram, where curve B represents the function relationship curve diagram of r = X + sY calculated in this embodiment. Curve C represents the function relationship curve diagram of r = X + sY calculated in this embodiment when the selector is an 8-to-1 selector.
[0049] The following takes a 4-to-1 selector as an example:
[0050] In the first case, Y is compared with X / 4. If Y < X / 4, s0 is selected as the output of the distortion coefficient s. Implementing X / 4 in hardware means shifting X two bits to the right (d). So, the value of X shifted two bits to the right and Y are used as the two input ports of a comparator for comparison, and s0 is selected as the output.
[0051] In the second case, Y is compared with 2X / 4. If Y < X / 2, s1 is selected as the output of the distortion coefficient s. Implementing X / 2 in hardware means shifting X one bit to the right. So, the value of X shifted one bit to the right and Y are used as the two input ports of a comparator for comparison, and s1 is selected as the output.
[0052] In the third case, Y is compared with 3X / 4. If Y < 3X / 4, s2 is selected as the output of the distortion coefficient s. Implementing 3X / 4 in hardware means the value of X shifted two bits to the right plus the value of X shifted one bit to the right. So, an adder is needed for the bit-taking calculation, and then the result of 3X / 4 calculated by the adder and Y are used as the two input ports of a comparator for comparison, and s2 is selected as the output.
[0053] In the fourth case, Y is compared with 4X / 4. If Y < X, s3 is selected as the output of the distortion coefficient s. X and Y are directly used as the two input ports of a comparator for comparison, and s3 is selected as the output, where s0 < s1 < s2 < s3.
[0054] After that, the distortion coefficient s is multiplied by Y, and then the output of the selector is used as one input port of the multiplier. The other input is directly introduced by the data Y in the second register. The multiplication gives a product result. Finally, the data X in the first register and the product result are input into the adder to obtain the final result of r in r = X + sY, which is the value of r.
[0055] In summary, the method of the present invention is simple to implement. In terms of hardware resource consumption, it consumes very little. The present invention only uses one multiplier and at most two adders. Compared with the original method that requires two multipliers, five adders, and several selectors, the advantages are obvious. In terms of calculation speed, the timing implementation of the present invention is fast, only consuming one to two clock cycles of the multiplier itself. The traditional method requires 15-stage pipelining. In comparison, the advantages of the present invention are also obvious. The magnitude of the error in the embodiment of the present invention is related to the number of equal parts of X. For example:
[0056] Divide X into 2 equal parts, 4 equal parts, and 8 equal parts (X / Y) respectively. That is, Y is compared with the X value of each equal part to select the required distortion coefficient s value. For example, for 4 equal parts: when Y is less than or equal to X / 4, select s0 for output; when Y is less than or equal to 2X / 4, select s1 for output; when Y is less than or equal to 3X / 4, select s2 for output; when Y is less than or equal to 4X / 4, select s3 for output. See the specific hardware circuit in Figure 5 . The specific errors are statistically as follows: (the darker the color, the greater the error). For 2 equal parts, as shown in Figure 6 (a) in, the maximum error is 7%; for 4 equal parts, as shown in Figure 6 (b) in, the maximum error is 4%; for 8 equal parts, as shown in Figure 6 (c) in, the maximum error is 1.7%; for 16 equal parts, as shown in Figure 6 (d) in, the maximum error is 1%; It can be seen that in the method of the polynomial distortion model, the presentation effect of 8 equal parts on the actual image can already reach the normal level and is acceptable, and there is no obvious difference in human eye vision.
[0057] Based on the above method embodiments, the embodiments of the present invention also provide a lens distortion correction method, as shown in Figure 7 , including the following steps:
[0058] Step 1, input the first-direction distance and the second-direction distance between the pixel points of the distorted image and the pixel center into the comparator.
[0059] Step 2, save the larger value output from the output interface of the comparator in the first register, and save the smaller value output from the output interface of the comparator in the second register; input the data saved in the second register into the first input interface of the selector, and input the data saved in the first register or the processed data into the second input interface of the selector, where the processed data is the data obtained by performing a shift or addition operation on the data saved in the first register.
[0060] Step 3, obtain the distortion coefficient determined according to the comparison result from the output interface of the selector; input the distortion coefficient into the first input interface of the multiplier, and input the smaller value output from the output interface of the comparator into the second input interface of the multiplier.
[0061] Step 4, obtain the output product result from the multiplier; input the data in the first register and the product result into the input interface of the first adder; obtain the distance between the pixel point and the optical axis center output from the output interface of the first adder.
[0062] In one possible embodiment, when the processed data is input to the second input interface of the selector, the data input to the second input interface of the selector is the output data of the second adder, and the input of the second adder is the data stored in the second register.
[0063] Although the embodiments of the present invention have been described in detail above, it is obvious to those skilled in the art that various modifications and changes can be made to these embodiments. However, it should be understood that such modifications and changes are all within the scope and spirit of the present invention described in the claims. Moreover, the present invention described herein can have other embodiments and can be implemented or realized in various ways.
Claims
1. A lens distortion correction circuit, characterized in that Including a comparator, a register, a selector, a multiplier, and a first adder: The input interface of the comparator is used to input the first-direction distance and the second-direction distance between the pixel points of the distorted image and the pixel center; the larger value output by the output interface of the comparator is stored in the first register, and the smaller value output by the output interface of the comparator is stored in the second register; The first input interface of the selector is used to input the data stored in the second register, the second input interface of the selector is used to input the data stored in the first register or the processed data, the processed data is the data obtained by performing a shift or addition operation on the data stored in the first register, and the output interface of the selector is used to output the distortion coefficient determined according to the comparison result; The first input interface of the multiplier inputs the distortion coefficient, and the second input interface of the multiplier inputs the smaller value output by the output interface of the comparator; The multiplier outputs a product result; The input interface of the first adder is used to input the data in the first register and the product result, and the output interface of the first adder is used to output the distance between the pixel point and the optical axis center.
2. The calibration circuit according to claim 1, wherein When the second input interface of the selector is used to input the processed data, the input data of the second input interface of the selector is the output data of the second adder, and the input of the second adder is the data stored in the second register.
3. The calibration circuit according to claim 1 or 2, characterized in that, The value of the distortion coefficient is related to the ratio between the first-direction distance and the second-direction distance.
4. The calibration circuit according to claim 1 or 2, characterized in that The distance r between the pixel point and the optical axis center satisfies the following formula: r = X + sY Where X is the larger value output by the output interface of the comparator, Y is the smaller value output by the output interface of the comparator, and s is the distortion coefficient.
5. The calibration circuit according to claim 1 or 2, characterized in that When the selector is a 1-out-of-N selector and N is 2 n , where n is a positive integer greater than or equal to 1, the selector is configured to: When the data in the second register is less than i / N of the data in the first register, the distortion coefficient corresponding to i is output, and the value of i is an integer from 1 to N.
6. The calibration circuit according to claim 5, wherein N is 4, 8, or 16.
7. A method for lens distortion correction, characterized in that, Including: Inputting the first-direction distance and the second-direction distance between the pixel points of the distorted image and the pixel center to the comparator; Storing the larger value output by the output interface of the comparator in the first register, and storing the smaller value output by the output interface of the comparator in the second register; Inputting the data stored in the second register to the first input interface of the selector, and inputting the data stored in the first register or the processed data to the second input interface of the selector, the processed data is the data obtained by performing a shift or addition operation on the data stored in the first register; Obtaining the distortion coefficient determined according to the comparison result from the output interface of the selector; Inputting the distortion coefficient to the first input interface of the multiplier, and inputting the smaller value output by the output interface of the comparator to the second input interface of the multiplier; Obtaining the output product result from the multiplier; Inputting the data in the first register and the product result to the input interface of the first adder; Obtaining the output distance between the pixel point and the optical axis center from the output interface of the first adder.
8. The calibration method according to claim 7, wherein When processed data is input to the second input interface of the selector, the data input to the second input interface of the selector is the output data of the second adder, and the input of the second adder is the data stored in the second register.