Camera adaptive calibration method based on lens distortion

Abstract

The invention discloses a camera adaptive calibration method based on lens distortion, and the method comprises the steps: carrying out the fitting processing of an image collected by a camera, and obtaining a fitting evaluation value; wherein the fitting evaluation value is used for reflecting the distortion condition of the lens; the lens is assembled in the camera; and selecting a matched calibration model based on the fitting evaluation value, and calibrating the camera by using the matched calibration model so as to calculate the internal reference of the camera. According to the invention, the calibration model is adaptively selected to carry out camera calibration calculation based on different lens distortions, and the internal reference matched with the currently calibrated camera is obtained, so that the matched calibration model is automatically selected to calibrate the camera under the condition of correcting fisheye lenses with small and large distortions. The problem of modeling errors caused by the change of lens models (distortion) is reduced, and the adaptability of correction of different types of camera lenses and the accuracy of internal parameter calculation are improved.

Description

Technical Field

[0001] This application relates to the technical field of sensor calibration, and particularly to a camera adaptive calibration method based on lens distortion. Background Art

[0002] In order to obtain a larger detection range, a line laser measurement system usually uses a fish-eye camera with a larger field of view angle, and the fish-eye camera is equipped with a fish-eye lens. The calibration principles of both the fish-eye camera and the pinhole camera are to use the mathematical interface functions set in the fixed camera calibration model for geometric modeling and calculating the internal parameters of the camera; however, for the fish-eye camera, due to the non-linear imaging characteristics of the fish-eye lens, the objects in the world coordinate system are stretched and deformed in the imaging plane of the fish-eye lens, and usually distortion correction is also required to compensate for a more accurate image.

[0003] During the production process of the lens, the magnitude of the edge distortion of the lens is proportional to the magnitude of the field of view angle. When using the fixed camera calibration model to correct the lens with relatively small distortion or relatively large distortion, modeling errors often occur, and the adaptability to correct lenses of different models is poor, resulting in inaccurate calculation of the internal parameters. Summary of the Invention

[0004] This application aims to provide a camera adaptive calibration method based on lens distortion, and the specific technical solution is as follows: A camera adaptive calibration method based on lens distortion includes: performing fitting processing on the images collected by the camera to obtain a fitting evaluation value; wherein, the fitting evaluation value is used to reflect the distortion situation of the lens; the lens is assembled in the camera; selecting a matching calibration model based on the fitting evaluation value, and then using the matching calibration model to calibrate the camera to calculate the internal parameters of the camera. This application adaptively selects a calibration model for camera calibration calculation based on different lens distortions, obtains the internal parameters that match the currently calibrated camera, so as to automatically select a matching calibration model to calibrate the camera in the case of correcting fish-eye lenses with relatively small distortion and relatively large distortion, reduce the problem of modeling errors caused by changes in lens models (distortion), and improve the adaptability to correct different types of camera lenses and the accuracy of internal parameter calculation.

[0005] Further, before performing the fitting process on the image collected by the camera, that is, before performing step A, it further includes: controlling the camera to capture an image of the target to obtain a to-be-calibrated image; the to-be-calibrated image shows a distorted target; selecting multiple to-be-fitted corner points from the to-be-calibrated image to represent the corner points distributed on the same straight line; wherein, the target is composed of multiple cells arranged row by row or column by column in one direction; wherein, the to-be-fitted corner points are the corner points located at the boundary of the target among the cells in the column closest to the boundary of the to-be-calibrated image; or, the to-be-fitted corner points are the corner points located at the boundary of the target among the cells in the row closest to the boundary of the to-be-calibrated image; or, the to-be-fitted corner points are the corner points of the cells in the column closest to the boundary of the to-be-calibrated image inside the target; or, the to-be-fitted corner points are the corner points of the cells in the row closest to the boundary of the to-be-calibrated image inside the target; wherein, the corner points are pixel points in the to-be-calibrated image. Thus, a group of corner points closest to the boundary of the to-be-calibrated image can be selected as a group of to-be-fitted corner points for the aforementioned fitting process, improving the calculation accuracy of the fitting evaluation value.

[0006] Further, the method for selecting a matching calibration model based on the fitting evaluation value includes: detecting the distortion situation of the lens according to the magnitude relationship between the fitting evaluation value and a preset threshold; selecting a matching calibration model according to the distortion situation of the lens to correct the distortion of the lens. Thus, the problems of modeling and calibration accuracy caused by the distortion of the lens are considered. It will not cause modeling errors because a calibration model with a relatively large operation parameter scale is still used when the distortion of the lens is small, nor will it cause modeling errors or low calibration efficiency because a calibration model with a relatively small operation parameter scale is still used when the distortion of the lens is large, improving the accuracy of camera calibration.

[0007] Further, the method for performing a fitting process on the image collected by the camera to obtain a fitting evaluation value includes: using the least squares method to perform a straight line fitting on the selected multiple to-be-fitted corner points to obtain a target fitting straight line; calculating the distances from each to-be-fitted corner point to the target fitting straight line, then averaging the distances from each to-be-fitted corner point to the target fitting straight line, and then setting the obtained average value as the fitting evaluation value. Thus, the average value of the distances from all to-be-fitted corner points to the target fitting straight line is used to describe the trend of the target fitting straight line, to determine the deviation of the target fitting straight line from the connection line formed by all to-be-fitted corner points mapped into the world coordinate system, and further to reflect the distortion situation of the lens.

[0008] Further, the method for detecting the distortion of the lens according to the magnitude relationship between the fitting evaluation value and the preset threshold includes: when obtaining the fitting evaluation value by performing linear fitting on multiple selected corner points to be fitted, the preset threshold used for detecting the distortion of the lens is the first preset threshold; when the fitting evaluation value is greater than the first preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter, indicating that the distortion of the lens is relatively large; when the fitting evaluation value is less than or equal to the first preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, indicating that the distortion of the lens is relatively small, so as to prompt the camera or the ranging module where the camera is located to call a calibration model different from that in the case of a relatively large distortion of the lens, reduce the misjudgment of the distortion situation, and meet the calibration accuracy in the case where the distortion of the lens is less than or equal to the preset distortion parameter.

[0009] Further, the method for performing fitting processing on the image collected by the camera to obtain the fitting evaluation value includes: performing curve fitting on multiple selected corner points to be fitted based on the least squares method to form an Nth-degree polynomial equation, and determining the fitted target fitting curve; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable; setting the intersection point of the target fitting curve and the central axis of the image to be calibrated as the middle point; calculating the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the middle point to obtain the first first partial derivative value; calculating the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the middle point to obtain the first second partial derivative value; calculating the second-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the middle point to obtain the second first partial derivative value; calculating the second-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the middle point to obtain the second second partial derivative value; calculating the curvature of the middle point based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value; setting the curvature of the middle point as the fitting evaluation value. Therefore, the fitting evaluation value can describe the maximum bending degree of the connection line (a straight line segment parallel to the boundary of the target) of a group of corner points to be fitted mapped into the world coordinate system within the image to be calibrated, and further reflect the maximum distortion of the lens, serving as the basis for selecting the calibration method, so as to facilitate selecting a matching calibration model in step B.

[0010] Further, the method for performing fitting processing on the images collected by the camera to obtain a fitting evaluation value includes: performing curve fitting on a plurality of selected corner points to be fitted based on the least squares method to form an Nth-degree polynomial equation, and determining the target fitting curve obtained by fitting; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable; sequentially selecting a plurality of pixel points from among the pixel points passed by the target fitting curve within the image to be calibrated; for the plurality of selected pixel points, there are: calculating the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain a first first partial derivative value; calculating the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain a first second partial derivative value; calculating the second-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain a second first partial derivative value; calculating the second-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain a second second partial derivative value; then calculating the curvature of the same pixel point based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value; after the curvature of each of the selected pixel points is calculated, calculating the average value of the curvatures of all the selected pixel points; and then setting the calculated average value as the fitting evaluation value. Thus, after the curvature of each of the selected pixel points is calculated, calculating the average value of the curvatures of all the selected pixel points; and then setting the calculated average value as the fitting evaluation value. Among them, the average value of the curvatures of each of the selected pixel points is not necessarily the maximum curvature of the target fitting curve within the image to be calibrated, but can be used to identify the average degree of bending that occurs after the boundary of the target in the world coordinate system is projected onto the pixel coordinate system through the lens, and reflects the average distortion degree of the lens.

[0011] Further, N is equal to the value 3 to form a cubic polynomial equation; the first independent variable included in the cubic polynomial equation is used to represent the abscissa of the pixel coordinate system, and the second independent variable included in the cubic polynomial equation is used to represent the ordinate of the pixel coordinate system, so that the first-order partial derivative of the cubic polynomial equation with respect to the first independent variable represents the tangent slope of the target fitting curve in the horizontal axis direction of the pixel coordinate system, and the first-order partial derivative of the cubic polynomial equation with respect to the second independent variable represents the tangent slope of the target fitting curve in the vertical axis direction of the pixel coordinate system. In summary, the present application can introduce a cubic polynomial for curvature calculation, quickly obtain the fitting evaluation value, reduce the time for fitting processing, and improve the efficiency of judging lens distortion.

[0012] Further, based on the first partial derivative value, the second partial derivative value, the third partial derivative value, and the fourth partial derivative value, the method for calculating the curvature includes: controlling the multiplication of the first partial derivative value and the fourth partial derivative value to obtain a first product; controlling the multiplication of the second partial derivative value and the third partial derivative value to obtain a second product; calculating the sum of the squares of the first partial derivative value and the second partial derivative value to obtain a first sum of squares; calculating the ratio between the absolute value of the difference between the first product and the second product and the square root of the cube of the first sum of squares to obtain the curvature. Therefore, this application provides a basis for determining the degree of bending that occurs when the boundary of the target in the world coordinate system is projected through the lens onto the pixel coordinate system.

[0013] Further, the method for detecting the distortion of the lens according to the magnitude relationship between the fitting evaluation value and the preset threshold includes: in the case of performing curve fitting on a plurality of selected corner points to be fitted, if the curvature of the middle point is set as the fitting evaluation value, the preset threshold used for detecting the distortion of the lens is the second preset threshold; when the fitting evaluation value is greater than the second preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter; when the fitting evaluation value is less than or equal to the second preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter; wherein, the middle point is the intersection point of the target fitting curve and the central axis of the image to be calibrated; in the case of performing curve fitting on a plurality of selected corner points to be fitted, if the average value of the curvatures of a plurality of pixel points selected from within the target fitting curve is set as the fitting evaluation value, the preset threshold used for detecting the distortion of the lens is the third preset threshold; when the fitting evaluation value is greater than the third preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter; when the fitting evaluation value is less than or equal to the third preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter; wherein, the third preset threshold is less than the second preset threshold. The third preset threshold and the second preset threshold respectively correspond to the actual calibration situations of the lenses of different models of fish-eye cameras.

[0014] Further, the method for selecting a matching calibration model according to the distortion situation of the lens includes: when it is determined that the distortion of the lens is greater than the preset distortion parameter, the selected matching calibration model is a fish-eye lens distortion calibration model for geometric modeling and calculating the camera internal parameters; when it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, the selected matching calibration model is a pinhole imaging distortion calibration model for geometric modeling and calculating the camera internal parameters. Thus, it is possible to select corresponding camera calibration methods according to different distortions, enhance the calibration adaptability of lenses with different distortions, improve the accuracy of internal parameter calculation, and therefore, when collecting images using the corresponding lenses after adaptive correction, the measurement errors generated meet the control standards of the camera ranging performance.

[0015] Further, the camera adaptive calibration method further includes: converting the calculated camera internal parameters into parameters compatible with the data transmission interface, so that the transmission of the camera internal parameters output by different calibration models is compatible with the data transmission interface; wherein, each calibration model matches a camera and its internal parameters. By executing the camera adaptive calibration method, the camera adaptive calibration method can be set as a general calibration software, improving the data transmission compatibility effect and calibration adaptability of the software for different camera lenses, and meeting the camera adaptive calibration effect.

[0016] Further, the method for converting the calculated camera internal parameters into parameters compatible with the data transmission interface includes: marking the camera internal parameters calculated by the pinhole imaging distortion calibration model as the first internal parameters; marking the camera internal parameters calculated by the fish-eye lens distortion calibration model as the second internal parameters; after the camera internal parameters are calculated by the pinhole imaging distortion calibration model, expanding the data bit width of the corresponding projection parameters or adding new projection parameters by zero-padding the first internal parameters, and then setting the zero-padded first internal parameters as the parameters compatible with the data transmission interface; after the camera internal parameters are calculated by the fish-eye lens distortion calibration model, not zero-padding the second internal parameters, and setting the second internal parameters as the parameters compatible with the data transmission interface. This can not only reduce the modeling error caused by the lack of stored and transmitted parameters, but also meet the transmission and storage requirements of the data transmission interface.

[0017] Further, the number of projection parameters included in the first internal parameters is less than the number of projection parameters included in the second internal parameters, so that there are no partial projection parameters required for correction by the fish-eye lens distortion calibration model among the projection parameters required for correction by the pinhole imaging distortion calibration model; the bit width of the projection parameters included in the first internal parameters is less than the bit width of the same type of projection parameters included in the second internal parameters, so that the projection parameters required for correction by the pinhole imaging distortion calibration model are different from the same type of projection parameters required for correction by the fish-eye lens distortion calibration model; wherein, the bit width of the data transmission interface is configured to be equal to the bit width of the second internal parameters; both the first internal parameters and the second internal parameters include projection parameters, and the projection parameters are used to represent the projection relationship from the world coordinate system to the pixel coordinate system. By comparing the size and quantity relationship between the first internal parameters and the second internal parameters, the difference between the projection parameters required for correction by the pinhole imaging distortion calibration model and the projection parameters required for correction by the fish-eye lens distortion calibration model is determined, providing a judgment basis related to the calibration model for subsequent zero-padding.

[0018] Further, the method for padding zeros to the first internal parameter includes: when some of the projection parameters to be corrected in the pinhole imaging distortion calibration model do not exist in the partial projection parameters to be corrected in the fisheye lens distortion calibration model, new projection parameters in the first internal parameter are composed of binary zeros with multiple bits, and the bit width of the new projection parameters is set to be equal to the bit width of the partial projection parameters to be corrected in the fisheye lens distortion calibration model, and each bit in the new projection parameters is marked with a zero-padding bit; when the bit width of the projection parameters included in the first internal parameter is less than the bit width of the same type of projection parameters included in the second internal parameter, binary zeros are padded bit by bit starting from the first bit of the projection parameters included in the first internal parameter until the bit width of the projection parameters included in the first internal parameter after padding zeros is equal to the bit width of the same type of projection parameters included in the second internal parameter, and the value of the projection parameters included in the first internal parameter after padding zeros is equal to the value of the projection parameters included in the first internal parameter before padding zeros, and then each bit padded is marked with a zero-padding bit.

[0019] In summary, after calculating the camera internal parameter using the matched calibration model, in order to adapt to the generality of different lenses, it is necessary to process the data according to the data transmission interface. The number of parameters of the lens calibrated by selecting the pinhole imaging distortion calibration model is less than that of the lens calibrated by selecting the fisheye lens distortion calibration model, and the values of the parameters of the lens calibrated by selecting the pinhole imaging distortion calibration model are different from those of the lens calibrated by selecting the fisheye lens distortion calibration model. Then, the projection parameters included in the first internal parameter are less than those included in the second internal parameter, and the first internal parameter is different from the second internal parameter. Therefore, zeros are padded to the parameters of the lens calibrated by selecting the pinhole imaging distortion calibration model during data transmission and storage to be compatible with the data transmission interface, so that the data transmission interface can adapt to the generality of different lenses; in the subsequent scenario of using the first internal parameter or the second internal parameter for ranging calculation, different lens types are determined by checking the zero-padding bits set at certain bits in the camera internal parameter and judging whether their values are binary zeros, including distinguishing between fisheye lenses and the lenses of pinhole cameras. Description of the Drawings

[0020] Figure 1 is a schematic flowchart of a method for camera adaptive calibration based on lens distortion disclosed in an embodiment of the present application.

[0021] Figure 2 is a schematic diagram showing the distribution of corner points to be fitted in the image of a target in another embodiment of the present application. Among them, the target is Figure 2 displayed as a distorted checkerboard within the rectangular image area of Figure 2 and the checkerboard in Figure 2It consists of cells filled with 3 right-leaning dashed lines and cells filled with white. Embodiment

[0022] Next, the technical solutions in the embodiments of the present invention will be described in detail with reference to the accompanying drawings in the embodiments of the present invention. To further illustrate each embodiment, the present invention provides accompanying drawings. These accompanying drawings are a part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principle of the embodiments. With reference to these contents, those of ordinary skill in the art should be able to understand other possible embodiments and the advantages of the present invention.

[0023] Different camera models correspond to different types of physical lenses, but in reality, physical lenses cannot perfectly correspond to a certain camera model. This imperfection includes the installation error of the lens itself and the expression limitation of the imaging model itself. In fact, one of the main tasks of lens manufacturers is to design and manufacture lenses that are as close as possible to the camera model. Therefore, the difference between the imaging model and the imaging of the real lens can be defined as the aforementioned distortion. Distortion can also be described by a certain mathematical function. It can be simply understood that distortion is to insert a function into the ideal camera imaging model, so that the finally obtained pixel coordinates are inconsistent with the coordinates obtained by the camera projection relationship. Among them, the lens is an optical device composed of multiple groups of lenses. The thickness of the center and edge of the lens is different. A convex lens is thick in the middle and thin at the edges, and a concave lens is thin in the middle and thick at the edges. Generally speaking, light is orthogonal to the main axis and passes through the main axis without distortion. The farther the light is from the main axis, the greater the distortion. When the magnification increases with the increase of the incident angle, it is called positive distortion; that is, pincushion distortion; when the magnification decreases with the increase of the incident angle, it is negative distortion, that is, barrel distortion. Lens distortion is actually the general term for the inherent perspective distortion of optical lenses, that is, the distortion caused by perspective reasons. Due to the inherent characteristics of this kind of lens, such as the optical reaction of convex lenses converging light and concave lenses diverging light.

[0024] To be compatible with camera calibration operations under different lens distortions, the present application discloses a camera adaptive calibration method based on lens distortion; the execution subject of the camera adaptive calibration method is a control unit for calibrating a fish-eye camera or a non-fish-eye camera. The control unit can be arranged inside or outside the ranging module where the camera is located and automatically execute the camera adaptive calibration method.

[0025] The lens is assembled in the camera, and the camera is detachably installed in the ranging module. The camera disclosed in the present application is an electronic device with image acquisition capabilities.

[0026] Referring to Figure 1 It can be seen that the camera adaptive calibration method includes: Step A: Perform fitting processing on the images collected by the camera to obtain a fitting evaluation value; then execute Step B; in this application, the obtained fitting evaluation value is used to describe the bending degree of the imaging line segment of a straight line in the physical world within the lens or the deviation degree of a straight line in the physical world (world coordinate system) relative to the fitting straight line. Therefore, the fitting evaluation value is used to reflect the distortion situation of the lens and is used as a reliable parameter for evaluating the lens distortion size in this application; the methods of fitting processing include, but are not limited to, fitting a curve equation and solving the curvature, or fitting a straight line and solving the point-line distance, or other methods that can extract the value representing the bending degree of the imaging of a straight line within the lens, etc.

[0027] Step B: Select a matching calibration model based on the fitting evaluation value to determine that a lens with corresponding distortion matches a calibration model; then use the matching calibration model to calibrate the camera to calculate the camera internal parameters, and preferably obtain the mapping coefficients required for fisheye lens distortion correction; each calibration model matches a lens to correct the distortion of the corresponding lens. Thus, it can be used as a set of general calibration methods to adaptively detect camera lenses with different distortions.

[0028] Therefore, this application adaptively selects a calibration model for camera calibration calculation using different lens distortions to obtain the internal parameters that match the currently calibrated camera. By executing Step A and Step B for different models of cameras or lenses with different distortions, the requirements for adaptive calibration of different models of cameras or lenses with different distortions are met. Thus, when correcting fisheye lenses with relatively small and relatively large distortions, a matching calibration model is automatically selected to calibrate the camera, reducing the problem of modeling errors caused by changes in lens models (distortions), and improving the adaptability of correcting different types of camera lenses and the accuracy of internal parameter calculation.

[0029] As an embodiment, before performing fitting processing on the image collected by the camera, it further includes: controlling the camera to capture an image of the target to obtain a to-be-calibrated image; the target is within the field of view of the camera; the to-be-calibrated image shows a distorted target; then, multiple to-be-fitted corner points are selected from the to-be-calibrated image to represent the corner points distributed on the same straight line in the world coordinate system. For example, the corner points of the target distributed along the boundary of a row or a column of cell areas in the world coordinate system. Due to distortion, the corner points distributed on the same straight line in the world coordinate system become distributed along a curve or discretely after being projected onto the pixel coordinate system, and fitting processing is required. In this embodiment, at least three to-be-fitted corner points are selected to form a group of corner points to participate in the fitting processing, that is: using a group of to-be-fitted corner points for fitting processing; wherein, the to-be-fitted corner points are the corner points located at the boundary of the target among the columns of cells closest to the boundary of the to-be-calibrated image; or, the to-be-fitted corner points are the corner points located at the boundary of the target among the rows of cells closest to the boundary of the to-be-calibrated image; or, the to-be-fitted corner points are the corner points of the column of cells closest to the boundary of the to-be-calibrated image inside the target; or, the to-be-fitted corner points are the corner points of the row of cells closest to the boundary of the to-be-calibrated image inside the target.

[0030] It should be noted that the corner points are pixel points in the to-be-calibrated image. The target can be a checkerboard, and it is the checkerboard set on the surface of the calibration board in front of the camera; the target can also be other objects with straight boundary lines.

[0031] Schematically, the to-be-calibrated image is Figure 2 a rectangular image in which multiple cells have barrel distortion, and the to-be-calibrated image encloses Figure 2 all the cells in Figure 2 The barrel distortion in

[0032] is the distortion generated when the target is projected onto the pixel plane of the camera, and this rectangular image can be defined as the image of the target. For example, the rectangle in front of the fish-eye lens will generate distortion after focusing imaging, including barrel distortion. Figure 2 Since the target is composed of multiple cells arranged row by row or column by column in one direction, so Figure 2 the area composed of the cells with filled diagonal lines ( Figure 2 the cells with 3 right-tilted dotted lines filled in Figure 2Among them, the target can be regarded as being composed of 48 cells arranged row by row along the V-axis direction, where the number of cells arranged in each row is 8; the target can also be regarded as being composed of 48 cells arranged column by column along the U-axis direction, where the number of cells arranged in each column is 6.

[0033] In some embodiments, the corner point to be fitted is the corner point located at the boundary of the target among the cells in the column closest to the boundary of the image to be calibrated. Schematically, as Figure 2 shown, the corner point to be fitted is located on the boundary of the target area composed of cells in 6 rows and 8 columns; specifically, the corner point to be fitted is the corner point located at the right boundary of the target among the cells in the column closest to the right boundary of the image to be calibrated (the last column of cells counted from left to right); correspondingly, the corner point to be fitted is the black dot distributed on the right boundary of the target; all the black dots distributed on the right boundary of the target (which can be regarded as being distributed along the V-axis direction) form a set of corner points to be fitted, and this set of corner points to be fitted are sequentially distributed as corner point c9, corner point d1, corner point d2, corner point d3, corner point d4, corner point d5, and corner point d6. It should be noted that when the camera is equipped with a fish-eye lens, due to the relatively large distortion at the edge of the fish-eye lens, a set of corner points closest to the boundary of the image to be calibrated is selected as a set of corner points to be fitted to perform the fitting process described in step A, so as to improve the calculation accuracy of the fitting evaluation value.

[0034] In some embodiments, the corner point to be fitted is the corner point located at the boundary of the target among the cells in the row closest to the boundary of the image to be calibrated; schematically, as Figure 2 shown, the corner point to be fitted is located on the boundary of the target area composed of cells in 6 rows and 8 columns; specifically, the corner point to be fitted is the corner point located at the upper boundary of the target among the cells in the row closest to the upper boundary of the image to be calibrated (the last row of cells counted from bottom to top); correspondingly, the corner point to be fitted is the black dot distributed on the upper boundary of the target; all the black dots distributed on the upper boundary of the target (which can be regarded as being distributed along the U-axis direction) form a set of corner points to be fitted, and this set of corner points to be fitted are sequentially distributed as corner point c1, corner point c2, corner point c3, corner point c4, corner point c5, corner point c6, corner point c7, corner point c8, and corner point c9. It should be noted that when the camera is equipped with a fish-eye lens, due to the relatively large distortion at the edge of the fish-eye lens, a set of corner points closest to the boundary of the image to be calibrated is selected as a set of corner points to be fitted to perform the fitting process described in step A, so as to improve the calculation accuracy of the fitting evaluation value.

[0035] In some embodiments, the corner point to be fitted is the corner point of the cells in the column closest to the boundary of the image to be calibrated inside the target; therefore, the corner point to be fitted is not located on the boundary of the target. Schematically, asFigure 2 As shown, it is located within the target (a target area composed of cells in 6 rows and 8 columns), and the corner point to be fitted is located on the boundary of the area composed of cells in 4 rows and 6 columns; specifically, within the target, the corner point to be fitted is the corner point of the column of cells that is closest to the right boundary of the image to be calibrated; correspondingly, the corner point to be fitted is Figure 2 the black dot that is not on the boundary of the target in the last column of cells counted from left to right (along the positive direction of the U axis) in ; all the black dots that are not on any boundary of the target in the column of cells occupying the right boundary of the target form a set of corner points to be fitted; this set of corner points to be fitted are sequentially distributed as corner point e7, corner point f1, corner point f2, corner point f3, and corner point f4; it should be noted that when the camera is equipped with a fish-eye lens, since the distortion is greater the closer it is to the boundary of the image to be calibrated, a set of corner points that are closest to the boundary of the image to be calibrated but not on the boundary of the target is selected as a set of corner points to be fitted for the fitting process described in step A to improve the calculation accuracy of the fitting evaluation value.

[0036] In some embodiments, the corner point to be fitted is the corner point of the row of cells that is closest to the boundary of the image to be calibrated within the target; therefore, the corner point to be fitted is not on the boundary of the target. Schematically, as Figure 2 shown, it is located within the target (a target area composed of cells in 6 rows and 8 columns), and the corner point to be fitted is located on the boundary of the area composed of cells in 4 rows and 6 columns; specifically, within the target, the corner point to be fitted is the corner point of the row of cells that is closest to the upper boundary of the image to be calibrated; correspondingly, the corner point to be fitted is Figure 2 the black dot that is not on the boundary of the target in the last row of cells counted from bottom to top (along the positive direction of the V axis) in ; all the black dots that are not on any boundary of the target in the row of cells occupying the upper boundary of the target form a set of corner points to be fitted; this set of corner points to be fitted are sequentially distributed as corner point e1, corner point e2, corner point e3, corner point e4, corner point e5, corner point e6, and corner point e7. It should be noted that when the camera is equipped with a fish-eye lens, since the distortion is greater the closer it is to the boundary of the image to be calibrated, a set of corner points that are closest to the boundary of the image to be calibrated but not on the boundary of the target is selected as a set of corner points to be fitted for the fitting process described in step A to improve the calculation accuracy of the fitting evaluation value.

[0037] Combined with Figure 2 it can be seen that the cells with filled diagonal lines set at intervals ( Figure 2 the cells with 3 right-leaning dashed line segments filled in ) and the cells filled with white form the target; the corner points are the intersection points of the edge lines of the cells with filled diagonal lines, or the intersection points of the edge lines of the white cells, or the two end points of the common side of two adjacent cells. InFigure 2 It can be represented by using black dots, which can be regarded as two-dimensional coordinates on the pixel plane. Preferably, the number of cells forming the target is linearly related to the number of corner points searched out. Preferably, the Harris corner detection method is used to search for corner points within the complete target. It should be noted that the Harris corner detection method mainly searches for corner points based on the gradient of the change in the gray value of image pixels. Specifically, in the area near the corner point of the cell, it is an area where the change in its pixel gray value is very large, and its gray value gradient is also very large. In other words, in the neighborhood of non-corner positions, the pixel values of each point change little, or even are almost equal, and their gradients are relatively small.

[0038] As an embodiment, in the step B, the method for selecting a matching calibration model based on the fitting evaluation value includes: detecting the distortion situation of the lens according to the magnitude relationship between the fitting evaluation value and a preset threshold; thus, when the fitting evaluation value is greater than the preset threshold, it is determined that the lens distortion is large; when the fitting evaluation value is less than the preset threshold, it is determined that the lens distortion is small. Among them, as the assembly position of the target relative to the camera changes, the preset threshold can be adaptively changed; different lens models produce different distortions, so the setting of the preset threshold is also different; specifically, the preset threshold can be set according to actual test data.

[0039] Then, a matching calibration model is selected according to the distortion situation of the lens to correct the distortion of the lens and achieve the calibration of the camera. Thus, when the distortion of the lens is large, a calibration model with a relatively large scale of operation parameters is automatically selected as the modeling and internal parameter calculation function required for calibrating the camera; when the distortion of the lens is small, a calibration model with a relatively small scale of operation parameters is automatically selected as the modeling and internal parameter calculation function required for calibrating the camera; thus, considering the modeling and calibration accuracy problems brought by the distortion of the lens, it will not cause modeling errors because a calibration model with a relatively large scale of operation parameters is still used when the distortion of the lens is small, nor will it cause modeling errors or low calibration efficiency because a calibration model with a relatively small scale of operation parameters is still used when the distortion of the lens is large, improving the calibration accuracy of the camera.

[0040] In some embodiments, the image collected by the camera is subjected to fitting processing to obtain a fitting curve; then, the curvature of the pixel point that is equal to the abscissa (or ordinate) of the midpoint of the boundary of the to-be-calibrated image and lies on the fitting curve can be used as the fitting evaluation value; among them, the pixel point that is equal to the abscissa (or ordinate) of the midpoint of the boundary of the to-be-calibrated image and lies on the fitting curve is the midpoint; according to the image distortion characteristics, the pixel points on the central axis of the to-be-calibrated image are the pixel points with the maximum curvature within the to-be-calibrated image. Then, by comparing the curvature of the midpoint with a preset threshold, it is more capable of judging the extreme curvature level of the surface / curve fitted by the to-be-calibrated image, that is, reflecting the maximum distortion that the lens can generate. By selecting a matching calibration model using the maximum distortion that the lens can generate, the influence of misjudgment can be reduced, and a more suitable calibration model can be selected for camera calibration to ensure the accuracy of camera calibration.

[0041] As Embodiment 1, for the aforementioned step A, the method for performing fitting processing on the image collected by the camera to obtain a fitting evaluation value includes: Use the least squares method to perform linear fitting on multiple selected corner points to be fitted to obtain a target fitting line, forming a fitting result of the linear fitting; the number of the multiple selected corner points to be fitted is greater than or equal to three, arranged as a group of corner points to be fitted or a group of corner points, which is convenient for performing linear fitting using the first-order term function (target fitting line equation) of the least squares method, calculating the coefficients and constant terms of each order required for the linear equation, thereby constituting the target fitting line equation. Then, a target fitting line is obtained in the pixel coordinate system, and this target fitting line is used as the best fitting line that approximates a pre-selected group of corner points to be fitted. The connection line of the group of corner points to be fitted is parallel to a boundary of the target in the target, and the connection line of the group of corner points to be fitted mapped into the world coordinate system is a straight line segment.

[0042] After determining the target fitting line, that is, after obtaining the fitting result of the linear fitting, calculate the distances from each corner point to be fitted to the target fitting line. Specifically, use the coefficients and constant terms in the target fitting line equation to construct the point-to-line equation, and then substitute the coordinates of each corner point to be fitted into the point-to-line equation in turn to calculate the distances from each corner point to be fitted to the target fitting line. Then, calculate the average value of the distances from each corner point to be fitted to the target fitting line, and then set the calculated average value as the fitting evaluation value. Thus, use the average value of the distances from all corner points to be fitted to the target fitting line to describe the trend of the target fitting line, so as to determine the deviation formed by the target fitting line relative to the connection line of all corner points to be fitted mapped into the world coordinate system, and further reflect the distortion situation of the lens.

[0043] Specifically, the method for detecting the distortion of the lens according to the magnitude relationship between the fitting evaluation value and the preset threshold includes: in the case of obtaining the fitting evaluation value by performing linear fitting on a plurality of selected corner points to be fitted, corresponding to the fitting process in the foregoing Embodiment 1, the preset threshold used for detecting the distortion of the lens is the first preset threshold; when the fitting evaluation value is greater than the first preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter, indicating that the distortion of the lens is relatively large; when the fitting evaluation value is less than or equal to the first preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, indicating that the distortion of the lens is relatively small, so as to prompt the camera or the ranging module where the camera is located to call a calibration model different from the case where the distortion of the lens is relatively large, reduce the misjudgment of the distortion situation, and meet the calibration accuracy in the case where the distortion of the lens is less than or equal to the preset distortion parameter.

[0044] The preset distortion parameter is used as the distortion reference value of different types of lenses to describe the distortion degree of the lens and is used as the basis for the distortion parameter for selecting the matching calibration model.

[0045] As Embodiment 2, for the foregoing step A, the method for performing fitting processing on the image collected by the camera to obtain the fitting evaluation value includes: Performing curve fitting on a plurality of selected corner points to be fitted based on the least squares method to form an Nth-degree polynomial equation, and determining the target fitting curve to form the fitting result of the curve fitting within the image to be calibrated; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable; the first independent variable can be represented by Figure 2 the abscissa U, and the second independent variable can be represented by Figure 2 the ordinate V, and both the first independent variable and the second independent variable can be combined with the coefficients of each term to form a monomial of the corresponding degree in the Nth-degree polynomial equation; in this embodiment, at least 3 of the plurality of preselected corner points to be fitted are arranged in a group of corner points to be fitted or a group of corner points; in this embodiment, curve fitting is performed based on a multiple function or a multiple polynomial (Nth-degree polynomial equation) of the least squares method, and the coefficients of each term and the constant term of the optimal multiple function are calculated, so that an Nth-degree polynomial equation can be constructed, which is also defined as an Nth-degree surface equation.

[0046] Setting the intersection point of the target fitting curve and the central axis of the image to be calibrated as the middle point. Specifically, the coordinates of the middle point in the pixel coordinate system UOV can be obtained by combining the equation of the central axis and the Nth-degree polynomial equation, and are denoted as the coordinates of the middle point; schematically, as Figure 2As shown, the central axis of the image to be calibrated can extend along the U-axis direction or along the V-axis direction. Among them, based on the image distortion characteristics, the curvature of the target fitting curve is the largest at the central axis or center of the image to be calibrated, so that the curvature generated by the intermediate point with respect to the pixel points or corner points distributed on both sides of the central axis within the target fitting curve is large; in some specific scenarios, the intermediate point can be the corner point to be fitted.

[0047] After determining the Nth-degree polynomial equation and the coordinates of the intermediate point, the following partial derivative calculations exist: Calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the intermediate point to obtain the first first partial derivative value; calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the intermediate point to obtain the first second partial derivative value; thus, obtain the first-order partial derivative result of the Nth-degree polynomial equation. It can be understood that substituting the coordinates of the intermediate point into the expression of the first derivative of the Nth-degree polynomial equation with respect to U and V (the first independent variable and the second independent variable) to obtain the first first partial derivative value and the first second partial derivative value.

[0048] Calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the intermediate point to obtain the second first partial derivative value; calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the intermediate point to obtain the second second partial derivative value; thus, obtain the second-order partial derivative result of the Nth-degree polynomial equation. It can be understood that substituting the coordinates of the intermediate point into the expression of the second derivative of the Nth-degree polynomial equation with respect to U and V to obtain the second first partial derivative value and the second second partial derivative value.

[0049] Then, based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value, calculate the curvature of the intermediate point; during the implementation of the calculation process, substitute the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value into the curvature formula to convert the curvature of the intermediate point in the target fitting curve, so as to represent the curvature of the intermediate point on the curve fitted by the Nth-degree polynomial equation. Among them, the curvature formula is preset to calculate the curvature of the Nth-degree polynomial equation at a certain pixel point by using the first-order partial derivative and the second-order partial derivative; then set the curvature of the intermediate point as the fitting evaluation value. Generally, the curvature of the intermediate point is greater than the curvature of the pixel points passed by the target fitting curve distributed on both sides of the intermediate point.

[0050] Therefore, the fitting evaluation value can describe the maximum bending degree of the connection line (a straight line segment parallel to the boundary of the target) of a group of corner points to be fitted mapped into the world coordinate system within the image to be calibrated, and further reflect the maximum distortion of the lens, which is used as the basis for selecting the calibration method to facilitate the selection of a matching calibration model in step B.

[0051] As Embodiment 3, for the aforementioned Step A, the method for performing fitting processing on the image collected by the camera to obtain a fitting evaluation value includes: Based on the least squares method, curve fitting is performed on multiple corner points to be fitted that are selected, to form an Nth-degree polynomial equation, and the target fitting curve is determined, so as to form a fitting result of curve fitting within the image to be calibrated; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable. The first independent variable can be represented by Figure 2 the abscissa U, and the second independent variable can be represented by Figure 2 the ordinate V, and both the first independent variable and the second independent variable can be combined with the coefficients of each term to form the Nth-degree polynomial equation; in this embodiment, at least 3 of the multiple corner points to be fitted that are pre-selected exist and are arranged into a group of corner points to be fitted; in this embodiment, curve fitting is performed based on a multiple function (Nth-degree polynomial equation) of the least squares method, and the coefficients of each term and the constant term that are optimal for the multiple function are calculated, so that an Nth-degree polynomial equation can be constructed, which is also defined as an Nth-degree surface equation.

[0052] Among the pixel points passed by the target fitting curve within the image to be calibrated, multiple pixel points are sequentially selected; it should be noted that the distribution positions of the multiple selected pixel points within the image to be calibrated are not limited, and they can be symmetric about the central axis of the image to be calibrated, or can be sequentially distributed along the same coordinate axis direction at a certain coordinate interval in the target fitting curve, or can be from corner points, or even multiple pixel points randomly selected from the target fitting curve; where the number of the multiple selected pixel points can be 3 or more.

[0053] For the multiple selected pixel points, there is: Calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain a first partial derivative value; calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain a second partial derivative value; thereby obtaining the first-order partial derivative results of the Nth-degree polynomial equation at each pixel point passed by the target fitting curve. It can be understood that by substituting the coordinates of each selected pixel point into the expression of the first derivative of the Nth-degree polynomial equation with respect to U and V (the first independent variable and the second independent variable), the first partial derivative value and the second partial derivative value of each pixel point in the Nth-degree polynomial equation are respectively obtained.

[0054] Calculate the second partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain the first second partial derivative value; calculate the second partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain the second second partial derivative value; thereby obtaining the second partial derivative results of the Nth-degree polynomial equation at each pixel point passed by the target fitting curve. It can be understood that substituting the coordinates of each selected pixel point into the expression of the second derivative of the Nth-degree polynomial equation with respect to U and V (the first independent variable and the second independent variable), the first second partial derivative value and the second second partial derivative value of each pixel point in the Nth-degree polynomial equation are obtained respectively.

[0055] Then, based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value and the second second partial derivative value, calculate the curvature of the same pixel point; during the implementation of the calculation, substitute the first first partial derivative value, the first second partial derivative value, the second first partial derivative value and the second second partial derivative value into the curvature formula to convert the curvature of the same pixel point, so as to represent the curvature of the same pixel point on the curve fitted by the Nth-degree polynomial equation, where the curvature formula is preset to calculate the curvature of the Nth-degree polynomial equation at a certain pixel point by using the first-order partial derivative and the second-order partial derivative.

[0056] After the curvature of each selected pixel point is converted, calculate the average value of the curvatures of all the selected pixel points; then set the calculated average value as the fitting evaluation value. Among them, the average value of the curvatures of each selected pixel point is not necessarily the maximum curvature of the target fitting curve in the image to be calibrated, but can play a role in identifying the average bending degree of the boundary of the target in the world coordinate system after being projected by the lens into the pixel coordinate system, and reflect the average distortion degree of the lens.

[0057] On the basis of the foregoing Embodiment 2 and Embodiment 3, N is equal to the value 3 to form a cubic polynomial equation, and a cubic polynomial surface is determined to be fitted. As for the general form of the cubic polynomial equation, those skilled in the art can combine the coefficients of each term and the constant term calculated during the curve fitting process to combine a cubic polynomial to construct the Nth-degree polynomial equation. Preferably, the coefficients of each term and the constant term all represent the fitting parameters. The coefficients of each term include the coefficient of the linear term, the coefficient of the quadratic term, the coefficient of the cubic term and the coefficient of the cross term, which belong to the coefficients of each term of the curve equation calculated based on the least squares method. The curve fitting method is used for fitting processing to judge the distortion size of the lens and serve as the basis for selecting the calibration method. Since there is a threshold setting for the curvature later, a specific calibration model can be selected through a range judgment.

[0058] The first independent variable included in the cubic polynomial equation is used to represent the abscissa of the pixel coordinate system, and the second independent variable included in the cubic polynomial equation is used to represent the ordinate of the pixel coordinate system, so that the first-order partial derivative of the cubic polynomial equation with respect to the first independent variable represents the tangent slope of the target fitting curve in the horizontal axis direction of the pixel coordinate system, and the first-order partial derivative of the cubic polynomial equation with respect to the second independent variable represents the tangent slope of the target fitting curve in the vertical axis direction of the pixel coordinate system.

[0059] In summary, the present application can introduce a cubic polynomial for curvature calculation, quickly obtain the fitting evaluation value, reduce the time of fitting processing, and improve the efficiency of judging lens distortion.

[0060] In the foregoing Embodiment 2 and Embodiment 3, based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value, the method for calculating the curvature includes: controlling the multiplication of the first first partial derivative value and the second second partial derivative value to obtain a first product; controlling the multiplication of the first second partial derivative value and the second first partial derivative value to obtain a second product; calculating the sum of the squares of the first first partial derivative value and the first second partial derivative value to obtain a first sum of squares; and then calculating the ratio between the absolute value of the difference between the first product and the second product and the square root of the cube of the first sum of squares to obtain the curvature; when the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value are all partial derivatives at the intermediate point, the calculated curvature is the curvature of the intermediate point. When the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value are all partial derivatives at one of the pre-selected pixel points, the calculated curvature is the curvature of one of the pre-selected pixel points. Therefore, the present application provides an evaluation numerical basis for judging and identifying the degree of bending of the boundary of the target in the world coordinate system after being projected by the lens onto the pixel coordinate system.

[0061] Specifically, the method for detecting the distortion condition of the lens according to the magnitude relationship between the fitting evaluation value and the preset threshold includes: In the case of performing curve fitting on multiple corner points to be fitted, if the curvature of the middle point is set to the fitting evaluation value, corresponding to the fitting process of the foregoing Embodiment 2, the preset threshold used to detect the distortion of the lens is the second preset threshold; when the fitting evaluation value is greater than the second preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter, indicating that the distortion of the lens is relatively large; when the fitting evaluation value is less than or equal to the second preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, indicating that the distortion of the lens is relatively small, so as to prompt the camera or the ranging module where the camera is located to call a calibration model different from the calibration model used in the case of a relatively large distortion of the lens, reducing the misjudgment of the distortion situation. The preset distortion parameter is used as the distortion reference value for different models of lenses to describe the distortion degree of the lens and is used as the basis for the distortion parameter for selecting the matching calibration model.

[0062] It should be noted that the middle point is the intersection point of the target fitting curve and the central axis of the image to be calibrated; according to the image distortion characteristics, the pixel points on the central axis of the image to be calibrated are the pixel points with the largest curvature in the image to be calibrated. Therefore, comparing the curvature of the middle point with the preset threshold can better judge the extreme curvature level of the surface / curve fitted by the image to be calibrated, that is, reflect the maximum distortion that the lens can generate. Therefore, comparing the curvature of the middle point with the second preset threshold reduces the misjudgment of the distortion situation and meets the calibration accuracy in the case of excessive distortion of the lens.

[0063] On the other hand, in the case of performing curve fitting on multiple corner points to be fitted, if the average value of the curvatures of multiple pixel points selected from the target fitting curve is set to the fitting evaluation value, corresponding to the fitting process of the foregoing Embodiment 3, the preset threshold used to detect the distortion of the lens is the third preset threshold; when the fitting evaluation value is greater than the third preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter, indicating that the distortion of the lens is relatively large; when the fitting evaluation value is less than or equal to the third preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, indicating that the distortion of the lens is relatively small, so as to prompt the camera or the ranging module where the camera is located to call a calibration model different from the calibration model used in the case of a relatively large distortion of the lens, adapting to the correction of lenses with relatively small distortion.

[0064] It should be noted that the second preset threshold is not equal to the first preset threshold; the third preset threshold is not equal to the first preset threshold; the third preset threshold is less than the second preset threshold. Among them, the fitting evaluation value obtained by the fitting process in the foregoing Embodiment 3 is less than the fitting evaluation value obtained by the fitting process in the foregoing Embodiment 2, and the determination of the distortion size of the camera lens can be achieved, but different thresholds need to be set so that the third preset threshold and the second preset threshold respectively correspond to the actual calibration conditions of the lenses of different models of fisheye cameras. Adaptively, when the distortion of the lenses of the cameras participating in the selection of the calibration model causes the curvature of the pixel points on the imaging plane to be greater than the first preset threshold and / or the third preset threshold, the second preset threshold is selected as the preset threshold for detecting the distortion of the lens. When the distortion of the lenses of the cameras participating in the selection of the calibration model causes the curvature of the pixel points on the imaging plane to be greater than the first preset threshold but less than the second preset threshold, the third preset threshold is selected as the preset threshold for detecting the distortion of the lens. If more values of the preset threshold are set, more lens types can be distinguished by the preset threshold, that is, more lens types with different distortions can be identified to be compatible with the calibration of more distorted cameras.

[0065] Based on the foregoing embodiments, the method for selecting a matching calibration model according to the distortion of the lens includes: when it is determined that the distortion of the lens is greater than the preset distortion parameter, the selected matching calibration model is a fisheye lens distortion calibration model for geometric modeling and calculating the camera internal parameters, including calculating the calibrated camera internal parameters using the fisheye lens distortion correction function. When it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, the selected matching calibration model is a pinhole imaging distortion calibration model for geometric modeling and calculating the camera internal parameters, including calculating the calibrated camera internal parameters using the pinhole imaging distortion correction function. Among them, the preset distortion parameter is used as the distortion reference value for different models of lenses to describe the distortion degree of the lens and is used as the basis for the distortion parameter for selecting the matching calibration model, specifically related to whether to use the fisheye lens distortion correction function or the pinhole imaging distortion correction function for camera calibration. When the matching calibration model outputs the camera internal parameters, it is determined that the calibration of the camera is completed. Thus, corresponding camera calibration methods can be selected according to different distortions, enhancing the calibration adaptability of different distorted lenses and improving the accuracy of internal parameter calculation. Therefore, when images are collected using the corresponding lenses after adaptive correction, the measurement errors generated meet the control standards of the camera ranging performance.

[0066] The matched calibration model belongs to the camera calibration model, which is derived from the mathematical analysis and operation software publicly available in the prior art. The mathematical interface function set in the camera calibration model calculates the pixel points of the image to be calibrated, and can calculate the internal parameters of the camera and the external parameters from the calibration target to the camera, realizing the calibration of the camera before using the camera for ranging. Preferably, the camera calibration model takes into account the distortion coefficient of the lens and uses the distortion coefficient elimination constructor to overcome the image distortion error caused by the distortion in the corresponding coordinate axis direction.

[0067] For the internal parameter calibration of a fish-eye camera, the OCamCalib full-view camera model calibration and correction algorithm proposed by David Scaramuzza can be used, or the calculation method based on the fish-eye lens distortion calibration model proposed by Juho Kannala and Sami Brandt in the open-source algorithm library OpenCV can be used to calculate the internal parameters of the camera.

[0068] Generally, the imaging model of the lens is used to describe the projection relationship of the real camera lens, and there will be a gap with the reality. Therefore, in this application, different imaging models are used to describe the projection relationships of different lenses, such as the aforementioned pinhole imaging distortion calibration model and the fish-eye lens distortion calibration model. In this application, as long as the lens with a slightly larger imaging angle (which can be imagined as a fish-eye lens with a larger distortion), if the simple pinhole model is still used to describe it, the ranging deviation will be very large. Therefore, different camera models need to be selected to describe it.

[0069] The results output by the matched calibration model include the internal parameters of the camera. Based on the internal parameters of the camera, the conversion from the camera coordinate system to the pixel coordinate system can be achieved, and the internal parameters output by the matched calibration model can also explain the stretching and deformation of the graphics in the world coordinate system by the fish-eye lens (the lens of the fish-eye camera) or the pinhole camera lens. The corresponding internal parameter values can be compensated for the misalignment from the sensor to the lens in the form of a stretching matrix.

[0070] As an embodiment, after calculating the internal parameters of the camera, in order to adapt to the versatility of different lenses, it is necessary to process the data according to the data transmission interface. Combining Figure 1 it can be seen that the camera adaptive calibration method further includes: Step C: Convert the calculated camera internal parameters into parameters compatible with the data transmission interface, so that the transmission of camera internal parameters output by different calibration models is compatible with the data transmission interface; therefore, the data transmission interface is adapted to the fisheye lens and / or the lens of the pinhole camera in software control; wherein, Step C starts to be executed after Step B is executed. When using a general interface for pixel data transmission, by configuring and executing Steps A to C, the camera self-adaptive calibration method can be set as a general calibration software, improving the data transmission compatibility effect and calibration adaptability of the software for different camera lenses, and meeting the camera self-adaptive calibration effect.

[0071] Combined with the foregoing embodiments, each calibration model matches a camera and its internal parameters. Among them, it can be considered that each calibration model matches a lens to correct the distortion of the corresponding lens. In the present application, the fisheye lens distortion calibration model matches a fisheye lens so that the camera where the fisheye lens is located selects the fisheye lens distortion calibration model for camera calibration, and the pinhole imaging distortion calibration model matches a pinhole lens so that the camera where the pinhole lens is located selects the pinhole imaging distortion calibration model for camera calibration. The camera that selects the fisheye lens distortion calibration model can be defined as a fisheye camera, and the fisheye camera is equipped with the fisheye lens; the camera that selects the pinhole imaging distortion calibration model can be defined as a pinhole camera, and the pinhole camera is equipped with the pinhole lens. The same data transmission interface can be compatible with the internal parameter data transmission and storage required by the fisheye lens and the pinhole lens, and can avoid the problem of modeling errors caused by the parameter bit width (storage address bit width) being less than the bit width of the same data transmission interface when the distortion is small, improving the compatibility of data storage and transmission for camera calibration with different calibration models.

[0072] As an embodiment, the method for converting the calculated camera internal parameters into parameters compatible with the data transmission interface includes: Mark the camera internal parameters calculated by the pinhole imaging distortion calibration model as the first internal parameters; mark the camera internal parameters calculated by the fisheye lens distortion calibration model as the second internal parameters; since the parameter scale required for the fisheye lens distortion calibration model to correct the camera internal parameters is larger than the parameter scale required for the pinhole imaging distortion calibration model to correct the camera internal parameters (for example, the size and quantity of distortion parameters, the coordinate type of the focal position), the number of parameters required to be corrected by the fisheye lens distortion calibration model is more than the number of parameters required to be corrected by the pinhole imaging distortion calibration model (for example, coordinate system mapping parameters), so the number of the first internal parameters is less than the number of the second internal parameters, and the first internal parameters and the second internal parameters are not completely the same.

[0073] After the camera internal parameters are calculated by the pinhole imaging distortion calibration model, zero-padding processing is performed on the first internal parameters to expand the data bit width of the corresponding projection parameters or add new projection parameters, and then the first internal parameters after zero-padding processing are set as the parameters compatible with the data transmission interface; wherein, the zero-padding processing on the first internal parameters is a binary zero-value filling operation set for data storage and data transmission scenarios; in some embodiments, zero-padding processing can be performed on other calibration-related parameters to be compatible with the transmission and storage requirements of the data transmission interface, including the address bit width requirements. Thereby reducing the modeling error caused by the lack of storage and transmission parameters.

[0074] After the camera internal parameters are calculated by the fisheye lens distortion calibration model, zero-padding is not performed on the second internal parameters, and the second internal parameters are set as the parameters compatible with the data transmission interface; wherein, the parameters output by the fisheye lens distortion calibration model are more complex than the parameters output by the pinhole imaging distortion calibration model, which are sufficient to meet the transmission and storage requirements of the data transmission interface, at least meeting the address bit width requirements of the data transmission interface.

[0075] In some embodiments, the number of projection parameters included in the first internal parameters is less than the number of projection parameters included in the second internal parameters, so that there are no partial projection parameters required for calibration by the fisheye lens distortion calibration model among the projection parameters required to be corrected by the pinhole imaging distortion calibration model. That is, in the scenario of calibrating the same image, the pinhole imaging distortion calibration model calls fewer projection parameters than the fisheye lens distortion calibration model. The parameters of the lens selected for calibration by the pinhole imaging distortion calibration model (corresponding to the projection parameters required to be corrected by the pinhole imaging distortion calibration model) are fewer than the parameters of the lens selected for calibration by the fisheye lens distortion calibration model (corresponding to the projection parameters required to be corrected by the fisheye lens distortion calibration model). It can be considered that the pinhole imaging distortion calibration model lacks parameters relative to the fisheye lens distortion calibration model during data storage and needs to be zero-padded; wherein, both the first internal parameters and the second internal parameters include projection parameters, and the projection parameters are used to represent the projection relationship from the world coordinate system to the pixel coordinate system.

[0076] In some embodiments, the bit width of the projection parameters included in the first internal parameter is less than the bit width of the same type of projection parameters included in the second internal parameter, such that the projection parameters to be corrected by the pinhole imaging distortion calibration model are different from the same type of projection parameters to be corrected by the fisheye lens distortion calibration model. That is, the parameters of the lens selected for calibration using the pinhole imaging distortion calibration model (corresponding to the projection parameters to be corrected by the pinhole imaging distortion calibration model) are different from the parameters of the lens selected for calibration using the fisheye lens distortion calibration model (corresponding to the same type of projection parameters to be corrected by the fisheye lens distortion calibration model). In the scenario of calibrating the same image, the projection parameters called by the pinhole imaging distortion calibration model are less than the projection parameters called by the fisheye lens distortion calibration model; wherein, both the first internal parameter and the second internal parameter include projection parameters, and the projection parameters are used to represent the projection relationship from the world coordinate system or the camera coordinate system to the pixel coordinate system. Preferably, the bit width of the data transmission interface is configured to be equal to the bit width of the second internal parameter, so that the bit width of the data transmission interface is configured to be compatible with the second internal parameter, enabling the second internal parameter to fill the data transmission interface during the process of passing through the preset data transmission interface.

[0077] In summary, by comparing the magnitude and quantity relationship between the first internal parameter and the second internal parameter, the difference between the projection parameters to be corrected by the pinhole imaging distortion calibration model and the projection parameters to be corrected by the fisheye lens distortion calibration model is determined, providing a judgment basis related to the calibration model for subsequent zero-padding.

[0078] In some embodiments, the method of zero-padding the first internal parameter includes: In the case where some of the projection parameters to be corrected by the fisheye lens distortion calibration model do not exist among the projection parameters to be corrected by the pinhole imaging distortion calibration model, it can be confirmed that the projection parameters called by the pinhole imaging distortion calibration model are fewer than the projection parameters called by the fisheye lens distortion calibration model, and it can be understood that the internal camera parameters output by the pinhole imaging distortion calibration model are missing parameters at the data transmission interface. Then, new projection parameters in the first internal parameter are formed by binary zero values with multiple bits, and the bit width of the new projection parameters is set to be equal to the bit width of some of the projection parameters to be corrected by the fisheye lens distortion calibration model to be compatible with the data transmission interface; and each bit in the new projection parameters is marked with a zero-padding bit. Among the new projection parameters here, the number of bits can be equal to the bit width of the same type of projection parameters in the fisheye lens distortion calibration model to make up for the defect of the missing parameters of the internal camera parameters output by the pinhole imaging distortion calibration model at the data transmission interface.

[0079] When the bit width of the projection parameter included in the first internal parameter is less than the bit width of the same type of projection parameter included in the second internal parameter, it can be determined that the projection parameter to be corrected by the pinhole imaging distortion calibration model is different from the same type of projection parameter to be corrected by the fisheye lens distortion calibration model. It can be understood that the projection parameter called by the pinhole imaging distortion calibration model is less than the projection parameter called by the fisheye lens distortion calibration model. Then, binary zero values need to be filled bit by bit starting from the first bit of the projection parameter included in the first internal parameter until the bit width of the projection parameter included in the first internal parameter after zero filling is equal to the bit width of the same type of projection parameter included in the second internal parameter, and the value of the projection parameter included in the first internal parameter after zero filling is equal to the value of the projection parameter included in the first internal parameter. That is, zero filling the same internal parameter does not change the original numerical size and will not introduce new errors to the camera; then, mark each filled bit as a zero-filled bit.

[0080] In summary, after calculating the camera internal parameters using the matched calibration model, in order to adapt to the generality of different lenses, it is necessary to process the data according to the data transmission interface. The number of parameters of the lens selected for calibration using the pinhole imaging distortion calibration model is less than the number of parameters of the lens selected for calibration using the fisheye lens distortion calibration model, and the value of the parameters of the lens selected for calibration using the pinhole imaging distortion calibration model is different from the value of the parameters of the lens selected for calibration using the fisheye lens distortion calibration model. Then, the projection parameter included in the first internal parameter is less than the projection parameter included in the second internal parameter, and the first internal parameter is different from the second internal parameter; therefore, zero filling is performed on the parameters of the lens selected for calibration using the pinhole imaging distortion calibration model during data transmission and storage to be compatible with the data transmission interface, so that the data transmission interface can adapt to the generality of different lenses; thereby making full use of the data transmission interface to improve the transmission efficiency and storage efficiency during the calibration of different distortion lenses under the matched calibration model, and ensuring that the data transmission and storage requirements of different distortion lenses under the matched calibration model are compatible with the data transmission interface, adapting to the camera internal parameter storage corresponding to different lenses, and providing a hardware interface environment with strong compatibility for the adaptive calibration of different distortion lenses.

[0081] In the subsequent scenario of using the first internal parameter or the second internal parameter for ranging calculation, different lens types are determined by checking the zero-filled bits set at certain bit positions in the camera internal parameter and judging whether their values are binary zero values, including distinguishing between fisheye lenses and the lenses of pinhole cameras.

[0082] Next, several coordinate systems and coordinate transformation principles involved in this application will be introduced in detail.

[0083] World coordinate system: A reference coordinate system is also selected in the environment to describe the positions of the camera and the object, and this coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix R and the translation vector t.

[0084] Pixel coordinate system (U, V): It is a two-dimensional rectangular coordinate system that reflects the arrangement of pixel points in the imaging plane of the camera. The origin O is located at the upper left corner of the image, and the U-axis and V-axis are parallel to the two sides of the image plane respectively.

[0085] The above embodiments are only for illustrating the technical concept and features of the present invention. The purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A camera adaptive calibration method based on lens distortion, characterized in that, it includes: Performing fitting processing on the images collected by the camera to obtain a fitting evaluation value; wherein, the fitting evaluation value is used to reflect the distortion condition of the lens; the lens is assembled in the camera; Based on the fitting evaluation value, selecting a matching calibration model, and then using the matching calibration model to calibrate the camera to calculate the internal parameters of the camera.

2. The camera adaptive calibration method according to claim 1, characterized in that, Before performing fitting processing on the images collected by the camera, it further includes: Controlling the camera to capture an image of a target to obtain an image to be calibrated; Selecting a plurality of corner points to be fitted from the image to be calibrated; wherein, the target is composed of a plurality of cells arranged row by row or column by column in one direction; wherein, the corner points to be fitted are the corner points located at the boundary of the target among the columns of cells closest to the boundary of the image to be calibrated; or, the corner points to be fitted are the corner points located at the boundary of the target among the rows of cells closest to the boundary of the image to be calibrated; or, the corner points to be fitted are the corner points of the columns of cells closest to the boundary of the image to be calibrated inside the target; or, the corner points to be fitted are the corner points of the rows of cells closest to the boundary of the image to be calibrated inside the target.

3. The camera adaptive calibration method according to claim 2, characterized in that, The method for selecting a matching calibration model based on the fitting evaluation value includes: Detecting the distortion condition of the lens according to the magnitude relationship between the fitting evaluation value and a preset threshold; Selecting a matching calibration model according to the distortion condition of the lens to correct the distortion of the lens.

4. The camera adaptive calibration method according to claim 3, characterized in that, The method for performing fitting processing on the images collected by the camera to obtain a fitting evaluation value includes: Using the least squares method to perform linear fitting on the selected plurality of corner points to be fitted to obtain a target fitting line; Calculating the distances from each corner point to be fitted to the target fitting line, then averaging the distances from each corner point to be fitted to the target fitting line, and then setting the obtained average value as the fitting evaluation value.

5. The camera adaptive calibration method according to claim 4, characterized in that, The method for detecting the distortion condition of the lens according to the magnitude relationship between the fitting evaluation value and a preset threshold includes: In the case of performing linear fitting on the selected plurality of corner points to be fitted to obtain a fitting evaluation value, the preset threshold used for detecting the distortion condition of the lens is a first preset threshold; when the fitting evaluation value is greater than the first preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter; when the fitting evaluation value is less than or equal to the first preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter.

6. The camera adaptive calibration method according to claim 3, characterized in that, The method for performing fitting processing on the images collected by the camera to obtain a fitting evaluation value includes: Curve fitting is performed on multiple corner points to be fitted selected based on the least squares method to form an Nth-degree polynomial equation, and the target fitting curve is determined; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable. The intersection point of the target fitting curve and the central axis of the to-be-calibrated image is set as the middle point. Calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the middle point to obtain the first first partial derivative value. Calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the middle point to obtain the first second partial derivative value. Calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the middle point to obtain the second first partial derivative value. Calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the middle point to obtain the second second partial derivative value. Based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value, calculate the curvature of the middle point. Set the curvature of the middle point as the fitting evaluation value.

7. The camera adaptive calibration method according to claim 3, wherein, the method for performing fitting processing on the image collected by the camera to obtain a fitting evaluation value includes: Curve fitting is performed on multiple corner points to be fitted selected based on the least squares method to form an Nth-degree polynomial equation, and the target fitting curve is determined; where N is an integer greater than 2; the Nth-degree polynomial equation includes a polynomial composed of a first independent variable, coefficients of each term, and a second independent variable. Successively select multiple pixel points from among the pixel points passed through by the target fitting curve within the to-be-calibrated image. Among the multiple selected pixel points, there are: calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain the first first partial derivative value; calculate the first-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain the first second partial derivative value; calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the first independent variable at the same pixel point to obtain the second first partial derivative value; calculate the second-order partial derivative of the Nth-degree polynomial equation with respect to the second independent variable at the same pixel point to obtain the second second partial derivative value; then based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value, calculate the curvature of the same pixel point. After the curvature of each selected pixel point is calculated, calculate the average value of the curvatures of all the selected pixel points; then set the calculated average value as the fitting evaluation value.

8. The camera adaptive calibration method according to claim 6 or 7, wherein, N is equal to the value 3 to form a cubic polynomial equation. The first independent variable included in the cubic polynomial equation is used to represent the abscissa of the pixel coordinate system, and the second independent variable included in the cubic polynomial equation is used to represent the ordinate of the pixel coordinate system, so that the first-order partial derivative of the cubic polynomial equation with respect to the first independent variable represents the tangent slope of the target fitting curve in the horizontal axis direction of the pixel coordinate system, and the first-order partial derivative of the cubic polynomial equation with respect to the second independent variable represents the tangent slope of the target fitting curve in the vertical axis direction of the pixel coordinate system.

9. The camera adaptive calibration method according to claim 6 or 7, characterized in that Based on the first first partial derivative value, the first second partial derivative value, the second first partial derivative value, and the second second partial derivative value, the ways to calculate the curvature include: Control the multiplication of the first first partial derivative value and the second second partial derivative value to obtain a first product; Control the multiplication of the first second partial derivative value and the second first partial derivative value to obtain a second product; Calculate the sum of the squares of the first first partial derivative value and the first second partial derivative value to obtain a first sum of squares; Calculate the ratio between the absolute value of the difference between the first product and the second product and the square root of the cube of the first sum of squares to obtain the curvature.

10. The camera adaptive calibration method according to claim 6 or 7, characterized in that The method for detecting the distortion of the lens according to the magnitude relationship between the fitting evaluation value and the preset threshold includes: In the case of curve fitting for a selected plurality of corner points to be fitted, if the curvature of the middle point is set as the fitting evaluation value, the preset threshold used for detecting the distortion of the lens is the second preset threshold; when the fitting evaluation value is greater than the second preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter; when the fitting evaluation value is less than or equal to the second preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter; wherein, the middle point is the intersection point of the target fitting curve and the central axis of the image to be calibrated; In the case of curve fitting for a selected plurality of corner points to be fitted, if the average value of the curvatures of a plurality of pixel points selected from the target fitting curve is set as the fitting evaluation value, the preset threshold used for detecting the distortion of the lens is the third preset threshold; when the fitting evaluation value is greater than the third preset threshold, it is determined that the distortion of the lens is greater than the preset distortion parameter; when the fitting evaluation value is less than or equal to the third preset threshold, it is determined that the distortion of the lens is less than or equal to the preset distortion parameter; wherein, the third preset threshold is less than the second preset threshold.

11. The camera adaptive calibration method according to claim 3, characterized in that The method for selecting a matching calibration model according to the distortion of the lens includes: When it is determined that the distortion of the lens is greater than the preset distortion parameter, the selected matching calibration model is a fisheye lens distortion calibration model to perform geometric modeling and calculate the camera internal parameters; When it is determined that the distortion of the lens is less than or equal to the preset distortion parameter, the selected matching calibration model is a pinhole imaging distortion calibration model to perform geometric modeling and calculate the camera internal parameters.

12. The camera adaptive calibration method according to claim 11, characterized in that The camera adaptive calibration method further includes: converting the calculated camera internal parameters into parameters compatible with the data transmission interface, so that the transmission of the camera internal parameters output by different calibration models is compatible with the data transmission interface; wherein, each calibration model matches a camera and its internal parameters.

13. The camera adaptive calibration method according to claim 12, wherein, the method for converting the calculated camera internal parameters into parameters compatible with the data transmission interface includes: marking the camera internal parameters calculated by the pinhole imaging distortion calibration model as the first internal parameters; marking the camera internal parameters calculated by the fisheye lens distortion calibration model as the second internal parameters; after the camera internal parameters are calculated by the pinhole imaging distortion calibration model, expanding the data bit width of the corresponding projection parameters or adding new projection parameters by zero-padding the first internal parameters, and then setting the zero-padded first internal parameters as the parameters compatible with the data transmission interface; after the camera internal parameters are calculated by the fisheye lens distortion calibration model, not zero-padding the second internal parameters, and setting the second internal parameters as the parameters compatible with the data transmission interface.

14. The camera adaptive calibration method according to claim 13, wherein, the number of projection parameters included in the first internal parameters is less than the number of projection parameters included in the second internal parameters, so that there are no partial projection parameters required for correction by the fisheye lens distortion calibration model among the projection parameters required for correction by the pinhole imaging distortion calibration model; the bit width of the projection parameters included in the first internal parameters is less than the bit width of the same type of projection parameters included in the second internal parameters, so that the projection parameters required for correction by the pinhole imaging distortion calibration model are different from the same type of projection parameters required for correction by the fisheye lens distortion calibration model; wherein, the bit width of the data transmission interface is configured to be equal to the bit width of the second internal parameters; both the first internal parameters and the second internal parameters include projection parameters, and the projection parameters are used to represent the projection relationship from the world coordinate system to the pixel coordinate system.

15. The camera adaptive calibration method according to claim 14, wherein, the way of zero-padding the first internal parameters includes: when there are no partial projection parameters required for correction by the fisheye lens distortion calibration model among the projection parameters required for correction by the pinhole imaging distortion calibration model, forming new projection parameters in the first internal parameters with multi-bit binary zero values, setting the bit width of the new projection parameters to be equal to the bit width of the partial projection parameters required for correction by the fisheye lens distortion calibration model, and marking each bit in the new projection parameters with a zero-padding bit; when the bit width of the projection parameters included in the first internal parameters is less than the bit width of the same type of projection parameters included in the second internal parameters, sequentially padding binary zero values starting from the first bit of the projection parameters included in the first internal parameters until the bit width of the projection parameters included in the zero-padded first internal parameters is equal to the bit width of the same type of projection parameters included in the second internal parameters, and the value of the projection parameters included in the zero-padded first internal parameters is equal to the value of the projection parameters included in the first internal parameters before zero-padding, and then marking each padded bit with a zero-padding bit.

Images