Image distortion correction method, device, electronic device and storage medium

Abstract

The present invention provides an image distortion correction method, device, electronic device and storage medium. The image distortion correction method comprises: obtaining a first image of a calibration plate taken by a camera; for each preset feature point, determining the current feature point in the first image corresponding to the preset feature point, wherein the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera takes the first image, and mapping the feature point on the calibration plate to the feature point in the image taken by the camera; determining the value of the distortion parameter of the camera according to the preset feature point and the predicted feature point, wherein the predicted feature point is represented by the current feature point in the first image and the distortion parameter of the camera with an undetermined value; and correcting the first image using the distortion parameter of the camera with a determined value to obtain a second image. This solution can accurately perform distortion correction on the image taken by the camera.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image distortion correction method, an image distortion correction device, an electronic device, a storage medium and a computer program product. Background Art

[0002] Lens distortion refers to the geometric deformation of image information caused by the optical characteristics of the lens used to capture the image, and mainly includes radial distortion and tangential distortion. Radial distortion usually manifests itself as barrel distortion (image information at the edge of the image bulges outward) and pincushion distortion (image information at the edge of the image sinks inward), while tangential distortion is an asymmetric distortion caused by the lens and the imaging plane not being completely parallel.

[0003] Lens distortion can cause straight lines in the image to bend and produce image distortion, which affects the geometric accuracy of the image and subsequent computer vision tasks. In order to correct these distortions, lens distortion calibration is usually required to use the calibration accuracy of camera internal and external parameters. During calibration, multiple stereo calibration blocks or calibration plate images in multiple postures need to be taken for estimating distortion parameters.

[0004] When the camera depth of field is small, the tilted calibration plate will be out of focus, which is not conducive to calibrating the internal and external parameters of the camera, and will lead to poor distortion calibration accuracy and affect the image distortion correction effect. Summary of the invention

[0005] The present invention has been made in view of the above-mentioned problems.

[0006] According to a first aspect of the present invention, a method for image distortion correction is provided. The method comprises: acquiring a first image of a calibration plate taken by a camera; for each current feature point in the first image, determining a preset feature point corresponding to the current feature point, wherein the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera takes the first image, and mapping the feature point on the calibration plate to the feature point in the first image; determining the value of the distortion parameter of the camera according to the preset feature point and the predicted feature point, wherein the predicted feature point is represented by the current feature point in the first image and the distortion parameter of the camera with an undetermined value; and correcting the first image using the distortion parameter of the camera with a determined value to obtain a second image.

[0007] Exemplarily, determining the value of the distortion parameter of the camera based on the preset feature points and the predicted feature points includes: determining a minimization function for the unknown quantity of the distortion parameter based on the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point; performing at least one line search on the minimization function so that the minimization function obtains a minimum value; when the minimization function obtains the minimum value, using the current value of the distortion parameter as the value of the distortion parameter.

[0008] Exemplarily, performing at least one line search on the minimization function so that the minimization function obtains a minimum value includes: using a quasi-Newton method to determine a search direction for each line search; performing at least one line search on the minimization function according to the determined search direction, and updating the value of the distortion parameter after each line search until the minimization function obtains the minimum value.

[0009] Exemplarily, the method of correcting the first image by using the distortion parameters of the camera with determined values ​​to obtain the second image includes: determining the corrected pixels in the first image according to the distortion parameters with determined values; determining the second image according to the corrected pixels, wherein, for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined according to the pixel values ​​of pixels adjacent to the corrected pixel.

[0010] Exemplarily, the first image is taken by the camera when the optical axis of the camera is perpendicular to the calibration plate.

[0011] Exemplarily, the flange interface of the camera is provided with a reflector, and the method further includes: irradiating the reflector with a laser autocollimator and receiving a reflected light signal reflected from the reflector, wherein the laser autocollimator is configured so that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located; and determining whether the optical axis of the camera is perpendicular to the calibration plate based on the reflected light signal received by the laser autocollimator.

[0012] Exemplarily, the calibration plate is arranged such that an arrangement direction of checkerboards of the calibration plate in the first image is parallel to a horizontal direction of a plane where the first image is located.

[0013] Exemplarily, the method also includes: determining the standard deviation between the length of each chessboard side of the calibration plate in the second image and the average value of the length of each chessboard side; when the standard deviation is less than a preset threshold, using the distortion parameters with the determined values ​​to correct other images taken by the camera in the same posture.

[0014] Exemplarily, the unknown distortion parameters include camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

[0015] According to a second aspect of the present invention, there is also provided an image distortion correction device, comprising:

[0016] An image receiving module, used to obtain a first image of the calibration plate taken by a camera;

[0017] a feature point matching module, configured to determine, for each preset feature point, a current feature point in the first image corresponding to the preset feature point, wherein the preset feature point is based on a positional relationship between the camera and the calibration plate when the camera captures the first image, and maps the feature point on the calibration plate to a feature point in the image captured by the camera;

[0018] a parameter determination module, configured to determine a value of a distortion parameter of the camera according to the preset feature points and the predicted feature points, wherein the predicted feature points are represented by current feature points in the first image and distortion parameters of the camera with undetermined values;

[0019] The image correction module is used to correct the first image by using the distortion parameter of the camera with a determined value to obtain a second image.

[0020] According to a third aspect of the present invention, there is also provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned image distortion correction method when the processor is running.

[0021] According to a fourth aspect of the present invention, a storage medium is further provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned image distortion correction method when running.

[0022] According to a fifth aspect of the present invention, a computer program product is also provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned image distortion correction method when running.

[0023] In the above technical solution, for each current feature point in the first image, a preset feature point corresponding to the current feature point is determined, wherein the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera captures the first image, and the feature point on the calibration plate is mapped to the feature point in the first image. Then, based on the feature point predicted by using the current feature point in the first image and the distortion parameter of the camera with an undetermined value, and the preset feature point, the value of the distortion parameter of the camera is determined. Then, using the distortion parameter of the camera with a determined value, the first image is corrected to obtain the second image. In this way, the image captured by the camera can be accurately distorted.

[0024] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0026] Figure 1 A schematic flow chart of an image distortion correction method according to an embodiment of the present invention is shown;

[0027] Figure 2 A schematic flow chart of determining the value of a distortion parameter of a camera according to preset feature points and predicted feature points according to an embodiment of the present invention is shown;

[0028] Figure 3 A schematic flow chart showing at least one line search of a minimization function so that the minimization function obtains a minimum value according to an embodiment of the present invention;

[0029] Figure 4 A schematic flow chart showing at least one line search of a minimization function so that the minimization function obtains a minimum value according to yet another embodiment of the present invention;

[0030] Figure 5 A schematic flow chart of correcting a first image to obtain a second image using distortion parameters of a camera with determined values ​​according to an embodiment of the present invention is shown;

[0031] Figure 6 A schematic flow chart showing whether the optical axis of a camera is perpendicular to a calibration plate according to an embodiment of the present invention is shown;

[0032] Figure 7 A schematic diagram showing a method of using a laser autocollimator to determine whether the optical axis of a camera is perpendicular to a calibration plate according to an embodiment of the present invention is shown;

[0033] Figure 8 A schematic diagram showing a first image according to an embodiment of the present invention;

[0034] Fig. 9A schematic flow chart of correcting an image using distortion parameters that meet requirements according to an embodiment of the present invention is shown;

[0035] Fig.10 A schematic flow chart of an image distortion correction method according to another embodiment of the present invention is shown;

[0036] Fig.11 A schematic block diagram of an image distortion correction device according to an embodiment of the present invention is shown; and

[0037] Fig.12 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0039] In order to at least partially solve the above problems, an image distortion correction method is proposed. The method determines, for each current feature point in a first image, a preset feature point corresponding to the current feature point, wherein the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera captures the first image, and maps the feature point on the calibration plate to the feature point in the first image. Then, based on the feature point predicted by using the current feature point in the first image and the distortion parameter of the camera with an undetermined value, and the preset feature point, the value of the distortion parameter of the camera is determined. Then, the first image is corrected using the distortion parameter of the camera with a determined value to obtain a second image.

[0040] Figure 1 FIG. 2 shows a schematic flow chart of an image distortion correction method according to an embodiment of the present invention. Figure 1 As shown, the image distortion correction method may include steps S110 to S140.

[0041] In step S110, a first image of the calibration plate captured by a camera is acquired.

[0042] The camera may be a shallow depth of field camera, such as a microscope camera. The camera may be used to photograph a designated area of ​​the calibration plate to obtain a first image. The designated area of ​​the calibration plate may be the entire calibration plate or a partial area of ​​the calibration plate. When the camera photographs the first image of the calibration plate, the position between the camera and the calibration plate should be fixed, so that the steps of calibrating the camera's internal and external parameters can be reduced, thereby reducing the influence of the camera's internal and external parameter calibration on the distortion correction accuracy.

[0043] When the camera is used to capture the first image, the angle between the plane where the calibration plate is located and the camera optical axis can be as close to vertical as possible, so as to minimize the impact of the camera internal and external parameter calibration on the distortion correction accuracy. The image information in the first image should be clear and accurate.

[0044] Exemplarily, the first image is taken by the camera when the optical axis of the camera is perpendicular to the calibration plate.

[0045] When the optical axis of the camera is perpendicular to the calibration plate, the distortion parameters are determined with a relatively ideal camera posture, and there is no need to calibrate the camera's internal and external parameters. This can reduce the error caused by the calibration of the camera's internal and external parameters, and the distortion parameters are more accurate.

[0046] In step S120, for each current feature point in the first image, a preset feature point corresponding to the current feature point is determined, where the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera captures the first image, and the feature point on the calibration plate is mapped to the feature point in the first image.

[0047] The current feature point in the first image is a point that characterizes the image information of the first image, such as a corner point, a key point, a feature point extracted by a feature, etc. The feature point may also be an artificially specified point. After correcting the image distortion of the first image, the size of the first image does not change, so the pixel coordinate range of the first image and the second image is the same in the image coordinate system corresponding to the camera used to capture the first image. For example, the current feature point in the first image may be a corner point in the first image. For a calibration plate in the first image, the corner points in the first image are usually the corner points of the checkerboard of the calibration plate.

[0048] According to the positional relationship between the camera and the calibration plate when the camera takes the first image, an image coordinate system can be established with the plane where the first image is located, and an image coordinate system can be established with the plane where the calibration plate is located, so as to establish a camera imaging model. According to the designated area of ​​the calibration plate in the first image, the calibration plate area corresponding to the designated area is determined in the world coordinate system where the calibration plate is located. For each feature point in the calibration plate area, according to the mapping relationship between the calibration plate coordinate system and the image coordinate system of the first image when the camera takes the first image, each feature point in the calibration plate area can be determined to be mapped to the preset feature point in the first image, and the pixel coordinates of these preset feature points can be determined. Among them, the preset feature point is not a real feature point in the first image, but only used as a virtual feature point corresponding to the current feature point in the first image. It can be understood that the current feature point in the first image corresponds to the feature point of the calibration plate area corresponding to the designated area of ​​the calibration plate in the first image, and the feature point of the calibration plate area corresponding to the designated area of ​​the calibration plate in the first image corresponds to the preset feature point. Therefore, when the current feature point and the preset feature point both correspond to the same feature point of the calibration plate area corresponding to the designated area of ​​the calibration plate in the first image, the current feature point and the preset feature point also correspond. After determining the current feature point and the preset feature point, the user may specify the current feature point and the corresponding preset feature point.

[0049] When the camera that captures the first image does not have lens distortion, there is no image distortion in the first image. At this time, the pixel coordinates of the current feature point in the first image and the preset feature point corresponding to the current feature point are the same. However, because the first image has image distortion, the image distortion will affect the pixel coordinates of the current feature point. Therefore, the pixel coordinates of the current feature point and the preset feature point corresponding to the current feature point are different, but the image distortion may not have a significant impact on the pixel coordinates of the current feature point. For example, based on the distance between the pixel coordinates of the current feature point of the first image and the preset feature point corresponding to the current feature point, the preset feature point with the smallest distance to the current feature point can be determined as the preset feature point corresponding to the current feature point.

[0050] In step S130, the values ​​of the distortion parameters of the camera are determined according to the preset feature points and the predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the distortion parameters of the camera with undetermined values.

[0051] The difference between the current feature point and the predicted feature point characterizes the influence of the distortion parameter on the image information of the first image, that is, the degree of image distortion. The predicted feature point is the point corresponding to the current feature point after the image distortion is corrected. Because the value of the distortion parameter is undetermined, the position of the predicted feature point corresponding to the current feature point is also undetermined at this time. The position to which the current feature point needs to be corrected (for example, the position of the preset feature point corresponding to the current feature point in the first image) can be used as the target position where the predicted feature point should be located, and then the value of the unknown distortion parameter can be inferred based on the target position and the position of the current feature point.

[0052] Exemplarily, the distortion parameters include camera principal point coordinates, camera focal length, radial distortion coefficient, and tangential distortion coefficient.

[0053] The pixel coordinates of the current feature point can be converted into the corresponding normalized camera coordinates, and then the normalized camera coordinates of the predicted feature points corresponding to the current feature points can be determined based on the normalized camera coordinates of the current feature points and the distortion parameters of the camera. Then, the pixel coordinates corresponding to the predicted feature points can be determined based on the normalized coordinates corresponding to the predicted feature points.

[0054] For example, the normalized camera coordinates of the current feature point can be determined according to the following formula 1 and formula 2 ( ):

[0055]

[0056] Among them, u represents the horizontal pixel coordinate of the current feature point, and v represents the vertical pixel coordinate of the current feature point. represents the pixel coordinates of the camera principal point (the intersection of the optical axis with the image plane corresponding to the first image), are the principal point coordinates in the horizontal direction, are the vertical coordinates of the principal point, usually located at the center of the image. Indicates the focal length of the camera, in pixels. Indicates the horizontal focal length of the camera. Indicates the vertical focal length of the camera.

[0057] The camera that captures the first image may have radial distortion and tangential distortion, so the normalized camera coordinates corresponding to the predicted feature points corresponding to the current feature points may be determined according to the above formula 1, the radial distortion coefficient, and the tangential distortion coefficient.

[0058] For example, when determining the normalized camera coordinates of the current feature point After that, the normalized camera coordinates corresponding to the predicted feature points corresponding to the current feature points can be determined according to the following formulas 3, 4 and 5: :

[0059]

[0060] in, represents the radial distortion parameter, represents the tangential distortion parameter, Indicates the radial distance from the normalized camera coordinate point of the current feature point to the camera optical axis.

[0061] After determining the normalized camera coordinates corresponding to the predicted feature points ( ), it is also necessary to convert it into the corresponding pixel coordinates ( ).

[0062] For example, the pixel coordinates corresponding to the predicted feature points may be determined according to the following formulas 6 and 7:

[0063]

[0064] For each predicted feature point, the position of the preset feature point corresponding to the predicted feature point in the first image can be used as the target position where the predicted feature point should be located in the second image. The target position can be used as the position of the predicted feature point, and the value of the distortion parameter corresponding to the current feature point corresponding to the predicted feature point in the first image can be determined based on the position difference between the predicted feature point and the current feature point corresponding to the predicted feature point.

[0065] However, there are multiple current feature points in the first image. If each current feature point is corrected to the position of its corresponding preset feature point, the values ​​of the determined distortion parameters may be different. Therefore, in order to correct the image distortion in the first image, the optimal distortion parameter value can be determined based on all the multiple current feature points to make the predicted feature point corresponding to each current feature point as close to its corresponding preset feature point as possible, so as to ensure that the correction effect of the image distortion in the first image is the best.

[0066] In step S140 , the first image is corrected using the distortion parameters of the camera with determined values ​​to obtain a second image.

[0067] The distortion parameter of the determined value characterizes the influence of the image distortion on the pixels in the first image. The current feature point corresponding to the predicted feature point can be corrected according to the distortion parameter of the determined value and the above formula. In this way, the current feature point corresponding to the predicted feature point can be adjusted to the position of the predicted feature point, thereby reducing the influence of the image distortion on the current feature point corresponding to the predicted feature point.

[0068] Similarly, for each pixel in the first image, after determining the value of the distortion parameter, the pixel in the first image is substituted into the above formula as the current feature point in the above formula to determine the corrected pixel coordinates of the pixel in the first image, thereby eliminating the influence and obtaining the corrected first image as the second image.

[0069] The determined distortion parameter value can also be set in the shallow depth of field camera when it leaves the factory. As long as the lens and camera of the shallow depth of field camera are not disassembled or replaced, the camera automatically corrects the image distortion in the captured image when capturing the image, and outputs the corrected image.

[0070] In the above technical solution, for each current feature point in the first image, a preset feature point corresponding to the current feature point is determined, wherein the preset feature point is based on the positional relationship between the camera and the calibration plate when the camera captures the first image, and the feature point on the calibration plate is mapped to the feature point in the first image. Then, based on the feature point predicted by using the current feature point in the first image and the distortion parameter of the camera with an undetermined value, and the preset feature point, the value of the distortion parameter of the camera is determined. Then, using the distortion parameter of the camera with a determined value, the first image is corrected to obtain the second image. In this way, the image captured by the camera can be accurately distorted.

[0071] Figure 2 FIG. 1 is a schematic flow chart of determining the value of the distortion parameter of the camera according to the preset feature points and the predicted feature points according to an embodiment of the present invention. Figure 2 As shown, the above step S130 may include steps S210 to S230.

[0072] Step S210: determining a minimization function for distortion parameters according to the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point.

[0073] For example, there are N current feature points in the first image. For the i-th current feature point, the pixel coordinates of the i-th predicted feature point corresponding to the current feature point can be determined according to the above formula: ,in . When determining the pixel coordinates of the i-th preset feature point corresponding to the i-th predicted feature point After that, the minimization function of the distortion parameters can be determined according to the following formula 8:

[0074]

[0075] Step S220, performing at least one line search on the minimization function so that the minimization function obtains a minimum value.

[0076] The parameter vector can be determined according to the following formula 9 :

[0077]

[0078] Assume the search step is , the search direction is , after multiple searches, the minimum point is found , the predicted feature point obtained at this time is the optimal predicted feature point.

[0079] For example, the above parameter vector The search step length for each line search is determined by the following formula 10: :

[0080]

[0081] Search step length The selection adopts the line search method, first let , which contains the minimum point . Among them, for , if exists Monotonically decreasing, Monotonically increasing. Select the initial point in the interval [a,b] and initial step length .from Start Search ,if , then increase Continue searching. If in the first step, , then use to search for it.

[0082] For example, the search direction may be determined based on the steepest descent direction, Newton direction, quasi-Newton direction, conjugate gradient direction, etc. .

[0083] Step S230: When the minimization function obtains the minimum value, the current value of the distortion parameter is used as the value of the distortion parameter.

[0084] When the minimization function reaches the minimum value, the pixel coordinates of all predicted feature points are the optimal correction effect of the pixel coordinates of the current feature point of the first image, so the current value of the determined distortion parameter can make the correction effect of the first image the optimal result.

[0085] In the above technical solution, a minimization function for the distortion parameter is determined according to the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point, and then at least one line search is performed on the minimization function so that the minimization function obtains a minimum value. When the minimization function obtains a minimum value, the current value of the distortion parameter is used as the value of the distortion parameter. In this way, the value of the distortion parameter that makes the correction effect of the first image the optimal result can be obtained.

[0086] Figure 3 FIG. 1 is a schematic flow chart showing at least one line search of a minimization function so that the minimization function obtains a minimum value according to an embodiment of the present invention. Figure 3 As shown, the above step S220 may include steps S310 to S320.

[0087] In step S310, the search direction of each line search is determined using the quasi-Newton method.

[0088] The search direction can be determined according to the following formula 11 :

[0089]

[0090] in , indicating the current search direction , The first derivative (gradient) at the current point, is the inverse of the approximate Hessian matrix.

[0091] In step S320, at least one line search is performed on the minimization function according to the determined search direction, and the value of the distortion parameter is updated after each line search until the minimization function obtains a minimum value.

[0092] After determining the search step size and search direction of each line search, the pixel coordinates of the predicted feature points can be corrected once for each line search. After multiple searches, the minimum point is found. , the predicted feature point obtained at this time is the optimal predicted feature point, and the value of the distortion parameter at this time makes the minimization function obtain the minimum value.

[0093] For example, the initial search position may be selected as , which can be All are 1, represents the coordinates of the center point of the image, Set to 0, The unit matrix can be used as the initial value of the inverse of the approximate Hessian matrix. If the minimum point is found at this time , then according to the minimum point The corresponding parameter vector Determine the affine transformation matrix. Otherwise, calculate the descent direction Then follow the direction Perform a search and use the above formula 10 to determine the search step length ,make . At the same time, correct the parameters To obtain the corrected For example, the DFP (Davidon-Fletcher-Powell) algorithm and the BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm can be used to correct If the minimum point is found in this line search , then according to the minimum point The corresponding parameter vector Determine the value of the distortion parameter. If the minimum point is not found in this line search , k value plus 1, so as to use the modified As a parameter The above line search process is executed repeatedly to perform the next line search. Each line search can correct the pixel coordinates of the predicted feature points and determine the value of the distortion parameter determined after the line search until the minimum point is determined. , thereby determining the value of the distortion parameter that makes the minimization function reach the minimum value.

[0094] In the above technical solution, the quasi-Newton method is used to determine the search direction of each line search, and the minimization function is searched at least once according to the determined search direction, and the value of the distortion parameter is updated after each line search until the minimization function obtains the minimum value. In this way, the value of the distortion parameter that makes the minimization function obtain the minimum value can be determined more accurately.

[0095] Figure 4 A schematic flow chart of performing at least one line search on a minimization function so that the minimization function obtains a minimum value according to yet another embodiment of the present invention is shown.

[0096] like Figure 4As shown, the search range of the distortion parameter value set according to the camera manual and experience can be determined first. Then the quasi-Newton method and the above formula 11 are used to determine the search direction of the current sub-line search, and the above formula 10 and the line search method are used to determine the search step of the current sub-line search. After that, a distortion correction can be performed. In this distortion correction, the pixel coordinates of the predicted feature points can be corrected once according to the results after the current sub-line search, and the value of the distortion parameter at this time can be determined. Then it can be determined whether the minimum point is found. If the minimum point is found after the current line search, the distortion parameters at this time can be saved. If the minimum point is not found after the current line search, the next line search is resumed from the determination of the search step.

[0097] Figure 5 FIG. 4 is a schematic flow chart of correcting a first image to obtain a second image using distortion parameters of a camera with determined values ​​according to an embodiment of the present invention. Figure 5 As shown, the above step S140 may include steps S510 to S520.

[0098] In step S510, the corrected pixels in the first image are determined according to the distortion parameters of the determined values.

[0099] According to the influence of the distortion parameter with the determined value on the pixels in the first image, the pixels in the first image whose positions are adjusted to eliminate the influence can be determined as the corrected pixels in the first image.

[0100] For example, the pixel coordinates in the first image can be used as the current feature points in the above formula 1-7, and the corrected pixels in the first image can be substituted into the above formula 1-7 as predicted feature points. Then, the corrected pixels in the first image can be determined based on the distortion parameters with determined values ​​and the above formula 1-7.

[0101] In step S520, a second image is determined based on the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, a pixel value of a corresponding pixel in the second image is determined based on pixel values ​​of pixels adjacent to the corrected pixel.

[0102] Pixel coordinates should all be integers, but the corrected pixel coordinates may not be integers. For example, the corrected pixel coordinates may be (23.7, 45.3). The pixel values ​​of these non-integer positions can be calculated based on the pixel values ​​of the pixels adjacent to the corrected pixel and the interpolation algorithm. Among them, the interpolation algorithm can be nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. When the real-time requirement is high, the interpolation algorithm can choose nearest neighbor interpolation or bilinear interpolation. When the image quality requirement is high, the interpolation algorithm can choose bicubic interpolation or Lanczos interpolation.

[0103] In the above technical solution, the corrected pixels in the first image are determined according to the distortion parameters of the determined values, the second image is determined according to the corrected pixels, and for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined according to the pixel values ​​of the pixels adjacent to the corrected pixel. In this way, the situation that the pixel coordinates of the corrected pixels are not integers can be taken into account, thereby obtaining an accurate second image.

[0104] Exemplarily, a reflector is disposed on the flange interface of a camera for capturing the first image. Figure 6 FIG. 2 shows a schematic flow chart of determining whether the optical axis of a camera is perpendicular to a calibration plate according to an embodiment of the present invention. Figure 6 As shown, the above-mentioned image distortion correction method may further include step S610 to step S620.

[0105] In step S610, a laser autocollimator is used to illuminate a reflector, and a reflected light signal reflected from the reflector is received, wherein the laser autocollimator is configured so that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located.

[0106] The basic structure of a laser autocollimator may include a laser light source and a reflector, an imaging system, and a detector. The laser autocollimator's high-precision detector (such as a position sensitive detector or a charge coupled device) can be used to accurately measure the offset between the emitted laser and the received reflected signal to determine the angle change. The laser autocollimator can be used to emit a laser to illuminate the reflector. If the angle of the reflector changes, the returning laser beam will also be offset. By detecting the offset of the returning beam, the angle change of the reflector can be calculated.

[0107] In step S620, it is determined whether the optical axis of the camera is perpendicular to the calibration plate according to the reflected light signal received by the laser autocollimator.

[0108] Figure 7 FIG. 2 is a schematic diagram showing a method of using a laser autocollimator to determine whether the optical axis of a camera is perpendicular to a calibration plate according to an embodiment of the present invention. Figure 7 As shown, since the reflector is installed on the camera flange, it can be used Figure 7 The Y axis of the camera coordinate system represents the plane where the reflector is located. The Z axis (optical axis) of the camera coordinate system points to the plane where the calibration plate is located. The laser emitted by the laser autocollimator perpendicular to the plane where the calibration plate is located can be used as the incident laser of the reflector. Then, the laser autocollimator can be used to receive the reflected laser returned by the reflector from the reflector. Then, according to the angles of the incident laser and the reflected laser, it can be determined whether the optical axis of the camera is perpendicular to the plane where the calibration plate is located, thereby determining whether the optical axis of the camera is perpendicular to the plane where the calibration plate is located.

[0109] For example, whether the optical axis of the camera is perpendicular to the calibration plate can be determined according to the following formula 12:

[0110]

[0111] in, It represents the offset between the reflected laser and the incident laser determined by the laser autocollimator, and L represents the distance between the laser autocollimator and the reflector. When , it means that the optical axis of the camera is perpendicular to the plane where the calibration plate is located, that is, the optical axis of the camera is perpendicular to the calibration plate.

[0112] In the above technical solution, a laser autocollimator is used to illuminate the reflector at the camera flange interface, and a reflected light signal reflected from the reflector is received, wherein the laser autocollimator is configured so that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located. Then, based on the reflected light signal received by the laser autocollimator, it is determined whether the optical axis of the camera is perpendicular to the calibration plate. In this way, it is possible to more accurately determine whether the optical axis of the camera is perpendicular to the calibration plate, so as to determine the camera posture when the first image is captured when the optical axis of the camera is perpendicular to the calibration plate.

[0113] Exemplarily, the calibration plate is arranged such that the arrangement direction of the checkerboard of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located.

[0114] Figure 8 FIG. 1 is a schematic diagram of a first image according to an embodiment of the present invention. The calibration plate is arranged so that the arrangement direction of the checkerboard of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located. Figure 8 As shown, for the plane where the first image is located, the image coordinate system where the plane is located can be determined, wherein the direction of the x-axis of the image coordinate system can be used as the horizontal direction of the plane where the first image is located. For example, Figure 8 As shown, when the first image is a rectangular image, the upper and lower edges of the first image are parallel to the horizontal direction of the plane where the first image is located. The area of ​​the calibration plate in the first image contains a plurality of checkerboards, and the arrangement direction of the checkerboards is parallel to the horizontal direction of the plane where the first image is located. At this time, the arrangement direction of the checkerboards of the first image can also be parallel to the direction of the upper and lower edges of the first image.

[0115] For example, the calibration plate may be fixed on an optical rotating stage, and the angle of the rotating stage may be adjusted to ensure that the arrangement direction of the checkerboard of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located.

[0116] The checkerboard in the first image may be bent due to image distortion, but the image distortion has little effect on the image information near the center of the first image. Therefore, it is possible to determine whether the arrangement direction of the checkerboard of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located based on the arrangement direction of the middle row of checkerboards in the first image.

[0117] When the arrangement direction of the chessboard is parallel to the horizontal direction of the plane where the first image is located, the coordinates of the current feature point in the first image can be easily determined, and calculation errors that may be caused by the tilted arrangement direction of the chessboard in the first image can be avoided.

[0118] Fig. 9 FIG. 4 shows an exemplary flow chart of correcting an image using distortion parameters that meet the requirements according to an embodiment of the present invention. Fig. 9 As shown, the above-mentioned image distortion correction method may further include step S910 to step S920.

[0119] In step S910, the standard deviation between the side length of each checkerboard of the calibration plate in the second image and the average value of the side length of each checkerboard is determined.

[0120] The standard deviation characterizes the correction effect of the second image. It can be understood that the side length of each chessboard of the calibration plate is the same. Therefore, in the ideal image, the side length of each chessboard of the calibration plate is also the same, and is the same as the average value of the side length. Although the pixel coordinates of the feature points in the second image have been corrected, there may still be partial image distortion in the second image, and there is still a difference between the second image and the ideal image. Therefore, the side length of each chessboard of the calibration plate in the second image is different from the average value of the side length of each chessboard. The better the correction effect of the second image, the smaller the value of the standard deviation.

[0121] In step S920, when the standard deviation is less than the preset threshold, other images captured by the camera at the same posture are corrected using the distortion parameter with the determined value.

[0122] When the standard deviation is less than the preset threshold, it means that the correction effect of the second image meets the requirements. When other images are taken with the same posture using the camera that took the first image, the distortion parameter value used to correct the first image to the second image can be directly used to correct the image, thereby reducing the image distortion in the image.

[0123] In the above technical solution, the standard deviation between the side length of each chessboard of the calibration plate in the second image and the average value of each chessboard side length is determined, and then when the standard deviation is less than a preset threshold, the distortion parameter with the determined value is used to correct other images taken with the camera in the same posture. In this way, it can be avoided that the correction effect of the second image is too poor, so that when the determined distortion parameter value is used to correct other images, a better correction effect can be achieved.

[0124] Fig.10 FIG. 4 is a schematic flow chart of an image distortion correction method according to another embodiment of the present invention. Fig.10 As shown, the verticality of the camera can be corrected by a laser autocollimator so that the optical axis of the camera is perpendicular to the calibration plate. Then the optical rotating stage flattens the calibration plate to ensure that the checkerboard in the ideal image is horizontally aligned with the direction of the edge of the ideal image. Then the first image can be captured by the camera, and the value of the distortion parameter can be determined according to the feature points in the first image. Then the corrected pixel in the first image can be determined according to the distortion parameter whose value has been determined. And the second image is determined according to the corrected pixel, wherein for each corrected pixel whose position coordinate is not an integer, the image interpolation calculation is performed according to the pixel value of the pixel adjacent to the corrected pixel to determine the pixel value of the corresponding pixel in the second image. Then the accuracy evaluation can be performed according to the side length of each checkerboard of the calibration plate in the second image and the average value of each checkerboard side length to determine the correction effect of the second image. When the result of the accuracy evaluation is unqualified, the image difference and accuracy evaluation are re-performed starting from the determination of the value of the distortion parameter. When the result of the accuracy evaluation is qualified, the second image can be output as a corrected image.

[0125] Fig.11 FIG. 4 is a schematic block diagram of an image distortion correction device according to an embodiment of the present invention. Fig.11 As shown, the image distortion correction device includes an image receiving module 1110 , a feature point matching module 1120 , a parameter determination module 1130 , and an image correction module 1140 .

[0126] The image receiving module 1110 is used to obtain a first image of the calibration plate taken by a camera.

[0127] The feature point matching module 1120 is used to determine, for each preset feature point, the current feature point in the first image corresponding to the preset feature point. The preset feature point is based on the positional relationship between the camera and the calibration plate when the camera takes the first image, and maps the feature point on the calibration plate to the feature point in the image taken by the camera.

[0128] The parameter determination module 1130 is used to determine the values ​​of the distortion parameters of the camera according to the preset feature points and the predicted feature points, wherein the predicted feature points are represented by the current feature points in the first image and the distortion parameters of the camera with undetermined values.

[0129] The image correction module 1140 is used to correct the first image using the distortion parameters of the camera with determined values ​​to obtain the second image.

[0130] Exemplarily, the parameter determination module 1130 includes a function calculation submodule, a line search submodule and a parameter value determination submodule. The function calculation submodule is used to determine the minimization function of the distortion parameter according to the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point. The line search submodule is used to perform at least one line search on the minimization function so that the minimization function obtains a minimum value. The parameter value determination submodule is used to use the current value of the distortion parameter as the value of the distortion parameter when the minimization function obtains a minimum value.

[0131] Exemplarily, the line search submodule includes a search direction determination submodule and a minimum value determination submodule. The search direction determination submodule is used to determine the search direction of each line search using the quasi-Newton method. The minimum value determination submodule is used to perform at least one line search on the minimization function according to the determined search direction, and update the value of the distortion parameter after each line search until the minimization function obtains a minimum value.

[0132] Exemplarily, the image correction module 1140 includes a pixel determination submodule and a second image determination submodule. The pixel determination submodule is used to determine the corrected pixels in the first image according to the distortion parameters of the determined values. The second image determination submodule is used to determine the second image according to the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined according to the pixel values ​​of the pixels adjacent to the corrected pixel.

[0133] Exemplarily, the first image is taken by the camera when the optical axis of the camera is perpendicular to the calibration plate.

[0134] Exemplarily, the flange interface of the camera is provided with a reflector, and the image distortion correction device further comprises a laser autocollimator control submodule and a verticality determination module. The laser autocollimator control submodule is used to illuminate the reflector with a laser autocollimator and receive a reflected light signal reflected from the reflector, wherein the laser autocollimator is configured so that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located. The verticality determination module is used to determine whether the optical axis of the camera is perpendicular to the calibration plate according to the reflected light signal received by the laser autocollimator.

[0135] Exemplarily, the calibration plate is arranged such that the arrangement direction of the checkerboard of the calibration plate in the first image is parallel to the horizontal direction of the plane where the first image is located.

[0136] Exemplarily, the image distortion correction device further includes a standard deviation determination submodule and a distortion correction submodule. The standard deviation determination submodule is used to determine the standard deviation between the length of each chessboard side of the calibration plate in the second image and the average value of each chessboard side. The distortion correction submodule is used to correct other images taken by the camera in the same posture using the distortion parameter with the determined value when the standard deviation is less than a preset threshold.

[0137] Exemplarily, the unknown distortion parameters include the coordinates of the camera principal point, the camera focal length, the radial distortion coefficient, and the tangential distortion coefficient.

[0138] According to another aspect of the present invention, an electronic device is provided. Fig.12 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Fig.12 As shown, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used by the processor to execute the image distortion correction method as described above when the processor is running.

[0139] In addition, according to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the above-mentioned image distortion correction method according to the embodiment of the present invention, and is used to implement the corresponding modules in the above-mentioned image distortion correction device according to the embodiment of the present invention. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0140] According to another aspect of the present invention, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned image distortion correction method when running.

[0141] A person skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned image distortion correction device, electronic device, storage medium and computer program product by reading the above-mentioned specific description of the image distortion correction method. For the sake of brevity, they will not be described in detail here.

[0142] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0143] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0145] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0146] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0147] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0148] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0149] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the image distortion correction device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0150] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0151] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for image distortion correction, characterized in that: The method comprises: Acquire a first image of the calibration plate captured by the camera; For each current feature point in the first image, determine a preset feature point corresponding to the current feature point, wherein the preset feature point is based on a mapping relationship between an image coordinate system of a plane where the first image is located when the camera captures the first image and an image coordinate system of a plane where the calibration plate is located, and maps the feature point on the calibration plate to the feature point in the first image; Determining the value of the distortion parameter of the camera according to the preset feature point and the predicted feature point, wherein the predicted feature point is represented by the current feature point in the first image and the distortion parameter of the camera with an undetermined value, the predicted feature point is a point corresponding to the current feature point after the image distortion is corrected, the value of the determined distortion parameter makes the minimization function of the distortion parameter obtain a minimum value, and the minimization function of the distortion parameter is a function determined according to the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point; The first image is corrected using the distortion parameter of the camera with a determined value to obtain a second image.

2. The method according to claim 1, characterized in that The step of determining the value of the distortion parameter of the camera according to the preset feature points and the predicted feature points includes: Determining a minimization function for the distortion parameter according to a variance of coordinates between the predicted feature point and a preset feature point corresponding to the predicted feature point; Performing at least one line search on the minimization function so that the minimization function obtains a minimum value; When the minimization function obtains the minimum value, the current value of the distortion parameter is used as the value of the distortion parameter.

3. The method according to claim 2, characterized in that The performing at least one line search on the minimization function so that the minimization function obtains a minimum value comprises: Determining the search direction of each line search using a quasi-Newton method; The minimization function is subjected to at least one line search according to the determined search direction, and the value of the distortion parameter is updated after each line search until the minimization function obtains the minimum value.

4. The method according to any one of claims 1 to 3, characterized in that: The step of correcting the first image by using the distortion parameter of the camera with a determined value to obtain the second image comprises: Determining corrected pixels in the first image according to the distortion parameters of the determined values; The second image is determined based on the corrected pixels, wherein for each corrected pixel whose position coordinates are not integers, the pixel value of the corresponding pixel in the second image is determined based on the pixel values ​​of pixels adjacent to the corrected pixel.

5. The method according to any one of claims 1 to 3, characterized in that: The first image is taken by the camera when the optical axis of the camera is perpendicular to the calibration plate.

6. The method according to claim 5, characterized in that The flange interface of the camera is provided with a reflector, The method further comprises: irradiating the reflector with a laser autocollimator and receiving a reflected light signal reflected from the reflector, wherein the laser autocollimator is configured so that the laser emitted by the laser autocollimator is perpendicular to the plane where the calibration plate is located; According to the reflected light signal received by the laser autocollimator, it is determined whether the optical axis of the camera is perpendicular to the calibration plate.

7. The method according to any one of claims 1 to 3, characterized in that: The calibration plate is arranged so that an arrangement direction of checkerboards of the calibration plate in the first image is parallel to a horizontal direction of a plane where the first image is located.

8. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Determine a standard deviation between a side length of each checkerboard of the calibration plate in the second image and an average value of the side length of each checkerboard; When the standard deviation is less than a preset threshold, other images captured by the camera at the same posture are corrected using the distortion parameter with the determined value.

9. The method according to any one of claims 1 to 3, characterized in that: The unknown distortion parameters include the camera principal point coordinates, camera focal length, radial distortion coefficient and tangential distortion coefficient.

10. An image distortion correction device, characterized in that: include: An image receiving module, used to obtain a first image of the calibration plate taken by a camera; a feature point matching module, configured to determine, for each preset feature point, a current feature point in the first image corresponding to the preset feature point, wherein the preset feature point is based on a mapping relationship between an image coordinate system of a plane where the first image is located when the camera captures the first image and an image coordinate system of a plane where the calibration plate is located, and maps the feature point on the calibration plate to a feature point in the image captured by the camera; a parameter determination module, configured to determine the value of the distortion parameter of the camera according to the preset feature point and the predicted feature point, wherein the predicted feature point is represented by the current feature point in the first image and the distortion parameter of the camera with an undetermined value, the predicted feature point is a point corresponding to the current feature point after the image distortion is corrected, the value of the determined distortion parameter makes the minimization function of the distortion parameter obtain a minimum value, and the minimization function of the distortion parameter is a function determined according to the variance of the coordinates between the predicted feature point and the preset feature point corresponding to the predicted feature point; The image correction module is used to correct the first image by using the distortion parameter of the camera with a determined value to obtain a second image.

11. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the image distortion correction method according to any one of claims 1 to 9 when the processor runs the computer program instructions.

12. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the image distortion correction method according to any one of claims 1 to 9 when running.

13. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the image distortion correction method according to any one of claims 1 to 9 when running.

Images