Image conversion device and method

Abstract

To provide an image conversion device and method.SOLUTION: The image conversion device includes: a first image processing apparatus which subjects an original image to first image processing on the basis of a value of a first parameter of the original image to acquire an intermediate image; a repeating device which repeatedly subjects the intermediate image to the first image processing on the basis of a value of the first parameter of the intermediate image to acquire a final intermediate image; an estimation device which estimates an estimate of the first parameter on the basis of a conversion relation between a distorted image and its corresponding transparent image, the original image, and the final intermediate image; and a second image processing apparatus which subjects the original image to second image processing on the basis of the estimate of the first parameter to acquire a converted image.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to the technical field of image processing.

Background Art

[0002] Computer vision technology has already been widely applied in all aspects of production and life. Camera calibration is an important part of computer vision technology. Camera calibration methods mainly include an internal camera parameter calibration method and an external camera parameter calibration method. Among them, the internal camera parameter calibration technology also includes a non-linear distortion coefficient caused by the camera lens.

[0003] In the conventional camera calibration method, it is necessary to use a calibration object with a known size. By establishing the correspondence between the points with known coordinates on the calibration object and their image points, the internal and external parameters of the camera model are obtained using a predetermined algorithm. Since a calibration object is always required in the calibration process, the manufacturing accuracy of the calibration object may affect the calibration result. Also, when the placement of the calibration object is not appropriate, the method may be limited.

[0004] Therefore, a camera calibration method that does not require a calibration object, for example, a method for estimating the internal parameters of a camera using an RGB image based on a deep learning method for camera calibration, has been proposed. Thereby, the application range of camera calibration can be expanded.

[0005] Note that the introduction of the above background art is for clearly and completely explaining the technical solution of the present invention and for facilitating the understanding of those skilled in the art. These technical solutions should not be construed as being well-known to those skilled in the art just because they are described in the background art of the present invention.

Summary of the Invention

Problems to be Solved by the Invention

[0006] The inventor has discovered the following. That is, in the prior art, when the distortion of an image is relatively small or there is no distortion, the accuracy of the camera parameters obtained by using the deep learning method is not high. When there is relatively large distortion in the image, when the image distortion removal process is executed only once using the parameters estimated by the deep learning method, there are still many distortion components in the image after distortion removal, and when the distortion removal process is executed multiple times using the parameters, the image quality of the image after distortion removal deteriorates.

[0007] In view of at least one of the above problems, embodiments of the present invention provide an image conversion device and method.

Means for Solving the Problems

[0008] According to a first aspect of an embodiment of the present invention, an image conversion device is provided, and the device includes: A first image processing device that performs first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; An iteration device that repeatedly executes first image processing on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; An estimation device that estimates an estimated value of the first parameter based on the conversion relationship between the distorted image and the corresponding perspective image, the original image, and the final intermediate image; and A second image processing apparatus is included that performs second image processing on the original image based on the estimated value of the first parameter to obtain a converted image.

[0009] According to a second aspect of an embodiment of the present invention, an image conversion method is provided, and the method includes: performing first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; repeatedly performing first image processing on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; estimating an estimated value of the first parameter based on a conversion relationship between a distorted image and a corresponding perspective image, the original image, and the final intermediate image; and performing second image processing on the original image based on the estimated value of the first parameter to obtain a converted image.

Advantages of the Invention

[0010] The advantageous effects of the embodiments of the present invention are at least as follows. That is, by performing image processing on the original image a plurality of times to obtain a plurality of intermediate images, and estimating the estimated value of the first parameter using the original image and the plurality of intermediate images as training data, a high-precision first parameter can be obtained. Therefore, when removing the distortion of the original image, the image distortion removal accuracy can be improved. Also, by performing image processing on the original image based on the high-precision parameter, an image with good distortion removal effect and high image quality can be obtained. Furthermore, when adding distortion to the original image, a distortion image (distorted image) close to the actual situation can be obtained, so that it can be applied to different scenes.

[0011] Note that terms such as "including / having", when used in this specification, refer to the presence of features, elements, steps, or assemblies, but also refer to not precluding the presence or addition of one or more other features, elements, steps, or assemblies.

Brief Description of the Drawings

[0012] The elements and features described in one drawing or one embodiment of the present invention can be combined with the elements and features shown in one or more other drawings or embodiments. Also, in the drawings, like reference numerals indicate corresponding parts in several drawings and are also used to indicate corresponding parts used in multiple embodiments.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0013] By referring to the accompanying drawings and the following description, the foregoing and other features of the present invention will become apparent. Although specific embodiments of the present invention are disclosed in the specification and drawings, they are only some examples that can adopt the principles of the present invention. It should be understood that the present invention is not limited to the described embodiments, that is, the present invention also includes all changes, modifications, and substitutions belonging to the scope of the appended claims.

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0015] <Example of the first side> In an embodiment of the present invention, an image conversion method is provided. FIG. 1 is a diagram showing the image conversion method in an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps (operations).

[0016] 101: Perform first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; 102: Repeatedly perform first image processing on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; 103: Estimate an estimated value of the first parameter based on the conversion relationship between the distorted image and the corresponding perspective image, the original image, and the final intermediate image; and 104: Perform second image processing on the original image based on the estimated value of the first parameter to obtain a converted image.

[0017] Thereby, by performing image processing on the original image a plurality of times to obtain a plurality of intermediate images, and estimating the estimated value of the first parameter using the original image and the plurality of intermediate images as training data, a high-precision first parameter can be obtained.

[0018] In some embodiments, the image conversion method in the embodiments of the present invention can be applied to a scene where distortion removal processing is performed on a distorted image. For example, the original image is a distorted image (hereinafter abbreviated as "distortion image"), and the first parameter of the original image includes the focal length f and the distortion coefficient ξ of the camera corresponding to the original image. Hereinafter, for the convenience of explanation, the first parameter of the distortion image may be referred to as the "distortion parameter".

[0019] For example, in step 101, an intermediate image is obtained by performing a first image process on the distorted image based on the value of the distortion parameter of the distorted image; in step 102, a final intermediate image is obtained by repeatedly performing the first image process on the intermediate image based on the value of the distortion parameter of the intermediate image; in step 103, an estimated value of the distortion parameter is estimated based on the conversion relationship between the distorted image and the final intermediate image, and the value of the distortion parameter of the distorted image or the value of the distortion parameter of the final intermediate image; and in step 104, an undistorted image is obtained by performing a second image process on the distorted image based on the estimated value of the distortion parameter.

[0020] Thereby, by performing multiple distortion removal processes on the distorted image to obtain a plurality of intermediate images, and estimating the estimated value of the distortion parameter using the distorted image and the plurality of intermediate images as training data, a high-precision distortion parameter can be obtained, so that the image distortion removal accuracy can be improved.

[0021] FIG. 2 is a diagram for explaining the distortion of an image due to the distortion of a camera during spherical projection. In FIG. 2, for the sake of convenience of explanation, the focal length f when no distortion occurs is set to 1.

[0022] Hereinafter, the distortion of the image will be described taking FIG. 2 as an example.

[0023] As shown in FIG. 2(a), when there is no distortion, the distance between the projection of the point (X, Y, Z) in the world coordinate system on the imaging plane and the center point of the imaging plane is r. However, due to the existence of distortion, the distance between the projection of the point in the world coordinate system on the imaging plane and the center point of the imaging plane becomes rd, and the focus of the image changes from point Op to Od. Here, the distance between Op and Od is denoted as "distortion coefficient ξ", and the relationship between these points is as shown in FIG. 2(b). The distortion coefficient ξ and the field of view angle θ satisfy the following formula (1).

[0024]

Equation

[0025]

Equation

[0026] In the embodiments of the present invention, in order to facilitate the calculation, normalization processing may be performed on the "distortion coefficient ξ", and the numerical range of the "distortion coefficient ξ" after the normalization processing is 0 to 1. When the distortion coefficient ξ is 0, it indicates that there is no distortion in the image. When the distortion coefficient is 1, it indicates that there is distortion in the image. The larger the distortion coefficient ξ, the greater the degree of distortion, and vice versa. In the following description of the embodiments, the pre-defined value of the distortion coefficient may be any value between 0 and 1. When the image is an undistorted image or a perspective image, its distortion coefficient can be pre-defined as 0. When the image is a distorted image, its distortion coefficient can be pre-defined as 1, and vice versa.

[0027] FIG. 3 is a diagram showing one implementation method of the first image processing in the embodiments of the present invention. In the example shown in FIG. 3, distortion exists in the original image F, and hereinafter, this image is abbreviated as the "distortion image F". FIG. 3 shows the process of performing the first image processing on the distortion image F.

[0028] In this example, the first image processing is a process of obtaining an intermediate image by performing distortion removal processing on a distorted image. As shown in FIG. 3, the process of performing the first image processing on the distortion image F may include the following steps.

[0029] 301: Estimate the magnification ratio based on the size of the distorted image and perform an enlargement process on the canvas; 302: Perform bilinear interpolation processing on the enlarged canvas based on the values of the distortion parameters of the distorted image to obtain a first interpolated image; 303: Perform trimming processing on the first interpolated image to obtain a first trimmed image with an aspect ratio equal to the aspect ratio of the distorted image; and 304: Perform scaling (enlarging / reducing) processing on the first trimmed image to obtain an intermediate image with the same size as the size of the distorted image.

[0030] As shown in FIG. 3, image F represents the original distorted image, images F1 to FN-1 represent the intermediate images obtained in the distortion removal process, image FN represents the final intermediate image, and N is a natural number greater than 1. The process of performing the first image processing on the distorted image F is as follows, that is, estimate the magnification ratio based on the size of the distorted image F (the length is W and the width is H) to obtain an enlarged canvas P, and perform bilinear interpolation processing on the enlarged canvas P based on the distortion parameters f and ξ of the distorted image F to obtain a first interpolated image P1, trim the first interpolated image P1 to obtain a first trimmed image P11 with an aspect ratio equal to the aspect ratio of the distorted image F, and scale the first trimmed image to obtain an intermediate image F1 with the same size as the size of the distorted image F.

[0031] In this example, the magnification ratio of the canvas P can be predefined. For example, it is 1.5 times, or calculate the position of at least one vertex of the four vertices of the distorted image F within the canvas, and determine the magnification ratio of the canvas based on this position. It should be noted that for the implementation method of enlarging the canvas, relevant technologies can be referred to, and the embodiments of the present invention do not limit it.

[0032] In this example, in the first interpolated image P1 obtained by bilinear interpolation processing, there is a maximum rectangular area A1. The midpoints of the four sides of A1 correspond to the midpoints of the four sides of the distortion image F. Cut out the maximum range within A1 according to the aspect ratio of the distortion image F. For example, cut out the first trimming image P11 with a length of W1 and a width of H1, where H1 / W1 = H / W. Note that for the implementation methods of bilinear interpolation processing and equal-scale scaling (expansion and contraction) processing, relevant technologies can be referred to, and the embodiments of the present invention do not limit them in this regard.

[0033] In the embodiments of the present invention, the process of performing the first image processing on the intermediate image is the same as the process of performing the first image processing on the distortion image, that is, estimating the magnification based on the size of the intermediate image F1 and expanding the canvas; performing bilinear interpolation processing on the expanded canvas based on the distortion parameters f1 and ξ1 of the intermediate image F1 to obtain an interpolated image; trimming the interpolated image to obtain a trimming image with an aspect ratio equal to the aspect ratio of the intermediate image F1; and scaling the trimming image to obtain an intermediate image F2 with a size equal to the size of the intermediate image F1. Also, until the final intermediate image FN is obtained, the above process is repeatedly executed for the intermediate images F2 to FN-1.

[0034] In this example, the description is made by taking the example of obtaining at least three intermediate images. However, the embodiments of the present invention are not limited thereto, and the number of intermediate images and the selection of the final intermediate image are related to the end conditions of the iterative execution.

[0035] In some embodiments, the end conditions of the iterative execution include that the change trend of the value of the first parameter starts to reverse or a predetermined number of iterations is reached.

[0036] When repeatedly performing distortion removal processing on the distortion image, the first parameter (i.e., the distortion parameter) is the focal length focal and the distortion coefficient ξ of the camera corresponding to the distortion image. The end conditions of the iterative execution include that the value of the distortion parameter starts to increase or a predetermined number of iterations is reached. The following will be described respectively.

[0037] In some embodiments, when the value of the distortion parameter begins to increase in the iterative process, the iteration is terminated.

[0038] For an image, the relationship between the viewing angle θ, size H, and focal length focal of the image can be expressed as follows.

[0039] Tan(θ)=(0.5H) / focal (4) After performing the distortion removal operation on the image, since the size H of the image remains unchanged, the viewing angle θ of the image after distortion removal becomes smaller. Therefore, when repeating the distortion removal operation, it is necessary to reduce the focal length focal, thereby ensuring that the size of the viewing angle θ corresponding to the size H remains unchanged and balancing the reduction factor of the initial viewing angle θ.

[0040] When repeatedly performing the distortion removal operation on the image, if the predicted value of the focal length f or the distortion coefficient ξ becomes larger than the previous value, the image can be considered to be over-calibrated, and at this time, the iterative process of further calibration should be stopped.

[0041] FIG. 4 and FIG. 5 are diagrams for explaining the end conditions of the iterative execution, respectively. FIG. 4 shows the change of the focal length when performing 10 iterations on one distorted image A, and FIG. 5 shows the change of the focal length when performing 10 iterations on another distorted image B.

[0042] As shown in FIGS. 4 and 5, the initial focal length of the distorted image is f0, the focal length of the image after the first execution of the first image processing becomes f1, the focal length of the image after the second execution of the first image processing becomes f2, the focal length of the image after the third execution of the first image processing becomes f3, the focal length of the image after the fourth execution of the first image processing becomes f4, the focal length of the image after the fifth execution of the first image processing becomes f5, the focal length of the image after the sixth execution of the first image processing becomes f6, the focal length of the image after the seventh execution of the first image processing becomes f7, the focal length of the image after the eighth execution of the first image processing becomes f8, the focal length of the image after the ninth execution of the first image processing becomes f9, and then, the tenth execution of the first image processing can be performed based on the focal length f9.

[0043] As shown in FIG. 4, from the focal length f0 to the focal length f4, the value decreases sequentially, and from the focal length f4 to the focal length f5, the value increases. Therefore, the iteration ends after the fifth execution of the first image processing (for example, step 401 shown in FIG. 4), and the intermediate image after the fourth execution of the first image processing is output as the final intermediate image (for example, step 402 shown in FIG. 4).

[0044] In other words, in some embodiments, when ending the execution of the iteration when the value of the distortion parameter starts to increase, the intermediate image obtained after the previous execution of the first image processing is used as the final intermediate image. Also, when the value of the distortion parameter of the intermediate image obtained after executing the first image processing once on the distorted image is greater than the value of the distortion parameter of the distorted image, the distorted image is used as the final intermediate image.

[0045] For example, as shown in FIG. 5, since the value increases from the focal length f0 to the focal length f1, the iteration ends after the first execution of the image processing (for example, step 501 shown in FIG. 5), and the distorted image B is output as the final intermediate image (for example, step 502 shown in FIG. 5).

[0046] Although the above description has been given by taking the focal length as an example, the same applies to the distortion coefficient ξ, and thus the detailed description thereof is omitted here.

[0047] In some embodiments, the iteration ends after reaching a predetermined number of iterations. In the embodiments of the present invention, when the iteration ends after reaching the predetermined number of iterations, the intermediate image obtained after the execution of the last first image processing is used as the final intermediate image. Note that the predetermined number of iterations may be set according to needs, but in the embodiments of the present invention, the predetermined number of iterations is not limited.

[0048] In the embodiments of the present invention, as shown in FIG. 1, the method may further include the following steps.

[0049] 105: Obtain the value of the first parameter of the original image and the value of the first parameter of the intermediate image.

[0050] When the original image is a distorted image, the original image and the intermediate image can be regarded as having distortion, and since the first parameter thereof is the focal length and the distortion coefficient of the camera corresponding to the image, the first parameter of the distorted image is referred to as the "distortion parameter". In step 105, the value of the distortion parameter of the original image and the value of the distortion parameter of the intermediate image are obtained. For example, the distortion parameter of the image may be predicted based on a deep learning model. For example, the focal length and the distortion coefficient of the image can be predicted using the DeepCalib model. Note that for the specific implementation of the DeepCalib model, reference can be made to related technologies, such as the papers 《DeepCalib: A Deep Learning Approach for Automatic Intrinsic Calibration of Wide Field-of-View Cameras》 by Bogdan O, Eckstein V, Rameau F, etc. Also, for the specific implementation method, reference can be made to related technologies, and the embodiments of the present invention do not limit this.

[0051] The "value of the distortion parameter of the original image" obtained in step 105 can be used in step 101. For example, the method can be used to obtain the distortion parameters f and ξ for the distorted image F in FIG. 3. The "value of the distortion parameter of the intermediate image" obtained in step 105 can be used in step 102. For example, the method can be used to obtain the distortion parameters fn and ξn for the intermediate images F1 to FN-1 in FIG. 3. Among them, n is a natural number greater than or equal to 1 and less than or equal to N-1, and N is a natural number greater than 1.

[0052] In an embodiment of the present invention, an estimated value of the distortion parameter is obtained based on a predicted value of the distortion parameter, and distortion removal processing is performed on the image based on the estimated value. Compared with the image after distortion removal obtained by repeatedly performing distortion removal processing on the distorted image, the clarity of the image after distortion removal obtained by performing distortion removal processing on the distorted image once with the estimated value is higher, and the distortion removal effect is better.

[0053] In an embodiment of the present invention, a forward conversion function may be adopted to estimate an estimated value of the first parameter, or an inverse conversion function may be adopted to estimate an estimated value of the first parameter. Forward Transform (FT) refers to the point coordinate conversion from a perspective image to a distorted image, which can be used in a scene where the focal length and distortion coefficient (also called "distortion parameter") of the original image are estimated when the original image has distortion. Backward Transform (BT) refers to the point coordinate conversion from a distorted image to a perspective image, which can be used in a scene where the focal length of the original image is estimated when the original image has no distortion. The following will explain them respectively.

[0054] In some embodiments, when there is distortion in the original image, the forward conversion function FT-Mf can be adopted to estimate the estimated value of the distortion parameter of the original image, and the difference function for estimation is the residual function of the forward conversion function.

[0055] The forward transformation function FT-Mf can be expressed as follows.

[0056] xqi = Mfx(f, ξ; xpi, ypi) yqi = Mfy(f, ξ; xpi, ypi) (5) The residual function can be expressed as follows.

[0057] dxi = Mfx(xpi, ypi) - xqi dyi = Mfy(xpi, ypi) - yqi (6) Among them, Mf represents the function expression of the forward transformation, Mfx corresponds to the expression in the x direction (for example, the width direction of the image), Mfy corresponds to the expression in the y direction (for example, the height direction of the image), xqi and yqi represent the x-direction coordinates and y-direction coordinates of each pixel point qi of the distorted image, xpi and ypi represent the x-direction coordinates and y-direction coordinates of each pixel point pi of the perspective image, f includes the focal length of the perspective image and the focal length of the distorted image, ξ represents the distortion coefficient of the distorted image, and dxi and dyi represent the residuals between the distorted image corresponding to the perspective image and the distorted image.

[0058] When estimating the estimated value of the distortion parameter using the forward transformation function FT-Mf, input the image coordinates FN(xpi, ypi) of the final intermediate image FN into the function FT-Mf to obtain the image coordinates Fn(xqi, yqi) of the theoretical distorted image Fn corresponding to the final intermediate image FN. That is, input the final intermediate image as the perspective image into the forward transformation function FT-Mf, and the output result of the function is the distorted image corresponding to the final intermediate image. Alternatively, the focal length fn of the final intermediate image FN can be input into the forward transformation function FT-Mf. Also, when it is necessary to input the distortion coefficient ξ of the perspective image (i.e., the final intermediate image), the distortion coefficient ξ can be preset as 0.

[0059] In this example, by using the residual function, the residuals dxi and dyi between the image coordinates Fn(xqi, yqi) of the distorted image corresponding to the final intermediate image and the image coordinates F(xqi, yqi) of the original image (i.e., the distorted image) are calculated. When the residuals dxi and dyi are minimized, it indicates that the perspective image corresponding to the final intermediate image is closest to the distortion image, that is, the values of the distortion coefficients at this time are closest to the real values.

[0060] Also, in this example, the description is given by taking the input of the pixel coordinates of the image into the function FT-Mf as an example. However, the embodiments of the present invention are not limited thereto. In actual applications, corresponding parameters may be input based on the actual structure of the function FT-Mf. For example, the function FT-Mf may further include the field of view angle θ, rotation angle φ, etc. of the image. In this case, the field of view angle θ, rotation angle φ, etc. of the image may be further input into the function FT-Mf. Also, the pixel coordinates after normalization processing may be used when constructing the function FT-Mf. In this case, subsequent processing may be performed after performing normalization processing on the pixel coordinates of the input image. Note that the specific expression of the function FT-Mf is not limited in the embodiments of the present invention, and specifically, related technologies can be referred to.

[0061] In some embodiments, the least squares method (LSM) may be used to estimate the first parameter (for example, the distortion parameter). For example, the total residual S may be S=(1 / 2)*SUM(dxi^2+dyi^2) expressed as. In this case, the least squares method is used to estimate the values of the distortion parameters f and ξ corresponding to when the total residual S is minimized, and these values are the estimated values f* and ξ* of the distortion parameters.

[0062] In this example, in step 104, based on the estimated values f* and ξ* of the distortion parameters, second image processing is performed on the original distorted image to obtain a converted image after distortion removal. The process of performing second image processing on the original image is substantially the same as the process of performing first image processing on the original image. The difference lies in that the distortion parameters used in the second image processing are the estimated values of the distortion parameters.

[0063] In other words, the second image processing may include the following, that is, estimating an enlargement ratio based on the size of the original image F and enlarging the canvas; performing bilinear interpolation processing on the enlarged canvas based on the estimated values f* and ξ* of the first parameters to obtain a second interpolated image; trimming the second interpolated image to obtain a second trimmed image with an aspect ratio equal to the aspect ratio of the original image; and scaling the second trimmed image to obtain the converted image with the same size as the original image, that is, the image after distortion removal. Thereby, by performing relatively few times of second image processing (for example, one time of second image processing) on the original distorted image based on the highly accurate distortion parameters obtained by estimation, an image after distortion removal with high image quality can be obtained. Therefore, in the present invention, an image after distortion removal with good distortion removal effect and high image quality can be obtained.

[0064] As described above, the process of performing distortion removal on the original image when there is distortion in the original image has been described. However, the embodiments of the present invention are not limited thereto. The embodiments of the present invention can further be applied to a scenario where distortion is added to the original image when there is no distortion in the original image or the degree of distortion of the original image is within an acceptable range. For example, a relatively realistic distorted image may be obtained based on the original distortion-free image, and then the distorted image obtained by conversion may be used for the next step of research, or the converted image may be used as learning data for machine learning. Note that the embodiments of the present invention do not limit this.

[0065] The following describes the process of adding distortion to the original image.

[0066] FIG. 6 is a diagram showing another implementation manner of the first image processing in an embodiment of the present invention.

[0067] In the example shown in FIG. 6, it can be said that there is no distortion in the original image F', or the degree of distortion of the original image F' is within an acceptable range (hereinafter, for convenience of explanation, the original image F' may sometimes be referred to as an undistorted image F'). FIG. 6 shows the process of performing the first image processing on the original image F'.

[0068] In this example, the first image processing is a process of obtaining an intermediate image by adding distortion to an undistorted image. When repeatedly performing the distortion addition process on the undistorted image, the first parameter is the focal length f of the camera corresponding to the undistorted image, and the distortion coefficient ξ used when adding distortion is preset. For example, the distortion coefficient ξ is preset as an arbitrary value between 0 and 1. As shown in FIG. 6, the process of performing the first image processing on the original image F' may include the following steps.

[0069] 601: Estimate the magnification based on the size of the original image and expand the canvas; 602: Perform distortion addition processing on the expanded canvas based on the value of the first parameter of the original image and a predetermined distortion coefficient to obtain a first distortion image; 603: Trim the first distortion image to obtain a first trimmed image with an aspect ratio equal to the aspect ratio of the original image; and 604: Scale the first trimmed image to obtain the intermediate image with a size equal to the size of the original image.

[0070] As shown in FIG. 6, the image F’ represents an image without original distortion, the images F1’ to FN-1’ represent intermediate images obtained by the distortion addition process, the image FN’ represents the final intermediate image, and N is a natural number greater than 1. The process of performing the first image processing on the distortion-free image F’ is as follows. That is, an enlargement ratio is estimated based on the size of the distortion-free image F’ (with length W and width H) to obtain an enlarged canvas P’, and distortion addition processing is performed on the enlarged canvas P’ based on the focal length f of the distortion-free image F’ and a predetermined distortion coefficient ξ0 to obtain a first distortion image P1’. Trimming processing is performed on the first distortion image P1’ to obtain a first trimmed image P11’ with an aspect ratio equal to the aspect ratio of the distortion-free image F’. Then, the first trimmed image P11’ is scaled to obtain an intermediate image F1’ with the same size as the distortion-free image F’.

[0071] In this example, the enlargement ratio of the canvas P’ may be predefined. For example, it may be 1.5 times, or, in the case of a predetermined distortion coefficient ξ0, the position of at least one vertex of the four vertices of the distortion-free image F’ within the canvas is calculated, and the enlargement ratio of the canvas is determined based on this position. The predetermined distortion coefficient ξ0 can take any value between 0 and 1. Note that for the implementation method of enlarging the canvas, reference can be made to related technologies, and the embodiments of the present invention are not limited thereto.

[0072] In this example, there is a largest rectangular region A1’ in the first distortion image P1’ obtained by the distortion addition process. The midpoints of the four sides of A1’ correspond to the midpoints of the four sides of the distortion image F. The largest range within A1 is cut out according to the aspect ratio of the distortion-free image F’. For example, a first trimmed image P11’ with length W1 and width H1 is cut out, and H1 / W1 = H / W. Note that for the implementation methods of the distortion addition process and the equal magnification scaling process, reference can be made to related technologies, and the embodiments of the present invention are not limited thereto.

[0073] In an embodiment of the present invention, the process of performing the first image processing on the intermediate images F1' to FN-1' is the same as the process of performing the first image processing on the undistorted image F', that is, the magnification is estimated based on the size of the intermediate image F1' and the canvas is enlarged; distortion addition processing is performed on the enlarged canvas based on the focal length f1 of the intermediate image F1' and a predetermined distortion coefficient ξ1 to obtain an intermediate distorted image; trimming processing is performed on the intermediate distorted image to obtain a trimmed image with an aspect ratio equal to the aspect ratio of the intermediate image F1'; and the trimmed image is scaled to obtain an intermediate image F2' with the same size as the intermediate image F1'. Further, until the final intermediate image FN' is obtained, the above process is repeatedly executed on the intermediate images F2' to FN-1'.

[0074] In this example, the end conditions for repeated execution include that the value of the focal length f starts to decrease or the predetermined number of repetitions is reached.

[0075] In an embodiment of the present invention, when the iteration is terminated when the value of the focal length f starts to decrease, the intermediate image obtained after the previous execution of the first image processing is used as the final intermediate image. Also, when the value of the focal length f of the intermediate image obtained after performing the first image processing once on the undistorted image is smaller than the value of the focal length f of the undistorted image, the undistorted image is used as the final intermediate image.

[0076] Also, when the iteration is terminated after reaching the predetermined number of repetitions, the intermediate image obtained after the last execution of the first image processing is used as the final intermediate image. Note that the predetermined number of repetitions may be set according to needs, and the embodiment of the present invention does not limit the predetermined number of repetitions.

[0077] In this example, in step 105, only the focal length value of the camera corresponding to the original image and the focal length value of the camera corresponding to the intermediate image may be obtained. Note that for the implementation method of obtaining the focal length value of the camera corresponding to the image, related technologies can be referred to, and the embodiment of the present invention does not limit this.

[0078] In some embodiments, when there is no distortion in the original image or when the degree of distortion of the original image is within an acceptable range, the inverse transformation function BT-Mf can be adopted to estimate the estimated value of the focal length of the original image, and the differential function for estimation is the residual function of the inverse transformation function.

[0079] The inverse transformation function BT-Mf can be expressed as follows.

[0080] xpi = Mfx(f, ξ; xqi, yqi) ypi = Mfy(f, ξ; xqi, yqi) (7) The residual function can be expressed as follows.

[0081] dxi = Mfx(xqi, yqi) - xpi dyi = Mfy(xqi, yqi) - ypi (8) Among them, Mf represents the functional expression of the inverse transformation, Mfx corresponds to the expression in the x direction (for example, the width direction of the image), Mfy corresponds to the expression in the y direction (for example, the height direction of the image), xpi and ypi represent the x-direction coordinate and y-direction coordinate of each pixel point pi of the perspective image, xqi and yqi represent the x-direction coordinate and y-direction coordinate of each pixel point qi of the distortion image, f includes the focal length of the perspective image and the focal length of the distortion image, ξ represents the distortion coefficient of the distortion image, and dxi and dyi represent the residual between the perspective image corresponding to the distortion image and the perspective image.

[0082] Estimating the estimated value of the focal length f of the camera using the inverse transformation function BT-Mf is the same as estimating the estimated values of the focal length f and the distortion coefficient ξ using the forward transformation function FT-Mf. Input the image coordinates Fn'(xqi, yqi) of the final intermediate image FN' into the function BT-Mf to obtain the image coordinates Fn'(xpi, ypi) of the theoretical undistorted image Fn' corresponding to the final intermediate image FN', that is, the image coordinates Fn'(xpi, ypi) of the perspective image corresponding to the final intermediate image FN'. That is, input the final intermediate image as a distorted image into the inverse transformation function BT-Mf, and the output result of the function is the undistorted perspective image corresponding to the final intermediate image. Also, the focal length fn and the distortion coefficient ξn-1 of the final intermediate image FN may be input into the inverse transformation function BT-Mf.

[0083] In this example, by using the residual function, the residuals dxi and dyi between the image coordinates Fn'(xpi, ypi) of the perspective image corresponding to the final intermediate image FN' and the image coordinates F'(xpi, ypi) of the original image F' are calculated. When the residuals dxi and dyi are minimized, it indicates that the perspective image (image without theoretical distortion) corresponding to the final intermediate image is closest to the original distortion-free image, that is, the value of the focal length at this time is closest to the real value.

[0084] Also, similar to the forward transformation function FT-Mf, in actual applications, corresponding parameters may be input based on the actual structure of the function BT-Mf. Note that the above description is given by taking the input of pixel coordinates as an example, but the embodiments of the present invention are not limited to this, and the viewing angle θ, rotation angle φ, etc. of the image may be input. Also, subsequent processing may be performed after normalizing the pixel coordinates, but the embodiments of the present invention do not limit this.

[0085] In this example, the least squares method may be used to estimate the distortion parameters. For example, the total residual S is S=(1 / 2)*SUM(dxi^2+dyi^2) It can be expressed as such. In this case, the least squares method is used to estimate the value of the focal length f corresponding to the minimum total residual S, and this value is the estimated value f* of the focal length f.

[0086] Also, in the embodiments of the present invention, further, other estimation methods, such as maximum likelihood estimation, etc., may be used to estimate the distortion parameters. Specifically, relevant technologies can be referred to, and detailed descriptions thereof are omitted here.

[0087] Also, in the embodiments of the present invention, the forward conversion function FT-Mf and / or the inverse conversion function BT-Mf may be linear functions after the distortion parameters f and ξ have undergone one optimization. Note that the embodiments of the present invention are not limited thereto.

[0088] As above, only each step or process related to the present invention has been described, but the present invention is not limited thereto. The image conversion method may further include other steps or processes, and the specific contents of these steps or processes can be referred to the prior art. Also, as above, some structures of the models used in the image conversion method are exemplified to illustrate the embodiments of the present invention, but the present invention is not limited to these structures, and furthermore, appropriate modifications can be made to these structures. Note that all implementation manners of these modifications are included in the scope of the embodiments of the present invention.

[0089] The above-described embodiments are for exemplarily explaining the embodiments of the present invention, but the present invention is not limited thereto, and furthermore, appropriate modifications can be made based on the above-described embodiments. For example, the above-described embodiments can be used alone, or a plurality of the above-described embodiments can be combined and used.

[0090] As can be seen from the above embodiments, by performing image processing on the original image multiple times to obtain a plurality of intermediate images, and estimating the estimated value of the first parameter using the original image and the plurality of intermediate images as training data, a high-precision first parameter can be obtained. Therefore, when removing the distortion of the original image, the image distortion removal accuracy can be improved. Also, by performing image processing on the original image based on the high-precision parameter, an image with a good distortion removal effect and high image quality can be obtained. Furthermore, when adding distortion to the original image, a distortion image close to the actual situation can be obtained, so that it can be applied to different scenes.

[0091] <Embodiment of the second aspect> In an embodiment of the present invention, an image conversion device is further provided, and the device corresponds to the image conversion method in the embodiment of the first aspect. FIG. 7 is a diagram showing the image conversion device in the embodiment of the present invention. As shown in FIG. 7, the image conversion device 700 includes the following.

[0092] First image processing device 701: performing first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; Iteration device 702: repeatedly performing first image processing on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; Estimation device 703: estimating the estimated value of the first parameter based on the conversion relationship between the distorted image and the corresponding perspective image, the original image, and the final intermediate image; and Second image processing device 704: performing second image processing on the original image based on the estimated value of the first parameter to obtain a converted image.

[0093] In some embodiments, the end condition for the iteration device 702 to perform the iteration includes that the change trend of the value of the first parameter starts to reverse or reaches a predetermined number of iterations.

[0094] In some embodiments, the conversion relationship includes a first conversion relationship from a perspective image to a distorted image. The estimation device 703 converts the final intermediate image into a corresponding distorted image based on the first conversion relationship, and obtains an estimated value of the first parameter when the value of a first difference function is minimized based on the distorted image corresponding to the final intermediate image and the original image. The first difference function represents the difference between the distorted image corresponding to the final intermediate image and the original image.

[0095] In some embodiments, the conversion relationship includes a second conversion relationship from a distorted image to a perspective image. The estimation device 703 converts the final intermediate image into a corresponding perspective image based on the second conversion relationship, and obtains an estimated value of the first parameter when the value of a second difference function is minimized based on the perspective image corresponding to the final intermediate image and the original image. The second difference function represents the difference between the perspective image corresponding to the final intermediate image and the original image.

[0096] In some embodiments, the estimation device 703 calculates the estimated value of the first parameter by using the least squares method.

[0097] In some embodiments, the first parameter includes the focal length and / or distortion coefficient of the camera corresponding to the image.

[0098] In some embodiments, for the first image processing device 701 to perform the first image processing: estimating an enlargement ratio based on the size of the original image and enlarging the canvas; performing bilinear interpolation processing on the enlarged canvas based on the value of the first parameter of the original image to obtain a first interpolated image; performing trimming processing on the first interpolated image to obtain a first trimmed image with an aspect ratio equal to the aspect ratio of the original image; and performing scaling processing on the first trimmed image to obtain the intermediate image with the same size as the size of the original image.

[0099] In some embodiments, the first image processing device 701 performing the first image processing includes: estimating an enlargement ratio based on the size of the original image and enlarging a canvas; performing distortion addition processing on the enlarged canvas based on the value of the first parameter of the original image and a predetermined distortion coefficient to obtain a first distorted image; performing trimming processing on the first distorted image to obtain a first trimmed image having an aspect ratio equal to the aspect ratio of the original image; and performing scaling processing on the first trimmed image to obtain the intermediate image having the same size as the size of the original image.

[0100] In some embodiments, the second image processing device 704 performing the second image processing includes: estimating an enlargement ratio based on the size of the original image and enlarging a canvas; performing bilinear interpolation processing on the enlarged canvas based on the estimated value of the first parameter to obtain a second interpolated image; trimming the second interpolated image to obtain a second trimmed image having an aspect ratio equal to the aspect ratio of the original image; and scaling the second trimmed image to obtain the converted image having the same size as the size of the original image.

[0101] In some embodiments, the second image processing device 704 performing the second image processing includes: estimating an enlargement ratio based on the size of the original image and enlarging a canvas; performing distortion addition processing on the enlarged canvas based on the estimated value of the first parameter and a predetermined distortion coefficient to obtain a second interpolated image; trimming the second interpolated image to obtain a second trimmed image having an aspect ratio equal to the aspect ratio of the original image; and Including obtaining the converted image having the same size as the original image by scaling the second trimming image.

[0102] In some embodiments, as shown in FIG. 7, the image conversion device 700 further includes the following.

[0103] Parameter acquisition device 705: obtaining the value of the first parameter of the original image and the value of the first parameter of the intermediate image.

[0104] In this embodiment, since the specific implementation methods of the above-described devices are the same as those described in the embodiments of the first aspect, the detailed description thereof is omitted here.

[0105] Note that only the components or modules related to the present invention have been described above, but the present invention is not limited thereto. The image conversion device 700 may further include other components or modules. For the specific content of these components or modules, reference may be made to related technologies.

[0106] Also, for convenience, FIG. 7 only shows the connection relationship or signal direction between each component or module. However, as can be understood by those skilled in the art, various related technologies such as bus connection may be adopted. Furthermore, the above-described components or modules may be implemented by hardware such as a processor, a memory, etc., but the embodiments of the present invention are not limited thereto.

[0107] The above-described embodiments are for exemplarily explaining the embodiments of the present invention, but the present invention is not limited thereto, and furthermore, appropriate modifications can be made based on the above-described embodiments. For example, the above-described embodiments may be used alone, or a plurality of the above-described embodiments may be used in combination.

[0108] As can be seen from the above embodiments, by performing image processing on the original image multiple times to obtain a plurality of intermediate images, and estimating the estimated value of the first parameter using the original image and the plurality of intermediate images as training data, a high-precision first parameter can be obtained. Therefore, when removing the distortion of the original image, the image distortion removal accuracy can be improved. Also, by performing image processing on the original image based on the high-precision parameter, an image with a good distortion removal effect and high image quality can be obtained. Furthermore, when adding distortion to the original image, a distortion image close to the actual situation can be obtained, so it is possible to cope with applications in different scenes..

[0109] <Embodiment of the third aspect> In an embodiment of the present invention, an electronic device is further provided. FIG. 8 is a diagram showing the electronic device in the embodiment of the present invention. As shown in FIG. 8, the electronic device 800 includes an image conversion device 801. The structure and function of the image conversion device 801 are the same as those of the image conversion device 700 in the embodiment of the second aspect. For specific details, reference can be made to the description in the embodiment of the second aspect, and the detailed description thereof is omitted here.

[0110] FIG. 9 is a block diagram showing the system configuration of the electronic device in the embodiment of the present invention. As shown in FIG. 9, the electronic device 900 may include a processor 901 and a memory 902, and the memory 902 is connected to the processor 901. Note that this figure is only an example, and furthermore, by using other types of structures to supplement or replace this structure, an electric communication function or other functions may be realized.

[0111] In some embodiments, the processor 901 includes at least one of a central processing unit (CPU) and a graphics processing unit (GPU (graphics processing unit)).

[0112] As shown in FIG. 9, the electronic device 900 may further include an input device 903, a display 904, a power supply 905, etc.

[0113] In one embodiment, the functions of the image conversion device can be integrated into the processor 901. The processor 901 may be configured as follows, that is, perform first image processing on the distorted image based on the value of the distortion parameter of the distorted image to obtain an intermediate image; repeat and execute the first image processing on the intermediate image based on the value of the distortion parameter of the intermediate image to obtain a final intermediate image; estimate an estimated value of the distortion parameter based on the conversion relationship between the distorted image and the final intermediate image, and the value of the distortion parameter of the distorted image or the value of the distortion parameter of the final intermediate image; and perform second image processing on the distorted image based on the estimated value of the distortion parameter to obtain an undistorted image. For details of other embodiments, reference can be made to the examples of the first aspect, and the detailed description thereof is omitted here.

[0114] In another embodiment, the image conversion device may be arranged separately from the processor 901. For example, the image conversion device may be configured as a chip connected to the processor 901, and the functions of the image conversion device may be realized under the control of the processor 901.

[0115] In this embodiment, the electronic device 900 does not necessarily include all the components shown in FIG. 9.

[0116] As shown in FIG. 9, the processor 901 may sometimes be referred to as a controller or an operation control. For example, it may include a microprocessor or other processing device and / or logic device, and the processor 901 can receive an input and control the operation of each component of the electronic device 900.

[0117] The memory 902 may include, for example, one or more of a buffer, flash memory, HDD, removable media, volatile memory, non-volatile memory, or other suitable devices. Further, the processor 901 can execute the program stored in the memory 902 to realize storage, processing, etc. of information. Note that the functions of other components are the same as those in the prior art, and detailed descriptions thereof are omitted here. Further, each component of the electronic device 900 may be realized by dedicated hardware, firmware, software, or a combination thereof, and all of them belong to the scope of the present invention.

[0118] As can be seen from the above embodiments, by performing image processing on the original image multiple times to obtain a plurality of intermediate images, and estimating the estimated value of the first parameter using the original image and the plurality of intermediate images as training data, a high-precision first parameter can be obtained. Therefore, when removing the distortion of the original image, the image distortion removal accuracy can be improved. Further, by performing image processing on the original image based on the high-precision parameter, an image with a good distortion removal effect and high image quality can be obtained. Furthermore, when adding distortion to the original image, a distortion image close to the actual situation can be obtained, so that it is possible to cope with applications in different scenes.

[0119] In an embodiment of the present invention, a computer-readable program is further provided. When the program is executed by an image conversion device or an electronic device, the program causes the computer to execute the image conversion method described in the embodiment of the first aspect by the image conversion device or the electronic device.

[0120] In an embodiment of the present invention, a storage medium storing a computer-readable program is further provided. The computer-readable program causes the computer to execute the image conversion method described in the embodiment of the first aspect by an image conversion device or an electronic device.

[0121] In addition, the above-described apparatus and method may be implemented by software or hardware, or may be implemented by a combination of hardware and software. The present invention further relates to a computer-readable program as follows, that is, when the program is executed by a logic component, the logic component realizes the above-described apparatus or component, or the logic component realizes the above-described various methods or steps. The logic component may be, for example, an FPGA (Field Programmable Gate Array), a microprocessor, a processor used in a computer, or the like. The present invention further relates to a storage medium storing the above-described program, for example, a hard disk, a magnetic disk, an optical hard disk, a DVD, a flash memory, or the like.

[0122] Furthermore, one or more combinations of the functional blocks described in the drawings and / or one or more combinations of the functional blocks may be implemented as a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic components, discrete gates or transistor logic components, discrete hardware assemblies, or any other suitable combination for performing the functions described herein. Also, one or more combinations of the functional blocks described in the drawings and / or one or more combinations of the functional blocks may further be configured as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors connected in communication with the DSP, or any other configuration combination.

[0123] Regarding the above-described embodiments and the like, the following additional remarks are further disclosed.

[0124] (Supplementary Note 1) An image conversion method, comprising: S1: Performing first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; S2: Repeatedly perform a first image process on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; S3: Estimate an estimated value of the first parameter based on the conversion relationship between the distorted image and the corresponding perspective image, the original image, and the final intermediate image; and S4: A method including performing a second image process on the original image based on the estimated value of the first parameter to obtain a converted image.

[0125] (Appendix 2) The method according to Appendix 1, The end condition of the repeated execution in step S2 includes that the change trend of the value of the first parameter begins to reverse or a predetermined number of repetitions is reached.

[0126] (Appendix 3) The method according to Appendix 2, When ending the repeated execution when the change trend of the value of the first parameter begins to reverse, the intermediate image obtained after performing the previous first image process is used as the final intermediate image.

[0127] (Appendix 4) The method according to Appendix 1, The conversion relationship includes a first conversion relationship from the perspective image to the distorted image, Step S3 includes converting the final intermediate image into a corresponding distorted image based on the first conversion relationship, and obtaining an estimated value of the first parameter when the value of the first difference function is minimized based on the distorted image corresponding to the final intermediate image and the original image. The first difference function represents the difference between the distorted image corresponding to the final intermediate image and the original image.

[0128] (Appendix 5) The method according to Appendix 1, The conversion relationship includes a second conversion relationship from the distorted image to the perspective image, Step S3 includes converting the final intermediate image into a corresponding perspective image based on the second conversion relationship, and obtaining an estimated value of the first parameter when the value of the second difference function is minimized based on the perspective image corresponding to the final intermediate image and the original image. The second difference function represents the difference between the perspective image corresponding to the final intermediate image and the original image, method.

[0129] (Appendix 6) The method according to Appendix 4 or 5, Calculating an estimated value of the distortion parameter using the least squares method, method.

[0130] (Appendix 7) The method according to Appendix 1, The first parameter is the focal length and distortion coefficient of the image, method.

[0131] (Appendix 8) The method according to Appendix 4, Step S1 is Estimating an enlargement ratio based on the size of the original image and enlarging the canvas; Performing bilinear interpolation processing on the enlarged canvas based on the value of the first parameter of the original image to obtain a first interpolated image; Trimming the first interpolated image to obtain a first trimmed image with an aspect ratio equal to the aspect ratio of the original image; and Including scaling the first trimmed image to obtain an intermediate image with the same size as the original image, method.

[0132] (Appendix 9) The method according to Appendix 5, Step S1 is Estimating an enlargement ratio based on the size of the original image and enlarging the canvas; Performing distortion addition processing on the enlarged canvas based on the value of the first parameter of the original image and a predetermined distortion coefficient to obtain a first distortion image; Trimming the first distortion image to obtain a first trimmed image having an aspect ratio equal to the aspect ratio of the original image; and A method comprising obtaining an intermediate image having the same size as the original image by scaling the first trimmed image.

[0133] (Appendix 10) The method according to Appendix 4, wherein Step S4 is estimating an enlargement ratio based on the size of the original image and enlarging the canvas; performing bilinear interpolation processing on the enlarged canvas based on the estimated value of the first parameter to obtain a second interpolated image; performing trimming processing on the second interpolated image to obtain a second trimmed image having an aspect ratio equal to the aspect ratio of the original image; and performing scaling processing on the second trimmed image to obtain a converted image having the same size as the original image.

[0134] (Appendix 11) The method according to Appendix 5, wherein Step S4 is estimating an enlargement ratio based on the size of the original image and enlarging the canvas; performing distortion addition processing on the enlarged canvas based on the estimated value of the first parameter and a predetermined distortion coefficient to obtain a second interpolated image; performing trimming processing on the second interpolated image to obtain a second trimmed image having an aspect ratio equal to the aspect ratio of the original image; and scaling the second trimmed image to obtain a converted image having the same size as the original image.

[0135] (Appendix 12) A storage medium storing a computer-readable program, wherein the computer-readable program causes a processor connected to the storage medium to perform an image conversion method, wherein Performing first image processing on the original image based on the value of the first parameter of the original image to obtain an intermediate image; Repeating and executing first image processing on the intermediate image based on the value of the first parameter of the intermediate image to obtain a final intermediate image; Estimating an estimated value of the first parameter based on the conversion relationship between the distorted image and its corresponding perspective image, the original image, and the final intermediate image; and A method including performing second image processing on the original image based on the estimated value of the first parameter to obtain a converted image A storage medium for causing the above to be executed.

[0136] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to such embodiments, and any changes to the present invention belong to the technical scope of the present invention as long as they do not depart from the spirit of the present invention.

Claims

1. A device for converting images, comprising: a first image processing device that performs a first image processing on the original image based on a value of a first parameter of the original image to obtain an intermediate image; an iterative device for repeatedly performing a first image processing on the intermediate images based on values ​​of the first parameter of the intermediate images to obtain a final intermediate image; an estimator for estimating an estimate of the first parameter based on a transformation relationship between a distorted image and a corresponding perspective image, the original image, and the final intermediate image; and and a second image processor that performs a second image process on the original image based on the estimate of the first parameter to obtain a transformed image.

2. 2. The device of claim 1, The device, wherein a condition for terminating the iterations performed by the iterative device includes a condition in which a trend of change in the value of the first parameter begins to reverse or a condition in which a predetermined number of iterations is reached.

3. 2. The device of claim 1, the transformation relationship includes a first transformation relationship for transforming a perspective image to a distorted image; the estimation device transforms the final intermediate image into a corresponding distorted image based on the first transformation relationship, and obtains an estimate of the first parameter when a value of a first difference function is minimum based on the distorted image corresponding to the final intermediate image and the original image; The first difference function represents the difference between a distorted image corresponding to the final intermediate image and the original image.

4. 2. The device of claim 1, The transformation relationship includes a second transformation relationship for transforming the distorted image to the perspective image; the estimation device transforms the final intermediate image into a corresponding perspective image based on the second transformation relationship, and obtains an estimate of the first parameter when a value of a second difference function is minimum based on the perspective image corresponding to the final intermediate image and the original image; The second difference function represents the difference between a perspective image corresponding to the final intermediate image and the original image.

5. 5. The device according to claim 3 or 4, The estimator calculates the estimate of the first parameter using a least squares method.

6. 2. The device of claim 1, The first parameters include a focal length and / or distortion coefficients of a camera corresponding to the image.

7. 4. The device of claim 3, The first image processing device performs the first image processing, Enlarging the canvas by estimating a magnification ratio based on the size of the original image; performing a bilinear interpolation process on the enlarged canvas based on the value of the first parameter of the original image to obtain a first interpolated image; performing a cropping process on the first interpolated image to obtain a first cropped image having an aspect ratio equal to an aspect ratio of the original image; and performing a scaling process on the first cropped image to obtain the intermediate image having the same size as the original image.

8. 4. The device of claim 3, The second image processing device performs the second image processing, Enlarging the canvas by estimating a magnification ratio based on the size of the original image; performing a bilinear interpolation process on the enlarged canvas based on the estimated value of the first parameter to obtain a second interpolated image; performing a cropping process on the second interpolated image to obtain a second cropped image having an aspect ratio equal to an aspect ratio of the original image; and performing a scaling process on the second cropped image to obtain the transformed image of the same size as the original image.

9. 2. The device of claim 1, The apparatus further includes a parameter acquisition device for acquiring a value of a first parameter of the original image and a value of a first parameter of the intermediate image.

10. 1. A method of transforming an image, comprising the steps of: performing a first image processing on the original image based on a value of a first parameter of the original image to obtain an intermediate image; iteratively performing a first image process on the intermediate image based on values ​​of the first parameter of the intermediate image to obtain a final intermediate image; estimating an estimate of the first parameter based on a transformation relationship between a distorted image and a corresponding perspective image, the original image, and the final intermediate image; and performing a second image process on the original image based on the estimate of the first parameter to obtain a transformed image.

Images