Virtual simulation method and device for camera distortion

Abstract

The present invention provides a virtual simulation method and device for camera distortion, the method comprising: obtaining calibrated distortion parameters, internal parameters and a real shot image of a camera to be simulated; determining the shrinkage ratio of preset pixel points in the imaging distortion process according to the calibrated distortion parameters and the internal parameters; determining the image resolution magnification ratio that can meet the nominal resolution after distortion, adjusting the resolution of the real shot image, and obtaining the image to be virtually simulated; obtaining the adjusted distortion parameters of the camera to be simulated, performing virtual simulation of the camera distortion, and obtaining a simulation result of the camera distortion to be simulated, and reasonably adjusting the size and distortion parameters of the image to be virtually simulated based on the camera imaging principle and characteristics and relying on a mathematical model, so as to achieve a realistic and reasonable simulation image in principle and in visualization effect, so that the simulation result is more in line with the distortion imaging effect of the actual camera, thereby improving the accuracy of the simulation result.

Description

Technical Field

[0001] The present invention relates to the field of virtual simulation technology, and in particular to a virtual simulation method and device for camera distortion. Background Art

[0002] Currently, various types of cameras are widely used in industrial manufacturing, autonomous driving, traffic security and other fields. The demand for high-fidelity simulation of cameras is also increasing. However, the industry currently has the following problems in virtual simulation of cameras, especially virtual simulation of camera imaging defects:

[0003] The existing technical solutions are mainly based on the barrel or pincushion distortion model of the camera to perform simple and direct pixel position indentation, which visually meets the expected distortion effect. However, in terms of the relationship between the pixel range and the FOV (Field Of View) of the simulated camera, the simulation effect is not satisfactory, which will cause distortion that is inconsistent with the actual imaging effect. Summary of the invention

[0004] The object of the present invention is to provide a virtual simulation method and device for camera distortion that can meet the actual camera distortion imaging effect.

[0005] To achieve the above object, the present invention provides a virtual simulation method of camera distortion, which is used to improve the accuracy of simulation results and better fit the distortion imaging effect of an actual camera, and comprises:

[0006] Obtaining the calibration distortion parameters, internal parameters and the image actually taken by the camera to be simulated;

[0007] According to the calibration distortion parameters and internal parameters, the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated is determined; according to the shrinkage ratio, the image resolution magnification ratio of the image actually shot after distortion that can meet the nominal resolution of the camera to be simulated is determined; according to the magnification ratio, the resolution of the image actually shot is adjusted to obtain the image to be virtually simulated;

[0008] Based on the distortion model function characteristics of the camera to be simulated, and according to the maximum field of view of the camera to be simulated, the calibrated distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated;

[0009] According to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, a virtual simulation of the camera distortion is performed to obtain a simulation result of the camera distortion to be simulated.

[0010] In a specific embodiment, according to the calibration distortion parameters and the internal parameters, determining the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated includes:

[0011] According to the internal parameter matrix of the camera to be simulated, the distance between the preset pixel point and the image center in the image actually taken by the camera to be simulated is determined;

[0012] According to the distance and the distortion model of the camera to be simulated, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined.

[0013] Furthermore, if the camera to be simulated is a pinhole camera, the shrinkage ratio of the preset pixel point in the imaging distortion process of the camera to be simulated is determined according to the following formula based on the distance and the distortion model of the camera to be simulated:

[0014] I=(1+k1r 2 +k2r 4 +k3r 6 ) / r

[0015] Among them, r represents the distance between the preset pixel point and the center of the image; I represents the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated; k1, k2 and k3 represent the calibrated distortion parameters of the camera to be simulated.

[0016] In a specific implementation, if the camera to be simulated is a fisheye camera, the shrinkage ratio of the preset pixel point in the imaging distortion process of the camera to be simulated is determined according to the following formula based on the distance and the distortion model of the camera to be simulated:

[0017] I = (tan(θ) / θ)(1+k1r 2 +k2r 4 +k3r 6 ) / r

[0018] Among them, r represents the distance between the preset pixel point and the center of the image; I represents the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated; θ represents the angle between the incident light of the camera to be simulated and the main axis of the lens; k1, k2 and k3 represent the calibrated distortion parameters of the camera to be simulated.

[0019] In a specific embodiment, according to the magnification ratio, the resolution of the real captured image is adjusted to obtain the image to be virtually simulated, including:

[0020] According to the magnification ratio and the nominal resolution of the camera to be simulated, the minimum image resolution of the real captured image that can meet the nominal resolution of the camera to be simulated after distortion is obtained;

[0021] Based on the minimum image resolution, the resolution of the image actually shot is adjusted to determine the image to be virtually simulated.

[0022] Specifically, based on the distortion model function characteristics of the camera to be simulated, according to the maximum field of view of the camera to be simulated, the calibration distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated, including:

[0023] According to the maximum field of view of the camera to be simulated, determine the distance value between the pixel point corresponding to the maximum field of view and the image center;

[0024] Draw an image of the distortion model function of the camera to be simulated, and determine the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the center of the image;

[0025] Adjust the calibrated distortion parameters of the camera to be simulated so that the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance value between the pixel point corresponding to the maximum field of view angle and the image center, and obtain the adjusted distortion parameters of the camera to be simulated.

[0026] Furthermore, in order to further improve the accuracy of the simulation, the virtual simulation method of camera distortion provided in the specific embodiment also includes:

[0027] The image to be virtually simulated is used as the initial image, and the optimal image to be virtually simulated is determined cyclically. The following operations are performed in each cycle:

[0028] Based on the calibrated distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image;

[0029] Cutting out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistorting and restoring the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image;

[0030] Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to a preset value, the initial image is determined as the optimal image to be virtually simulated; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle until the similarity is no lower than the preset value.

[0031] The present invention also provides a virtual simulation device for camera distortion, which is used to improve the accuracy of simulation results and more closely match the distortion imaging effect of an actual camera, and comprises:

[0032] A data acquisition module is used to obtain the calibration distortion parameters, internal parameters and the image actually taken by the camera to be simulated;

[0033] The original image determination module is used to determine the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibration distortion parameters and internal parameters; according to the shrinkage ratio, determine the image resolution magnification ratio of the image actually shot after distortion that can meet the nominal resolution of the camera to be simulated; according to the magnification ratio, adjust the resolution of the image actually shot to obtain the image to be virtually simulated;

[0034] The distortion parameter adjustment module is used to adjust the calibrated distortion parameters of the camera to be simulated based on the distortion model function characteristics of the camera to be simulated and the maximum field of view of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated;

[0035] The distortion simulation module is used to perform virtual simulation of camera distortion according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, so as to obtain the simulation result of the distortion of the camera to be simulated.

[0036] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned virtual simulation method of camera distortion when executing the computer program.

[0037] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program for executing the above-mentioned virtual simulation method of camera distortion.

[0038] In an embodiment of the present invention, calibrated distortion parameters, internal parameters and an image actually shot by the camera to be simulated are obtained; based on the calibrated distortion parameters and the internal parameters, a shrinkage ratio of preset pixels in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated is determined; based on the shrinkage ratio, an image resolution magnification ratio of the image actually shot that can meet the nominal resolution of the camera to be simulated after distortion is determined, and based on the magnification ratio, the resolution of the image actually shot is adjusted to obtain the image to be virtually simulated; based on the distortion model function characteristics of the camera to be simulated, the calibrated distortion parameters of the camera to be simulated are adjusted according to the maximum field of view of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated; based on the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, a virtual simulation of the camera distortion is performed to obtain a simulation result of the distortion of the camera to be simulated. Compared with the technical solution in the prior art that performs simple and direct pixel position indentation based on the barrel or pincushion distortion model of the camera, based on the camera imaging principle and characteristics, relying on the mathematical model, the shrinkage ratio of the preset pixel point in the imaging distortion process of the camera to be simulated is determined, and the size of the image to be virtually simulated is reasonably adjusted based on the shrinkage ratio. The distortion parameters of the camera to be simulated are adjusted based on the distortion model function characteristics of the camera to be simulated. This can achieve true and reasonable image simulation in principle and visualization effect, make the simulation result more consistent with the distortion imaging effect of the actual camera, and thus improve the accuracy of the simulation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The following drawings are intended only to illustrate and explain the present invention, and are not intended to limit the scope of the present invention.

[0040] Figure 1 Schematic diagram of the implementation process of the virtual simulation method of camera distortion according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of a specific implementation process of step 102 in a specific embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of another specific implementation process of step 102 in a specific embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of a specific implementation process of step 103 in a specific embodiment of the present invention;

[0044] Figure 5 is a comparison diagram of the distortion model function image of the pinhole camera in a specific embodiment of the present invention and the y=x straight line;

[0045] Figure 6 is a schematic diagram of a specific implementation process of step 104 in a specific embodiment of the present invention;

[0046] Figure 7 It is a schematic diagram of the implementation process of a virtual simulation method for camera distortion according to a specific embodiment of the present invention;

[0047] Figure 8 is a schematic diagram of the imaging (original image) of a pinhole camera model without distortion in a specific example of the present invention;

[0048] Fig. 9 is a schematic diagram of an image (with black edges) simulated in a traditional manner in a specific example of the present invention;

[0049] Fig.10 is a schematic diagram of an image having the same principal point as the original image but with a larger field angle in a specific example of the present invention;

[0050] Fig.11 In the specific example of the present invention Fig.10 Schematic diagram after distortion processing;

[0051] Fig.12 In the specific example of the present invention, Fig.11 Schematic diagram of an image cut out at its original resolution;

[0052] Fig.13 In the specific example of the present invention, Fig.12 Schematic diagram after restoration using the de-distortion algorithm;

[0053] Fig.14 In the specific example of the present invention, Fig.13 Schematic diagram of the area of ​​the original resolution size copied from the center part;

[0054] Fig.15 It is a structural schematic diagram of a virtual simulation device for camera distortion according to an embodiment of the present invention;

[0055] Fig.16 It is a structural schematic diagram of a virtual simulation device for camera distortion according to a specific embodiment of the present invention;

[0056] Fig.17 It is a structural schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present application is further described in detail below through the accompanying drawings and examples. Through these descriptions, the characteristics and advantages of the present application will become clearer and more specific. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise noted.

[0059] In addition, the technical features involved in different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0060] In the prior art, cameras have some imaging defects, such as distortion, when imaging. Different cameras have different degrees of distortion, mainly including the following two types:

[0061] Pinhole camera model: The most widely used classic pinhole camera model in fields such as machine vision is as follows:

[0062]

[0063] x′=x / z

[0064] y′=y / z

[0065] Through the rotation matrix R and the translation matrix t, the world coordinate system X, Y, Z values ​​of the imaged point can be converted to the x, y, z values ​​of the camera coordinate system, and then normalized to the pixel coordinate system. x' and y' represent the ideal pinhole projection coordinates.

[0066] It is generally believed that the distortion produced by optical cameras during the imaging process can also be reflected in the following ways:

[0067]

[0068]

[0069] The actual distorted pixel corresponding coordinates x″, y″ can be obtained by the above transformation of the ideal pinhole projection coordinates x′ and y′, where k1, k2, k3, k4, k5, k6 are distortion parameters that describe the actual distortion effect. In the actual calibration process, k4, k5, k6 are generally regarded as 0, so the key to the distortion effect is 1+k1r 2 +k2r 4 +k3r 6 This proportionality factor.

[0070] The perspective projection model of a pinhole camera can be simplified as:

[0071] r = f tan q

[0072] Where q is the angle between the incident light and the principal axis of the lens; r is the distance between the image pixel and the image center (principal point); and f is the equivalent focal length of the lens.

[0073] Fisheye camera model: Since the imaging principle of the fisheye camera does not completely follow the pinhole imaging model, the distortion model must also be specifically analyzed.

[0074] Fisheye cameras are usually designed to follow one of the following models:

[0075] r′=2f tan(q / 2)(stereoscopic projection)

[0076] r′=fq(equidistant projection)

[0077] r′=2f sin(q / 2) (equiangular projection)

[0078] r′=f sin q (orthogonal projection)

[0079] The most common projection model is the equidistant projection, so its core mathematical model can be equivalent to:

[0080] r′=Mq(1+k1r 2 +k2r 4 +k3r 6 ) / tan(q)

[0081] M is the pixel coordinate of the original undistorted image (it can also be regarded as the coordinate of the image projected directly from the three-dimensional world coordinates through the pinhole camera model);

[0082] r′ is the pixel coordinate of the image actually displayed by the fisheye camera with distortion parameters k1, k2, and k3;

[0083] q is a very critical parameter in the fisheye camera imaging model, which is the angle between the incident light from a three-dimensional point and the optical axis.

[0084] This is a pixel shrinkage process that is more significant than the pinhole camera distortion model. In fact, the most significant distortion effect of the fisheye lens does not come from the distortion parameters, but from the model r′=Mq / tan(q) itself.

[0085] From the properties of trigonometric functions, it can be concluded that among the four common models of fisheye cameras, the orthogonal projection model has the strongest distortion, the stereoscopic projection has the weakest, and the equidistant projection is between the two.

[0086] Therefore, the prior art performs simple and direct pixel position indentation based on the above mathematical model, that is, based on the barrel or pincushion distortion model of the camera.

[0087] However, the inventors discovered that if the simulation of the camera distortion effect is simply regarded as the indentation of the pixel position, the original image will have a larger indentation at the four corners and less indentation at the four sides, and the overall image will appear "barrel-shaped". Although a rectangular image can be obtained by using the de-distortion algorithm at this time, if the goal is to simulate the actual camera shooting picture, then such a "barrel-shaped" picture is obviously an unacceptable distortion.

[0088] In addition, the inventors also noticed that since the distortion parameters are only an approximate mathematical description of the actual distortion effect, and the calibration of the distortion parameters in the calibration stage cannot be consistent with the actual parameters, the ordinary calibration method is directly used to simulate the distortion effect of the camera. Sometimes, the pixels at the edge of the image will be rolled into the back of the image due to excessive "shrinkage" (especially obvious in the simulation process of fisheye cameras), which is obviously a distortion that is inconsistent with the actual imaging effect.

[0089] In order to solve the above-mentioned principle distortion problem in the prior art, an embodiment of the present invention provides a virtual simulation method of camera distortion, which is used to improve the accuracy of the simulation result and better fit the distortion imaging effect of the actual camera. Figure 1 As shown, the method includes:

[0090] Step 101: obtaining calibrated distortion parameters, internal parameters, and images actually captured by the camera to be simulated;

[0091] Step 102: determining the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibration distortion parameters and the internal parameters; determining the image resolution magnification ratio of the image actually shot after distortion that can meet the nominal resolution of the camera to be simulated according to the shrinkage ratio; adjusting the resolution of the image actually shot according to the magnification ratio to obtain the image to be virtually simulated;

[0092] Step 103: Based on the distortion model function characteristics of the camera to be simulated and according to the maximum field of view of the camera to be simulated, adjusting the calibrated distortion parameters of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated;

[0093] Step 104: Perform virtual simulation of camera distortion according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, and obtain a simulation result of the camera distortion to be simulated.

[0094] Depend on Figure 1It can be seen from the shown process that in the embodiment of the present invention, the calibrated distortion parameters, internal parameters and the image actually shot by the camera to be simulated are obtained; according to the calibrated distortion parameters and the internal parameters, the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated is determined; according to the shrinkage ratio, the image resolution magnification ratio of the actually shot image that can meet the nominal resolution of the camera to be simulated after the distortion is determined, and according to the magnification ratio, the resolution of the actually shot image is adjusted to obtain the image to be virtually simulated; based on the distortion model function characteristics of the camera to be simulated, according to the maximum field of view of the camera to be simulated, the calibrated distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated; according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, a virtual simulation of the camera distortion is performed to obtain the simulation result of the distortion of the camera to be simulated. Compared with the technical solution in the prior art that performs simple and direct pixel position indentation based on the barrel or pincushion distortion model of the camera, based on the camera imaging principle and characteristics, relying on the mathematical model, the shrinkage ratio of the preset pixel point in the imaging distortion process of the camera to be simulated is determined, and the size of the image to be virtually simulated is reasonably adjusted based on the shrinkage ratio. The distortion parameters of the camera to be simulated are adjusted based on the distortion model function characteristics of the camera to be simulated. This can achieve true and reasonable image simulation in principle and visualization effect, make the simulation result more consistent with the distortion imaging effect of the actual camera, and thus improve the accuracy of the simulation result.

[0095] In the specific implementation, first obtain the calibrated distortion parameters, internal parameters and the image actually taken by the camera to be simulated. By calibrating the distortion parameters of the camera to be simulated, the calibrated distortion parameters and internal parameters of the camera to be simulated are obtained. Specifically, the internal parameters and distortion parameters of the camera to be simulated are obtained from the calibration plate picture taken by the real camera. The distortion parameters are mainly radial distortion. First, the camera to be simulated is used to shoot the standard black and white chessboard calibration plate. According to the pixel coordinates of the corresponding black and white intersections in multiple photos at different angles, a set of equations is established to solve the camera's external parameter matrix, internal parameter matrix and various distortion parameters, so as to obtain the calibrated distortion parameters and internal parameters of the camera to be simulated. At the same time, obtain the image actually taken by the camera to be simulated, that is, obtain the image taken by the camera to be simulated in a real scene, as the image material for simulation.

[0096] After obtaining the calibrated distortion parameters and internal parameters of the camera to be simulated, the shrinkage ratio of the pixel during the distortion process is determined according to the calibrated distortion parameters and internal parameters of the camera to be simulated. The specific implementation process is as follows: Figure 2 As shown, including:

[0097] Step 201: determining the distance between a preset pixel point and the center of the image actually captured by the camera to be simulated according to the internal parameter matrix of the camera to be simulated;

[0098] Step 202: According to the above distance and the distortion model of the camera to be simulated, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined.

[0099] The inventor discovered that, for a specific camera, it has its own internal parameter matrix Determines the angle between the incident light and the main axis and the specific pixel at which the light will be projected. x Represents the ratio of the horizontal focal length to the pixel size (which can be simply understood as the focal length); f y represents the ratio of longitudinal focal length to pixel size; c x Indicates the horizontal center of the image; c y Indicates the vertical center of the image.

[0100] From the perspective projection model of the pinhole camera, it can be seen that the incident light angle is positively correlated with the distance r from the specific pixel on the photosensitive element to the center of the image. Among them, x0 represents the horizontal resolution, that is, the maximum horizontal pixel value, and y0 represents the vertical resolution, that is, the maximum r value corresponding to the pixels at the four corners.

[0101] Furthermore, if the camera to be simulated is a pinhole camera, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined according to the following formula based on the above distance and the distortion model of the camera to be simulated:

[0102] I=(1+k1r 2 +k2r 4 +k3r 6 ) / r

[0103] Among them, r represents the distance between the preset pixel point and the center of the image; I represents the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated; k1, k2 and k3 represent the calibrated distortion parameters of the camera to be simulated.

[0104] If the camera to be simulated is a fisheye camera, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined according to the following formula based on the above distance and the distortion model of the camera to be simulated:

[0105] I = (tan(θ) / θ)(1+k1r 2 +k2r 4 +k3r 6 ) / r

[0106] Among them, r represents the distance between the preset pixel and the center of the image; I represents the shrinkage ratio of the preset pixel during the imaging distortion process of the simulated camera; θ represents the angle between the incident light of the simulated camera and the main axis of the lens; k1, k1 and k3 represent the calibrated distortion parameters of the simulated camera. The maximum value of θ is half of the nominal field of view of the fisheye camera, so the simulated distorted image will be circular, which is consistent with the actual imaging of the fisheye camera.

[0107] In specific implementation, the image obtained after the barrel distortion is corrected by the distortion correction algorithm will generally be a "pincushion-shaped" image, that is, the valid pixels in the rectangular image will return to their original positions under the standard pinhole camera model after "extrapolation" by the correction algorithm. Since the real camera with barrel distortion actually captures a "pincushion-shaped" area in the real environment, that is, the diagonal field of view is larger than the nominal field of view, then from the perspective of the field of view, using an image that meets the field of view of the original camera to simulate the effect after distortion will obviously result in "black corners", because some pixels that would enter the camera's field of view in real situations are not provided to the distortion simulation algorithm.

[0108] In order to ensure that the simulated distorted image is not distorted, it is necessary to prepare an image with a larger field of view to ensure that after the "contraction" caused by the distortion effect, the effective pixels can still fill the nominal resolution of the camera. Therefore, the next step is to calculate the magnification ratio.

[0109] In a specific embodiment, after determining the shrinkage ratio of preset pixels in the image actually taken by the camera to be simulated during the imaging distortion process of the camera to be simulated, the image resolution magnification ratio that can meet the nominal resolution of the camera to be simulated after the distortion of the actually taken image is determined based on the shrinkage ratio.

[0110] According to the magnification ratio, the resolution of the real captured image is adjusted to obtain the image to be virtually simulated. The specific process is as follows: Figure 3 As shown, including:

[0111] Step 301: according to the magnification ratio and the nominal resolution of the camera to be simulated, obtaining the minimum image resolution that can meet the nominal resolution of the camera to be simulated after the distortion of the real captured image;

[0112] Step 302: Based on the minimum image resolution, the resolution of the image actually captured is adjusted to determine the image to be virtually simulated.

[0113] Specifically, since the pixels at the four corners of the camera correspond to the maximum r value, the minimum magnification ratio can be obtained by substituting the current r into the formula of the shrinkage ratio of the pixels corresponding to the camera type during the distortion process. The minimum magnification ratio can determine the above-mentioned minimum image resolution, so the preset pixel point in this embodiment generally refers to any one of the pixel points at the four corners of the image actually taken by the camera to be simulated, but those skilled in the art can understand that, by taking any pixel point in the image actually taken by the camera to be simulated as the preset pixel point, the magnification ratio obtained by solving is greater than the above-mentioned minimum magnification ratio, so that the resolution of the actually taken image is adjusted, and the obtained image can meet the needs of simulation, so the preset pixel points in the embodiment of the present invention are not limited here.

[0114] Taking a pinhole camera as an example, the specific implementation process is: calculate the image resolution magnification ratio that can meet the nominal resolution of the camera to be simulated after the distortion of the real captured image according to the following formula:

[0115]

[0116] Among them, r max represents the maximum distance between the pixel point in the camera and the center of the image; I′ represents the image resolution magnification ratio that can meet the nominal resolution of the camera to be simulated after the real image is distorted; x 0max Indicates the maximum horizontal pixel coordinate; y 0max Indicates the maximum vertical pixel coordinate; c x represents the horizontal pixel center; c y represents the vertical pixel center; f x Indicates the horizontal focal length pixel size ratio; f y Indicates the ratio of vertical focal length to pixel size; c x 、c y 、f x 、f y The value of is obtained from the internal parameter matrix of the camera to be simulated; k1, k2 and k3 represent the calibration distortion parameters of the camera to be simulated.

[0117] By multiplying the image resolution magnification ratio I′ obtained above and the nominal resolution of the camera to be simulated, the minimum image resolution of the real image that can meet the nominal resolution of the camera to be simulated after distortion can be obtained. The resolution of the real image is adjusted so that its resolution is greater than or equal to the minimum resolution obtained above, and the image to be virtually simulated can be determined.

[0118] In order to avoid the phenomenon of excessive shrinkage of image edges and pixel overlap caused by inaccurate distortion parameters, after determining the image to be virtually simulated, it is necessary to adjust the calibrated distortion parameters to make them as close to the actual parameters as possible. In specific implementation, based on the distortion model function characteristics of the camera to be simulated, according to the maximum field of view FOV of the camera to be simulated, the calibrated distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated. The specific process is as follows: Figure 4 As shown, including:

[0119] Step 401: Determine the distance between the pixel point corresponding to the maximum FOV and the image center according to the maximum FOV of the camera to be simulated;

[0120] Step 402: draw an image of the distortion model function of the camera to be simulated, and determine the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the center of the image;

[0121] Step 403: Adjust the calibrated distortion parameters of the camera to be simulated so that the distance between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance between the pixel point corresponding to the maximum FOV and the image center, and obtain the adjusted distortion parameters of the camera to be simulated.

[0122] Taking the pinhole camera as an example, combined with the image analysis of its distortion model function, Figure 5 Under the action of two sets of distortion parameters with different values, 1+k1r 2 +k2r 4 +k3r 6 The graph of the function and the graph of y=x. Comparing the three curves, we can see that the key lies in 1+k1r 2 +k2r 4 +k3r 6 Monotonicity and monotonic intervals of functions.

[0123] In fact, y=x can be regarded as the image produced by the classic pinhole projection model without any distortion. Therefore, it can be seen that the degree to which these two function curves are lower than the y=x straight line at different independent variable values ​​is the degree to which the pixel points in the camera image under the action of this set of distortion parameters are "indented" at the corresponding distance from the principal point. The closer to the principal point, the less distortion there is, or the weaker the indentation effect is. The farther away from the principal point, the closer to the edge, the more obvious the indentation is. Since a polynomial function is used to approximate this "indentation", as the pixel position moves away from the principal point, this approximation will gradually fail, that is, the error will become larger and larger, or even completely distorted.

[0124] In reality, as the pixel position moves away from the principal point, the actual distorted image boundary will gradually tend to be circular (assuming that the photosensitive element is infinite, that is, the resolution determined by the actual camera photosensitive element is not considered), and pixels will not overlap each other. However, if the function drawn according to the distortion parameter changes from monotonically increasing to monotonically decreasing as r (the distance between the pixel position and the principal point, normalized) gradually increases, this distortion phenomenon that is inconsistent with the actual physical process will occur.

[0125] Of course, it is also necessary to determine whether the r corresponding to the inflection point will appear in the actual picture resolution. If the r corresponding to the maximum FOV (i.e., the diagonal FOV) is smaller than the independent variable value corresponding to the inflection point of the distortion model function, then it means that within the image size range determined by the photosensitive element, the pixel position is normal, that is, the simulation effect at this time meets our needs. If the r corresponding to the maximum FOV is larger than the inflection point, it means that "involution" will definitely occur in the simulation picture. At this time, it is necessary to recalibrate the distortion parameters or make certain adjustments to the calibration process to obtain an image that is not excessively involution. That is, adjust the calibrated distortion parameters of the camera to be simulated so that the distance between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance between the pixel point corresponding to the maximum FOV and the image center, and obtain the adjusted distortion parameters of the camera to be simulated. In the specific implementation, in order to be closer to reality and further improve the accuracy of the simulation, it is best to make the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center much larger than the distance value between the pixel point corresponding to the maximum FOV and the image center, and let the distance value between the pixel point corresponding to the maximum FOV and the image center be distributed as much as possible in the first half of the function image.

[0126] After obtaining the adjusted distortion parameters of the camera to be simulated, the camera distortion can be virtually simulated according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, and the simulation result of the camera distortion can be obtained. Figure 6 As shown, including:

[0127] Step 601: Based on the adjusted distortion parameters of the camera to be simulated, a virtual simulation of camera distortion is performed on the image to be virtually simulated to obtain a simulated image;

[0128] Step 602: Cut out an image of the nominal resolution size of the camera to be simulated from the simulated image as a simulation result of the distortion of the camera to be simulated.

[0129] Since the original image after magnification will still be curled after being distorted by the camera, the "barrel-shaped" area obtained by distorting the enlarged original image can ensure that the original resolution of the camera to be simulated is covered. Therefore, the distorted image required for simulation is inside this "barrel-shaped" area. It is only necessary to cut out an image of the nominal resolution size of the camera to be simulated with the principal point as the center in the simulated image, which can be regarded as the image actually displayed by the camera to be simulated when shooting the target image.

[0130] Preferably, in order to further improve the accuracy of the simulation results, in a specific embodiment of the present invention, a verification and adjustment process is also added, such as Figure 7 As shown, in Figure 1 On the basis of

[0131] The image to be virtually simulated is used as the initial image, and the optimal image to be virtually simulated is determined cyclically. The following operations are performed in each cycle:

[0132] Step 701: Based on the calibrated distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image;

[0133] Step 702: cutting out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistorting and restoring the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image;

[0134] Step 703: Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to the preset value, determine the initial image as the optimal image to be virtually simulated; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle until the similarity is no lower than the preset value.

[0135] That is, after step 102, the simulation is verified to be reasonable through the step of distortion restoration, and the optimal image to be virtually simulated is determined, and then the simulation result of the distortion of the camera to be simulated is obtained through steps 103 and 104. Specifically, the distortion correction algorithm in the commonly used computer vision algorithm can be used to restore the image of the nominal resolution size of the camera to be simulated, and the calibrated distortion parameters are used to restore the restored image. From the specific actual operation, it can be known that a "pincushion-shaped" image slightly larger than the nominal resolution of the camera to be simulated can be obtained. This is the complete field of view of the camera when the simulated camera shoots the target scene. Of course, if only a rectangular image of the nominal resolution size of the camera to be simulated is needed, it is only necessary to copy the pixel part of the nominal resolution size of the camera to be simulated in the center part, and compare the copied image with the target image shot by the camera to be simulated. If the similarity between the two is not less than the preset value, for example, set to 90%, the rationality of the simulation can be verified at this time. If the similarity between the two is lower than the preset value, the resolution of the real shot image is readjusted, and the image of the nominal resolution size of the camera to be simulated is recut until the similarity is not lower than the preset value.

[0136] Similarly, in order to seek the best simulation result, in a specific embodiment, a new step may be added after step 104:

[0137] The image to be virtually simulated is used as the initial image, and the distortion parameters of the camera to be simulated after adjustment are used as the initial distortion parameters. The optimal image to be virtually simulated and the optimal distortion parameters are determined cyclically. The following operations are performed in each cycle:

[0138] Based on the initial distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image;

[0139] Cut out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistort and restore the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image;

[0140] Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to the preset value, the initial image is determined as the optimal image to be virtually simulated, and the initial distortion parameters are determined as the optimal distortion parameters; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle, and readjust the distortion parameters as the initial distortion parameters for the next cycle until the similarity is not lower than the preset value.

[0141] Based on the optimal image to be virtually simulated and the optimal distortion parameter, step 104 is re-executed to obtain the optimal simulation result.

[0142] The following is a specific example to illustrate how to perform virtual simulation of camera distortion in an embodiment of the present invention. This example is applied to a resolution of 1920×1080 and an internal parameter matrix of Virtual simulation of a real camera with calibrated distortion parameters k1 = -0.0355, k2 = -0.0114 and k3 = 0.0097.

[0143] So it can be obtained, We get I′=(1+k1r max 2 +k2r max 4 +k3r max 6 ) / r max ≈1.04, that is, the image actually required should be at least 1.04 times the resolution of the target image.

[0144] by Figure 8 The camera shown needs to capture a standard black and white chessboard, and perform a virtual simulation of camera distortion, where: Figure 8 The resolution of the image shown is consistent with the above-mentioned real camera resolution, which is 1920×1080. It is assumed that this is an image obtained by the camera to be simulated taking pictures of the real world, that is, the image actually taken by the camera to be simulated.

[0145] According to the traditional distortion simulation method, the obtained simulated image is as follows Fig. 9 As shown, the resolution remains unchanged at 1920×1080. It can be clearly seen that if the original resolution is maintained, there will inevitably be black edges in the image area.

[0146] Next, prepare an image with a larger FOV and the same imaging center, such as Fig.10 As shown in the figure, to simplify the calculation and show the idea more clearly, the resolution is directly doubled to 3840×2160, that is, the magnification ratio is 2, which is much larger than the 1.04 times of the above solution. Fig.10 It does not stretch the original image, but expands the imaging range, that is, assuming that the real camera keeps the pixel size unchanged (that is, f in the internal parameter matrix x and f y It remains unchanged), but has a higher resolution (more pixels). It can be understood that the camera originally used a photosensitive element with the same pixel size but a higher resolution.

[0147] Based on the distortion model function characteristics of the camera to be simulated, the calibration distortion parameters of the camera to be simulated are adjusted according to the maximum FOV of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated. Fig.10 Perform simulation and get Fig.11 From the simulation results shown, we can see that although the distorted image cannot fill the original resolution (3840×2160), after calculation, the enlarged "barrel-shaped" area can ensure that it covers the original resolution (1920×1080) of the camera to be simulated. Therefore, the required distorted simulation image is inside this "barrel-shaped" area. Fig.11 The center cuts out the original resolution size (1920×1080) of the picture, Fig.12 According to the optical principle of distortion, Fig.12 It requires a simulated camera to shoot Figure 8 The image actually displayed by the camera when the scene is shown.

[0148] The distortion reduction step is used to verify whether the above simulation is reasonable. The distortion correction algorithm in the commonly used computer vision algorithm can be used to Fig.12 Using the above distortion parameters to restore Fig.13 , note that in order to express and Fig.11 relationship, specifically used Fig.11 The resolution is 3840×2160. As you can see, a "pincushion-shaped" image slightly larger than the original resolution (1920×1080) is obtained. This is the image captured by the simulated camera. Figure 8 When the scene shown is actually within the camera's full field of view.

[0149] Of course, if you only need a rectangular image of the original resolution (1920×1080), you only need to copy the pixel portion of the original resolution (1920×1080) in the center, such as Fig.14 As shown, we can see that we have obtained the Figure 8 The completely consistent images verify the rationality of the simulation method used in this specific example.

[0150] It can be seen that Fig.12 In order to simulate the complete image required by the physical principles of distortion, if this image is used for subsequent de-distortion, restoration of scene objects and other tasks, it can meet the requirements and the results will not produce additional deviations.

[0151] After successful verification, the determined distortion parameters and the above-mentioned simulation steps can be input into the image rendering program in a specific programming language. When simulating the camera, the simulation result of the camera can be obtained by directly calling it.

[0152] Compared with the existing camera simulation methods, the method provided in this specific example reasonably adjusts the traditional camera simulation method based on the camera imaging principles and characteristics and relies on mathematical models to achieve realistic and reasonable simulation images in both principle and visualization effects, so that it can be better applied in various industrial simulations, algorithm experiments and solution demonstration links.

[0153] The implementation of the above specific application is only an example, and the remaining implementation methods will not be described in detail one by one.

[0154] Based on the same inventive concept, an embodiment of the present invention further provides a virtual simulation device for camera distortion. Since the principle of solving the problem by the virtual simulation device for camera distortion is similar to that of the virtual simulation method for camera distortion, the implementation of the virtual simulation device for camera distortion can refer to the implementation of the virtual simulation method for camera distortion, and the repeated parts will not be repeated. The specific structure is as follows: Fig.15 As shown:

[0155] The data acquisition module 1501 is used to acquire the calibration distortion parameters, internal parameters and the image actually taken by the camera to be simulated;

[0156] The original image determination module 1502 is used to determine the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibration distortion parameters and the internal parameters; according to the shrinkage ratio, determine the image resolution magnification ratio of the image actually shot after distortion that can meet the nominal resolution of the camera to be simulated; according to the magnification ratio, adjust the resolution of the image actually shot to obtain the image to be virtually simulated;

[0157] The distortion parameter adjustment module 1503 is used to adjust the calibrated distortion parameters of the camera to be simulated based on the distortion model function characteristics of the camera to be simulated and the maximum field of view of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated;

[0158] The distortion simulation module 1504 is used to perform virtual simulation of camera distortion according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, so as to obtain a simulation result of the distortion of the camera to be simulated.

[0159] In a specific embodiment, the original image determination module 1502 is specifically used to:

[0160] According to the internal parameter matrix of the camera to be simulated, the distance between the preset pixel point and the image center in the image actually taken by the camera to be simulated is determined;

[0161] According to the distance and the distortion model of the camera to be simulated, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined.

[0162] Further, in a specific embodiment, the original image determination module 1502 is specifically used to:

[0163] According to the magnification ratio and the nominal resolution of the camera to be simulated, the minimum image resolution that can meet the nominal resolution of the camera to be simulated after the distortion of the real captured image is obtained;

[0164] Based on the minimum image resolution, the resolution of the actually captured image is adjusted to determine the image to be virtually simulated.

[0165] In a specific embodiment, the distortion parameter adjustment module 1503 is specifically used to:

[0166] According to the maximum FOV of the camera to be simulated, determine the distance value between the pixel point corresponding to the maximum FOV and the center of the image;

[0167] Draw an image of the distortion model function of the camera to be simulated, and determine the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the center of the image;

[0168] Adjust the calibrated distortion parameters of the camera to be simulated so that the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance value between the pixel point corresponding to the maximum FOV and the image center, and obtain the adjusted distortion parameters of the camera to be simulated.

[0169] In a specific embodiment, in order to further improve the accuracy of the simulation, a virtual simulation device for camera distortion is provided, such as Fig.16 As shown, in Fig.15 On the basis of

[0170] The adjustment and correction module 1601 is used to:

[0171] The image to be virtually simulated is used as the initial image, and the optimal image to be virtually simulated is determined cyclically. The following operations are performed in each cycle:

[0172] Based on the calibrated distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image;

[0173] Cutting out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistorting and restoring the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image;

[0174] Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to the preset value, the initial image is determined as the optimal image to be virtually simulated; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle until the similarity is no lower than the preset value.

[0175] An embodiment of the present invention further provides a computer device, Fig.17 Schematic diagram of a computer device in an embodiment of the present invention. The computer device can implement all steps in the virtual simulation method of camera distortion in the above embodiment. The computer device specifically includes the following contents:

[0176] Processor (processor) 1701, memory (memory) 1702, communication interface (CommunicationsInterface) 1703 and communication bus 1704;

[0177] The processor 1701, the memory 1702, and the communication interface 1703 communicate with each other through the communication bus 1704; the communication interface 1703 is used to realize information transmission between related devices;

[0178] The processor 1701 is used to call the computer program in the memory 1702, and the processor implements the virtual simulation method of camera distortion in the above embodiment when executing the computer program.

[0179] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program for executing the above-mentioned virtual simulation method of camera distortion.

[0180] In summary, the camera distortion virtual simulation method and device provided by the embodiments of the present invention have the following advantages:

[0181] By obtaining the calibrated distortion parameters, internal parameters and the image actually taken by the camera to be simulated; determining the shrinkage ratio of preset pixel points in the image actually taken by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibrated distortion parameters and internal parameters; determining the image resolution magnification ratio of the image actually taken after distortion that can meet the nominal resolution of the camera to be simulated according to the shrinkage ratio, adjusting the resolution of the image actually taken according to the magnification ratio to obtain the image to be virtually simulated; based on the distortion model function characteristics of the camera to be simulated, adjusting the calibrated distortion parameters of the camera to be simulated according to the maximum field of view of the camera to be simulated, and obtaining the adjusted distortion parameters of the camera to be simulated; performing virtual simulation of camera distortion according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, and obtaining the simulation result of the distortion of the camera to be simulated. Compared with the technical solution in the prior art that performs simple and direct pixel position indentation based on the barrel or pincushion distortion model of the camera, based on the camera imaging principle and characteristics, relying on the mathematical model, the shrinkage ratio of the preset pixel point in the imaging distortion process of the camera to be simulated is determined, and the size of the image to be virtually simulated is reasonably adjusted based on the shrinkage ratio. The distortion parameters of the camera to be simulated are adjusted based on the distortion model function characteristics of the camera to be simulated. This can achieve true and reasonable image simulation in principle and visualization effect, make the simulation result more consistent with the distortion imaging effect of the actual camera, and thus improve the accuracy of the simulation result.

[0182] Although the present invention provides method operation steps as described in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment).

[0183] It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, devices (systems) or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0184] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0187] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. The orientation or position relationship indicated by the terms "upper", "lower", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise expressly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other in the absence of conflict. The present invention is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or permutation of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of the present invention can be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A virtual simulation method for camera distortion, characterized in that: include: Obtaining the calibration distortion parameters, internal parameters and the image actually taken by the camera to be simulated; According to the calibration distortion parameters and internal parameters, determining the shrinkage ratio of the preset pixel points in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated; According to the contraction ratio, determine the image resolution magnification ratio of the real captured image after distortion to meet the nominal resolution of the camera to be simulated, and adjust the resolution of the real captured image according to the magnification ratio to obtain the image to be virtually simulated; Based on the distortion model function characteristics of the camera to be simulated, and according to the maximum field of view of the camera to be simulated, the calibrated distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated; According to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, a virtual simulation of the camera distortion is performed to obtain a simulation result of the camera distortion to be simulated.

2. The virtual simulation method of camera distortion according to claim 1, characterized in that: Determining the shrinkage ratio of a preset pixel point in an image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibrated distortion parameter and the internal parameter, including: According to the internal parameter matrix of the camera to be simulated, the distance between the preset pixel point and the image center in the image actually taken by the camera to be simulated is determined; According to the distance and the distortion model of the camera to be simulated, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined.

3. The virtual simulation method of camera distortion according to claim 2, characterized in that: If the camera to be simulated is a pinhole camera, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined according to the following formula based on the distance and the distortion model of the camera to be simulated: I=(1+k1r 2 +k2r 4 +k3r 6 ) / r Among them, r represents the distance between the preset pixel point and the center of the image; I represents the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated; k1, k2 and k3 represent the calibrated distortion parameters of the camera to be simulated.

4. The virtual simulation method of camera distortion according to claim 2, characterized in that: If the camera to be simulated is a fisheye camera, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined according to the following formula based on the distance and the distortion model of the camera to be simulated: I=(tan(θ) / θ)(1+k1r 2 +k2r 4 +k3r 6 ) / r Among them, r represents the distance between the preset pixel point and the center of the image; I represents the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated; θ represents the angle between the incident light of the camera to be simulated and the main axis of the lens; k1, k2 and k3 represent the calibrated distortion parameters of the camera to be simulated.

5. The virtual simulation method of camera distortion according to claim 1, characterized in that: According to the magnification ratio, the resolution of the real captured image is adjusted to obtain the image to be virtually simulated, including: According to the magnification ratio and the nominal resolution of the camera to be simulated, the minimum image resolution of the real captured image that can meet the nominal resolution of the camera to be simulated after distortion is obtained; Based on the minimum image resolution, the resolution of the image actually shot is adjusted to determine the image to be virtually simulated.

6. The virtual simulation method of camera distortion according to claim 1, characterized in that: Based on the distortion model function characteristics of the camera to be simulated, and according to the maximum field of view of the camera to be simulated, the calibration distortion parameters of the camera to be simulated are adjusted to obtain the adjusted distortion parameters of the camera to be simulated, including: According to the maximum field of view of the camera to be simulated, determine the distance value between the pixel point corresponding to the maximum field of view and the image center; Draw an image of the distortion model function of the camera to be simulated, and determine the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the center of the image; Adjust the calibrated distortion parameters of the camera to be simulated so that the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance value between the pixel point corresponding to the maximum field of view and the image center, and obtain the adjusted distortion parameters of the camera to be simulated.

7. The virtual simulation method of camera distortion according to claim 1, characterized in that: Also includes: The image to be virtually simulated is used as the initial image, and the optimal image to be virtually simulated is determined cyclically. The following operations are performed in each cycle: Based on the calibrated distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image; Cutting out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistorting and restoring the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image; Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to a preset value, the initial image is determined as the optimal image to be virtually simulated; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle until the similarity is no lower than the preset value.

8. A virtual simulation device for camera distortion, characterized in that: include: A data acquisition module is used to obtain the calibration distortion parameters, internal parameters and the image actually taken by the camera to be simulated; An original image determination module, used to determine the shrinkage ratio of a preset pixel point in the image actually shot by the camera to be simulated during the imaging distortion process of the camera to be simulated according to the calibration distortion parameter and the internal parameter; According to the contraction ratio, determine the image resolution magnification ratio of the real captured image after distortion to meet the nominal resolution of the camera to be simulated, and adjust the resolution of the real captured image according to the magnification ratio to obtain the image to be virtually simulated; The distortion parameter adjustment module is used to adjust the calibrated distortion parameters of the camera to be simulated based on the distortion model function characteristics of the camera to be simulated and the maximum field of view of the camera to be simulated to obtain the adjusted distortion parameters of the camera to be simulated; The distortion simulation module is used to perform virtual simulation of camera distortion according to the image to be virtually simulated and the adjusted distortion parameters of the camera to be simulated, so as to obtain the simulation result of the distortion of the camera to be simulated.

9. The virtual simulation device for camera distortion according to claim 8, characterized in that: The original image determination module is specifically used for: According to the internal parameter matrix of the camera to be simulated, the distance between the preset pixel point and the image center in the image actually taken by the camera to be simulated is determined; According to the distance and the distortion model of the camera to be simulated, the shrinkage ratio of the preset pixel point during the imaging distortion process of the camera to be simulated is determined.

10. The virtual simulation device for camera distortion according to claim 8, characterized in that: The original image determination module is specifically used for: According to the magnification ratio and the nominal resolution of the camera to be simulated, the minimum image resolution of the real captured image that can meet the nominal resolution of the camera to be simulated after distortion is obtained; Based on the minimum image resolution, the resolution of the image actually shot is adjusted to determine the image to be virtually simulated.

11. The virtual simulation device for camera distortion according to claim 8, characterized in that: The distortion parameter adjustment module is specifically used for: According to the maximum field of view of the camera to be simulated, determine the distance value between the pixel point corresponding to the maximum field of view and the image center; Draw an image of the distortion model function of the camera to be simulated, and determine the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the center of the image; Adjust the calibrated distortion parameters of the camera to be simulated so that the distance value between the pixel point corresponding to the inflection point of the distortion model function of the camera to be simulated and the image center is greater than the distance value between the pixel point corresponding to the maximum field of view and the image center, and obtain the adjusted distortion parameters of the camera to be simulated.

12. The virtual simulation device for camera distortion according to claim 8, characterized in that: Also includes: Adjustment and correction module for: The image to be virtually simulated is used as the initial image, and the optimal image to be virtually simulated is determined cyclically. The following operations are performed in each cycle: Based on the calibrated distortion parameters, a virtual simulation of camera distortion is performed on the initial image to obtain a simulated image; Cutting out an image of the nominal resolution size of the camera to be simulated from the simulated image, and dedistorting and restoring the cut image of the nominal resolution size of the camera to be simulated according to the calibrated distortion parameter to obtain a restored image; Compare the restored image with the image actually taken by the camera to be simulated. If the similarity between the two is higher than or equal to a preset value, the initial image is determined as the optimal image to be virtually simulated; if the similarity between the two is lower than the preset value, readjust the resolution of the actually taken image to obtain a new image to be virtually simulated as the initial image for the next cycle until the similarity is no lower than the preset value.

13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 7.

Images