Methods for eliminating image distortion, electronic devices, storage media, and vehicles

Abstract

This application provides a method, electronic device, storage medium, and vehicle for eliminating image distortion. The method includes: calibrating multiple preset points in a constructed calibration coordinate system; determining a first pose relationship between the preset real camera coordinate system and the calibration coordinate system using the pixel coordinates of each of the multiple preset points in a preset real camera image; determining the shooting height of the real camera image based on the first pose relationship; calibrating the multiple preset points in a preset virtual camera coordinate system; determining a second pose relationship between the real camera coordinate system and the virtual camera coordinate system using the pixel coordinates of each of the multiple preset points; constructing a projection transformation relationship between the virtual camera and the real camera at the shooting height based on the second pose relationship; and mapping the real camera image to the virtual camera coordinate system and presenting it using the projection transformation relationship.

Description

Technical Field

[0001] Embodiments of this application relate to the field of image processing technology, and more particularly to a method for eliminating image distortion, electronic devices, storage media, and vehicles. Background Technology

[0002] For cameras with wide-angle capabilities, such as fisheye cameras used in vehicles, the images captured often contain huge distortions. On the other hand, in order to capture more information around the vehicle body, fisheye cameras have a downward tilt angle.

[0003] As a result of the above, after removing the distortion of the wide-angle camera itself, the image will have obvious perspective distortion.

[0004] Therefore, a solution is needed that can remove the distortion and perspective distortion inherent in wide-angle cameras. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method, electronic device, storage medium and vehicle for eliminating image distortion, so as to eliminate the distortion and perspective distortion generated by the wide-angle camera itself.

[0006] To achieve the above objectives, this application provides a method for eliminating image distortion, comprising:

[0007] Multiple preset points are calibrated in the constructed calibration coordinate system. The first pose relationship between the preset real camera coordinate system and the calibration coordinate system is determined by using the pixel coordinates of each of the multiple preset points in the preset real camera image.

[0008] Based on the first pose relationship, determine the shooting height of the real camera image;

[0009] In a preset virtual camera coordinate system, the plurality of preset points are marked, and the second pose relationship between the real camera coordinate system and the virtual camera coordinate system is determined using the pixel coordinates of each of the plurality of preset points.

[0010] Based on the second pose relationship, a projection transformation relationship between the virtual camera and the real camera at the shooting height is constructed, and the real camera image is mapped to the virtual camera coordinate system and presented using the projection transformation relationship.

[0011] Furthermore, before calibrating multiple preset points in the constructed calibration coordinate system, the following steps are taken:

[0012] Set the plurality of preset points;

[0013] The calibration coordinate system is constructed using any preset point as the origin;

[0014] The calibration coordinates of each of the plurality of preset points are determined in the calibration coordinate system.

[0015] Further, determining the first pose relationship between the preset real camera coordinate system and the calibration coordinate system includes:

[0016] Using preset camera intrinsic parameters, a first perspective projection relationship is constructed between the real camera coordinate system and the calibration coordinate system;

[0017] Based on the first perspective projection relationship, multiple first linear equations about the first camera extrinsic parameters are established using multiple pixel coordinates and multiple calibration coordinates, and the first camera extrinsic parameters are determined using the multiple first linear equations.

[0018] The first pose relationship is determined based on the first camera extrinsic parameters.

[0019] Further, determining the second pose relationship between the real camera coordinate system and the virtual camera coordinate system includes:

[0020] Using the camera intrinsic parameters, a second perspective projection relationship is constructed between the real camera coordinate system and the virtual coordinate system;

[0021] Based on the second perspective projection relationship, multiple second linear equations about the second camera extrinsic parameters are established using multiple pixel coordinates and multiple virtual camera coordinates, and the second camera extrinsic parameters are determined using the multiple second linear equations.

[0022] The second pose relationship is determined based on the second camera extrinsic parameters;

[0023] The coordinates of the multiple virtual cameras are obtained by calibrating the multiple preset points in the virtual camera coordinate system.

[0024] Furthermore, constructing the projection transformation relationship between the virtual camera and the real camera at the shooting height includes:

[0025] Determine the perpendicular vector between the origin of the real camera coordinate system and the preset plane;

[0026] A projection transformation matrix is ​​constructed using the vertical vector, the camera intrinsic parameters, the shooting height, and the second camera extrinsic parameters;

[0027] The projection transformation relationship is determined based on the projection transformation matrix.

[0028] Furthermore, mapping the real camera image to the virtual camera coordinate system using the aforementioned projection transformation relationship and then presenting it includes:

[0029] Determine the pixel coordinates of all pixels in the real camera image;

[0030] The pixel coordinates of all pixels in the real camera coordinate system are transformed to the virtual camera coordinate system according to the projection transformation relationship.

[0031] The image is then presented according to the converted pixel coordinates.

[0032] Based on the same inventive concept, this application also provides an apparatus for eliminating image distortion, including: a first pose relationship determination module, a shooting height determination module, a second pose relationship determination module, and a projection transformation module;

[0033] The first pose relationship determination module is configured to calibrate multiple preset points in the constructed calibration coordinate system, and use the pixel coordinates of the multiple preset points in the preset real camera image to determine the first pose relationship between the preset real camera coordinate system and the calibration coordinate system.

[0034] The shooting height determination module is configured to determine the shooting height of the real camera image based on the first pose relationship.

[0035] The second pose relationship determination module is configured to mark the plurality of preset points in a preset virtual camera coordinate system, and use the pixel coordinates of each of the plurality of preset points to determine the second pose relationship between the real camera coordinate system and the virtual camera coordinate system.

[0036] The projection transformation module is configured to construct a projection transformation relationship between the virtual camera and the real camera at the shooting height based on the second pose relationship, and to use the projection transformation relationship to map the real camera image onto the virtual camera coordinate system and present it.

[0037] Based on the same inventive concept, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image distortion elimination method as described in any of the above claims.

[0038] Based on the same inventive concept, this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions for causing the computer to perform the image distortion elimination method described above.

[0039] Based on the same inventive concept, this application also provides a vehicle, the vehicle including an image distortion correction device and an electronic device, the electronic device performing the image distortion correction method as described in any of the above claims.

[0040] As can be seen from the above, the image distortion elimination method provided in this application can establish a first pose relationship between the real camera coordinate system and the calibration coordinate system based on multiple preset points and a calibration coordinate system during real camera shooting. Through this first pose relationship, the shooting height of the real camera in the calibration coordinate system can be determined. At the same time, based on multiple preset points and a set virtual camera, a second pose relationship between the real camera coordinate system and the virtual camera coordinate system can be established. Based on this, by combining the second pose relationship and the shooting height, the projection transformation relationship between the real camera and the virtual camera can be determined. By using this projection transformation relationship, the real camera image captured by the real camera can be mapped onto the virtual camera to obtain an image with distortion eliminated. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A schematic diagram of the camera setup for a vehicle according to an embodiment of this application;

[0043] Figure 2 This is a flowchart of a method for eliminating image distortion according to an embodiment of this application;

[0044] Figure 3 This is a schematic diagram illustrating the setup of a virtual camera according to an embodiment of this application;

[0045] Figure 4 This is a flowchart illustrating the determination of the first pose relationship in an embodiment of this application;

[0046] Figure 5 This is a flowchart illustrating the determination of the second pose relationship in an embodiment of this application;

[0047] Figure 6 This is a flowchart illustrating the determination of projection transformation relationships in an embodiment of this application;

[0048] Figure 7 The original wide-angle camera image without distortion correction is an embodiment of this application.

[0049] Figure 8 The image provided is an embodiment of this application without perspective distortion correction.

[0050] Figure 9 This is a diagram illustrating the effect of image distortion removal according to an embodiment of this application.

[0051] Figure 10This is a schematic diagram of the structure of the image distortion elimination device according to an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the electronic device structure according to an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0054] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] As described in the background section, existing methods for eliminating image distortion are still insufficient to meet users' actual needs.

[0056] In the process of implementing this application, the applicant discovered that the main problem with the relevant methods for eliminating image distortion is that for cameras with wide-angle capabilities, such as fisheye cameras used in vehicles, the images captured often contain huge distortions. On the other hand, in order to capture more information around the vehicle body, fisheye cameras have a downward tilt angle.

[0057] Based on the above, after removing the distortion of the wide-angle camera itself, the image will have obvious perspective distortion, that is, the objects appear larger when they are closer and smaller when they are farther away. This manifests as objects that are perpendicular to the ground in the real world appearing severely tilted in the image.

[0058] The applicant also found in the research that the following three methods are often chosen for distortion correction: 1. Only perform distortion correction on the images captured by the fisheye camera; 2. Perform distortion correction on the images captured by the fisheye camera to a certain extent.

[0059] One drawback of method 1 is that the image after distortion correction still has perspective distortion. That is, objects that should be upright become slanted after distortion correction. For example, objects such as pillars appear tilted in the image, which can cause confusion for users.

[0060] Furthermore, for option 2, although some distortion correction was performed on the fisheye image, making the objects in the image appear curved, the overall visual effect is better, but it still does not conform to human visual habits.

[0061] Based on this, one or more embodiments of this application provide a method for eliminating image distortion to address the issue that images captured by a wide-angle camera contain both distortion from the camera itself and perspective distortion.

[0062] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0063] In embodiments of this application, a vehicle, as a specific example, is equipped with at least one camera, i.e., a camera. Figure 1 In the example shown, the vehicle has four cameras: a front camera, a rear camera, a left camera, and a right camera, which can capture front, rear, left, and right images from four different directions.

[0064] Each camera has its own built-in real camera coordinate system, with the camera's optical center serving as the origin of that system.

[0065] In this embodiment, taking any camera as an example, in order to enable the driver to observe a wider range through the camera, the camera can be a fisheye camera with wide-angle function, so that the captured image has a larger field of view than that of an ordinary camera.

[0066] Furthermore, when setting up cameras, taking the rear camera as an example, in order to enable it to collect more information around the vehicle body, the rear camera is set at a downward tilt angle.

[0067] refer to Figure 2 One embodiment of this application describes a method for eliminating image distortion, applied to a wide-angle fisheye camera, and specifically includes the following steps:

[0068] Step S201: In the constructed calibration coordinate system, multiple preset points are calibrated. Using the pixel coordinates of each of the multiple preset points in the preset real camera image, the first pose relationship between the preset real camera coordinate system and the calibration coordinate system is determined.

[0069] In the embodiments of this application, based on the constructed calibration coordinate system, the preset real camera coordinate system in the camera, and multiple preset points marked in advance, the PNP algorithm (Prespective N Point algorithm) can be used to determine the external parameters between the calibration coordinate system and the real camera coordinate system, that is, the first pose relationship between the calibration coordinate system and the real camera coordinate system.

[0070] Specifically, when the aforementioned rear camera is used as a real camera, a real camera image will be generated in the real camera coordinate system.

[0071] The real camera image contains a real image coordinate system, which can be set according to the real camera coordinate system. That is, through the pre-set camera intrinsic parameters, a fixed transformation relationship is formed between the real image coordinate system and the real camera coordinate system.

[0072] Based on this, by calibrating multiple preset points in the real camera coordinate system, the pixel coordinates of each preset point in the real image coordinate system can be determined.

[0073] In this embodiment, a calibration plate, also called a calibration cloth, is set around the vehicle. Figure 3 The image shows a calibration plate positioned behind the vehicle, near the rear camera, with the calibration plate laid on the ground.

[0074] Furthermore, in Figure 3 In the calibration board, select four corner points of each of the two black squares as preset points, that is, select a total of eight preset points.

[0075] In some other embodiments, other numbers may be selected, such as 6 preset points, to implement this method.

[0076] Furthermore, based on the multiple preset points selected above, a three-dimensional calibration coordinate system can be constructed by selecting any one of the preset points as the origin. It can be seen that this calibration coordinate system can be regarded as a coordinate system in the real world.

[0077] In a specific example, based on the selected 8 preset points, one can select... Figure 3 The top left corner is taken as the origin, and the direction perpendicular to the calibration plate, that is, the direction perpendicular to the ground, is taken as a dimension of the calibration coordinate system, that is, the direction of one of the coordinate axes. Based on this, it can be determined that the directions of the other two coordinate axes are parallel to the ground.

[0078] As can be seen, by selecting the origin and setting the direction of the coordinate axes, a calibration coordinate system can be obtained.

[0079] Furthermore, based on the constructed calibration coordinate system, after calibrating each preset point in the calibration coordinate system, the coordinates of each preset point in the calibration coordinate system can be obtained and used as the calibration coordinates.

[0080] In this embodiment, based on the pixel coordinates of each of the multiple preset points and the calibration coordinates of each of the multiple preset points, the PNP algorithm can be used to determine the extrinsic parameters between the real camera coordinate system and the calibration coordinate system, and the pose relationship between the real camera coordinate system and the calibration coordinate system, that is, the first pose relationship, can be determined using the extrinsic parameters.

[0081] Specifically, in a real camera, camera intrinsics can be preset or determined in advance.

[0082] Furthermore, based on this camera intrinsic parameter, a first perspective projection relationship between the real camera coordinate system and the calibration coordinate system can be constructed, and multiple first linear equations can be established using this first perspective projection relationship.

[0083] The first linear equation can be considered as a linear equation between the real camera coordinate system and the calibration coordinate system, relating to the extrinsic parameters of the first camera.

[0084] Specifically, the extrinsic parameters of the first camera include the first rotation matrix R1 and the first translation matrix t1.

[0085] Furthermore, by combining multiple first linear equations, the first camera extrinsic parameters can be determined. The first camera extrinsic parameters represent the calculation relationship between the two coordinate systems during the transformation. When this calculation relationship is substituted into the two coordinate systems, it is reflected as the first pose relationship between the two coordinate systems in terms of position.

[0086] As can be seen, the first pose relation indicates the positional relationship between the real camera coordinate system and the calibration coordinate system, and is specifically quantified by external parameters.

[0087] Step S202: Determine the shooting height of the real camera image based on the first pose relationship.

[0088] In the embodiments of this application, based on the first pose relationship determined above, the shooting height of the actual camera during shooting can be obtained by converting the corresponding first camera extrinsic parameters in the first pose relationship.

[0089] In a specific example, based on the first camera extrinsic parameters R1 and t1 mentioned above, by inverting the first rotation matrix R1, the first translation matrix t1 can be transformed into the calibration coordinate system, and its second dimension can be used as the shooting height of the rear camera when shooting.

[0090] Step S203: Mark the plurality of preset points in the preset virtual camera coordinate system, and use the pixel coordinates of each of the plurality of preset points to determine the second pose relationship between the real camera coordinate system and the virtual camera coordinate system.

[0091] In the embodiments of this application, based on the constructed virtual camera coordinate system, the preset real camera coordinate system in the camera, and the aforementioned multiple preset points, the PNP algorithm (Prespective N Point algorithm) can be used to determine the external parameters between the virtual camera coordinate system and the real camera coordinate system, that is, the first pose relationship between the virtual camera coordinate system and the real camera coordinate system.

[0092] Specifically, by setting up a virtual camera at the shooting position of a real camera, the perspective of a human eye looking straight ahead can be simulated.

[0093] Furthermore, based on the set virtual camera, the virtual camera coordinate system can be determined.

[0094] In a specific example, Figure 3 The instructions for setting up a virtual camera are shown, where the virtual camera is set to... Figure 3 The intersection point of the X, Y, and Z axes is the origin of the virtual camera coordinate system.

[0095] As can be seen, the virtual camera coordinate system is identical to the real camera coordinate system. Its X and Z axes are parallel to the ground, that is, parallel to the calibration plate, while its Y axis is perpendicular to the ground.

[0096] Based on this, multiple preset points can be calibrated in the virtual camera coordinate system, and the virtual camera coordinates of each preset point in the virtual camera coordinate system can be obtained respectively.

[0097] In this embodiment, based on the pixel coordinates of each of the multiple preset points and the virtual camera coordinates of each of the multiple preset points, the PNP algorithm can be used to determine the extrinsic parameters between the real camera coordinate system and the virtual camera coordinate system, and the pose relationship between the real camera coordinate system and the virtual camera coordinate system, i.e., the second pose relationship, can be determined using the extrinsic parameters.

[0098] Specifically, based on the intrinsic parameters of the real camera in the aforementioned steps, a second perspective projection relationship between the real camera coordinate system and the virtual camera coordinate system can be constructed, and multiple second linear equations can be established by further utilizing this second perspective projection relationship.

[0099] The second linear equation can be considered as a linear equation between the real camera coordinate system and the virtual camera coordinate system, relating to the second camera's extrinsic parameters.

[0100] Specifically, the extrinsic parameters of the second camera include the second rotation matrix R2 and the second translation matrix t2.

[0101] Furthermore, by combining multiple second linear equations, the second camera extrinsic parameters can be determined. These second camera extrinsic parameters represent the computational relationship between the two coordinate systems during the transformation. When this computational relationship is substituted into the two coordinate systems, it is reflected as the second pose relationship between the two coordinate systems in terms of position.

[0102] As can be seen, the second pose relationship indicates the positional relationship between the real camera coordinate system and the virtual camera coordinate system, and is specifically quantified by extrinsic parameters.

[0103] Step S204: Based on the second pose relationship, construct the projection transformation relationship between the virtual camera and the real camera at the shooting height, and use the projection transformation relationship to map the real camera image onto the virtual camera coordinate system and present it.

[0104] In the embodiments of this application, based on the second pose relationship determined above, a coordinate transformation relationship, i.e. a projection transformation relationship, can be constructed between the virtual camera coordinate system and the real camera coordinate system.

[0105] Specifically, based on the shooting height determined above, the projection transformation relationship can be described by constructing a projection transformation matrix.

[0106] In a specific example, since the virtual camera simulates the perspective of a human eye looking straight ahead, a plane perpendicular to the ground can be set as the target plane observed by the virtual camera that simulates the perspective of a human eye.

[0107] Furthermore, the target plane can be a plane at a distance from the shooting position, and the specific vertical distance between the target plane and the actual camera can be set according to the specific shooting requirements of the camera.

[0108] Furthermore, based on the vertical distance, the second camera extrinsic parameters, the shooting height, and the preset camera intrinsic parameters, a projection transformation matrix between the real camera and the virtual camera can be constructed.

[0109] The projection transformation matrix specifically describes the transformation relationship between the real camera image obtained by the real camera and the image expected to be presented by the virtual camera, i.e., the transformation relationship between the real camera image and the virtual camera image. In this embodiment, this transformation relationship is taken as the projection transformation relationship.

[0110] Specifically, based on a given real camera image, the pixel coordinates of all pixels in the real camera image can be determined.

[0111] Furthermore, through the projection transformation relationship between the real camera and the virtual camera, the pixel coordinates of all pixels in the real camera image can be transformed according to the proportional relationship of the projection transformation matrix, and new pixel coordinates can be obtained after mapping.

[0112] For example, the pixel coordinates of each pixel in the real camera image can be multiplied by the projection transformation matrix, and the resulting new pixel coordinates can be used as the pixel coordinates in the virtual camera image, and the image can be presented according to the new pixel coordinates.

[0113] As can be seen, through projection transformation, a real camera image can be mapped to a virtual camera image at a fixed ratio to remove image distortion in the real camera image.

[0114] In another embodiment of this application, such as Figure 4 As shown, determining the first pose relationship between the preset real camera coordinate system and the calibration coordinate system may include the following steps:

[0115] Step S401: Using preset camera intrinsic parameters, construct a first perspective projection relationship between the real camera coordinate system and the calibration coordinate system.

[0116] In the embodiments of this application, based on the determined camera intrinsic parameters of the real camera, a first perspective projection matrix between the real camera coordinate system and the calibration coordinate system can be established using the secondary coordinate method, so as to quantitatively describe the first perspective projection relationship.

[0117] Specifically, the coordinates of the preset point in the calibration coordinate system are represented as: [X w Y w Z w ] T The pixel coordinates are represented as: [u c v c ] T .

[0118] Based on this, the homogeneous coordinates of the calibration coordinates can be further represented as: [X w Y w Z w ,1] T The homogeneous coordinates of the pixel coordinates are represented as: [u, v, 1] T .

[0119] Furthermore, denoting the intrinsic parameter matrix of the real camera as K, for each of the multiple preset points, the first perspective projection matrix about that preset point can be constructed using the first rotation matrix R1 and the first translation matrix t1 in the camera's intrinsic parameters, as shown below:

[0120]

[0121] Among them, z c Indicates the position of the actual camera coordinates relative to u c and v c The coordinates of the preset point are on a different coordinate axis.

[0122] Based on this, the calculation relationship between the homogeneous coordinates of pixel coordinates and the homogeneous coordinates of calibration coordinates can be used to describe the transformation relationship between the real camera coordinate system and the calibration coordinate system, that is, the first perspective projection relationship.

[0123] Step S402: Based on the first perspective projection relationship, using multiple pixel coordinates and multiple calibration coordinates, establish multiple first linear equations about the extrinsic parameters of the first camera, and use the multiple first linear equations to determine the extrinsic parameters of the second camera.

[0124] In the embodiments of this application, based on the first perspective projection relationship determined above, the first camera extrinsic parameters, namely the first rotation matrix R1 and the first translation matrix t1, can be solved by combining multiple first perspective projection matrices after organizing the first perspective projection matrix.

[0125] Specifically, for the first perspective projection matrix mentioned above, the camera intrinsic parameter K is multiplied by... In the middle, it can be expanded into the following expanded form:

[0126]

[0127] Furthermore, where f represents multiplying K by... The intermediate parameters are then obtained, and their positions in the matrix are indicated by subscripts.

[0128] Furthermore, the above expansion can be rearranged into the following system of equations:

[0129] z c u c =f 11 X w +f 12 Y w +f 13 Z w +f 14

[0130] z c v c =f 21 X w +f 22 Y w +f 23 Z w+f 24

[0131] z c =f 31 X w +f 32 Y w +f 33 Z w +f 34

[0132] Based on this, for each preset point, the real camera coordinates in the real camera coordinate system and the corresponding pixel coordinates can be determined, thereby obtaining a set of matching points between three-dimensional and two-dimensional coordinates for that preset point.

[0133] Furthermore, based on multiple preset points, multiple sets of matching points can be determined. Using these multiple sets of matching points, multiple sets of the above-mentioned equations can be established. By solving these multiple sets of equations simultaneously, the first rotation matrix R1 and the first translation matrix t1 can be obtained, which means the first camera extrinsic parameters can be determined.

[0134] Step S403: Determine the first pose relationship based on the first camera extrinsic parameters.

[0135] In the embodiments of this application, based on the first camera extrinsic parameter determined above, the camera extrinsic parameter can be used to quantitatively describe the first pose relationship between the real camera coordinate system and the calibration coordinate system.

[0136] It can be seen that the first perspective projection matrix, constructed based on the homogeneous coordinates of the pixel coordinates and the homogeneous coordinates of the calibration coordinates, describes the computational transformation relationship between the real camera coordinate system and the calibration coordinate system, and specifically expresses the quantification of the relationship between the two coordinate systems in perspective projection. Based on this, the first camera extrinsic parameters can be solved, and the first pose relationship can be determined accordingly.

[0137] In another embodiment of this application, such as Figure 5 As shown, determining the second pose relationship between the real camera coordinate system and the virtual camera coordinate system may include the following steps:

[0138] Step S501: Using the camera intrinsic parameters, construct a second perspective projection relationship between the real camera coordinate system and the virtual coordinate system.

[0139] In the embodiments of this application, based on the determined camera intrinsic parameters of the real camera, a second perspective projection matrix between the real camera coordinate system and the virtual camera coordinate system can be established using secondary coordinates to quantitatively describe the second perspective projection relationship.

[0140] Specifically, the coordinates of the preset point in the virtual camera coordinate system are represented as: [Xv Y v Z v ] T The pixel coordinates are represented as: [u c v c ] T .

[0141] Based on this, the homogeneous coordinates of the virtual camera can be further represented as: [X v Y v Z v ,1] T The homogeneous coordinates of the pixel coordinates are represented as: [u, v, 1] T .

[0142] Furthermore, denoting the intrinsic parameter matrix of the real camera as K, for each of the multiple preset points, the second perspective projection matrix about that preset point can be constructed using the second rotation matrix R2 and the second translation matrix t2 in the camera's intrinsic parameters, as shown below:

[0143]

[0144] Among them, z c Indicates the position of the actual camera coordinates relative to u c and v c The coordinates of the preset point are on a different coordinate axis.

[0145] Based on this, the calculation relationship between the homogeneous coordinates of pixel coordinates and the homogeneous coordinates of virtual camera coordinates can be used to describe the transformation relationship between the real camera coordinate system and the virtual camera coordinate system, that is, the second perspective projection relationship.

[0146] Step S502: Based on the second perspective projection relationship, using multiple pixel coordinates and multiple virtual camera coordinates, establish multiple second linear equations about the second camera extrinsic parameters, and use the multiple second linear equations to determine the second camera extrinsic parameters.

[0147] In the embodiments of this application, based on the second perspective projection relationship determined above, the second camera extrinsic parameters, namely the second rotation matrix R2 and the second translation matrix t2, can be solved by combining multiple second perspective projection matrices after organizing the second perspective projection matrix.

[0148] Specifically, for the second perspective projection matrix mentioned above, the camera intrinsic parameter K is multiplied by... In the middle, it can be expanded into the following expanded form:

[0149]

[0150] Furthermore, where f′ represents multiplying K by... The intermediate parameters are then obtained, and their positions in the matrix are indicated by subscripts.

[0151] Furthermore, the above expansion can be rearranged into the following system of equations:

[0152] z c u c =f′ 11 X w +f′ 12 Y w +f′ 13 Z w +f′ 14

[0153] z c v c =f′ 21 X w +f′ 22 Y w +f′ 23 Z w +f′ 24

[0154] z c =f′ 31 X w +f′ 32 Y w +f′ 33 Z w +f′ 34

[0155] Based on this, for each preset point, the real camera coordinates in the real camera coordinate system and the corresponding pixel coordinates can be determined, thereby obtaining a set of matching points between three-dimensional and two-dimensional coordinates for that preset point.

[0156] Furthermore, based on multiple preset points, multiple sets of matching points can be determined. Using these multiple sets of matching points, multiple sets of the above-mentioned equations can be established. By solving these multiple sets of equations simultaneously, the second rotation matrix R2 and the second translation matrix t2 can be obtained, which means determining the extrinsic parameters of the second camera.

[0157] Step S503: Determine the second pose relationship based on the second camera extrinsic parameters.

[0158] In the embodiments of this application, based on the second camera extrinsic parameters determined above, the camera extrinsic parameters can be used to quantify and describe the second pose relationship between the real camera coordinate system and the virtual camera coordinate system.

[0159] It can be seen that the second perspective projection matrix, constructed based on the homogeneous coordinates of pixel coordinates and the homogeneous coordinates of virtual camera coordinates, describes the computational transformation relationship between the real camera coordinate system and the virtual camera coordinate system, and specifically expresses the quantification of the relationship between the two coordinate systems in perspective projection. Based on this, the second camera extrinsic parameters can be solved, and the second pose relationship can be determined accordingly.

[0160] In another embodiment of this application, such as Figure 6 As shown, constructing the projection transformation relationship between the virtual camera and the real camera at the shooting height may include the following steps:

[0161] Step S601: Determine the perpendicular vector between the origin of the real camera coordinate system and the preset plane.

[0162] In this embodiment, based on the preset plane, the vertical distance between the real camera and the preset plane can be determined, that is, the distance from the origin of the real camera coordinate system to the preset plane.

[0163] In the specific example calculation, this vertical distance can be regarded as a vertical vector describing the optical center of the real camera to the preset plane.

[0164] Step S602: Construct a projection transformation matrix using the vertical vector, the camera intrinsic parameters, the shooting height, and the second camera extrinsic parameters.

[0165] In this embodiment, based on the vertical vector, camera intrinsic parameters, and shooting height determined above, the following projection transformation matrix can be constructed:

[0166]

[0167] Where H represents the projection transformation matrix, K represents the camera intrinsic parameters, R2 represents the second rotation matrix, t2 represents the second translation matrix, T represents the transpose of the matrix, N represents the vertical vector, and d represents the shooting height.

[0168] Step S603: Determine the projection transformation relationship based on the projection transformation matrix.

[0169] In this embodiment, the projection transformation matrix determined above can be used to specifically quantify the calculation relationship when mapping from a real camera to a virtual camera, that is, the projection transformation relationship.

[0170] As can be seen, projection transformation considers the projection between planes. By selecting a plane that is perpendicular to the ground in the distance, it can directly affect the distorted parts in the real camera image. Selecting a plane that is perpendicular to the ground in the real world can make objects that are perpendicular to the ground appear perpendicular to the ground in the image after projection transformation, which is consistent with human visual habits.

[0171] As can be seen, the image distortion elimination method of the embodiments of this application, when shooting with a real camera, can establish a first pose relationship between the real camera coordinate system and the calibration coordinate system based on multiple preset points and a calibration coordinate system. Through this first pose relationship, the shooting height of the real camera in the calibration coordinate system can be determined. At the same time, based on multiple preset points and a set virtual camera, a second pose relationship between the real camera coordinate system and the virtual camera coordinate system can be established. Based on this, by combining the second pose relationship and the shooting height, the projection transformation relationship between the real camera and the virtual camera can be determined. By using this projection transformation relationship, the real camera image captured by the real camera can be mapped onto the virtual camera to obtain an image with distortion eliminated.

[0172] In specific image examples Figure 7 The image shown is an unprocessed raw image taken with a wide-angle fisheye camera. It can be seen that the objects in the raw image are greatly distorted without any processing.

[0173] Furthermore, Figure 8 The image shown is an image after the camera distortion of the wide-angle fisheye camera itself has been eliminated. It can be seen that before the perspective distortion was eliminated, objects in the image, especially distant planes, still have huge perspective distortion, to the point that walls that should be perpendicular to the ground appear skewed, causing viewers to misunderstand the actual environment when observing the image.

[0174] Furthermore, Figure 9 The image shown is the result of implementing the image distortion elimination method of this application. It can be seen that, based on Figure 8 Based on this, after eliminating perspective distortion, the planes perpendicular to the ground in the image, especially the walls in the distance, present an effect that conforms to human vision, restoring the real shooting environment.

[0175] It should be noted that the method of the embodiments of this application can be executed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method of the embodiments of this application, and the multiple devices will interact with each other to complete the method described.

[0176] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0177] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, the embodiments of this application also provide an apparatus for eliminating image distortion.

[0178] refer to Figure 10 The device for eliminating image distortion includes: a first pose relationship determination module 1001, a shooting height determination module 1002, a second pose relationship determination module 1003, and a projection transformation module 1004.

[0179] The first pose relationship determination module 1001 is configured to calibrate multiple preset points in the constructed calibration coordinate system, and use the pixel coordinates of each of the multiple preset points in the preset real camera image to determine the first pose relationship between the preset real camera coordinate system and the calibration coordinate system.

[0180] The shooting height determination module 1002 is configured to determine the shooting height of the real camera image based on the first pose relationship.

[0181] The second pose relationship determination module 1003 is configured to mark the plurality of preset points in a preset virtual camera coordinate system, and use the pixel coordinates of each of the plurality of preset points to determine the second pose relationship between the real camera coordinate system and the virtual camera coordinate system.

[0182] The projection transformation module 1004 is configured to construct a projection transformation relationship between the virtual camera and the real camera at the shooting height based on the second pose relationship, and to use the projection transformation relationship to map the real camera image onto the virtual camera coordinate system and present it.

[0183] As an optional embodiment, the first pose relationship determination module 1001 is specifically configured as follows:

[0184] Set the plurality of preset points;

[0185] The calibration coordinate system is constructed using any preset point as the origin;

[0186] The calibration coordinates of each of the plurality of preset points are determined in the calibration coordinate system.

[0187] Furthermore, using preset camera intrinsic parameters, a first perspective projection relationship is constructed between the real camera coordinate system and the calibration coordinate system;

[0188] Based on the first perspective projection relationship, multiple first linear equations about the first camera extrinsic parameters are established using multiple pixel coordinates and multiple calibration coordinates, and the first camera extrinsic parameters are determined using the multiple first linear equations.

[0189] The first pose relationship is determined based on the first camera extrinsic parameters.

[0190] As an optional embodiment, the second pose relationship determination module 1003 is specifically configured as follows:

[0191] Using the camera intrinsic parameters, a second perspective projection relationship is constructed between the real camera coordinate system and the virtual coordinate system;

[0192] Based on the second perspective projection relationship, multiple second linear equations about the second camera extrinsic parameters are established using multiple pixel coordinates and multiple virtual camera coordinates, and the second camera extrinsic parameters are determined using the multiple second linear equations.

[0193] The second pose relationship is determined based on the second camera extrinsic parameters;

[0194] The coordinates of the multiple virtual cameras are obtained by calibrating the multiple preset points in the virtual camera coordinate system.

[0195] As an optional embodiment, the projection transformation module 1004 is specifically configured as follows:

[0196] Determine the perpendicular vector between the origin of the real camera coordinate system and the preset plane;

[0197] A projection transformation matrix is ​​constructed using the vertical vector, the camera intrinsic parameters, the shooting height, and the second camera extrinsic parameters;

[0198] The projection transformation relationship is determined based on the projection transformation matrix.

[0199] Furthermore, the pixel coordinates of all pixels in the real camera image are determined;

[0200] The pixel coordinates of all pixels in the real camera coordinate system are transformed to the virtual camera coordinate system according to the projection transformation relationship.

[0201] The image is then presented according to the converted pixel coordinates.

[0202] For ease of description, the above apparatus is described in terms of its functions, divided into various modules. Of course, in implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0203] The apparatus of the above embodiments is used to implement the corresponding image distortion elimination method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0204] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for eliminating image distortion as described in any of the above embodiments.

[0205] Figure 11 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0206] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0207] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0208] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0209] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0210] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0211] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0212] The apparatus of the above embodiments is used to implement the corresponding image distortion elimination method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0213] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a vehicle, the vehicle including an image distortion correction device and an electronic device, the electronic device performing the image distortion correction method as described in any of the above claims.

[0214] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a computer-readable storage medium storing computer instructions for causing the computer to perform the image distortion elimination method as described in any of the above embodiments.

[0215] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0216] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to perform the image distortion elimination method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0217] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0218] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0219] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0220] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A method for eliminating image distortion, characterized in that, include: Multiple preset points are calibrated in the constructed calibration coordinate system. The first pose relationship between the preset real camera coordinate system and the calibration coordinate system is determined by using the pixel coordinates of each of the multiple preset points in the preset real camera image. Based on the first pose relationship, determine the shooting height of the real camera image; In a preset virtual camera coordinate system, the plurality of preset points are marked, and the second pose relationship between the real camera coordinate system and the virtual camera coordinate system is determined using the pixel coordinates of each of the plurality of preset points. Based on the second pose relationship, a projection transformation relationship between the virtual camera and the real camera at the shooting height is constructed, and the real camera image is mapped to the virtual camera coordinate system and presented using the projection transformation relationship; The construction of the projection transformation relationship between the virtual camera and the real camera at the shooting height includes: Using preset camera intrinsic parameters, a second perspective projection relationship is constructed between the real camera coordinate system and the virtual camera coordinate system; Based on the second perspective projection relationship, multiple second linear equations about the second camera extrinsic parameters are established using multiple pixel coordinates and multiple virtual camera coordinates. The second rotation matrix and the second translation matrix are then determined using the multiple second linear equations, and the second rotation matrix and the second translation matrix are used as the second camera extrinsic parameters. Determine the perpendicular vector between the origin of the real camera coordinate system and the preset plane; A projection transformation matrix is ​​constructed using the vertical vector, the camera intrinsic parameters, the shooting height, and the second camera extrinsic parameters; The formula for the projection transformation matrix H is: Where H represents the projection transformation matrix and K represents the camera intrinsic parameters. Let t1 represent the second rotation matrix, t2 represent the second translation matrix, T represent the transpose of the matrix, N represent the vertical vector, and d represent the shooting height. The projection transformation relationship is determined based on the projection transformation matrix.

2. The method according to claim 1, characterized in that, Before calibrating multiple preset points in the constructed calibration coordinate system, the process includes: Set the plurality of preset points; The calibration coordinate system is constructed using any preset point as the origin; The calibration coordinates of each of the plurality of preset points are determined in the calibration coordinate system.

3. The method according to claim 1, characterized in that, The determination of the first pose relationship between the preset real camera coordinate system and the calibration coordinate system includes: Using preset camera intrinsic parameters, a first perspective projection relationship is constructed between the real camera coordinate system and the calibration coordinate system; Based on the first perspective projection relationship, multiple first linear equations about the first camera extrinsic parameters are established using multiple pixel coordinates and multiple calibration coordinates, and the first camera extrinsic parameters are determined using the multiple first linear equations. The first pose relationship is determined based on the first camera extrinsic parameters.

4. The method according to claim 3, characterized in that, Determining the second pose relationship between the real camera coordinate system and the virtual camera coordinate system includes: Using the camera intrinsic parameters, a second perspective projection relationship is constructed between the real camera coordinate system and the virtual camera coordinate system; Based on the second perspective projection relationship, multiple second linear equations about the second camera extrinsic parameters are established using multiple pixel coordinates and multiple virtual camera coordinates, and the second camera extrinsic parameters are determined using the multiple second linear equations. The second pose relationship is determined based on the second camera extrinsic parameters; The coordinates of the multiple virtual cameras are obtained by calibrating the multiple preset points in the virtual camera coordinate system.

5. The method according to claim 1, characterized in that, The step of mapping the real camera image to the virtual camera coordinate system using the projection transformation relationship and then presenting it includes: Determine the pixel coordinates of all pixels in the real camera image; The pixel coordinates of all pixels in the real camera coordinate system are transformed to the virtual camera coordinate system according to the projection transformation relationship. The image is then presented according to the converted pixel coordinates.

6. An apparatus for eliminating image distortion, characterized in that, include: The module includes a first pose relationship determination module, a shooting height determination module, a second pose relationship determination module, and a projection transformation module. The first pose relationship determination module is configured to calibrate multiple preset points in the constructed calibration coordinate system, and use the pixel coordinates of the multiple preset points in the preset real camera image to determine the first pose relationship between the preset real camera coordinate system and the calibration coordinate system. The shooting height determination module is configured to determine the shooting height of the real camera image based on the first pose relationship. The second pose relationship determination module is configured to mark the plurality of preset points in a preset virtual camera coordinate system, and use the pixel coordinates of each of the plurality of preset points to determine the second pose relationship between the real camera coordinate system and the virtual camera coordinate system. The projection transformation module is configured to construct a projection transformation relationship between the virtual camera and the real camera at the shooting height based on the second pose relationship, and use the projection transformation relationship to map the real camera image onto the virtual camera coordinate system and present it. The projection transformation module is further configured to: Using preset camera intrinsic parameters, a second perspective projection relationship is constructed between the real camera coordinate system and the virtual camera coordinate system; Based on the second perspective projection relationship, multiple second linear equations about the second camera extrinsic parameters are established using multiple pixel coordinates and multiple virtual camera coordinates. The second rotation matrix and the second translation matrix are then determined using the multiple second linear equations, and the second rotation matrix and the second translation matrix are used as the second camera extrinsic parameters. Determine the perpendicular vector between the origin of the real camera coordinate system and the preset plane; A projection transformation matrix is ​​constructed using the vertical vector, the camera intrinsic parameters, the shooting height, and the second camera extrinsic parameters; The formula for the projection transformation matrix H is: Where H represents the projection transformation matrix and K represents the camera intrinsic parameters. Let t1 represent the second rotation matrix, t2 represent the second translation matrix, T represent the transpose of the matrix, N represent the vertical vector, and d represent the shooting height. The projection transformation relationship is determined based on the projection transformation matrix.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1 to 5.

9. A vehicle, characterized in that, Includes the image distortion elimination device as described in claim 6 or the electronic device as described in claim 7.

Images