Calibration method, panoramic image generation method, apparatus, equipment and storage medium

Abstract

This disclosure provides a calibration method, a method for generating a panoramic image, an apparatus, a device, and a storage medium. The calibration method includes acquiring point cloud data corresponding to a first feature point in a calibration object placed on the ground where the vehicle is located; acquiring an image of the calibration object captured by a fisheye camera mounted on the vehicle, and extracting a second feature point corresponding to the first feature point from the calibration object image; transforming the first feature point onto the unit sphere of the fisheye camera using its extrinsic parameters to obtain a first projection point; and projecting the second feature point onto the unit sphere of the fisheye camera using its intrinsic parameters to obtain a second projection point. By projecting the first and second feature points onto the unit sphere of the fisheye camera, the reprojection error is calculated. Compared with the reprojection error calculated by projecting the first feature point onto the acquired calibration object image, this method reduces the influence of fisheye camera distortion on the reprojection error and improves the accuracy of the calibration extrinsic parameters.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent driving technology, and in particular to calibration methods, surround view image generation methods, devices, equipment and storage media. Background Technology

[0002] A vehicle's surround-view system can monitor the area around the vehicle, assisting users in identifying blind spots during driving and parking. The system works by stitching together images captured by multiple fisheye cameras on the vehicle, after parameter conversion of the fisheye cameras.

[0003] In related technologies, fisheye camera parameters are usually calibrated by projecting manually measured ground coordinates onto the images acquired by the fisheye camera. This method has low accuracy and results in poor quality of the stitched panoramic images. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a calibration method, a method for generating panoramic images, an apparatus, a device, and a storage medium.

[0005] According to a first aspect of this disclosure, a calibration method for a fisheye camera is provided, wherein the fisheye camera is mounted on a vehicle, and a calibration object is placed on the ground where the vehicle is located, the calibration object being within the field of view of the fisheye camera, the method comprising:

[0006] Obtain the point cloud data corresponding to the first feature point in the calibration object;

[0007] Acquire the calibration object image captured by the fisheye camera, and obtain the second feature point corresponding to the first feature point from the calibration object image;

[0008] The first feature point is transformed onto the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain the first projection point, and the second feature point is projected onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point.

[0009] The extrinsic parameters of the fisheye camera are optimized based on the difference between the first projection point and the second projection point.

[0010] In any embodiment, optimizing the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point includes:

[0011] By changing the extrinsic parameters of the fisheye camera, the first projection point is changed;

[0012] The extrinsic parameter corresponding to the minimum difference between the first projection point and the second projection point is determined as the extrinsic parameter of the fisheye camera.

[0013] In any embodiment, the step of projecting the second feature point onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point includes:

[0014] The second feature point is projected onto the camera coordinate system based on the intrinsic parameters of the fisheye camera to obtain the distorted coordinates of the second feature point in the camera coordinate system.

[0015] The distortion correction coefficient is determined by iterating the projection incident angle of the second feature point multiple times.

[0016] The distortion coordinates are distorted according to the distortion correction coefficients to obtain the second projection point of the second feature point onto the unit sphere of the fisheye camera.

[0017] In any embodiment, the method further includes:

[0018] Acquire the image of the calibration object captured by the fisheye camera;

[0019] Obtain the second feature point from the calibration object image, and obtain the first calibration space point in the camera coordinate system corresponding to the second feature point;

[0020] The first calibration spatial point is projected onto the calibration object image using the intrinsic parameters and distortion of the fisheye camera to obtain the first calibration projection point. The distortion is determined based on the intrinsic parameters and the projection incident angle, and the projection incident angle is determined based on the inverse cosine of the distance between the first calibration spatial point and the optical center of the fisheye camera.

[0021] The intrinsic parameters of the fisheye camera are determined based on the second feature point and the first calibration projection point.

[0022] According to a second aspect of this disclosure, a method for generating a panoramic image is provided, the method comprising:

[0023] Acquire the first images captured by multiple fisheye cameras on the vehicle from their respective acquisition perspectives;

[0024] The first image of the fisheye camera is converted into a corresponding top view image according to the mapping relationship, wherein the mapping relationship is obtained according to the calibration method described in any of the above embodiments;

[0025] The top-view images corresponding to each first image are stitched together to obtain the panoramic image.

[0026] In any embodiment, the method further includes:

[0027] Establish a vehicle coordinate system with the vehicle as the origin;

[0028] Obtain the first pixel on the ground image;

[0029] The first pixel is transformed into the vehicle coordinate system to obtain the first spatial point of the first pixel in the vehicle coordinate system.

[0030] Based on the extrinsic parameters of the fisheye camera, the first spatial point is transformed into the camera coordinate system to obtain the second spatial point;

[0031] The second spatial point is converted into the fisheye image captured by the fisheye camera based on the camera's intrinsic parameters to obtain the second pixel point;

[0032] The third pixel corresponding to the second pixel in the image captured by the fisheye camera is obtained by bilinear interpolation.

[0033] Construct a mapping relationship between the first pixel on the ground image and the third pixel on the fisheye image.

[0034] According to a third aspect of this disclosure, a calibration device for a fisheye camera is provided, the fisheye camera being mounted on a vehicle, and a calibration object being placed on the ground where the vehicle is located, the calibration object being within the field of view of the fisheye camera, the device comprising:

[0035] The first acquisition unit is used to acquire point cloud data corresponding to the first feature point in the calibration object, as well as the calibration object image captured by the fisheye camera, and to acquire the second feature point corresponding to the first feature point from the calibration object image.

[0036] The conversion unit is used to convert the first feature point to the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain the first projection point, and to project the second feature point to the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point.

[0037] A calibration unit is used to optimize the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point.

[0038] According to a fourth aspect of this disclosure, a surround view image generation apparatus is provided, the apparatus comprising:

[0039] The second acquisition unit is used to acquire first images captured by multiple fisheye cameras on the vehicle from their respective acquisition perspectives;

[0040] A projection unit is used to convert the first image of the fisheye camera into a corresponding top view image according to a mapping relationship, wherein the mapping relationship is obtained according to the calibration method described in any of the above embodiments.

[0041] The stitching unit is used to stitch together the top view images corresponding to each first image to obtain a panorama image.

[0042] According to a fifth aspect of this disclosure, an electronic device is provided, the device comprising: a processor; and a memory for storing processor-executable instructions to perform a fisheye camera calibration method or a panoramic image generation method according to any embodiment of this disclosure.

[0043] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the fisheye camera calibration method or the panoramic image generation method according to any embodiment of this disclosure.

[0044] The technical solution provided in this disclosure may include the following beneficial effects: acquiring point cloud data corresponding to a first feature point in a calibration object placed on the ground where the vehicle is located; acquiring an image of the calibration object captured by a fisheye camera mounted on the vehicle, and acquiring a second feature point corresponding to the first feature point from the calibration object image; converting the first feature point to the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain a first projection point, and projecting the second feature point to the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain a second projection point; optimizing the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point, that is, calculating the reprojection error by projecting the first feature point and the second feature point to the unit sphere of the fisheye camera, compared with the reprojection error calculated by projecting the first feature point to the acquired calibration object image, can reduce the influence of fisheye camera distortion on the reprojection error and improve the accuracy of calibration extrinsic parameters.

[0045] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the technical solutions of this disclosure.

[0047] Figure 1 This disclosure is a flowchart illustrating a calibration method for a fisheye camera according to an exemplary embodiment.

[0048] Figure 2 This is a schematic diagram of a panoramic image shown according to an exemplary embodiment of the present disclosure.

[0049] Figure 3 This is a schematic diagram of the structure of a calibration device for a fisheye camera according to an exemplary embodiment of the present disclosure.

[0050] Figure 4 This is a schematic diagram of the structure of a panoramic image generation apparatus according to an exemplary embodiment of the present disclosure.

[0051] Figure 5 This is a schematic diagram of an electronic device structure for image processing according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0053] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0055] Currently, images captured by fisheye cameras at the front, rear, left, and right of a vehicle can be acquired. These images are then converted to a top-down view based on the extrinsic parameters of the fisheye cameras to obtain a top-down image. The top-down images corresponding to each fisheye camera are then stitched together to obtain a surround-view image, which can be used to assist driving.

[0056] In other words, the quality of the panoramic image is related to the accuracy of the fisheye camera's extrinsic parameters. However, the extrinsic parameters of the fisheye camera are usually calibrated by projecting manually measured ground coordinates onto the image acquired by the fisheye camera. Due to the large distortion of the fisheye camera, the accuracy of the extrinsic parameters calibrated by projecting onto the image acquired by the fisheye camera is low, resulting in a poor effect of the panoramic image formed by stitching.

[0057] In view of this, the present disclosure provides a calibration method for a fisheye camera. In the present disclosure, fisheye cameras can be set at the front, rear, left and right sides of a vehicle, and calibration objects can be placed on the ground around the vehicle. The calibration objects can be captured within the acquisition field of view of each fisheye camera. For example, the fisheye camera set at the front of the vehicle can capture the image corresponding to the calibration object placed on the ground at the front of the vehicle.

[0058] Since the calibration process is the same for each fisheye camera, the following embodiments will describe the calibration method of one of the fisheye cameras in conjunction with the accompanying drawings.

[0059] Figure 1 This disclosure is a flowchart illustrating a calibration method for a fisheye camera according to an exemplary embodiment, such as... Figure 1 As shown, the calibration method provided in this disclosure includes the following steps 101 to 104.

[0060] In step 101, the point cloud data corresponding to the first feature point in the calibration object is obtained.

[0061] In this embodiment, point cloud data corresponding to the first feature point in the calibration object collected by the radar on the vehicle can be obtained. The acquisition view of the radar partially overlaps with the acquisition view of the fisheye camera to be calibrated, that is, the radar and the fisheye camera to be calibrated can collect the same calibration object. The radar may include, for example, a lidar.

[0062] In some embodiments, since the point cloud data collected by the radar is relatively sparse, multiple frames of point cloud data collected by the radar during the vehicle's movement can be acquired to generate a dense point cloud. That is, subsequent frames of point cloud data can be superimposed onto the vehicle coordinate system in the initial frame to obtain a dense point cloud.

[0063] The vehicle coordinate system is established with the vehicle as the origin. For example, the direction of the vehicle's movement can be the positive direction of the x-axis, the left direction of the vehicle's movement can be the positive direction of the y-axis, and the direction of the vehicle's roof can be the positive direction of the z-axis.

[0064] The point cloud data corresponding to the first feature point is extracted from the point cloud data collected by the radar. The first feature point may include feature points on the calibration object and / or manually selected feature points. The calibration object may include a checkerboard calibration board.

[0065] When the calibration object is a chessboard calibration board, the first feature point can be a corner point on the chessboard.

[0066] In step 102, the calibration object image captured by the fisheye camera is acquired, and the second feature point corresponding to the first feature point is obtained from the calibration object image.

[0067] When the calibration object is a checkerboard calibration board, a corner point on the checkerboard can be recorded as corner point A. The point cloud data corresponding to corner point A can be obtained through step 101. In this step, the pixel point corresponding to corner point A can be determined in the calibration object image, and the pixel point corresponding to corner point A can be used as the second feature point corresponding to the first feature point.

[0068] In step 103, the first feature point is transformed onto the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain the first projection point, and the second feature point is projected onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point.

[0069] In this embodiment, the first feature point can be transformed to the camera coordinate system using the extrinsic parameters of the fisheye camera to obtain the transformed first feature point. The transformed first feature point is then projected onto the unit sphere of the fisheye camera to obtain the first projection point.

[0070] For example, the first feature point can be denoted as Q(x', y', z'). Using the extrinsic parameters of the fisheye camera, the coordinates are transformed to the camera coordinate system according to formula (1) to obtain the transformed first feature point, denoted as Q'. The transformed first feature point Q' is projected onto the unit sphere of the fisheye camera to obtain the first projection point, denoted as Q".

[0071] Q' = R*Q+T formula (1)

[0072] In formula (1), R and T are the extrinsic parameters of the fisheye camera, where R is the rotation matrix for transforming from the vehicle coordinate system to the camera coordinate system, and T is the translation matrix for transforming from the vehicle coordinate system to the camera coordinate system.

[0073] In this embodiment, the first feature point Q' after transformation is normalized to obtain the first projection point Q" and the coordinates of the first projection point Q" are determined according to the modulus of the first feature point Q' after transformation. The modulus of the first feature point Q' after transformation is determined according to formula (2).

[0074]

[0075] In formula (2) D q Let Q' be the modulus of the first feature point after transformation.

[0076] Given the modulus of the transformed first feature point Q', the coordinates of the first projection point Q" can be determined as follows: The modulus of the first projection point Q" is 1.

[0077] In this embodiment, the second feature point can be denoted as P'. The second feature point P' is projected onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point, denoted as P".

[0078] In step 104, the extrinsic parameters of the fisheye camera are optimized based on the difference between the first projection point and the second projection point.

[0079] The first projection point is also called the reprojection point. Therefore, the difference between the first projection point and the second projection point can be called the reprojection error. In other words, the extrinsic parameters of the fisheye camera can be obtained by calculating the reprojection error between the first projection point and the second projection point.

[0080] This embodiment acquires point cloud data corresponding to a first feature point in a calibration object placed on the ground where the vehicle is located; acquires an image of the calibration object captured by a fisheye camera mounted on the vehicle, and obtains a second feature point corresponding to the first feature point from the calibration object image; transforms the first feature point onto the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain a first projection point, and projects the second feature point onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain a second projection point. By projecting the first and second feature points onto the unit sphere of the fisheye camera to calculate the reprojection error, compared with the reprojection error calculated by projecting the first feature point onto the acquired calibration object image, the influence of fisheye camera distortion on the reprojection error can be reduced, thus improving the accuracy of the calibration extrinsic parameters.

[0081] In some embodiments, optimizing the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point may include: changing the extrinsic parameters of the fisheye camera to change the first projection point; and determining the extrinsic parameters corresponding to the minimum difference between the first projection point and the second projection point as the extrinsic parameters of the fisheye camera.

[0082] Given the first feature point Q(x', y', z'), the first feature point Q can be transformed to the camera coordinate system based on the extrinsic parameters R and T of the fisheye camera, resulting in the transformed first feature point Q'. The transformed first feature point Q' is then projected onto the unit sphere of the fisheye camera to obtain the first projected point Q'. Alternatively, the second projection point P" can be determined using the intrinsic parameters of the calibrated fisheye camera. In this process, R and T are unknowns that need to be solved.

[0083] The above problem can be solved based on the PNP (Perspective-n-Point) algorithm. The PNP algorithm can be used to determine the rotation matrix R and translation matrix T of the fisheye camera relative to the vehicle coordinate system when the three-dimensional coordinates of the plane are known.

[0084] In other words, the extrinsic parameters of a fisheye camera can be obtained by finding R and T that minimize ||Q"-P"||, which is conventionally measured using the 2-norm, i.e., the Euclidean norm. That is, by minimizing the 2-norm of ||Q"-P"||, the extrinsic parameters of the fisheye camera can be obtained.

[0085] In some embodiments, the intrinsic parameters of a fisheye camera can be calibrated according to the imaging formula of the fisheye camera. The calibration process may include the following steps.

[0086] In step a, the calibration object image captured by the fisheye camera is obtained.

[0087] In step b, a second feature point is obtained from the calibration object image, and a first calibration space point corresponding to the second feature point is obtained in the camera coordinate system.

[0088] When the calibration object is a checkerboard calibration board, the second feature point may include the corner points on the checkerboard.

[0089] Based on the imaging principle of the pinhole camera model, the first calibration space point corresponding to the second feature point in the camera coordinate system can be obtained.

[0090] In this embodiment, the first calibration spatial point can be denoted as P(x, y, z), and the second feature point can be denoted as P'(u, v).

[0091] In step c, the first calibration spatial point is projected onto the calibration object image using the intrinsic parameters and distortion variables of the fisheye camera to obtain the first calibration projection point.

[0092] The intrinsic parameters of a fisheye camera can include fx, fy, cx, cy, k1, k2, k3, and k4, where fx and fy are the focal lengths of the fisheye camera in the x and y directions, respectively; cx and cy are the optical center positions of the fisheye camera in the x and y directions, respectively; and k1, k2, k3, and k4 are distortion parameters. During the imaging process, the fisheye camera experiences distortion. According to the distortion model, the amount of distortion can be determined based on the distortion coefficients and the projected incident angle. The projected incident angle is the angle between the line connecting the first calibration point in space and the optical center of the fisheye camera and the optical axis (i.e., the z-axis in the camera coordinate system).

[0093] The distance D between the first calibration space point P and the optical center of the fisheye camera is determined according to formula (3). p .

[0094]

[0095] Normalize the first calibration point P to P1, and obtain the coordinates of P1 as follows:

[0096] In this embodiment, the projection incident angle is determined based on the inverse cosine of the distance between the first calibration spatial point and the optical center of the fisheye camera.

[0097] Specifically, the projection incident angle can be determined based on the distance z between the first calibration spatial point P and the xoy plane of the fisheye camera, and the distance D between the first calibration spatial point P and the optical center of the fisheye camera. p The inverse cosine can be determined, that is, it can be determined according to formula (4).

[0098]

[0099] In formula (4), θ is the incident angle of the projection, that is, the angle between POZ.

[0100] The projection incident angle exceeding 90° can be determined by the inverse cosine of the distance between the first calibration spatial point and the optical center of the fisheye camera, which can improve the accuracy of the projection incident angle compared to related technologies.

[0101] The distorted variable d is determined according to formula (5).

[0102] d=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9 Formula (5)

[0103] Therefore, the coordinates of the first calibration projection point, which projects the first calibration spatial point onto the calibration object image, can be determined by using the intrinsic parameters and distortion variables of the fisheye camera.

[0104] The x-coordinate of the first calibration projection point is

[0105] The ordinate of the first calibration projection point is

[0106] In step d, the intrinsic parameters of the fisheye camera are determined based on the second feature point and the first calibration projection point.

[0107] As can be seen from the imaging principle, the coordinates of the second feature point coincide with the coordinates of the first calibration projection point, that is, the second feature point P'(u, v) can be represented by the following formula.

[0108]

[0109]

[0110] Given the coordinates of the first calibration point P and the coordinates of the second feature point P', the camera's intrinsic parameters can be determined.

[0111] In some embodiments, projecting the second feature point onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain a second projection point may include: projecting the second feature point onto the camera coordinate system according to the intrinsic parameters of the fisheye camera to obtain the distortion coordinates of the second feature point in the camera coordinate system; performing multiple iterations on the projection incident angle of the second feature point to determine the distortion correction coefficient; and performing distortion correction processing on the distortion coordinates according to the distortion correction coefficient to obtain the second projection point of the second feature point projected onto the unit sphere of the fisheye camera.

[0112] Projecting a point from the camera coordinate system to the image coordinate system is usually called orthographic projection, while projecting a point from the image coordinate system to the camera coordinate system is called back projection. In other words, the second feature point can be back-projected onto the unit sphere of the fisheye camera.

[0113] In this embodiment, the second feature point P' can be back-projected onto the unit sphere of the fisheye camera through multiple iterations. The iteration process is as follows:

[0114] According to formula (6), the second feature point P' is projected onto the camera coordinate system to obtain the distorted coordinates of the second feature point in the camera coordinate system.

[0115]

[0116] In formula (6), x0 and y0 are the distorted coordinates of the second feature point in the camera coordinate system, and u and v are the coordinates of the second feature point P' in the image coordinate system.

[0117] The distorted projection incident angle θ is determined according to formula (7). d The projection incident angle θ is determined according to formula (8).

[0118]

[0119] θ = arctan(θ) d ) Formula (8)

[0120] Iterate ten times:

[0121] d=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9

[0122] d′=1+3k1θ 2 +5k2θ 4 +7k3θ 6 +9k4θ 8

[0123] θ=θ-(d-θ d) / d′

[0124] End the loop to obtain the distortion correction coefficients.

[0125] The distorted coordinates are distorted according to the distortion correction coefficients to obtain the second projection point P(x). d y d , z d ),in r = tan(θ), D is the distance from the second projection point P” to the origin.

[0126] This embodiment obtains the extrinsic parameters of the fisheye camera by minimizing the reprojection error between the first projection point Q" and the second projection point P".

[0127] This disclosure also provides a method for generating a surround view image, which may include: acquiring first images captured by multiple fisheye cameras on a vehicle from their respective acquisition perspectives; converting the first images of the fisheye cameras into corresponding top view images according to a mapping relationship, wherein the mapping relationship is obtained according to the calibration method described in the above embodiments; and stitching together the top view images corresponding to each first image to obtain a surround view image.

[0128] For example, the first images captured from the front, rear, left, and right fisheye cameras on the vehicle, each from its own perspective, can be obtained. Based on the mapping relationship between the fisheye image captured by the front fisheye camera and the ground image in front, a top-down view corresponding to the first image captured by the front fisheye camera can be obtained. Similarly, the top-down view corresponding to the first image captured by the rear fisheye camera, the left fisheye camera, and the right fisheye camera can be obtained. These top-down views from the front, rear, left, and right fisheye cameras are then stitched together according to weights to obtain the vehicle surround-view image. In other words, by stitching together the top-down views from the front, rear, left, and right fisheye cameras, overlapping areas are fused according to appropriate weights, and the overall image brightness is balanced, resulting in the final vehicle surround-view image.

[0129] In some embodiments, since the relationship between the fisheye image captured by the fisheye camera on the vehicle and the corresponding ground image is determined, the mapping relationship between each pixel in the fisheye image captured by the fisheye camera and each pixel in the corresponding ground image can be predetermined based on the intrinsic and extrinsic parameters of the fisheye camera. During the generation of the panoramic image, the first image captured by the fisheye camera can be converted into a top-view image corresponding to the first image based on the predetermined mapping relationship.

[0130] In this embodiment, the process of determining the mapping relationship may include establishing a vehicle coordinate system with the vehicle as the origin; acquiring a first pixel on the ground image; converting the first pixel to the vehicle coordinate system to obtain a first spatial point of the first pixel in the vehicle coordinate system; converting the first spatial point to the camera coordinate system according to the extrinsic parameters of the fisheye camera to obtain a second spatial point; converting the second spatial point to the fisheye image acquired by the fisheye camera according to the intrinsic parameters of the camera to obtain a second pixel; obtaining a third pixel in the image acquired by the fisheye camera corresponding to the second pixel through bilinear interpolation; and constructing a mapping relationship between the first pixel on the ground image and the third pixel on the fisheye image.

[0131] Figure 2 This is a schematic diagram of a panoramic image illustrated according to an exemplary embodiment of the present disclosure. Figure 2 For the surround view image 200 of vehicle 206, such as Figure 2 As shown, the panoramic image 200 may include a front ground image 201, a rear ground image 202, a left ground image 203, and a right ground image 204. During vehicle movement, the relative position of the fisheye camera on the vehicle to the ground is fixed; that is, the relative position of the fisheye image captured by the front fisheye camera to the front ground image 201 is determined. Therefore, the mapping relationship between each pixel in the fisheye image captured by the front fisheye camera and each pixel in the front ground image 201 can be predetermined.

[0132] For example, we can assume that the size of the surround view image is 2000cm*2000cm, and define a pixel point at 1cm intervals. Therefore, we can obtain the coordinates (u', v') of pixel point B in the front ground image 201, and denote it as the first pixel point. Then, the center of the surround view image is the center of the vehicle coordinate system (u'c, v'c). The length of pixel point B in space is a. The ground is a plane with a height of -h in the vehicle coordinate system. Therefore, the first spatial point corresponding to pixel point B in the vehicle coordinate system is denoted as B'((u'-u'c)*a, (v'-v'c)*a, -h).

[0133] Based on the extrinsic parameters of the fisheye camera, B' is transformed into the camera coordinate system to obtain the second spatial point. Based on the intrinsic parameters of the fisheye camera, the second spatial point is projected onto the fisheye image acquired by the fisheye camera to obtain the second pixel. Through bilinear interpolation, the third pixel corresponding to the second pixel in the image acquired by the fisheye camera is obtained, that is, the mapping relationship between pixel B on the front ground image and the pixel on the front fisheye image acquired by the fisheye camera.

[0134] For example, when the coordinates of the second pixel are (7.3, 8.4), the mapping relationship between the second pixel and the third pixel on the front fisheye image is shown in the table below.

[0135] 60％ (7，8) (8，8) 40％ (7，9) (8，9)

[0136] A mapping relationship is constructed between the fisheye images captured by the fisheye cameras in the front, back, left, and right directions and the corresponding ground images. Based on the constructed mapping relationship, when the first image captured by the fisheye camera is obtained, the top view image corresponding to the first image can be determined according to the mapping relationship.

[0137] Figure 3 This is a schematic diagram illustrating the structure of a calibration device for a fisheye camera according to an exemplary embodiment of the present disclosure. The fisheye camera is mounted on a vehicle, and a calibration object is placed on the ground where the vehicle is located. The calibration object is positioned within the field of view of the fisheye camera. Figure 3 The device shown includes:

[0138] The first acquisition unit 301 is used to acquire point cloud data corresponding to the first feature point in the calibration object, as well as the calibration object image captured by the fisheye camera, and to acquire the second feature point corresponding to the first feature point from the calibration object image.

[0139] The conversion unit 302 is used to convert the first feature point to the unit sphere of the fisheye camera using the extrinsic parameters of the fisheye camera to obtain the first projection point, and to project the second feature point to the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point.

[0140] The calibration unit 303 is used to optimize the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point.

[0141] In some embodiments, the calibration unit 303 is specifically used for:

[0142] By changing the extrinsic parameters of the fisheye camera, the first projection point changes; the extrinsic parameter corresponding to the minimum difference between the first projection point and the second projection point is determined as the extrinsic parameter of the fisheye camera.

[0143] In some embodiments, the conversion unit 302 is specifically used for:

[0144] The second feature point is projected onto the camera coordinate system based on the intrinsic parameters of the fisheye camera to obtain the distorted coordinates of the second feature point in the camera coordinate system; the projection incident angle of the second feature point is iterated multiple times to determine the distortion correction coefficient; the distortion correction coefficient is used to perform distortion correction processing on the distorted coordinates to obtain the second projection point of the second feature point on the unit sphere of the fisheye camera.

[0145] In some embodiments, the calibration unit 303 is further configured to: acquire a calibration object image captured by the fisheye camera; acquire a second feature point from the calibration object image and acquire a first calibration space point in the camera coordinate system corresponding to the second feature point; project the first calibration space point onto the calibration object image using the intrinsic parameters and distortion of the fisheye camera to obtain a first calibration projection point, wherein the distortion is determined according to the intrinsic parameters and the projection incident angle, and the projection incident angle is determined according to the inverse cosine of the distance between the first calibration space point and the optical center of the fisheye camera; and determine the intrinsic parameters of the fisheye camera according to the second feature point and the first calibration projection point.

[0146] Figure 4 This is a schematic diagram of the structure of a surround view image generation apparatus according to an exemplary embodiment of the present disclosure, as shown below. Figure 4 As shown, the surround view image generation device includes:

[0147] The second acquisition unit 401 is used to acquire first images captured by multiple fisheye cameras on the vehicle from their respective acquisition perspectives.

[0148] The projection unit 402 is used to convert the first image of the fisheye camera into a corresponding top view image according to the mapping relationship, wherein the mapping relationship is obtained according to the calibration method described in the above embodiment;

[0149] The stitching unit 403 is used to stitch together the top view images corresponding to each first image to obtain a panorama image.

[0150] In some embodiments, the surround view image generation apparatus further includes: a mapping relationship generation unit, configured to:

[0151] Establish a vehicle coordinate system with the vehicle as the origin and obtain the first pixel on the ground image;

[0152] The first pixel is transformed into the vehicle coordinate system to obtain the first spatial point of the first pixel in the vehicle coordinate system.

[0153] Based on the extrinsic parameters of the fisheye camera, the first spatial point is transformed into the camera coordinate system to obtain the second spatial point;

[0154] The second spatial point is converted into the fisheye image captured by the fisheye camera based on the camera's intrinsic parameters to obtain the second pixel point;

[0155] The third pixel corresponding to the second pixel in the image captured by the fisheye camera is obtained by bilinear interpolation.

[0156] Construct a mapping relationship between the first pixel on the ground image and the third pixel on the fisheye image.

[0157] Figure 5 A schematic diagram of an electronic device structure for image processing provided in at least one embodiment of this disclosure. (See diagram below.) Figure 5 As shown, the electronic device includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the fisheye camera calibration method or the panoramic image generation method according to any embodiment of the present disclosure when executing the computer instructions.

[0158] At least one embodiment of this disclosure also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the fisheye camera calibration method or the panoramic image generation method described in any embodiment of this disclosure.

[0159] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] In this specification, “and / or” means having at least one of two options. For example, “A and / or B” includes three options: A, B, and “A and B”.

[0161] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0162] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0163] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0164] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0165] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0166] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0167] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0168] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0169] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0170] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A calibration method for a fisheye camera, characterized in that, The fisheye camera is mounted on a vehicle, and a calibration object is placed on the ground where the vehicle is located. The calibration object is within the field of view of the fisheye camera. The method includes: Obtain the point cloud data corresponding to the first feature point in the calibration object; Acquire the calibration object image captured by the fisheye camera, and obtain the second feature point corresponding to the first feature point from the calibration object image; The first feature point is transformed from the vehicle coordinate system to the camera coordinate system using the extrinsic parameters of the fisheye camera, and the transformed first feature point is projected onto the unit sphere of the fisheye camera to obtain the first projection point. The second feature point is projected onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point. The vehicle coordinate system is established with the vehicle as the origin. The extrinsic parameters of the fisheye camera are optimized based on the difference between the first projection point and the second projection point.

2. The method according to claim 1, characterized in that, The step of optimizing the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point includes: By changing the extrinsic parameters of the fisheye camera, the first projection point is changed; The extrinsic parameter corresponding to the minimum difference between the first projection point and the second projection point is determined as the extrinsic parameter of the fisheye camera.

3. The method according to claim 1, characterized in that, The step of projecting the second feature point onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain the second projection point includes: The second feature point is projected onto the camera coordinate system based on the intrinsic parameters of the fisheye camera to obtain the distorted coordinates of the second feature point in the camera coordinate system. The distortion correction coefficient is determined by iterating the projection incident angle of the second feature point multiple times. The distortion coordinates are distorted according to the distortion correction coefficients to obtain the second projection point of the second feature point onto the unit sphere of the fisheye camera.

4. The method according to claim 1, characterized in that, The method further includes: Acquire the image of the calibration object captured by the fisheye camera; Obtain a second feature point from the calibration object image, and obtain a first calibration space point in the camera coordinate system corresponding to the second feature point; The first calibration spatial point is projected onto the calibration object image using the intrinsic parameters and distortion of the fisheye camera to obtain the first calibration projection point. The distortion is determined based on the intrinsic parameters and the projection incident angle, and the projection incident angle is determined based on the inverse cosine of the distance between the first calibration spatial point and the optical center of the fisheye camera. The intrinsic parameters of the fisheye camera are determined based on the second feature point and the first calibration projection point.

5. A method for generating a panoramic image, characterized in that, The method includes: Acquire the first images captured by multiple fisheye cameras on the vehicle from their respective acquisition perspectives; The first image of the fisheye camera is converted into a corresponding top-view image according to the mapping relationship, wherein the mapping relationship is obtained by the calibration method according to any one of claims 1 to 4; The top-view images corresponding to each first image are stitched together to obtain the panoramic image.

6. The method according to claim 5, characterized in that, The method further includes: Establish a vehicle coordinate system with the vehicle as the origin; Obtain the first pixel on the ground image; The first pixel is transformed into the vehicle coordinate system to obtain the first spatial point of the first pixel in the vehicle coordinate system. Based on the extrinsic parameters of the fisheye camera, the first spatial point is transformed into the camera coordinate system to obtain the second spatial point; The second spatial point is converted into the fisheye image captured by the fisheye camera based on the camera's intrinsic parameters to obtain the second pixel point; The third pixel corresponding to the second pixel in the image captured by the fisheye camera is obtained by bilinear interpolation. Construct a mapping relationship between the first pixel on the ground image and the third pixel on the fisheye image.

7. A calibration device for a fisheye camera, characterized in that, The fisheye camera is mounted on the vehicle, and a calibration object is placed on the ground where the vehicle is located. The calibration object is within the field of view of the fisheye camera. The device includes: The first acquisition unit is used to acquire point cloud data corresponding to the first feature point in the calibration object, as well as the calibration object image captured by the fisheye camera, and to acquire the second feature point corresponding to the first feature point from the calibration object image. The transformation unit is used to transform the first feature point from the vehicle coordinate system to the camera coordinate system using the extrinsic parameters of the fisheye camera, and project the transformed first feature point onto the unit sphere of the fisheye camera to obtain a first projection point, and project the second feature point onto the unit sphere of the fisheye camera using the intrinsic parameters of the fisheye camera to obtain a second projection point; wherein, the vehicle coordinate system is established with the vehicle as the origin; A calibration unit is used to optimize the extrinsic parameters of the fisheye camera based on the difference between the first projection point and the second projection point.

8. A device for generating a panoramic image, characterized in that, The device includes: The second acquisition unit is used to acquire first images captured by multiple fisheye cameras on the vehicle from their respective acquisition perspectives; A projection unit is used to convert the first image of the fisheye camera into a corresponding top view image according to a mapping relationship, wherein the mapping relationship is obtained by the calibration method according to any one of claims 1 to 4; The stitching unit is used to stitch together the top view images corresponding to each first image to obtain a panorama image.

9. An electronic device, characterized in that, The device includes: processor; A memory for storing processor-executable instructions to perform the calibration method of the fisheye camera as described in any one of claims 1 to 4, or the panoramic image generation method as described in any one of claims 5 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the calibration method for the fisheye camera as described in any one of claims 1 to 4, or the panoramic image generation method as described in any one of claims 5 to 6.

Images