Fisheye lens calibration method, device, electronic device and storage medium

Abstract

The present disclosure provides a fisheye lens calibration method, device, electronic device, and storage medium. The method includes: determining the first spherical coordinates corresponding to the image position of the center point of each reference object in a plane image corresponding to a calibration image captured by the fisheye lens to be calibrated; mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system; and determining the extrinsic parameters of the fisheye lens to be calibrated based on the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object. In this way, the extrinsic parameters of the fisheye lens to be calibrated can be calibrated, so that the requirements of image stitching and fusion can be met based on the extrinsic parameters of the calibrated fisheye lens, thereby improving the user experience.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a fisheye lens calibration method, device, electronic device, and storage medium. Background Art

[0002] Camera calibration is an essential technology in the field of machine vision. Calibration accuracy determines whether a machine vision system can effectively locate, detect, segment, and fuse regions of interest in an image. Camera calibration can generally be divided into two sub-processes: intrinsic parameter calibration and extrinsic parameter calibration. The camera's extrinsic parameters, also known as the camera pose, are generally represented in matrix form and can be further decomposed into a translation matrix (position) and a rotation matrix (pose). Camera extrinsic parameter calibration involves determining a rotation matrix and a translation matrix using a certain method to calculate the complete extrinsic parameter matrix. Multiplying a point in the world coordinate system by the extrinsic matrix returns it to the camera coordinate system. Conversely, multiplying a point in the camera coordinate system by the inverse matrix of the extrinsic matrix returns it to the world coordinate system.

[0003] Currently, computer vision systems using fisheye lenses have been widely used in many fields, including security monitoring, assisted driving, virtual reality, medical imaging, and intelligent robots. Some systems use two or more fisheye lenses for image stitching to achieve a visual range of more than 180°. However, images taken with fisheye lenses are distorted. If two fisheye images are to be stitched together, the fisheye lenses must be calibrated to achieve a good image stitching effect that meets user needs.

[0004] Therefore, it is very important to calibrate the fisheye lens to determine the external parameters of the fisheye lens. Summary of the Invention

[0005] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] The present disclosure proposes a fisheye lens calibration method, device, electronic device, and storage medium to achieve extrinsic parameter calibration of the fisheye lens to be calibrated by determining the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified from a plane image corresponding to a calibration image captured by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the marked spatial position of the center point of the reference object. Thus, based on the extrinsic parameters of the calibrated fisheye lens, the requirements for image stitching and fusion can be met, thereby improving the user experience.

[0007] According to a first aspect of the present disclosure, a method for calibrating a fisheye lens is provided, comprising: obtaining a calibration image captured by a fisheye lens to be calibrated, wherein the calibration image displays a plurality of reference objects in a calibration space; projecting the calibration image onto a set unit sphere to obtain a spherical image, and performing equidistant cylindrical projection on the spherical image to obtain a plane image; determining, based on the image position of the center point of each of the reference objects in the plane image, a first spherical coordinate of the center point of each of the reference objects in a spherical coordinate system in which the unit sphere is located; mapping the spatial position of the center point of each of the reference objects in a world coordinate system in the calibration space to the spherical coordinate system to obtain a second spherical coordinate of the center point of each of the reference objects in the spherical coordinate system; and determining the extrinsic parameters of the fisheye lens to be calibrated based on the difference between the first spherical coordinate and the second spherical coordinate corresponding to the center point of each of the reference objects.

[0008] Optionally, the plane image contains multiple grid areas, and the external parameters of the fisheye lens to be calibrated are determined based on the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of each reference object, including: determining the first reprojection error of the center point of each reference object in each grid area based on the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object; determining the reprojection error of the vertices of each grid area based on the first reprojection error of the center point of each reference object in each grid area; and determining the external parameters of the fisheye lens to be calibrated based on the reprojection error of the vertices of each grid area.

[0009] Optionally, determining the external parameters of the fisheye lens to be calibrated based on the reprojection errors of the vertices of each of the grid areas includes: determining the second reprojection errors of the center points of each of the reference objects in each of the grid areas based on the reprojection errors of the vertices of each of the grid areas; correcting the first spherical coordinates of the center points of each of the reference objects based on the second reprojection errors to obtain the corrected first spherical coordinates of the center points of each of the reference objects in each of the grid areas; constructing a reprojection error function based on the corrected first spherical coordinates of the center points of each of the reference objects in each of the grid areas; and determining the external parameters of the fisheye lens to be calibrated based on the reprojection error function.

[0010] Optionally, determining the second reprojection error of the center point of each reference object in each grid area based on the reprojection error of each vertex in each grid area includes: traversing each vertex in each grid area in turn to determine multiple reprojection errors corresponding to multiple vertices of each grid area; performing bilinear interpolation on the multiple reprojection errors corresponding to the multiple vertices of each grid area to obtain the second reprojection error of the center point of each reference object in each grid area.

[0011] Optionally, determining the first spherical coordinates of the center point of each reference object in the spherical coordinate system based on the image position of the center point of each reference object in the plane image includes: determining the projection transformation relationship between the plane image and the spherical image; and determining the first spherical coordinates of the center point of each reference object in the spherical coordinate system based on the projection transformation relationship and the image position of the center point of each reference object in the plane image.

[0012] Optionally, mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system includes: measuring the spatial position of the center point of each reference object in the world coordinate system of the calibration space by a set ranging device to obtain the spatial position of the center point of each reference object in the world coordinate system of the calibration space; performing a coordinate system transformation on the spatial position of the center point of each reference object in the world coordinate system of the calibration space to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system.

[0013] Optionally, before projecting the calibration image onto a set unit sphere to obtain a spherical image, and performing equidistant cylindrical projection on the spherical image to obtain a planar image, the method further includes: preprocessing the calibration image; wherein the preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing, and corrosion processing.

[0014] According to a second aspect of an embodiment of the present disclosure, a calibration device for a fisheye lens is provided, comprising: an acquisition module for acquiring a calibration image captured by the fisheye lens to be calibrated, wherein the calibration image displays a plurality of reference objects in a calibration space; a projection module for projecting the calibration image onto a set unit sphere to obtain a spherical image, and performing equidistant cylindrical projection on the spherical image to obtain a plane image; a first determination module for determining, based on the image position of the center point of each reference object in the plane image, a first spherical coordinate of the center point of each reference object in the spherical coordinate system in which the unit sphere is located; a mapping module for mapping the spatial position of the center point of each reference object in the world coordinate system in the calibration space to the spherical coordinate system to obtain a second spherical coordinate of the center point of each reference object in the spherical coordinate system; and a second determination module for determining the extrinsic parameters of the fisheye lens to be calibrated based on the difference between the first spherical coordinate and the second spherical coordinate corresponding to the center point of each reference object.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fisheye lens calibration method proposed in the embodiment of the first aspect of the present disclosure.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the fisheye lens calibration method proposed in the embodiment of the first aspect of the present disclosure.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program. When the computer program is executed by a processor of an electronic device, the electronic device is enabled to execute the fisheye lens calibration method proposed in the embodiment of the first aspect.

[0018] The technical solution disclosed herein obtains a calibration image captured by a fisheye lens to be calibrated, wherein the calibration image displays a plurality of reference objects in the calibration space; projects the calibration image onto a set unit sphere to obtain a spherical image, and performs equidistant cylindrical projection on the spherical image to obtain a plane image; determines the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere according to the image position of the center point of each reference object in the plane image; maps the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere; The second spherical coordinates of the center point in the spherical coordinate system; the external parameters of the fisheye lens to be calibrated are determined according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of the reference objects. Thus, the external parameters of the fisheye lens to be calibrated can be calibrated by the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified according to the plane image corresponding to the calibration image collected by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the real spatial position of the center point of the reference object in the world coordinate system. Therefore, according to the external parameters of the calibrated fisheye lens, the requirements of image stitching and fusion can be met, thereby improving the user experience.

[0019] Additional aspects and advantages of the present disclosure will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present disclosure will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 is a flow chart of a method for calibrating a fisheye lens according to an exemplary embodiment;

[0022] Figure 2 is a schematic flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment;

[0023] Figure 3 is a schematic flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment;

[0024] Figure 4 is a schematic flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment;

[0025] Figure 5 is a schematic diagram showing mapping the spatial position of the center point of a reference object in a world coordinate system to a spherical coordinate system according to an exemplary embodiment;

[0026] Figure 6 is a schematic diagram of performing equidistant cylindrical projection on a plane image according to an exemplary embodiment;

[0027] Figure 7 is a schematic structural diagram of a fisheye lens calibration device according to an exemplary embodiment;

[0028] Figure 8 The figure is a block diagram of an electronic device for calibrating a fisheye lens according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0030] In related technologies, fisheye lens calibration is achieved by mathematically modeling the distortion characteristics of the fisheye lens, and then calibrating the various parameters required for the model through experiments. The fisheye lens model uses a high-order polynomial function to fit the projection formula of the fisheye lens. Since the image resolution is larger and the accuracy requirement is higher, the order of the polynomial must be higher. However, for fisheye lenses, this calibration method that relies on high-order polynomial fitting is only effective in the center area of the image, and the effect is less stable at the edges of the image. If the selected polynomial order is very high, it is also easy for the model to fail to solve.

[0031] In response to the above problems, the present disclosure proposes a fisheye lens calibration method, device, electronic device, and storage medium.

[0032] The following describes in detail a fisheye lens calibration method, device, electronic device, and storage medium provided by the present disclosure in conjunction with the accompanying drawings.

[0033] Figure 1 The diagram is a flow chart of a method for calibrating a fisheye lens according to an exemplary embodiment. The method for calibrating a fisheye lens according to the embodiment of the present disclosure can be applied to any electronic device to enable the electronic device to perform the fisheye lens calibration function.

[0034] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, mobile terminal, server, etc. The mobile terminal can be, for example, a mobile phone, tablet computer, personal digital assistant, wearable device, etc., which are hardware devices with various operating systems, touch screens and / or display screens.

[0035] like Figure 1As shown in FIG, the fisheye lens calibration method includes the following steps:

[0036] Step 101: Obtain a calibration image captured by a fisheye lens to be calibrated.

[0037] The calibration image shows multiple reference objects in the calibration space.

[0038] In the embodiment of the present disclosure, the calibration space may be pre-set, the spatial shape of the calibration space may be regular, and there may be no debris in the calibration space. Reference objects are set around the calibration space, and the number of reference objects may be greater than a set threshold (e.g., 200). A world coordinate system is established with the center position of the calibration space as the origin. For example, the calibration space is a rectangular parallelepiped with an area of 40 square meters, and the length, width, and height of the rectangular parallelepiped are L, W, and H respectively. A world coordinate system is established with (L / 2, W / 2, H / 2) as the origin. It should be noted that the specifications of the multiple reference objects are the same, and the reference object may be a solid circular pattern with a diameter greater than or equal to a set length (e.g., 80 mm). The color of the reference object has a clear contrast with the background of the calibration space. The spacing between two adjacent reference objects may be determined according to the area of the calibration space. For example, the spacing between two adjacent reference objects is one meter.

[0039] In the embodiment of the present disclosure, an image containing a large number of reference objects may be taken using a fisheye lens to be calibrated as a calibration image.

[0040] Step 102 : Project the calibration image onto a set unit sphere to obtain a spherical image, and perform equidistant cylindrical projection on the spherical image to obtain a planar image.

[0041] As a possible implementation method of the embodiment of the present disclosure, the calibration image can be projected onto a set unit sphere using the theoretical projection formula of the fisheye lens to obtain a spherical image, where the radius of the unit sphere is 1, and the theoretical projection formula of the fisheye lens is r=f·θ, where f is the focal length of the fisheye lens, θ is the angle between the incident light and the Z axis of the camera coordinate system, and r is the distance between the image point on the photosensitive surface of the fisheye lens and the center point of the calibration image (the intersection of the optical axis and the photosensitive surface).

[0042] Then, the spherical image is projected onto an equidistant cylindrical surface to obtain a planar image.

[0043] Step 103 : determining the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located according to the image position of the center point of each reference object in the plane image.

[0044] In an embodiment of the present disclosure, the coordinates of the center point of each reference object are extracted from the plane image to obtain the image position of the center point of each reference object in the plane image. Furthermore, based on the image position of the center point of each reference object in the plane image and the projection transformation relationship between the plane image and the spherical image, the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located can be obtained.

[0045] Step 104 : Mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system.

[0046] As an example, the spatial positions of all reference objects relative to the origin of the world coordinate system can be measured by a set distance measuring device (such as a rangefinder), and the spatial position of each reference object can be mapped to a spherical coordinate system, so that the second spherical coordinates of the center point of each reference object in the spherical coordinate system can be obtained.

[0047] Step 105 : determining the extrinsic parameters of the fisheye lens to be calibrated according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of the reference objects.

[0048] In the embodiment of the present disclosure, the external parameters of the fisheye lens to be calibrated can be calibrated according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of the reference objects.

[0049] In summary, the calibration image collected by the fisheye lens to be calibrated is obtained; the calibration image is projected onto the set unit sphere to obtain a spherical image, and the spherical image is equidistantly projected to obtain a plane image; according to the image position of the center point of each reference object in the plane image, the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere are determined; the spatial position of the center point of each reference object in the world coordinate system of the calibration space is mapped to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system; according to the image position of each reference object in the plane image, the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere are determined; according to the image position of each reference object in the world coordinate system of the calibration space, the first spherical coordinates of the center point of each reference object in the spherical coordinate system are determined; according to the image position of each reference object in the plane image, the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere ... The extrinsic parameters of the fisheye lens to be calibrated are determined by the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of the reference object. Thus, the extrinsic parameters of the fisheye lens to be calibrated can be calibrated by the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified according to the plane image corresponding to the calibration image collected by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the real spatial position of the center point of the reference object in the world coordinate system. Therefore, according to the extrinsic parameters of the calibrated fisheye lens, the requirements of image stitching and fusion can be met, thereby improving the user experience.

[0050] In order to clearly illustrate how to determine the external parameters of the fisheye lens to be calibrated based on the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object, the following is combined with Figure 2 , the calibration method of the fisheye lens provided in the embodiment of the present disclosure is further explained.

[0051] Figure 2 FIG. 1 is a flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment. Figure 2 As shown in FIG, the fisheye lens calibration method includes the following steps:

[0052] Step 201: Obtain a calibration image captured by a fisheye lens to be calibrated.

[0053] The calibration image shows multiple reference objects in the calibration space.

[0054] In step 202 , the calibration image is projected onto a set unit sphere to obtain a spherical image, and the spherical image is subjected to equidistant cylindrical projection to obtain a planar image.

[0055] Step 203 : determining the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located according to the image position of the center point of each reference object in the plane image.

[0056] In step 204 , the spatial position of the center point of each reference object in the world coordinate system of the calibration space is mapped to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system.

[0057] Step 205 : determining a first reprojection error of the center point of each reference object in each grid area according to the difference between the first spherical coordinate and the second spherical coordinate corresponding to the center point of each reference object.

[0058] In an embodiment of the present disclosure, equidistant cylindrical projection and expansion are performed on the spherical image to generate a planar image containing multiple grid areas. Based on the reference objects in each grid area, the first reprojection error of the center point of each reference object in each grid area can be selected from the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object.

[0059] Step 206 : Determine the reprojection errors of the vertices of each mesh region according to the first reprojection errors of the center points of each reference object in each mesh region.

[0060] In the embodiment of the present disclosure, a bilinear interpolation method may be used to perform linear interpolation on the first reprojection errors of the center points of multiple reference objects in each grid area to obtain the reprojection errors of each vertex in each grid area.

[0061] It should be noted that in the embodiment of the present disclosure, the plane image is divided into n*n equal-sized grids, each grid has four vertices, and the plane image involves a total of (n+1)(n+1) grid vertices. Considering that all meridians on the sphere converge to the north and south poles (λ=±π / 2), the first and last rows of the grids correspond to one point each, and the entire grid has only (n*n+1) different vertices. Taking the South Pole vertex as The North Pole is For example, the subscripts of other mesh vertices can be addressed in ascending order from top to bottom and from left to right, starting from 1.

[0062] Step 207 : determining the extrinsic parameters of the fisheye lens to be calibrated according to the reprojection errors of the vertices of each mesh region.

[0063] As an example, based on the reprojection error of each vertex in each grid area, the second reprojection error of the center point of each reference object in each grid area is determined; based on the second reprojection error of the center point of each reference object, the first spherical coordinates of the center point of each reference object are corrected to obtain the corrected first spherical coordinates of the center point of each reference object in each grid area; based on the corrected first spherical coordinates of the center point of each reference object in each grid area, a reprojection error function is constructed; based on the reprojection error function, the external parameters of the fisheye lens to be calibrated are determined.

[0064] That is, in order to improve the reprojection error of the center point of each reference object, each vertex of each grid area can be traversed in sequence to determine the multiple reprojection errors corresponding to the multiple vertices of each grid area. Then, bilinear interpolation can be used to linearly fit the reprojection errors of the multiple vertices of each grid area to obtain the second reprojection error of the center point of each reference object in each grid area. Then, based on the second reprojection error of the center point of each reference object, the first spherical coordinate of the center point of each reference object is corrected. For example, the first spherical coordinate of the center point of each reference object is added to the corresponding second reprojection error to obtain the corrected first spherical coordinate. Then, based on the corrected first spherical coordinate of the center point of each reference object in each grid area, a reprojection error function is constructed. Then, based on the reprojection error function, the extrinsic parameters of the fisheye lens to be calibrated are determined. The more accurate the extrinsic parameters of the fisheye lens to be calibrated, the more accurate the reprojection error of the center point of each reference object, and the smaller the value of the reprojection error function. For example, an optimization algorithm (eg, Levenberg-Marquardt) may be used to iteratively optimize the extrinsic parameters of the fisheye lens to be calibrated, and ultimately the extrinsic parameters of the fisheye lens to be calibrated may minimize the reprojection error function value.

[0065] It should be noted that the execution process of steps 201 to 204 can be implemented in any of the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0066] In summary, the first reprojection error of the center point of each reference object in each grid area is determined according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object; the reprojection error of the vertices of each grid area is determined according to the first reprojection error of the center point of each reference object in each grid area, and the external parameters of the fisheye lens to be calibrated are determined according to the reprojection error of the vertices of each grid area. Therefore, by determining the reprojection error of each vertex of each grid area through the first reprojection error of the center point of each reference object in each grid area, the accuracy of the reprojection error of each vertex of each grid area can be improved. Furthermore, according to the reprojection error of each vertex of each grid area, the external parameters of the fisheye lens to be calibrated can be more accurately determined.

[0067] In order to clearly explain how to determine the first spherical coordinates of the center point of each reference object in the spherical coordinate system according to the image position of the center point of each reference object in the plane image, the following is combined with Figure 3 , the calibration method of the fisheye lens provided in the embodiment of the present disclosure is further explained.

[0068] Figure 3 FIG. 1 is a flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment. Figure 3 As shown in FIG, the fisheye lens calibration method includes the following steps:

[0069] Step 301: Obtain a calibration image captured by a fisheye lens to be calibrated.

[0070] The calibration image shows multiple reference objects in the calibration space.

[0071] In step 302 , the calibration image is projected onto a set unit sphere to obtain a spherical image, and the spherical image is subjected to equidistant cylindrical projection to obtain a planar image.

[0072] Step 303: Determine the projection transformation relationship between the planar image and the spherical image.

[0073] As an example, the projection transformation relationship between the planar image and the spherical image may be preset.

[0074] Step 304 : Determine the first spherical coordinates of the center point of each reference object in the spherical coordinate system according to the projection transformation relationship and the image position of the center point of each reference object in the plane image.

[0075] In the embodiment of the present disclosure, based on the projection transformation relationship between the plane image and the spherical image, the image position of the center point of each reference object in the plane image can be transformed into a spherical coordinate system, and the first spherical coordinates of the center point of each reference object in the spherical coordinate system can be obtained.

[0076] Step 305 : Mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system.

[0077] Step 306 : Determine the extrinsic parameters of the fisheye lens to be calibrated according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of the reference objects.

[0078] It should be noted that the execution process of steps 301 to 302 and step 306 can be implemented in any way in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0079] In summary, by determining the projection transformation relationship between the plane image and the spherical image; according to the projection transformation relationship and the image position of the center point of each reference object in the plane image, the first spherical coordinate of the center point of the reference object in the spherical coordinate system can be effectively determined.

[0080] In order to clearly illustrate how the above embodiment maps the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system, the following is combined with Figure 4 , the calibration method of the fisheye lens provided in the embodiment of the present disclosure is further explained.

[0081] Figure 4 FIG. 1 is a flow chart of a method for calibrating a fisheye lens according to another exemplary embodiment. Figure 4 As shown in FIG, the fisheye lens calibration method includes the following steps:

[0082] Step 401: Obtain a calibration image captured by a fisheye lens to be calibrated.

[0083] The calibration image shows multiple reference objects in the calibration space.

[0084] Step 402 : Project the calibration image onto a set unit sphere to obtain a spherical image, and perform equidistant cylindrical projection on the spherical image to obtain a planar image.

[0085] Step 403 : Determine the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located according to the image position of the center point of each reference object in the plane image.

[0086] In step 404 , the spatial position of the center point of each reference object in the world coordinate system of the calibration space is measured by a predetermined distance measuring device to obtain the spatial position of the center point of each reference object in the world coordinate system of the calibration space.

[0087] In the embodiment of the present disclosure, a set distance measuring device (such as a rangefinder) can be used to measure the spatial position of all reference objects relative to the origin of the world coordinate system, so as to obtain the spatial position of the center point of each reference object in the world coordinate system where the calibration space is located.

[0088] Step 405 : performing a coordinate system transformation on the spatial position of the center point of each reference object in the world coordinate system of the calibration space to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system.

[0089] As an example, the spatial position P of the center point of each reference object in the world coordinate system of the calibration space can be transformed into the camera coordinate system, and then the vector obtained by the transformation to the camera coordinate system is scale-normalized to obtain the spherical coordinate p′ corresponding to the spatial position P of the center point of each reference object in the world coordinate system of the calibration space, p′=μ·R(PT), where μ is the normalization coefficient, is the rotation matrix corresponding to the parameters of the fisheye lens, is the translation vector corresponding to the parameters of the fisheye lens (R and T are temporarily regarded as known quantities). It should be noted that when the camera coordinate system coincides with the world coordinate system, p′=μP, μ=((X w -T x ) 2 +(Y W -T y ) 2 +(Z w -T z ) 2 ) -1 / 2 .

[0090] Step 406 : Determine the extrinsic parameters of the fisheye lens to be calibrated according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center points of the reference objects.

[0091] It should be noted that the execution process of steps 401 to 403 and step 406 can be implemented in any way in the embodiments of the present disclosure, and the embodiments of the present disclosure do not limit this and will not be described in detail.

[0092] In summary, by measuring the spatial position of the center point of each reference object in the world coordinate system of the calibration space through the set ranging device, the spatial position of the center point of each reference object in the world coordinate system of the calibration space can be obtained; by performing a coordinate system transformation on the spatial position of the center point of each reference object in the world coordinate system of the calibration space, the second spherical coordinate of the center point of each reference object in the spherical coordinate system can be effectively obtained.

[0093] In any embodiment of the present disclosure, the fisheye lens calibration method of the embodiment of the present disclosure can also be implemented based on the following steps:

[0094] Step 1. Prepare the environment and facilities required for calibration, including a dedicated calibration room (calibration space). The calibration room is required to be in the shape of a standard rectangular parallelepiped, assuming that the length, width and height are L, W and H respectively, and there is no furniture or debris in the room; the four walls, ceiling and floor of the calibration room are evenly arranged with the markers (reference objects) required for calibration; a world coordinate system is established with the central position of the calibration room (L / 2, W / 2, H / 2) as the origin, and the three coordinate axes of the world coordinate system are parallel to the directions corresponding to the length, width and height of the calibration room respectively; a total station rangefinder is set up in the center of the calibration room, and the optical center of the rangefinder is adjusted to coincide with the origin of the world coordinate system; it should be noted that the marker can be a solid circular pattern with a diameter of not less than 80 mm, with no limit on color, and the total number of markers should be not less than 200. The spacing can be appropriately selected based on the area of the room, such as one meter;

[0095] Step 2: Use the rangefinder to measure the distance and orientation coordinates of all markers relative to the origin of the world coordinate system, and register the center coordinates (X w , Y w , Z w ) for standby; after the surveying and mapping is completed, the rangefinder is removed and the fisheye lens to be calibrated is set up in the original location; wherein the optical center of the fisheye lens to be calibrated can be as close as possible to the origin of the world coordinate system, and the three coordinate axes of the camera coordinate system are nearly parallel to the coordinate axes of the world coordinate system. For example, the camera coordinate system of the fisheye lens to be calibrated and the world coordinate system can completely coincide;

[0096] Step 3: Use the fisheye lens to be calibrated to capture an image, called I0. The image quality of I0 must be good, the markers on the screen must be clearly visible, and the optical center of the fisheye lens must intersect the camera imaging plane at pixel coordinates (u0, v0).

[0097] Step 4: Get the nominal focal length f of the lens, and project the image onto an imaginary unit sphere according to the theoretical projection formula of the fisheye lens. The radius of the imaginary unit sphere is 1, and the pixel (u, v) on the image I0 is projected onto the unit circle. point, where λ is the longitude value, is Latitude value; It should be noted that the theoretical projection formula of the fisheye lens is r = f·θ, where f is the focal length of the lens, θ is the angle between the incident light and the z-axis of the camera coordinate system, and r is the distance between the image point on the camera's photosensitive surface and the center point of the image (the intersection of the optical axis and the photosensitive surface);

[0098] Step 5: Perform equidistant cylindrical projection on the imaginary unit sphere described in step 4, and unfold the image on the imaginary unit sphere into a plane to obtain an actual projected image I1;

[0099] Step 6: Preprocess the image I1 to obtain image I2; the preprocessing of the image I1 may include any one or more of a variety of algorithm processing such as color space transformation, median filtering, bilateral filtering, binarization, morphological erosion, etc.;

[0100] Step 7: Analyze and process the image I2 to extract the center coordinates of all markers. Take a specific marker ξ as an example, let represents the coordinates of the center point of ξ on image I2, which can be uniquely associated with the imaginary unit sphere Point, that is, on image I2 Points and units on the sphere There is an equivalence relation between the points;

[0101] Step 8. Find the world coordinates of the center of the marker ξ based on the result of step 2 Define the external parameters of the fisheye lens, including a rotation matrix R and a translation vector T, where

[0102]

[0103] In the embodiment of the present disclosure, Figure 5 As shown, R and T can be temporarily regarded as known quantities, and P can be transformed into a coordinate system using R and T (e.g., Helmert transformation). The specific process is to first transform P into the camera coordinate system, and then scale-normalize the resulting vector to obtain the point on the unit sphere corresponding to point P. Specifically, it can be expressed as the following formula:

[0104] p′=μ·R(PT);

[0105] in, is a point on the unit sphere, represented here by rectangular coordinates, μ is the normalization coefficient, μ=((X w -T x ) 2 +(Y W -T y ) 2 +(Z w -T z ) 2 )-1 / 2 ; It should be noted that when the camera coordinate system and the world coordinate system completely coincide in step 2,

[0106] Then p′=μP,μ=(X w 2 +Y W 2 +Z w 2 ) -1 / 2 ;

[0107] Step 9: Temporarily treat (x′, y′, z′) as known quantities and calculate the spherical coordinates of point p′ The specific method is,

[0108]

[0109] Step 10: Temporarily As a known quantity, calculate the error of the coordinate value of the center of ξ obtained in step 7 and step 9,

[0110]

[0111] in, represents the measured value, represents the true value. Equivalently, Δ can also be expressed by (x) and (x′, y′, z′);

[0112] Step 11: Use Δ to establish a mapping relationship between the marker point P (true value) in the world coordinate system and the image coordinate point p (measured value), which can be expressed as the following formula:

[0113]

[0114] Where s = 1 / μ, M = R -1 , R and T are the external parameters of the fisheye lens to be solved, P and p and the corresponding μ are known quantities, and Δ is an unknown quantity (because p′ is unknown);

[0115] Step 12: Express the mapping equation P in step 11 as a homogeneous matrix.

[0116]

[0117] Rearranging the above equation, organizing the unknown elements of M and T into the unknown vector Y, and treating P, p, and Δ as known quantities to form the coefficient matrix G, we can obtain the following matrix equation:

[0118] G·Y=0;

[0119] According to the matrix equation, we only need to determine the Δ corresponding to each marker to determine the G matrix, and then solve Y, which is the equivalent form of the external parameters of the fisheye lens;

[0120] Step 13: Use a grid to divide the image I2 into nxn regions of equal area, involving a total of (n+1)(n+1) grid vertices. Considering that all meridians on the sphere converge to the north and south poles (λ=±π / 2), the first and last rows of the grid correspond to one point each, and the entire grid has only (n*n+1) different vertices. Let the north and south pole vertices be The subscripts of other vertices are addressed in ascending order from top to bottom and from left to right, starting from 1. Figure 6 As shown, the spherical image on the unit sphere is projected equidistantly to obtain a plane image, where the plane image includes 3x3 regions of equal area, involving a total of (3+1)(3+1) grid vertices, and the first row of the grid corresponds to The last line corresponds to The subscripts of other vertices are addressed in order from top to bottom and from left to right, starting from 1. Furthermore, an error is maintained for each vertex, using represents the error of the i-th vertex, then the total number of undetermined errors in the entire mesh is (n*n+1);

[0121] Step 14: Estimate the error parameters of each mesh vertex. The specific method is to take any region i, that is, the coordinate of the upper left corner is i, and let the errors at the four mesh vertices in the region be:

[0122]

[0123] According to the result of step 7, the region should contain a set of markers, assuming there are m of them. Then, the error of the jth marker can be fitted using the error of the four mesh vertices using the bilinear interpolation method. The specific method is:

[0124]

[0125]

[0126] Among them, a 1j 、a 2j 、a 3j and a 4j are the coefficients of bilinear interpolation, whose values are given by The positional relationship between the four mesh vertices is completely determined and satisfies the constraint a 1j +a 2j +a 3j +a 4j =1, and the matrix equation is:

[0127]

[0128] Let Λ represent the error of the four mesh vertices, Γ represent the true error of a marker point, and A represent the interpolation coefficient matrix. Temporarily treat the true error of the marker point as a known quantity (actually unknown, depending on M and T), then we have

[0129] A.Λ (i) =Γ (i) ,

[0130] Λ (i) =(A T A) -1 (A T Γ (i) ),

[0131] By traversing i in this way, we can get the values of each mesh vertex. Error, then, based on the errors of the four mesh vertices in a region, the error of any point in the region can be fitted using the bilinear interpolation method;

[0132] Step 15: Let Δ = 0, then the matrix G only contains known quantities, and SVD decomposition can be performed on G.

[0133] [SVD] = SVD_Decompose(G)

[0134] The three matrices obtained by decomposition satisfy,

[0135] G=S·V·D T ,

[0136] Among them, the last column of matrix D is a rough approximation of the Y vector to be found This includes an approximation of the T vector

[0137] use We can solve P = s·M·p+T and get the approximate solution of M

[0138]

[0139] Step 16: Define the reprojection error function,

[0140]

[0141] in,

[0142] ε(i)=P i -{s i (T)·M·f(p i,Λ)+T}, represents the reprojection error of the i-th marker, with a total of K markers,

[0143] P i represents the world coordinates of the i-th marker obtained by precise mapping method,

[0144] p i Represents the i-th marker extracted from the image Coordinate values,

[0145] f(p i ,Λ) represents the error information Λ of mesh vertices to p i Make corrections to get the corrected coordinate values And Λ is a function of M and T, M, T are the rotation matrix and translation vector, which are the camera external parameters to be solved.

[0146] s i (T) is the scaling factor of the i-th marker, which is a function of T.

[0147] ||.|| 2 represents the 2-norm operator,

[0148] Step 17: Solve the camera external parameters M, T and the grid error correction Λ. The specific process is that the reprojection error function in step 16 It contains a set of unknown parameters M, T and Λ, which can be solved by the Levenberg-Marquardt algorithm, where the Levenberg-Marquardt algorithm is an iterative method to find the optimal approximate solution of the unknown parameters in the least squares sense.

[0149] It should be noted that during iteration, the initial value of the undetermined parameter can be Λ=0, so f(p i ,Λ)=p i After each iteration, the latest round of M and T is used to calculate the Helmert transformation p′=μ·R(PT), and the latest projection error Δ=(p i -p i ′), then according to the method of step 14, the mesh vertex error correction amount Λ is calculated based on the latest Δ, and then the latest Λ is used to start the next round of iteration, and so on. When it has converged to a stable minimum, M, T, and Λ hardly change. At this time, M and T are the optimal camera extrinsics, and Λ is the optimal grid error correction. Based on Λ, an optimal correction value can be fitted for each pixel coordinate on the image. Based on the corrected image, an inverse fisheye transform and an inverse Helmert transform are performed to accurately calculate the position information (azimuth) of an object in real physical space.

[0150] The fisheye lens calibration method of the embodiment of the present disclosure obtains a calibration image captured by the fisheye lens to be calibrated; projects the calibration image onto a set unit sphere to obtain a spherical image, and performs equidistant cylindrical projection and expansion on the spherical image to obtain a plane image; determines the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located according to the image position of the center point of each reference object in the plane image; maps the spatial position of the center point of each reference object in the world coordinate system where the calibration space is located to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system. Spherical coordinates; according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object, the external parameters of the fisheye lens to be calibrated are determined. Thus, the external parameters of the fisheye lens to be calibrated can be calibrated by the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified according to the plane image corresponding to the calibration image collected by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the real spatial position of the center point of the reference object in the world coordinate system. Therefore, according to the external parameters of the calibrated fisheye lens, the requirements of image stitching and fusion can be met, thereby improving the user experience.

[0151] With the above Figures 1 to 6 Corresponding to the calibration method of the fisheye lens provided in the embodiment, the present disclosure further provides a calibration device for the fisheye lens. Figures 1 to 6 The calibration method of the fisheye lens provided in the embodiment corresponds to the embodiment, so the implementation of the calibration method of the fisheye lens is also applicable to the calibration device of the fisheye lens provided in the embodiment of the present disclosure, and will not be described in detail in the embodiment of the present disclosure.

[0152] Figure 7 The figure is a schematic structural diagram of a fisheye lens calibration device according to an exemplary embodiment.

[0153] like Figure 7 As shown, the fisheye lens calibration device 700 includes: an acquisition module 710 , a projection module 720 , a first determination module 730 , a mapping module 740 and a second determination module 750 .

[0154] Among them, the acquisition module 710 is used to acquire a calibration image captured by the fisheye lens to be calibrated, wherein the calibration image displays multiple reference objects in the calibration space; the projection module 720 is used to project the calibration image onto a set unit sphere to obtain a spherical image, and perform equidistant cylindrical projection on the spherical image to obtain a plane image; the first determination module 730 is used to determine the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere according to the image position of the center point of each reference object in the plane image; the mapping module 740 is used to map the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system; the second determination module 750 is used to determine the external parameters of the fisheye lens to be calibrated according to the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object.

[0155] As a possible implementation of the embodiment of the present disclosure, the plane image includes multiple grid areas, and the second determination module 750 is used to: determine the first reprojection error of the center point of each reference object in each grid area based on the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object; determine the reprojection error of each vertex in each grid area based on the first reprojection error of the center point of each reference object in each grid area; and determine the extrinsic parameters of the fisheye lens to be calibrated based on the reprojection error of each vertex in each grid area.

[0156] As a possible implementation method of the embodiment of the present disclosure, the second determination module 750 is further used to: determine the second reprojection error of the center point of each reference object in each grid area based on the reprojection error of each vertex of each grid area; correct the first spherical coordinates of the center point of each reference object based on the second reprojection error of the center point of each reference object to obtain the corrected first spherical coordinates of the center point of each reference object in each grid area; construct a reprojection error function based on the corrected first spherical coordinates of the center point of each reference object in each grid area; and determine the external parameters of the fisheye lens to be calibrated based on the reprojection error function.

[0157] As a possible implementation method of the embodiment of the present disclosure, the second determination module 750 is further used to: traverse each vertex of each grid area in turn to determine multiple reprojection errors corresponding to the multiple vertices of each grid area; perform bilinear interpolation on the multiple reprojection errors corresponding to the multiple vertices of each grid area to obtain a second reprojection error of the center point of each reference object in each grid area.

[0158] As a possible implementation method of an embodiment of the present disclosure, the first determination module 730 is used to: determine the projection transformation relationship between the plane image and the spherical image; and determine the first spherical coordinates of the center point of each reference object in the spherical coordinate system based on the projection transformation relationship and the image position of the center point of each reference object in the plane image.

[0159] As a possible implementation method of an embodiment of the present disclosure, the mapping module 740 is used to: measure the spatial position of the center point of each reference object in the world coordinate system of the calibration space through a set ranging device to obtain the spatial position of the center point of each reference object in the world coordinate system of the calibration space; perform a coordinate system transformation on the spatial position of the center point of each reference object in the world coordinate system of the calibration space to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system.

[0160] As a possible implementation of the embodiment of the present disclosure, the fisheye lens calibration device 700 further includes: a processing module.

[0161] The processing module is used to preprocess the calibration image, wherein the preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing and corrosion processing.

[0162] The fisheye lens calibration device of the embodiment of the present disclosure obtains a calibration image collected by the fisheye lens to be calibrated; projects the calibration image onto a set unit sphere to obtain a spherical image, and performs equidistant cylindrical projection and expansion on the spherical image to obtain a plane image; determines the first spherical coordinates of the center point of each reference object in the spherical coordinate system of the unit sphere according to the image position of the center point of each reference object in the plane image; maps the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system. Spherical coordinates; according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object, the external parameters of the fisheye lens to be calibrated are determined. Thus, the external parameters of the fisheye lens to be calibrated can be calibrated by the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified according to the plane image corresponding to the calibration image collected by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the real spatial position of the center point of the reference object in the world coordinate system. Therefore, according to the external parameters of the calibrated fisheye lens, the requirements of image stitching and fusion can be met, thereby improving the user experience.

[0163] In order to implement the above embodiments, the present disclosure also proposes an electronic device, such as Figure 8 As shown, Figure 8 FIG. 1 is a block diagram of an electronic device for calibrating a fisheye lens according to an exemplary embodiment. Figure 8 As shown, the electronic device 800 may include:

[0164] The memory 810 and the processor 820, a bus 830 connecting different components (including the memory 810 and the processor 820), the memory 810 stores a computer program, and when the processor 820 executes the program, the fisheye lens calibration method described in the embodiment of the present disclosure is implemented.

[0165] Bus 830 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0166] The electronic device 800 typically includes a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device 800, including volatile and non-volatile media, removable and non-removable media.

[0167] The memory 810 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 840 and / or cache memory 850. The electronic device 800 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 860 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 830 via one or more data medium interfaces. Memory 810 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present disclosure.

[0168] A program / utility 880 having a set (at least one) of program modules 870 may be stored, for example, in memory 810. Such program modules 870 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 870 generally implement the functions and / or methods of the embodiments described herein.

[0169] The electronic device 800 may also communicate with one or more external devices 890 (e.g., a keyboard, a pointing device, a display 891, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 892. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 893. Figure 8 As shown, the network adapter 893 communicates with other modules of the electronic device 800 via the bus 830. Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 800, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0170] The processor 820 executes various functional applications and data processing by running programs stored in the memory 810 .

[0171] It should be noted that the implementation process and technical principles of the electronic device of this embodiment can be found in the aforementioned explanation of the fisheye lens calibration method of the embodiment of the present disclosure, and will not be repeated here.

[0172] The electronic device provided by the embodiment of the present disclosure obtains a calibration image captured by a fisheye lens to be calibrated; projects the calibration image onto a set unit sphere to obtain a spherical image, and performs equidistant cylindrical projection and expansion on the spherical image to obtain a plane image; determines the first spherical coordinates of the center point of each reference object in the spherical coordinate system where the unit sphere is located according to the image position of the center point of each reference object in the plane image; maps the spatial position of the center point of each reference object in the world coordinate system where the calibration space is located to the spherical coordinate system to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system. Coordinates; according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object, the external parameters of the fisheye lens to be calibrated are determined. Thus, the external parameters of the fisheye lens to be calibrated can be calibrated by the difference between the first spherical coordinates corresponding to the image position of the center point of each reference object identified according to the plane image corresponding to the calibration image collected by the fisheye lens to be calibrated and the second spherical coordinates corresponding to the real spatial position of the center point of the reference object in the world coordinate system. Therefore, according to the external parameters of the calibrated fisheye lens, the requirements of image stitching and fusion can be met, thereby improving the user experience.

[0173] In order to implement the above embodiments, the embodiments of the present disclosure further provide a computer-readable storage medium.

[0174] When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the fisheye lens calibration method as described above.

[0175] In order to implement the above embodiments, the present disclosure further provides a computer program product. When the computer program is executed by a processor of an electronic device, the electronic device can execute the fisheye lens calibration method as described above.

[0176] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0177] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A fisheye lens calibration method, characterized in that: include: Acquire a calibration image captured by the fisheye lens to be calibrated, wherein the calibration image displays a plurality of reference objects in the calibration space; Projecting the calibration image onto a set unit sphere to obtain a spherical image, and performing equidistant cylindrical projection on the spherical image to obtain a planar image; wherein the planar image contains multiple grid areas; determining, according to the image position of the center point of each reference object in the plane image, a first spherical coordinate of the center point of each reference object in the spherical coordinate system where the unit sphere is located; Mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system; determining a first reprojection error of the center point of each reference object in each grid area according to a difference between the first spherical coordinate and the second spherical coordinate corresponding to the center point of each reference object; determining a reprojection error of each vertex of each mesh area according to a first reprojection error of a center point of each reference object in each mesh area; determining a second reprojection error of a center point of each reference object in each grid area according to a reprojection error of each vertex in each grid area; Correcting the first spherical coordinates of the center point of each reference object according to the second reprojection error of the center point of each reference object to obtain the corrected first spherical coordinates of the center point of each reference object in each grid area; constructing a reprojection error function according to the corrected first spherical coordinates of the center point of each reference object in each grid area; Determine the extrinsic parameters of the fisheye lens to be calibrated according to the reprojection error function.

2. The method according to claim 1, characterized in that Determining a second reprojection error of a center point of each reference object in each grid area according to the reprojection error of each vertex in each grid area includes: traversing each vertex of each mesh area in turn to determine a plurality of reprojection errors corresponding to a plurality of vertices of each mesh area; Bilinear interpolation is performed on a plurality of reprojection errors corresponding to a plurality of vertices of each of the mesh regions to obtain a second reprojection error of a center point of each of the reference objects in each of the mesh regions.

3. The method according to claim 1, characterized in that The determining, based on the image position of the center point of each reference object in the plane image, the first spherical coordinate of the center point of each reference object in the spherical coordinate system where the unit sphere is located, includes: determining a projection transformation relationship between the planar image and the spherical image; According to the projection transformation relationship and the image position of the center point of each reference object in the plane image, the first spherical coordinates of the center point of each reference object in the spherical coordinate system are determined.

4. The method according to claim 1, wherein Mapping the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system includes: Measuring the spatial position of the center point of each reference object in the world coordinate system of the calibration space by a predetermined distance measuring device to obtain the spatial position of the center point of each reference object in the world coordinate system of the calibration space; A coordinate system transformation is performed on the spatial position of the center point of each reference object in the world coordinate system of the calibration space to obtain the second spherical coordinates of the center point of each reference object in the spherical coordinate system.

5. The method according to any one of claims 1 to 4, characterized in that Before projecting the calibration image onto a set unit sphere to obtain a spherical image, and performing equidistant cylindrical projection on the spherical image to obtain a planar image, the method further includes: Preprocessing the calibration image; The preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing and corrosion processing.

6. A fisheye lens calibration device, characterized in that: include: An acquisition module, configured to acquire a calibration image captured by the fisheye lens to be calibrated, wherein the calibration image displays a plurality of reference objects in the calibration space; a projection module, configured to project the calibration image onto a set unit sphere to obtain a spherical image, and perform equidistant cylindrical projection on the spherical image to obtain a planar image; wherein the planar image contains a plurality of grid areas; a first determining module, configured to determine, based on an image position of the center point of each reference object in the plane image, a first spherical coordinate of the center point of each reference object in the spherical coordinate system where the unit sphere is located; a mapping module, configured to map the spatial position of the center point of each reference object in the world coordinate system of the calibration space to the spherical coordinate system, so as to obtain the second spherical coordinate of the center point of each reference object in the spherical coordinate system; The second determination module is used to determine the first reprojection error of the center point of each reference object in each grid area according to the difference between the first spherical coordinates and the second spherical coordinates corresponding to the center point of each reference object; determine the reprojection error of each vertex in each grid area according to the first reprojection error of the center point of each reference object in each grid area; determine the second reprojection error of the center point of each reference object in each grid area according to the reprojection error of each vertex in each grid area; correct the first spherical coordinate of the center point of each reference object according to the second reprojection error to obtain the corrected first spherical coordinate of the center point of each reference object in each grid area; construct a reprojection error function according to the corrected first spherical coordinates of the center point of each reference object in each grid area; and determine the extrinsic parameters of the fisheye lens to be calibrated according to the reprojection error function.

7. The device according to claim 6, characterized in that The second determining module is further configured to: traversing each vertex of each mesh area in turn to determine a plurality of reprojection errors corresponding to a plurality of vertices of each mesh area; Bilinear interpolation is performed on a plurality of reprojection errors corresponding to a plurality of vertices of each of the mesh regions to obtain a second reprojection error of a center point of each of the reference objects in each of the mesh regions.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the fisheye lens calibration method according to any one of claims 1 to 5.

9. A computer-readable storage medium, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the fisheye lens calibration method according to any one of claims 1 to 5.

Images