External parameter calibration method, device, storage medium and terminal device for a camera

Abstract

The present invention discloses an external parameter calibration method, device, storage medium and terminal device for a camera. The first image is acquired by the camera to be calibrated, and M first feature points in the first image and the first image coordinates of each first feature point are obtained; N second images corresponding to N different positions and the pose parameters of each second image are acquired by the camera on the mobile robot; the feature points in each second image are extracted and matched with the M first feature points; the second image coordinates corresponding to each first feature point in each second image are obtained according to the feature point matching result; the three-dimensional space coordinates of each first feature point are obtained according to the pose parameters of the N second images, the feature point matching result and the second image coordinates corresponding to the M first feature points; the pose parameters of the camera to be calibrated are obtained according to the first image coordinates and the three-dimensional space coordinates of the M first feature points; thereby, the accuracy of camera external parameter calibration can be improved, the calculation is simple, and it is easy to use.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular, to an external parameter calibration method, device, computer-readable storage medium, and terminal device for a camera. Background Art

[0002] The function of camera calibration is to obtain camera parameters. Camera parameters are divided into internal parameters and external parameters. Among them, internal parameters refer to the inherent parameters of the camera, such as the principal optical axis, focal length, and distortion coefficient, which are generally given by the manufacturer or pre-calibrated. External parameters refer to the pose of the camera relative to the world coordinate system, and the external parameters determine the relative position relationship between the camera coordinate and the world coordinate system.

[0003] Currently, there are mainly the following two methods for camera external parameter calibration: (1) The traditional external parameter calibration uses the checkerboard calibration method. This method requires using a checkerboard of a specific size as a calibration board. The user places the checkerboard in front of the camera in several different poses and allows the camera to capture pictures of the checkerboard in different poses, and calculates the external parameters of the camera through these checkerboards in different poses; (2) The 3D reconstruction method based on images. This method captures several unordered images from several perspectives in a room, and the camera to be calibrated also captures one image. These images are combined, and the Structure from Motion (SfM) algorithm is used to determine the poses corresponding to all the images.

[0004] However, the above method (1) is relatively troublesome and not conducive to practical use. Method (2) requires a sufficient number of images, and the calculation is very complex. Moreover, only through image reconstruction, the scale information (i.e., the real distance) cannot be restored, resulting in poor calibration accuracy. Summary of the Invention

[0005] The technical problem to be solved by the embodiments of the present invention is to provide an external parameter calibration method, device, computer-readable storage medium, and terminal device for a camera, which can improve the calibration accuracy of the camera's external parameters, and do not require the use of any calibration board or manual operation by the user, with simple calculation and easy to use.

[0006] To solve the above technical problem, an embodiment of the present invention provides an external parameter calibration method for a camera, including:

[0007] Obtain a first image through the camera to be calibrated, extract feature points from the first image, obtain M first feature points, and obtain the first image coordinate value of each first feature point; where M>0;

[0008] Obtain N second images corresponding to N different positions through the camera on the mobile robot, and obtain the pose parameters corresponding to each second image; where N>0;

[0009] Extract feature points from each of the second images respectively, and match the feature points in each second image with the M first feature points to obtain a feature point matching result;

[0010] Obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result;

[0011] According to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images, obtain the three-dimensional space coordinate values of each first feature point respectively;

[0012] Obtain the pose parameters of the camera to be calibrated according to the first image coordinate values and the three-dimensional space coordinate values of the M first feature points.

[0013] Further, the extracting feature points from each of the second images respectively, and matching the feature points in each second image with the M first feature points to obtain a feature point matching result specifically includes:

[0014] Use the SIFT algorithm or the SURF algorithm to extract feature points from each of the second images respectively;

[0015] Match the feature points in each second image with the M first feature points respectively;

[0016] When the i-th first feature point has a matching feature point in the j-th second image, set the corresponding matching result to a ij = 1;

[0017] When the i-th first feature point has no matching feature point in the j-th second image, set the corresponding matching result to a ij = 0; where i = 1, 2,..., M, j = 1, 2,..., N;

[0018] Obtain the feature point matching result according to all the matching results of the M first feature points in the N second images.

[0019] Further, the method further includes:

[0020] Obtain the total number of matching feature points of each first feature point in the N second images respectively;

[0021] When the total number of matching feature points of any one of the first feature points is less than a preset quantity threshold, delete the first feature point from the M first feature points.

[0022] Further, the method obtains the corresponding second image coordinate value of the \(i\)-th first feature point in the \(j\)-th second image according to the feature point matching result through the following steps:

[0023] When the \(i\)-th first feature point has a matching feature point in the \(j\)-th second image, take the image coordinate value of the matching feature point in the \(j\)-th second image as the corresponding second image coordinate value of the \(i\)-th first feature point in the \(j\)-th second image;

[0024] When the \(i\)-th first feature point has no matching feature point in the \(j\)-th second image, set the corresponding second image coordinate value of the \(i\)-th first feature point in the \(j\)-th second image to \(p\) ij =(0, 0); where \(i = 1, 2, \ldots, M\) and \(j = 1, 2, \ldots, N\).

[0025] Further, when the robot coordinate system corresponding to the mobile robot coincides with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional space coordinate value \(P\) of the \(i\)-th first feature point through the following steps i :

[0026] Solve according to the formula to correspondingly obtain the three-dimensional space coordinate value \(P\) of the \(i\)-th first feature point i ; where \(a\) ij represents the matching result of the \(i\)-th first feature point in the \(j\)-th second image, \(p\) ij represents the corresponding second image coordinate value of the \(i\)-th first feature point in the \(j\)-th second image, represents the image coordinate value after converting the three-dimensional space coordinate value \(P\) of the \(i\)-th first feature point i to the image plane, and \(K\) s represents the internal parameter matrix of the camera on the mobile robot, and \(H\) j represents the pose parameter corresponding to the \(j\)-th second image.

[0027] Further, when the robot coordinate system corresponding to the mobile robot does not coincide with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional space coordinate value \(P\) of the \(i\)-th first feature point through the following steps i :

[0028] Solve according to the formula to correspondingly obtain the three-dimensional space coordinate value \(P\) of the \(i\)-th first feature point i ; where \(a\) ij represents the matching result of the \(i\)-th first feature point in the \(j\)-th second image, \(p\) ij represents the corresponding second image coordinate value of the \(i\)-th first feature point in the \(j\)-th second image, Denote the three-dimensional spatial coordinate value \(P\) of the \(i\)-th first feature point i after being transformed to the image coordinate value in the image plane, \(K\) s denote the internal parameter matrix of the camera on the mobile robot, \(H\) j denote the pose parameter corresponding to the \(j\)-th second image, and \(H\) denote the transformation matrix between the camera coordinate system and the robot coordinate system.

[0029] Furthermore, obtaining the pose parameter of the camera to be calibrated according to the first image coordinate values and three-dimensional spatial coordinate values of the \(M\) first feature points specifically includes:

[0030] Solve according to the formula to correspondingly obtain the pose parameter \(H\) of the camera to be calibrated c ; where \(p\) i denote the first image coordinate value of the \(i\)-th first feature point, denote the image coordinate value after transforming the three-dimensional spatial coordinate value \(P\) of the \(i\)-th first feature point i to the image plane, \(K\) c denote the internal parameter matrix of the camera to be calibrated.

[0031] To solve the above technical problem, an embodiment of the present invention further provides an external parameter calibration device for a camera, including:

[0032] A first image acquisition and processing module, configured to acquire a first image through the camera to be calibrated, extract feature points from the first image, acquire \(M\) first feature points, and acquire the first image coordinate value of each first feature point; where \(M>0\);

[0033] A second image acquisition and processing module, configured to acquire \(N\) second images corresponding to \(N\) different positions through the camera on the mobile robot, and acquire the pose parameter corresponding to each second image; where \(N>0\);

[0034] A feature point extraction and matching module, configured to extract feature points from each second image respectively, and match the feature points in each second image with the \(M\) first feature points to obtain a feature point matching result;

[0035] A feature point image coordinate acquisition module, configured to acquire the second image coordinate value corresponding to each first feature point in each second image according to the feature point matching result;

[0036] A feature point spatial coordinate acquisition module, configured to respectively acquire the three-dimensional spatial coordinate value of each first feature point according to the pose parameters corresponding to the \(N\) second images, the feature point matching result, and the second image coordinate values corresponding to the \(M\) first feature points in the \(N\) second images;

[0037] A camera pose acquisition module, configured to acquire pose parameters of the camera to be calibrated according to first image coordinate values and three-dimensional space coordinate values of the M first feature points.

[0038] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program runs, it controls a device where the computer-readable storage medium is located to execute the external parameter calibration method of the camera described in any one of the above.

[0039] An embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the external parameter calibration method of the camera described in any one of the above.

[0040] Compared with the prior art, an embodiment of the present invention provides an external parameter calibration method, device, computer-readable storage medium, and terminal device for a camera. By using the camera to be calibrated to acquire a first image, extracting feature points from the first image to obtain M first feature points, and acquiring first image coordinate values of each first feature point; using a camera on a mobile robot to acquire N second images corresponding to N different positions, and acquiring pose parameters corresponding to each second image; respectively extracting feature points from each second image, and matching the feature points in each second image with the M first feature points to obtain a feature point matching result; obtaining second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result; respectively obtaining three-dimensional space coordinate values of each first feature point according to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images; and acquiring pose parameters of the camera to be calibrated according to the first image coordinate values and three-dimensional space coordinate values of the M first feature points. Therefore, the accuracy of external parameter calibration of the camera can be improved, and there is no need to use any calibration board or manual operation by the user, and the calculation is simple and easy to use. Description of the Drawings

[0041] Figure 1 is a flowchart of a preferred embodiment of an external parameter calibration method for a camera provided by the present invention;

[0042] Figure 2 is a structural block diagram of a preferred embodiment of an external parameter calibration device for a camera provided by the present invention;

[0043] Figure 3 is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Embodiment

[0044] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art in the technical field of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] The embodiments of the present invention provide an external parameter calibration method for a camera. Refer to Figure 1 As shown, it is a flowchart of a preferred embodiment of an external parameter calibration method for a camera provided by the present invention. The method includes steps S11 to S16:

[0046] Step S11: Obtain a first image through the camera to be calibrated, extract feature points from the first image, obtain M first feature points, and obtain the first image coordinate values of each first feature point; where M>0;

[0047] Step S12: Obtain N second images corresponding to N different positions through the camera on the mobile robot, and obtain the pose parameters corresponding to each second image; where N>0;

[0048] Step S13: Extract feature points from each second image respectively, and match the feature points in each second image with the M first feature points to obtain a feature point matching result;

[0049] Step S14: Obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result;

[0050] Step S15: Respectively obtain the three-dimensional space coordinate values of each first feature point according to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images;

[0051] Step S16: Obtain the pose parameters of the camera to be calibrated according to the first image coordinate values and the three-dimensional space coordinate values of the M first feature points.

[0052] Specifically, first, a first image is acquired by the camera to be calibrated, feature points are extracted from the obtained first image, and accordingly, M (M > 0) first feature points in the first image are obtained, and the first image coordinate values corresponding to each first feature point are obtained. Moreover, N second images corresponding to the mobile robot at N (N > 0) different positions are acquired by the camera on the mobile robot, and the pose parameters of the mobile robot in the world coordinate system corresponding to each second image are respectively obtained, and accordingly, N pose parameters of the mobile robot are obtained (i.e., N viewing parameters, representing the position information and orientation of the mobile robot); then, feature points are respectively extracted from each second image, the feature points extracted from each second image are matched with the M first feature points in the first image, and accordingly, a feature point matching result is obtained, and the second image coordinate values corresponding to each first feature point in each second image are obtained according to the obtained feature point matching result; then, according to the N pose parameters of the mobile robot, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images, the three-dimensional space coordinate values of each first feature point are respectively obtained; finally, according to the first image coordinate values and the three-dimensional space coordinate values of the M first feature points, the pose parameters of the camera to be calibrated are calculated; the pose parameters of the camera to be calibrated are the spatial transformation matrix of the camera to be calibrated or the pose of the camera to be calibrated relative to the world coordinate system, and a point on the image acquired by the camera to be calibrated can be converted into three-dimensional coordinates in the world coordinate system through the pose parameters.

[0053] Among them, the SIFT algorithm (Scale Invariant Feature Transform) or the SURF algorithm (Speeded Up Robust Features) can be used to extract feature points from the obtained first image, and accordingly, M first feature points in the first image are obtained, and based on the image coordinate system of the first image itself, M first image coordinate values corresponding to the M first feature points are obtained; the image coordinates are pixel coordinates. For example, assuming that the first image coordinate corresponding to a certain first feature point is (5, 5), it means that this first feature point is at the position of the 5th row and the 5th column in the first image.

[0054] A mobile robot and a SLAM algorithm (Simultaneous Localization and Mapping) can be used to build a map of the environment in the monitoring area of the camera to be calibrated, and accordingly obtain an environmental grid map corresponding to the monitoring area to determine the world coordinate system. The mobile robot can determine its pose parameters in the world coordinate system based on the built environmental grid map and the SLAM positioning function, so that the pose parameters of the mobile robot corresponding to each second image in the world coordinate system can be obtained accordingly.

[0055] It should be noted that what is actually required in the embodiments of the present invention are the images among the N second images that have a view angle overlap with the first image. If the view angles do not overlap, no matching feature points can be extracted when performing feature point matching between the second image and the first image. Therefore, in order to ensure that there are matching feature points between the N second images and the first image, the camera on the mobile robot can be used to capture images at different positions respectively, perform feature point matching with the first image, find the image with the most matching feature points, and then control the mobile robot to move near the position corresponding to the image with the most matching feature points, and continue to collect N second images corresponding to N different positions (the N different positions are near the position corresponding to the image with the most matching feature points).

[0056] A method for calibrating the external parameters of a camera provided by an embodiment of the present invention includes obtaining a first image through the camera to be calibrated, obtaining M first feature points and their corresponding first image coordinate values based on the first image, obtaining N second images corresponding to N different positions and N pose parameters of the mobile robot through the camera on the mobile robot, respectively matching the feature points in each second image with the M first feature points to obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result, and respectively obtaining the three-dimensional space coordinate values of each first feature point according to the N pose parameters of the mobile robot, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images. Thus, according to the first image coordinate values and the three-dimensional space coordinate values of the M first feature points, the pose parameters of the camera to be calibrated are calculated, which can improve the accuracy of calibrating the external parameters of the camera, and there is no need to use any calibration board or manual operation by the user, and the calculation is simple and easy to use.

[0057] In another preferred embodiment, the step of respectively extracting feature points from each second image, matching the feature points in each second image with the M first feature points, and obtaining the feature point matching result specifically includes:

[0058] Using the SIFT algorithm or the SURF algorithm to respectively extract feature points from each second image;

[0059] Match the feature points in each of the second images with the M first feature points respectively;

[0060] When the i-th first feature point has a matching feature point in the j-th second image, set the corresponding matching result as a ij = 1;

[0061] When the i-th first feature point has no matching feature point in the j-th second image, set the corresponding matching result as a ij = 0; where i = 1, 2,..., M, j = 1, 2,..., N;

[0062] Obtain the feature point matching result according to all the matching results of the M first feature points in the N second images.

[0063] Specifically, in combination with the above embodiments, the SIFT algorithm or the SURF algorithm (or other algorithms) can be used to extract feature points from each of the second images respectively, and match the feature points extracted from each of the second images with the M first feature points in the first image. Taking the matching situation between the i-th first feature point and the feature points extracted from the j-th second image as an example, when the i-th first feature point has a matching feature point in the j-th second image, set the matching result corresponding to the i-th first feature point and the feature points extracted from the j-th second image as a ij = 1; when the i-th first feature point has no matching feature point in the j-th second image, set the matching result corresponding to the i-th first feature point and the feature points extracted from the j-th second image as a ij = 0, where i = 1, 2,..., M, j = 1, 2,..., N; correspondingly, the matching results corresponding to each first feature point and the feature points extracted from each second image can be obtained, and according to all the matching results corresponding to the M first feature points and the feature points extracted from the N second images, the above feature point matching result can be obtained.

[0064] It should be noted that the feature point matching method can adopt a suitable matching method provided by the prior art, and the embodiments of the present invention do not make specific limitations.

[0065] In yet another preferred embodiment, the method further includes:

[0066] Obtain the total number of matching feature points of each first feature point in the N second images respectively;

[0067] When the total number of matching feature points of any one first feature point is less than a preset number threshold, delete the first feature point from the M first feature points.

[0068] Specifically, in combination with the above embodiments, after obtaining the feature point matching result, before obtaining the second image coordinate value corresponding to each first feature point in each second image according to the feature point matching result, the M first feature points can be screened accordingly according to the feature point matching result. In specific implementation, the total number of matching feature points corresponding to each first feature point in the N second images can be obtained first, and then the total number of obtained matching feature points is compared with a preset number threshold. When it is determined that the total number of matching feature points corresponding to any one of the first feature points is less than the preset number threshold, the first feature point is deleted from the M first feature points. After that, based on the M - 1 first feature points, the second image coordinate value corresponding to each first feature point in each second image is obtained according to the feature point matching result.

[0069] In yet another preferred embodiment, the method obtains the second image coordinate value corresponding to the i-th first feature point in the j-th second image through the following steps:

[0070] When the i-th first feature point has a matching feature point in the j-th second image, the image coordinate value of the matching feature point in the j-th second image is used as the second image coordinate value corresponding to the i-th first feature point in the j-th second image;

[0071] When the i-th first feature point has no matching feature point in the j-th second image, the second image coordinate value corresponding to the i-th first feature point in the j-th second image is set to p ij =(0, 0); where i = 1, 2,..., M and j = 1, 2,..., N.

[0072] Specifically, in combination with the above embodiments, the method for obtaining the second image coordinate value corresponding to each first feature point in each second image is the same. Here, taking the example of obtaining the second image coordinate value corresponding to the i-th first feature point in the j-th second image, in combination with the obtained feature point matching result, when the i-th first feature point has a matching feature point (i.e., a ij = 1) in the j-th second image, the image coordinate value of the matching feature point in the j-th second image that matches the i-th first feature point is used as the second image coordinate value corresponding to the i-th first feature point in the j-th second image; when the i-th first feature point has no matching feature point (i.e., a ij = 0) in the j-th second image, the second image coordinate value corresponding to the i-th first feature point in the j-th second image is set to p ij =(0, 0).

[0073] In yet another preferred embodiment, when the robot coordinate system corresponding to the mobile robot coincides with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional spatial coordinate value P of the i-th first feature point through the following steps i :

[0074] Solve according to the formula to correspondingly obtain the three-dimensional spatial coordinate value P of the i-th first feature point i ; where a ij represents the matching result of the i-th first feature point in the j-th second image, p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value P of the i-th first feature point i to the image plane, K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image

[0075] It should be noted that the mobile robot itself has a corresponding robot coordinate system (for example, if the positioning sensor of the mobile robot is a lidar, the robot coordinate system is the positioning coordinate system, that is, the coordinate system of the lidar), and the camera on the mobile robot itself also has a corresponding camera coordinate system. The camera itself is generally rotatable. Therefore, the robot coordinate system and the camera coordinate system may coincide or may not coincide

[0076] Specifically, in combination with the above embodiments, when the robot coordinate system corresponding to the mobile robot coincides with the camera coordinate system corresponding to the camera on the mobile robot, the embodiment of the present invention is based on the obtained N pose parameters corresponding to the mobile robot, the feature point matching results, and the second image coordinate values corresponding to the M first feature points in the N second images, and can be solved by the formula to correspondingly obtain the three-dimensional spatial coordinate value P of the i-th first feature point i , a ij represents the matching result of the i-th first feature point in the j-th second image, p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value P of the i-th first feature point i to the image plane, K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter of the mobile robot corresponding to the j-th second image, represents the Euclidean distance between two vectors

[0077] Among them, the internal parameter matrix of the camera on the mobile robot is generally a known parameter, specifically expressed as f α and f β u0 and v0 are all internal parameters of the camera in the camera imaging model. f α and f β respectively represent the focal lengths corresponding to the X-axis and Y-axis of the camera, and (u0, v0) represents the intersection point of the principal optical axis of the camera and the image plane.

[0078] In another preferred embodiment, when the robot coordinate system corresponding to the mobile robot does not coincide with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional spatial coordinate value Pi of the i-th first feature point through the following steps i :

[0079] Solve according to the formula to correspondingly obtain the three-dimensional spatial coordinate value Pi of the i-th first feature point i ; where a ij represents the matching result of the i-th first feature point in the j-th second image, and p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value Pi of the i-th first feature point i to the image plane, K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image, and H represents the transformation matrix between the camera coordinate system and the robot coordinate system.

[0080] Specifically, in combination with the above embodiments, when the robot coordinate system corresponding to the mobile robot does not coincide with the camera coordinate system corresponding to the camera on the mobile robot, the transformation matrix between the camera coordinate system corresponding to the camera on the mobile robot and the robot coordinate system corresponding to the mobile robot can be obtained and denoted as H. Then, based on the obtained N pose parameters, feature point matching results of the mobile robot, and the second image coordinate values corresponding to M first feature points in N second images, the present invention embodiment can solve through the formula to correspondingly obtain the three-dimensional spatial coordinate value Pi of the i-th first feature point i , a ij represents the matching result of the i-th first feature point in the j-th second image, and p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value Pi of the i-th first feature point i to the image plane, Ks represents the internal parameter matrix of the camera on the mobile robot, H j represents the pose parameters of the mobile robot corresponding to the j-th second image, and H represents the transformation matrix between the camera coordinate system corresponding to the camera on the mobile robot and the robot coordinate system corresponding to the mobile robot represents the Euclidean distance between two vectors

[0081] In yet another preferred embodiment, obtaining the pose parameters of the camera to be calibrated according to the first image coordinate values and three-dimensional space coordinate values of the M first feature points specifically includes:

[0082] Solve according to the formula to correspondingly obtain the pose parameters H of the camera to be calibrated c ; where p i represents the first image coordinate value of the i-th first feature point represents the image coordinate value after converting the three-dimensional space coordinate value P of the i-th first feature point i to the image plane, and K c represents the internal parameter matrix of the camera to be calibrated

[0083] Specifically, in combination with the above embodiments, after obtaining the first image coordinate values and three-dimensional space coordinate values corresponding to the M first feature points in the first image, the formula can be used for solving to correspondingly obtain the pose parameters H of the camera to be calibrated c , p i represents the first image coordinate value of the i-th first feature point represents the image coordinate value after converting the three-dimensional space coordinate value P of the i-th first feature point i to the image plane, and K c represents the internal parameter matrix of the camera to be calibrated, which is generally a known parameter

[0084] It should be noted that, given the first image coordinate values and three-dimensional space coordinate values corresponding to the M first feature points, by projecting the M three-dimensional space coordinate values onto the two-dimensional image, the transformation matrix H of the camera to be calibrated can be solved c .

[0085] The embodiment of the present invention also provides an external parameter calibration device for a camera. Refer to Figure 2 shown, which is a structural block diagram of a preferred embodiment of an external parameter calibration device for a camera provided by the present invention. The device includes:

[0086] The first image acquisition and processing module 11 is configured to acquire a first image through a camera to be calibrated, extract feature points from the first image, obtain M first feature points, and obtain the first image coordinate values of each first feature point; where M>0;

[0087] The second image acquisition and processing module 12 is configured to acquire N second images corresponding to N different positions through a camera on a mobile robot, and obtain the pose parameters corresponding to each second image; where N>0;

[0088] The feature point extraction and matching module 13 is configured to extract feature points from each second image respectively, and match the feature points in each second image with the M first feature points to obtain a feature point matching result;

[0089] The feature point image coordinate acquisition module 14 is configured to obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result;

[0090] The feature point spatial coordinate acquisition module 15 is configured to respectively obtain the three-dimensional spatial coordinate values of each first feature point according to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images;

[0091] The camera pose acquisition module 16 is configured to obtain the pose parameters of the camera to be calibrated according to the first image coordinate values and the three-dimensional spatial coordinate values of the M first feature points.

[0092] Preferably, the feature point extraction and matching module 13 specifically includes:

[0093] The feature point extraction unit is configured to extract feature points from each second image respectively by using the SIFT algorithm or the SURF algorithm;

[0094] The feature point matching unit is configured to match the feature points in each second image with the M first feature points respectively;

[0095] The first matching result setting unit is configured to set the corresponding matching result to a ij =1 when the i-th first feature point has a matching feature point in the j-th second image;

[0096] The second matching result setting unit is configured to set the corresponding matching result to a ij =0 when the i-th first feature point has no matching feature point in the j-th second image; where i = 1, 2, …, M, j = 1, 2, …, N;

[0097] A feature point matching result acquisition unit, configured to correspondingly obtain the feature point matching result according to all the matching results of the M first feature points in the N second images.

[0098] Preferably, the apparatus further includes a feature point screening module, configured to:

[0099] Obtain the total number of matching feature points of each first feature point in the N second images respectively;

[0100] When the total number of matching feature points of any one of the first feature points is less than a preset quantity threshold, delete the first feature point from the M first feature points.

[0101] Preferably, the feature point image coordinate acquisition module 14 specifically includes:

[0102] A first feature point image coordinate acquisition unit, configured to use the image coordinate value of the matching feature point in the j-th second image as the corresponding second image coordinate value of the i-th first feature point in the j-th second image when the i-th first feature point has a matching feature point in the j-th second image;

[0103] A second feature point image coordinate acquisition unit, configured to set the corresponding second image coordinate value of the i-th first feature point in the j-th second image to p ij =(0, 0); where i = 1, 2,..., M, j = 1, 2,..., N.

[0104] Preferably, when the robot coordinate system corresponding to the mobile robot coincides with the camera coordinate system corresponding to the camera on the mobile robot, the feature point spatial coordinate acquisition module 15 is specifically configured to:

[0105] Solve according to the formula to correspondingly obtain the three-dimensional spatial coordinate value P i of the i-th first feature point; where a ij represents the matching result of the i-th first feature point in the j-th second image, p ij represents the corresponding second image coordinate value of the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value P i of the i-th first feature point to the image plane, K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image.

[0106] Preferably, when the robot coordinate system corresponding to the mobile robot does not coincide with the camera coordinate system corresponding to the camera on the mobile robot, the feature point spatial coordinate acquisition module 15 is specifically configured to:

[0107] Solve according to the formula to obtain the three-dimensional spatial coordinate value P of the i-th first feature point accordingly i ; where a ij represents the matching result of the i-th first feature point in the j-th second image, and p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image, represents the image coordinate value after converting the three-dimensional spatial coordinate value P of the i-th first feature point i to the image plane, K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image, and H represents the transformation matrix between the camera coordinate system and the robot coordinate system.

[0108] Preferably, the camera pose acquisition module 16 is specifically configured to:

[0109] Solve according to the formula to obtain the pose parameter H of the camera to be calibrated accordingly c ; where p i represents the first image coordinate value of the i-th first feature point, represents the image coordinate value after converting the three-dimensional spatial coordinate value P of the i-th first feature point i to the image plane, and K c represents the internal parameter matrix of the camera to be calibrated.

[0110] It should be noted that an external parameter calibration device for a camera provided in an embodiment of the present invention can implement all the processes of the external parameter calibration method for a camera described in any of the above embodiments. The functions and achieved technical effects of each module and unit in the device respectively correspond to those of the external parameter calibration method for a camera described in the above embodiments, and will not be elaborated here.

[0111] An embodiment of the present invention also provides a computer-readable storage medium, and the computer-readable storage medium includes a stored computer program; wherein, the computer program controls the device where the computer-readable storage medium is located to execute the external parameter calibration method for a camera described in any of the above embodiments when running.

[0112] An embodiment of the present invention also provides a terminal device, see Figure 3As shown in the figure, it is a structural block diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 10, a memory 20, and a computer program stored in the memory 20 and configured to be executed by the processor 10. When the processor 10 executes the computer program, it implements the external parameter calibration method of the camera described in any of the above embodiments.

[0113] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2,...). The one or more modules / units are stored in the memory 20 and executed by the processor 10 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the terminal device.

[0114] The processor 10 can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 10 can also be any conventional processor. The processor 10 is the control center of the terminal device and connects various parts of the terminal device through various interfaces and circuits.

[0115] The memory 20 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc., and the data storage area can store relevant data, etc. In addition, the memory 20 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., or the memory 20 can also be other volatile solid-state storage devices.

[0116] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that Figure 3The structural block diagram is merely an example of the above terminal device, and does not constitute a limitation on the terminal device. It may include more or fewer components than those shown, or combine certain components, or different components.

[0117] In summary, for the external parameter calibration method, device, computer-readable storage medium, and terminal device of a camera provided in the embodiments of the present invention, a first image is obtained by the camera to be calibrated, and M first feature points and their corresponding first image coordinate values are obtained based on the first image. N second images corresponding to N different positions and N pose parameters of the mobile robot are obtained by the camera on the mobile robot, and the feature points in each second image are respectively matched with the M first feature points to obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result. Then, according to the N pose parameters of the mobile robot, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images, the three-dimensional space coordinate values of each first feature point are respectively obtained. Thus, according to the first image coordinate values and the three-dimensional space coordinate values of the M first feature points, the pose parameters of the camera to be calibrated are calculated, which can improve the accuracy of the external parameter calibration of the camera, and there is no need to use any calibration board or manual operation by the user, and the calculation is simple and easy to use.

[0118] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. An external parameter calibration method for a camera, characterized in that, Including: Obtain a first image through a camera to be calibrated, extract feature points from the first image, obtain M first feature points, and obtain the first image coordinate values of each first feature point; where M>0; Obtain N second images corresponding to N different positions through a camera on a mobile robot, and obtain the pose parameters corresponding to each second image; where N>0; Extract feature points from each second image respectively, and match the feature points in each second image with the M first feature points to obtain a feature point matching result; Obtain the second image coordinate values corresponding to each first feature point in each second image according to the feature point matching result; Obtain the three-dimensional space coordinate values of each first feature point respectively according to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images; Obtain the pose parameters of the camera to be calibrated according to the first image coordinate values and three-dimensional space coordinate values of the M first feature points; The step of obtaining the pose parameters of the camera to be calibrated according to the first image coordinate values and three-dimensional space coordinate values of the M first feature points specifically includes: Solve according to the formula to obtain the pose parameter H of the camera to be calibrated accordingly c ; where p i represents the first image coordinate value of the i-th first feature point, represents the image coordinate value after converting the three-dimensional space coordinate value P i of the i-th first feature point to the image plane, and K c represents the internal parameter matrix of the camera to be calibrated.

2. The external parameter calibration method of the camera according to claim 1, wherein, The step of extracting feature points from each second image respectively, and matching the feature points in each second image with the M first feature points to obtain a feature point matching result specifically includes: Use the SIFT algorithm or SURF algorithm to extract feature points from each second image respectively; Match the feature points in each second image with the M first feature points respectively; When the i-th first feature point has a matching feature point in the j-th second image, set the corresponding matching result to a ij = 1; When the i-th first feature point has no matching feature point in the j-th second image, set the corresponding matching result to a ij = 0; where i = 1, 2, …, M, and j = 1, 2, …, N; Obtain the feature point matching result according to all the matching results of the M first feature points in the N second images.

3. The external parameter calibration method of the camera according to claim 1, characterized in that, The method further includes: Obtain the total number of matching feature points of each first feature point in the N second images respectively; When the total number of matching feature points of any one first feature point is less than a preset number threshold, delete the first feature point from the M first feature points.

4. The external parameter calibration method of the camera according to claim 1, characterized in that The method obtains the second image coordinate values corresponding to the i-th first feature point in the j-th second image according to the feature point matching result through the following steps: When the i-th first feature point has a matching feature point in the j-th second image, use the image coordinate values of the matching feature points in the j-th second image as the second image coordinate values corresponding to the i-th first feature point in the j-th second image; When the i-th first feature point has no matching feature point in the j-th second image, set the second image coordinate value corresponding to the i-th first feature point in the j-th second image to p ij = (0, 0); where i = 1, 2,..., M and j = 1, 2,..., N.

5. The external parameter calibration method of the camera according to claim 1, characterized in that, When the robot coordinate system corresponding to the mobile robot coincides with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional spatial coordinate value P of the i-th first feature point through the following steps i : Solve according to the formula to obtain the three-dimensional spatial coordinate value P i of the i-th first feature point accordingly; Among them, a ij represents the matching result of the i-th first feature point in the j-th second image, and p ij represents the corresponding second image coordinate value of the i-th first feature point in the j-th second image. represents the image coordinate value after converting the three-dimensional space coordinate value P of the i-th first feature point i to the image plane, and K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image.

6. The external parameter calibration method of the camera according to claim 1, characterized in that, When the robot coordinate system corresponding to the mobile robot does not coincide with the camera coordinate system corresponding to the camera on the mobile robot, the method obtains the three-dimensional spatial coordinate value P of the i-th first feature point through the following steps i : Solve according to the formula to obtain the three-dimensional space coordinate value P of the i-th first feature point accordingly i ; where a ij represents the matching result of the i-th first feature point in the j-th second image, and p ij represents the second image coordinate value corresponding to the i-th first feature point in the j-th second image. represents the image coordinate value after converting the three-dimensional space coordinate value P of the i-th first feature point i to the image plane, and K s represents the internal parameter matrix of the camera on the mobile robot, and H j represents the pose parameter corresponding to the j-th second image. H represents the transformation matrix between the camera coordinate system and the robot coordinate system.

7. An external parameter calibration device for a camera, characterized in that, Including: A first image acquisition and processing module, configured to obtain a first image through a camera to be calibrated, extract feature points from the first image, obtain M first feature points, and obtain the first image coordinate values of each first feature point; where M>0; A second image acquisition and processing module, configured to obtain N second images corresponding to N different positions through a camera on a mobile robot, and obtain the pose parameters corresponding to each second image; where N>0; The feature point extraction and matching module is used to extract feature points from each second image respectively, and match the feature points in each second image with the M first feature points to obtain the feature point matching result; The feature point image coordinate acquisition module is used to obtain the second image coordinate value corresponding to each first feature point in each second image according to the feature point matching result; The feature point spatial coordinate acquisition module is used to obtain the three-dimensional spatial coordinate value of each first feature point respectively according to the pose parameters corresponding to the N second images, the feature point matching result, and the second image coordinate values corresponding to the M first feature points in the N second images; The camera pose acquisition module is used to obtain the pose parameters of the camera to be calibrated according to the first image coordinate values and three-dimensional spatial coordinate values of the M first feature points; Specifically, the camera pose acquisition module is used for: Solve according to the formula to obtain the pose parameter H of the camera to be calibrated accordingly c ; where p i represents the first image coordinate value of the i-th first feature point, represents the image coordinate value after converting the three-dimensional space coordinate value P i of the i-th first feature point to the image plane, and K c represents the internal parameter matrix of the camera to be calibrated.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, the computer program controls the device where the computer-readable storage medium is located to execute the external parameter calibration method of the camera according to any one of claims 1 to 6 when running.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. The processor implements the external parameter calibration method of the camera according to any one of claims 1 to 6 when executing the computer program.

Images