Camera calibration method and device, electronic device and storage medium

Abstract

The present application relates to a camera calibration method and device, an electronic device and a storage medium. The method includes: obtaining a target target image obtained by the camera to be calibrated performing image acquisition on a calibration plate; dividing the target target image into regions to obtain at least two non-overlapping image regions; obtaining sub-images in each image region of the target target image; obtaining parameter information of each marker point in the sub-image; based on the parameter information of each marker point in all sub-images corresponding to each image region, calculating the calibration parameters of the camera to be calibrated in each image region, and obtaining at least two calibration parameters corresponding to the camera to be calibrated. The present application can effectively overcome the problem that when a set of calibration parameters is used, the calibration accuracy of the camera to be calibrated in different image regions is different, resulting in a set of calibration parameters being unable to accurately calibrate the points to be calibrated in different image regions. Therefore, a plurality of sets of calibration parameters can be used to enable the camera to be calibrated to achieve a higher-precision calibration effect.

Description

Technical Field

[0001] The present application relates to the field of camera calibration technology, and in particular to a camera calibration method and device, an electronic device and a storage medium. Background Art

[0002] With the advancement of science and technology, the application of machine vision is widely used in fields such as measurement, medicine, security, biology, etc. Using cameras to obtain spatial information is particularly important for robots to achieve high-precision positioning of objects, accurate grasping and placement, and high-precision movement.

[0003] At present, high-precision calibration cameras are widely used in the positioning, detection, and measurement of industrial products. That is, the camera obtains the pixel information of the object, and then uses the internal parameters, external parameters, and camera distortion coefficients of the high-precision calibration camera to solve the spatial information of the measured object by back-projection. High-precision camera calibration is divided into 2D target calibration and 3D target calibration. Although 3D targets can collect images at one time to complete camera parameter calibration, 3D high-precision targets are difficult to process and expensive, and are not widely used. At present, most camera calibrations still use 2D plane calibration methods, and there is still a problem of poor calibration accuracy in the edge area of ​​the camera.

[0004] It can be seen from this that the camera calibration method in the related art has the problem of low calibration accuracy. Summary of the invention

[0005] In order to solve the technical problem of low calibration accuracy existing in the above-mentioned camera calibration method, the present application provides a camera calibration method and device, an electronic device and a storage medium.

[0006] In a first aspect, an embodiment of the present application provides a camera calibration method, comprising:

[0007] Acquire a target image obtained by the camera to be calibrated acquiring an image of a calibration plate, wherein the calibration plate has at least one marker position;

[0008] Dividing the target image into regions to obtain at least two non-overlapping image regions;

[0009] Acquire a sub-image within each image area in the target image;

[0010] Acquire parameter information of each marker point in the sub-image, wherein the marker point is obtained after performing image acquisition on the marker position;

[0011] Based on the parameter information of each of the marker points in all the sub-images corresponding to each of the image regions, the calibration parameters of the camera to be calibrated in each of the image regions are calculated to obtain at least two calibration parameters corresponding to the camera to be calibrated.

[0012] Optionally, as in the aforementioned method, dividing the target image into regions to obtain at least two non-overlapping image regions comprises:

[0013] Determine each first marker point in the target image;

[0014] For each of the first marker points, the pixel coordinates of the first marker point and the world coordinates of the first marker position corresponding to the first marker point are obtained, and a reprojection error of the first marker point is calculated according to the pixel coordinates of the first marker point and the world coordinates of the first marker position, wherein the first marker position corresponding to the first marker point is the marker position collected as the first marker point in the calibration plate;

[0015] According to the reprojection error size of each of the first marking points in the target image, the target image is divided into at least two image areas.

[0016] Optionally, as in the aforementioned method, dividing the target image into regions to obtain at least two non-overlapping image regions comprises:

[0017] Obtaining distortion information of the target image;

[0018] According to different distortion information, the target image is divided into at least two image areas.

[0019] Optionally, as in the aforementioned method, the calculating the calibration parameters of the camera to be calibrated in each image area based on the parameter information of each of the marker points in all the sub-images in each image area comprises:

[0020] For any image region, perform the following steps:

[0021] Determine each second marker point in all the sub-images of the image area, and a second marker position corresponding to each second marker point, wherein the second marker position is a marker position collected as the second marker point in the calibration plate;

[0022] Based on the pixel coordinates of the second marker point and the world coordinates of the second marker position, a first initial value of the target internal and external parameters is calculated, wherein, for each of the second marker points, the parameter information of the second marker point includes the pixel coordinates of the second marker point and the world coordinates of the second marker position corresponding to the second marker point, and the target internal and external parameters are the internal and external parameters corresponding to the camera to be calibrated and the image area;

[0023] According to the first initial value of the target internal and external parameters, a first target value of the target internal and external parameters and a second target value of the target distortion coefficient are calculated, wherein the target distortion coefficient is the distortion coefficient corresponding to the camera to be calibrated and the image area, and the calibration parameters are the target internal and external parameters and the target distortion coefficient.

[0024] Optionally, as in the aforementioned method, the calculating the first initial value of the target internal and external parameters based on the pixel coordinates of the second marker point and the world coordinates of the second marker position includes:

[0025] Determine a plurality of coordinate pairs corresponding to the image area, wherein each of the coordinate pairs includes a pixel coordinate of a second marker point and a world coordinate of the second marker corresponding to the second marker point;

[0026] The first initial value of the internal and external parameters of the target is obtained by calculating each of the coordinate pairs based on the Zhang Zhengyou calibration method.

[0027] Optionally, as in the aforementioned method, calculating the first target value of the target internal and external parameter and the second target value of the target distortion coefficient according to the first initial value of the target internal and external parameter includes:

[0028] An iterative calculation is performed based on the first initial value of the target internal and external parameter and the second initial value of the target distortion coefficient to obtain the first target value of the target internal and external parameter and the second target value of the target distortion coefficient, wherein the second initial value is 0.

[0029] In a second aspect, an embodiment of the present application provides a camera-based object calibration method, comprising:

[0030] Acquire a target image obtained after the target camera acquires an image of the target object, wherein the target camera is pre-calibrated by the camera calibration method as described in any of the preceding items, and the target camera includes at least two calibration parameters;

[0031] Determine the pixel coordinates of the points to be calibrated in the target image;

[0032] Determine, according to the pixel coordinates of the to-be-calibrated point, a target area in the target image where the to-be-calibrated point is located;

[0033] Acquiring target calibration parameters corresponding to the target area;

[0034] The target calibration parameters are used to calibrate the points to be calibrated.

[0035] In a third aspect, an embodiment of the present application provides a camera calibration device, including:

[0036] A first acquisition module is used to acquire a target image obtained by the camera to be calibrated acquiring an image of a calibration plate, wherein the calibration plate has at least one mark position;

[0037] A division module, used for dividing the target image into regions to obtain at least two non-overlapping image regions;

[0038] A second acquisition module, used for acquiring a sub-image in each image area of ​​the target image;

[0039] A third acquisition module is used to acquire parameter information of each marker point in the sub-image, wherein the marker point is obtained after performing image acquisition on the marker position;

[0040] The calibration parameter module is used to calculate the calibration parameters of the camera to be calibrated in each image area based on the parameter information of each of the marker points in all the sub-images corresponding to each image area, and obtain at least two calibration parameters corresponding to the camera to be calibrated.

[0041] In a fourth aspect, an embodiment of the present application provides a camera-based object calibration device, comprising:

[0042] A first acquisition module, configured to acquire a target image obtained after a target camera acquires an image of a target object, wherein the target camera is pre-calibrated by the camera calibration method described in any of the preceding items, and the target camera includes at least two calibration parameters;

[0043] A first determination module is used to determine the pixel coordinates of the to-be-calibrated points in the target image;

[0044] A second determination module is used to determine the target area where the point to be calibrated is located in the target image according to the pixel coordinates of the point to be calibrated;

[0045] A second acquisition module, used to acquire target calibration parameters corresponding to the target area;

[0046] The calibration module is used to calibrate the point to be calibrated using the target calibration parameters.

[0047] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0048] The memory is used to store computer programs;

[0049] The processor is used to implement any of the methods described above when executing the computer program.

[0050] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the storage medium includes a stored program, wherein the program executes a method as described in any of the preceding items when running.

[0051] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0052] The method provided in the embodiment of the present application adopts a regional calibration strategy to calculate at least two sets of calibration parameters for calibrating the camera to be calibrated, so as to obtain the calibration parameters corresponding to each imaging area. Then, when the calibrated point to be calibrated is calibrated by the calibrated camera, the position information of the point to be calibrated can be more accurately calculated based on the calibration parameters corresponding to the imaging area where the point to be calibrated is located. Therefore, by combining the two sets of calibration parameters, the problem that when a set of calibration parameters is used, the calibration accuracy of the camera to be calibrated in different image areas is different, resulting in a set of calibration parameters being unable to accurately calibrate the points to be calibrated in different image areas can be effectively overcome. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0055] Figure 1 A flowchart of a camera calibration method provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of a camera calibration method provided by another embodiment of the present application;

[0057] Figure 3 A flowchart of a camera calibration method provided by another embodiment of the present application;

[0058] Figure 4 A flowchart of a camera-based object calibration method provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of image area division provided in an embodiment of the present application;

[0060] Figure 6 A block diagram of a camera calibration device provided in an embodiment of the present application;

[0061] Figure 7 A block diagram of a camera-based object calibration device provided in an embodiment of the present application;

[0062] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0064] According to one aspect of an embodiment of the present application, a camera calibration method is provided. Optionally, in this embodiment, the camera calibration method can be applied to a terminal or a server.

[0065] The camera calibration method of the embodiment of the present application can be executed by a server, or by a terminal, or by both a server and a terminal. The camera calibration method of the embodiment of the present application can be executed by a terminal or by a client installed thereon.

[0066] Take the camera calibration method in this embodiment executed by the terminal as an example. Figure 1 As shown, the steps include:

[0067] Step S101, obtaining a target image obtained by acquiring an image of a calibration plate by a camera to be calibrated, wherein the calibration plate has at least one mark position.

[0068] The camera calibration method in this embodiment can be applied to the scene of calibrating a high-precision camera, and can also be applied to the scene of calibrating other types of cameras. In the embodiment of the present application, the camera calibration method is described by taking the calibration of a high-precision camera as an example. For other types of cameras, the camera calibration method is also applicable if there is no contradiction.

[0069] Taking the calibration of a high-precision camera as an example, the high-precision camera is used to calibrate different image areas of the high-precision camera by collecting various target images of calibration plates in multiple postures.

[0070] The calibration plate may be a plane with a checkerboard pattern of alternating black and white squares on the surface.

[0071] The calibration points may be diagonal points of each black and white grid on the calibration board.

[0072] Generally, the camera to be calibrated can be used to capture images of calibration plates in different postures (for example, at least three different postures, and the specific number of postures can be selected according to the actual application), and the calibration plate in each posture will capture a corresponding target image.

[0073] Step S102 , dividing the target image into regions to obtain at least two non-overlapping image regions.

[0074] The image area can be an area at a specific position in the image obtained when the camera to be calibrated shoots any object (for example, a calibration plate) within a specified distance range. In addition, the image area is independent of the content in the image. For any image taken by the camera to be calibrated under the above conditions, each image area is at the same position in each image.

[0075] Furthermore, there is no overlapping area between the image areas, and the sum of the image areas is equal to the total image area of ​​the target image.

[0076] For example, Figure 5 As shown, when the target image is divided into regions to obtain the first image region P and the second image region Q, the first image region can be a region at a specific position in the image obtained when the camera to be calibrated shoots any object (e.g., a calibration plate) within a specified distance interval, and the first image region is independent of the content in the image, and is at the same position in each image for any image shot by the camera to be calibrated under the above conditions. For example, when the image A shot is an image of 1000×500 pixels, the first image region can be a rectangular region of 600×300 pixels in the image, and the first image region is 200 pixels away from the top edge of image A, 200 pixels away from the bottom edge of image A, 100 pixels away from the left edge of image A, and 100 pixels away from the right edge of image A.

[0077] Similarly, the second image area can also be an area at a specific position in the image obtained when the camera to be calibrated shoots any object (for example, a calibration plate) within a specified distance interval, and, in general, there is no overlapping area between the second image area and the first image area, and the second image area is independent of the content in the image, and is at the same position in each image for any image taken by the camera to be calibrated under the above conditions. For example, other areas except the first image area in the captured image A can be used as the second image area; an area as the second image area can also be determined in image A as the second image area, and when more image areas need to be determined, the third image area, the fourth image area, and so on can be determined in this way.

[0078] Step S103, obtaining a sub-image in each image area of ​​the target image.

[0079] After obtaining each image area and the target target image, the sub-image within the image area can be determined in the target target image, and each target target image has a sub-image within the image area, so the number of sub-images within the image area is consistent with the number of target target images.

[0080] For example, when the first image area and the second image area are obtained by division, the first sub-image of the target target image located in the first image area and the second sub-image of the target target image located in the second image area can be determined based on the first image area. For example, the first sub-image and the second sub-image can be determined in the target target image based on the coordinates of the first pixel corresponding to the first image area and the coordinates of the second pixel corresponding to the second image area. Generally, a target target image corresponding to the calibration plate of each posture will be obtained, so there will be multiple target target images, and then there will be multiple first sub-images and multiple second sub-images.

[0081] Step S104, obtaining parameter information of each marker point in the sub-image, wherein the marker point is obtained after image acquisition of the marker position.

[0082] The marker point may be image information existing in the sub-image (ie, the target image) and corresponding one-to-one to the marker position.

[0083] For the parameter information of each marker point, the parameter information may include but is not limited to the pixel coordinates of the marker point and the world coordinates of the marker position corresponding to the marker point.

[0084] For example, when the target image A is obtained after image acquisition of the calibration plate of posture A, each marker point in the sub-image within the image area of ​​the target image A has a marker position in the calibration plate of posture A corresponding thereto.

[0085] Step S105 , based on the parameter information of each marker point in all sub-images corresponding to each image area, the calibration parameters of the camera to be calibrated in each image area are calculated to obtain at least two calibration parameters corresponding to the camera to be calibrated.

[0086] After all sub-images in each image area are obtained, for each image area, the calibration parameters of the camera to be calibrated can be calculated based on the parameter information of each marker point in all sub-images in the image area, and the calibration parameters of the camera to be calibrated corresponding to the image area are obtained. In this way, the calibration parameters of the camera to be calibrated corresponding to each image area can be obtained.

[0087] Furthermore, a correspondence between each image region and the calibration parameters can be established, so that the calibration parameters corresponding to the to-be-calibrated points can be determined by determining the image region where the to-be-calibrated points are located, thereby accurately calibrating the to-be-calibrated points.

[0088] For example, after dividing the first image area and the second image area, a first sub-image is obtained as a sub-image corresponding to the first image area, and the calibration parameters of the camera to be calibrated are calculated based on each marker point in the first sub-image, thereby obtaining the first calibration parameters of the camera to be calibrated in the first image area.

[0089] The calibration parameters include the intrinsic and extrinsic parameters of the camera and the lens distortion coefficient. Therefore, the first calibration parameters correspond to the intrinsic and extrinsic parameters of the camera to be calibrated in the first image area and the lens distortion coefficient.

[0090] A first sub-image corresponding to the first image area is obtained, and calibration parameters of the camera to be calibrated are calculated based on each marker point in the second sub-image, thereby obtaining second calibration parameters of the camera to be calibrated in the second image area.

[0091] Similar to the first calibration parameters, the second calibration parameters correspond to the intrinsic and extrinsic parameters and the lens distortion coefficient of the camera to be calibrated in the second image area.

[0092] After obtaining the first calibration parameter and the second calibration parameter, the first calibration parameter and the second calibration parameter can be used as calibration results, so as to facilitate the later position calibration of the to-be-calibrated points in the image captured by the to-be-calibrated camera based on the first calibration parameter and the second calibration parameter to determine the world coordinates of the to-be-calibrated points.

[0093] In summary, through the method in this embodiment, a regional calibration strategy is adopted to calculate at least two sets of calibration parameters for calibrating the camera to be calibrated. By combining the at least two sets of calibration parameters, it is possible to effectively overcome the problem that a set of calibration parameters cannot adapt to the different calibration accuracies of the camera to be calibrated in different image areas when used, resulting in a set of calibration parameters being unable to accurately calibrate the points to be calibrated in different image areas. Therefore, the at least two sets of calibration parameters can enable the camera to be calibrated to achieve a higher precision calibration effect; further, the method only needs to use a low-order distortion imaging model to complete the camera calibration, which reduces the complexity of calculating the camera parameters of the high-order distortion imaging model, and at the same time reduces the requirements for the accuracy of the 2D target, with low calculation complexity and easier implementation.

[0094] As an optional implementation, the method as described above includes at least two target images; the step S102 of dividing the target image into regions to obtain at least two non-overlapping image regions includes the following steps:

[0095] Step S201, determining each first marker point in each target image.

[0096] The first marker point may be a marker point existing in the target image. In addition, each target image may contain multiple marker points. Generally, the postures of the calibration plates collected as each target image are different.

[0097] For example, when there are target target image a, target target image b and target target image c, target target image a may be obtained by capturing an image of a calibration plate of posture A, target target image b may be obtained by capturing an image of a calibration plate of posture B, and target target image c may be obtained by capturing an image of a calibration plate of posture C. Since the target target image is used to calibrate the camera to be calibrated, generally, the calibration position in the calibration plate is captured, so each target target image has a first marker point, that is, each first marker point in each target target image can be determined.

[0098] Step S202, for each first marker point, obtaining the pixel coordinates of the first marker point and the world coordinates of the first marker position corresponding to the first marker point, and calculating the reprojection error of the first marker point according to the pixel coordinates of the first marker point and the world coordinates of the first marker position, wherein the first marker position corresponding to the first marker point is the marker position collected as the first marker point in the calibration plate;

[0099] For each first marker point, the reprojection error can be the error obtained by comparing the pixel coordinates of the first marker point (the observed projection position) with the position of the first marker position (i.e., the point position of the calibration plate in a certain posture in the actual space captured by the camera to be calibrated as the first marker point) projected according to the current estimated posture. Therefore, each first marker point has a corresponding reprojection error.

[0100] Step S203 , dividing the target image into at least two image areas according to the reprojection error size of each first marker point.

[0101] Since in general, the reprojection errors of each first marker point at different positions in the target target image are different, at least two image areas can be determined based on the reprojection errors of each first marker point in the target target image and the position of each first marker point in the target target image.

[0102] For example, the reprojection error of each first marker point can be determined first, and a first marker point set with the reprojection error in a preset interval can be determined; then the position of each first marker point in the first marker point set can be determined, and then the area corresponding to the first marker point with the reprojection error in the preset interval can be determined to determine the first image area, and then the area in the target image except the first image area can be used as the second image area.

[0103] Through the method in this embodiment, at least two image regions can be automatically divided based on the reprojection errors of each first marker point, thereby improving the accuracy of different image region divisions.

[0104] As an optional implementation, as in the above method, the step S102 of dividing the target image into regions to obtain at least two non-overlapping image regions includes the following steps:

[0105] Step S301, obtaining distortion information of a target image;

[0106] Step S302: dividing the target image into at least two image areas according to different distortion information.

[0107] The distortion information may be information for describing the degree of image distortion in a specified area.

[0108] In the image captured by the camera to be calibrated, the distortion degree of different areas is different due to reasons such as lens manufacturing. Therefore, the target image can be divided into at least two image areas based on the distortion information of each image area.

[0109] For example, Figure 5 As shown, in general, the distortion of the edge area of ​​the image captured by the camera to be calibrated is greater than the distortion of the central area of ​​the image captured by the camera to be calibrated. Therefore, the distortion information of the edge of the target target image is different from the distortion information of the central area of ​​the target target image. Therefore, based on the distortion information of different image areas, the target target image can be divided into a first image area P and a second image area Q, and the first image area P is located inside the second image area Q.

[0110] Through the method in this embodiment, multiple image regions with greatly different distortion conditions can be quickly divided based on the distortion information, so as to improve the efficiency of determining the calibration parameters corresponding to each image region.

[0111] like Figure 2 As shown, as an optional implementation, as in the aforementioned method, the step S105 of calculating the calibration parameters of the camera to be calibrated in each image area based on the parameter information of each of the marker points in all the sub-images in each image area includes the following steps:

[0112] For any image region, perform the following steps:

[0113] Step S401, determining each second marker point in all sub-images of the image area, and a second marker position corresponding to each second marker point, wherein the second marker position is a marker position collected as a second marker point in the calibration plate.

[0114] The second marker may be a marker in a calibration plate.

[0115] The second marker point is the marker point after the camera to be calibrated acquires the image of the calibration plate and acquires the second marker position as image information. Therefore, each second marker point has a second marker position corresponding to the second marker point in the calibration plate, that is, there is a one-to-one correspondence between the second marker point and the second marker position.

[0116] Step S402, based on the pixel coordinates of the second marker point and the world coordinates of the second marker position, calculate the first initial values ​​of the target internal and external parameters, wherein, for each second marker point, the parameter information of the second marker point includes the pixel coordinates of the second marker point and the world coordinates of the second marker position corresponding to the second marker point, and the target internal and external parameters are the internal and external parameters corresponding to the camera to be calibrated and the image area.

[0117] After obtaining the pixel coordinates of the second marker point and the world coordinates of the second marker in the image area, the calibration parameters of the calibrated camera in the image area can be calculated based on the pixel coordinates of the second marker point and the world coordinates of the second marker.

[0118] The second marking point is a marking point in the calibration plate used to calibrate the camera.

[0119] The pixel coordinates are the positions of the pixels in the image. Generally, the vertex at the upper left corner of the pixel coordinate system is taken as the origin, the horizontal rightward is the u axis, and the vertical downward is the v axis. That is to say, after determining the first marker point, the pixel coordinates of the first marker point can be determined based on the pixel coordinates of the pixel corresponding to the first marker point.

[0120] After the calibration plate is determined, a world coordinate system can be established based on the plane where the calibration plate is located, and then the world coordinates of each second marker point can be obtained based on the distance between each second marker point and the origin of the world coordinate system.

[0121] After obtaining the pixel coordinates of each first marking point and the world coordinates of each second marking point, the first initial value of the target internal and external parameter can be calculated. The first initial value of the target internal and external parameter is the value of the target internal and external parameter that needs to be further optimized.

[0122] like Figure 3As shown, as an optional implementation, as in the aforementioned method, the step S402 calculates the first initial value of the target internal and external parameters based on the pixel coordinates of the second marker point and the world coordinates of the second marker position, including the following steps:

[0123] Step S501, determining a plurality of coordinate pairs corresponding to the image area, wherein each coordinate pair includes a pixel coordinate of a second marker point and a world coordinate of a second marker corresponding to a first marker point.

[0124] After determining each second marker point in the image area, the second marker position corresponding to each second marker point can be determined, so a coordinate pair can be obtained based on the pixel coordinates of the second marker point and the world coordinates of the second marker position corresponding to the second marker point. Based on this, each second marker point in the image area has a corresponding coordinate pair.

[0125] Step S502, calculating each coordinate pair based on the Zhang Zhengyou calibration method to obtain the first initial value of the target internal and external parameters.

[0126] That is to say, by taking each coordinate pair as input data and calculating it using Zhang Zhengyou's calibration method, the first initial values ​​of the target internal and external parameters can be obtained.

[0127] For example, multiple internal and external parameters can be calculated using the following method:

[0128] The nonlinear camera model is used to describe the imaging process as follows:

[0129]

[0130]

[0131]

[0132]

[0133] Wherein, (Xw, Yw, Zw) is the world coordinate, (xc, yc, zc) is the camera coordinate, R is the rotation matrix, T is the translation matrix, x and y are the normalized camera coordinates, formula (1) is used to convert the world coordinate to the camera coordinate, formula (2) is used to normalize the camera coordinate, k1, k2, k3 are radial distortion coefficients, p1, p2 are tangential distortion coefficients, formula (3) is used to convert the camera coordinate to the image coordinate, that is, the imaging process, ideally it is a similar triangle, but in reality due to the reasons of camera manufacturing and assembly, the imaging process has distortion phenomenon; fx is the horizontal focal length and fy is the vertical focal length, u0 is the horizontal principal point, v0 is the vertical principal point, formula (4) is used to convert the image coordinate to the pixel coordinate; therefore, by combining formulas (1), (2), (3) and (4), the camera internal and external parameters and distortion coefficients when each world coordinate is converted to the pixel coordinate can be obtained.

[0134] To solve the optimal camera calibration parameters, the linear imaging model can be used to obtain the initial values ​​of the internal and external parameters, and then the LM iterative optimization can be used to solve the optimal camera calibration parameters. First, the camera calibration method of the linear imaging model is used to solve the homography matrix of each reference plane. Assuming that the world coordinate system is established on the target plane and Z = 0, the pinhole model and the above formulas (1), (2), (3) and (4) can be obtained:

[0135]

[0136] Where s is a constant factor, are the pixel coordinates and world coordinates in the same coordinate pair, the homography matrix H = λA[r1, r2, t], the translation vector t is the translation from the world coordinates to the optical camera coordinates, the rotation matrix R is a 3×3 orthogonal normalized matrix, r1 and r2 are the first and second columns of the rotation matrix R. H is the process of calculating the minimum residual between the pixel coordinates and the world coordinates, and the objective function is:

[0137]

[0138] According to the orthogonal normalization of the rotation matrix and H = λA[r1,r2,t], we can get:

[0139]

[0140] Further simplifying according to formula (7), let B = A -T A -1 According to the properties of matrix product, the B matrix is ​​a 3×3 symmetric matrix, so the B matrix contains only 6 independent unknowns. It can be further simplified. There are 6 independent variables in the B matrix, and the 6 unknowns are solved. The solution process is as follows: Let b = [B 11 B12 B 22 B 13 B 23 B 33 ] T , Substitute into equation (7), let the i-th column vector in H be h i =[h i1 h i2 h i3 ] T , so (7) can be simplified as:

[0141]

[0142] Let v ij =[h i1 h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ] T , then formula (8) can be simplified as:

[0143]

[0144] The solution of equation (9) is the matrix g = e T The eigenvector corresponding to the minimum eigenvalue of e, or by solving b through the singular decomposition of matrix e. Cholesky matrix method, decompose matrix b to get A-1, and then A-1 matrix

[0145] Find the inverse matrix and finally obtain the internal parameter matrix A.

[0146] After the internal parameter matrix A is solved, according to the orthogonality of the rotation matrix R, the modulus of column vectors r1 and r2 are both 1, so r1=1 / λA -1 h1 or r2 = 1 / λA -1 h2, we can get λ=1 / ||A -1 h1||=1 / ||A -1 h2||. By solving the corresponding single matrix Hn from the image obtained from each spatial posture of the chessboard, we can further solve the external parameter matrix B corresponding to each posture of the chessboard:

[0147]

[0148] Furthermore, based on the intrinsic parameter matrix A and the extrinsic parameter matrix B, the first initial values ​​of the target intrinsic and extrinsic parameters of the image area can be obtained.

[0149] Step S403, calculating a first target value of the target internal and external parameters and a second target value of the target distortion coefficient according to the first initial value of the target internal and external parameters, wherein the target distortion coefficient is the distortion coefficient corresponding to the camera to be calibrated and the image area, and the calibration parameters are the target internal and external parameters and the target distortion coefficient.

[0150] After obtaining the first initial value of the target internal and external parameter, optimization can be performed based on the first initial value of the target internal and external parameter to obtain the first target value of the target internal and external parameter and the second target value of the target distortion coefficient.

[0151] The first target value may be the final optimized value of the target internal and external parameters obtained after optimization.

[0152] The second target value may be a final optimized value of the target distortion coefficient obtained after optimization.

[0153] As an optional implementation, as in the above method, the step S403 calculates the first target value of the target internal and external parameter and the second target value of the target distortion coefficient according to the first initial value of the target internal and external parameter, including the following steps:

[0154] Step S601, performing iterative calculation according to a first initial value of a target internal and external parameter and a second initial value of a target distortion coefficient, to obtain a first target value of the target internal and external parameter and a second target value of the target distortion coefficient, wherein the second initial value is 0.

[0155] After obtaining the first initial values ​​of the target internal and external parameters, the target internal and external parameter data serving as the basis for optimization can be determined. At the same time, by setting the second initial value of the target distortion coefficient to 0, the initial values ​​of the calibration parameters used for optimization can be obtained (i.e., including the first initial values ​​of the target internal and external parameters and the second initial value of the target distortion coefficient).

[0156] Optionally, an LM (Levenberg-Marquarelt) iterative algorithm may be used to perform iterative calculation based on a first initial value of the target internal and external parameters and a second initial value of the target distortion coefficient to obtain a first target value of the target internal and external parameters and a second target value of the target distortion coefficient. The LM iterative algorithm is an algorithm for iteratively finding the extreme value of a function.

[0157] The method of calculating the first target value of the target internal and external parameters and the second target value of the target distortion coefficient using the LM iterative algorithm is as follows:

[0158] (J T (x)J(x)+uI)Δx=-J T (x)f(x) (11)

[0159]

[0160] Where u≥0, u represents the trust region radius. The iterative optimization process is as follows:

[0161] (1) Given an initial value x0 (i.e., the first initial value of the target intrinsic and extrinsic parameters and the second initial value of the target distortion coefficient), set the initial value u0 (u0 is set based on experience).

[0162] (2) At the kth iteration, the step length Δx is obtained according to the previous formula, and ρk is calculated at this time.

[0163] (3) According to ρ k The trust region radius is adjusted by adjusting the value of .

[0164] (4) If Δx k If it is small enough (i.e., smaller than a preset threshold), the iteration is stopped, and the first target value of the target internal and external parameters and the second target value of the target distortion coefficient are obtained; otherwise, according to ρ k Size judgment x k+1 How should we update and calculate x? k+1 Then return to step (2) and continue iterating.

[0165] like Figure 4 As shown, according to an embodiment of another aspect of the present application, a camera-based object calibration method is also provided, comprising the following steps:

[0166] Step S701, obtaining a target image obtained after the target camera acquires an image of the target object, wherein the target camera is calibrated in advance by the camera calibration method in any of the aforementioned embodiments, and the target camera includes at least two calibration parameters;

[0167] After the calibration of the camera to be calibrated is completed, the camera to be calibrated can be used as a target camera to calibrate the target object based on the calibrated target camera. Therefore, the target camera can be used to capture an image of the target object to obtain a target image.

[0168] Step S702, determining the pixel coordinates of the points to be calibrated in the target image;

[0169] After capturing the image of the target object, the target image corresponding to the target object can be obtained, and then the pixel coordinates of the points to be calibrated in the image can be obtained.

[0170] Step S703, determining the target area where the point to be calibrated is located in the target image according to the pixel coordinates of the point to be calibrated;

[0171] Step S704, obtaining target calibration parameters corresponding to the target area;

[0172] Step S705: calibrate the points to be calibrated using the target calibration parameters.

[0173] Since the image area is the area at a specific position in the image obtained when the camera to be calibrated shoots any object (e.g., calibration plate) within the specified distance interval, and the image area is independent of the content in the image, any image taken by the camera to be calibrated under the above conditions is at the same position in each image. Therefore, the pixel coordinate range of each image area in the image can be determined.

[0174] Therefore, based on the pixel coordinates of the points to be calibrated and the pixel coordinate ranges corresponding to each image region, a pixel coordinate range including the pixel coordinates of the points to be calibrated can be determined.

[0175] When the pixel coordinates of the point to be calibrated are included in the pixel coordinate range of a certain image area, the calibration parameters corresponding to the target camera and the image area are recorded as target calibration parameters, and then the point to be calibrated is calibrated according to the target calibration parameters. Therefore, the purpose of more accurate calibration of the point to be calibrated in the image area can be achieved.

[0176] For example, when the image captured by the target camera is divided into a first image area and a second image area, the pixel coordinate range including the pixel coordinate of the point to be calibrated can be determined based on the pixel coordinate of the point to be calibrated, the first pixel coordinate range of the first image area, and the second pixel coordinate range of the second image area.

[0177] When the pixel coordinates of the point to be calibrated are included in the first pixel coordinate range, the calibration parameters corresponding to the target camera and the first image area are the first calibration parameters, and then the point to be calibrated is calibrated according to the first calibration parameters.

[0178] When the pixel coordinates of the point to be calibrated are included in the second pixel coordinate range, the calibration parameters corresponding to the target camera and the second image area are the second calibration parameters, and then the point to be calibrated is calibrated according to the second calibration parameters.

[0179] In summary, by combining the calibration parameters of multiple sets of cameras, an overall high-precision calibration result can be obtained.

[0180] like Figure 6 As shown, according to an embodiment of another aspect of the present application, a camera calibration device is also provided, including:

[0181] The first acquisition module 11 is used to acquire a target image obtained by the camera to be calibrated acquiring an image of a calibration plate, wherein the calibration plate has at least one mark position;

[0182] A division module 12, used for dividing the target image into regions to obtain at least two non-overlapping image regions;

[0183] A second acquisition module 13 is used to acquire a sub-image in each image area of ​​the target image;

[0184] The third acquisition module 14 is used to acquire parameter information of each marker point in the sub-image, wherein the marker point is obtained after image acquisition of the marker position;

[0185] The calibration parameter module 15 is used to calculate the calibration parameters of the camera to be calibrated in each image area based on the parameter information of each marker point in all sub-images corresponding to each image area, and obtain at least two calibration parameters corresponding to the camera to be calibrated.

[0186] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0187] like Figure 7 As shown, according to an embodiment of another aspect of the present application, a camera-based object calibration device includes:

[0188] A first acquisition module 21 is used to acquire a target image obtained after the target camera acquires an image of the target object, wherein the target camera is calibrated in advance by any of the above camera calibration methods, and the target camera includes at least two calibration parameters;

[0189] A first determination module 22, used to determine the pixel coordinates of the point to be calibrated in the target image;

[0190] A second determination module 23 is used to determine the target area where the point to be calibrated is located in the target image according to the pixel coordinates of the point to be calibrated;

[0191] A second acquisition module 24 is used to acquire target calibration parameters corresponding to the target area;

[0192] The calibration module 25 is used to calibrate the points to be calibrated using target calibration parameters.

[0193] Specifically, the specific process of each module in the device of the embodiment of the present invention realizing its function can be referred to the relevant description in the method embodiment, which will not be repeated here.

[0194] According to another embodiment of the present application, there is also provided an electronic device, including: Figure 8As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0195] Memory 1503, used for storing computer programs;

[0196] The processor 1501 is used to implement the steps of the above method embodiment when executing the program stored in the memory 1503.

[0197] The bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0198] The communication interface is used for communication between the above electronic device and other devices.

[0199] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0200] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0201] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0202] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0203] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A camera calibration method, characterized in that: include: Acquire a target image obtained by the camera to be calibrated acquiring an image of a calibration plate, wherein the calibration plate has at least one marking position; Dividing the target image into regions to obtain at least two non-overlapping image regions; Acquire a sub-image within each image area in the target image; Acquire parameter information of each marker point in the sub-image, wherein the marker point is obtained after performing image acquisition on the marker position; Based on the parameter information of each of the marker points in all the sub-images corresponding to each of the image regions, calculating the calibration parameters of the camera to be calibrated in each of the image regions, and obtaining at least two calibration parameters corresponding to the camera to be calibrated; The step of dividing the target image into at least two non-overlapping image regions comprises: Determine each first marker point in the target image; For each of the first marker points, the pixel coordinates of the first marker point and the world coordinates of the first marker position corresponding to the first marker point are obtained, and a reprojection error of the first marker point is calculated according to the pixel coordinates of the first marker point and the world coordinates of the first marker position, wherein the first marker position corresponding to the first marker point is the marker position collected as the first marker point in the calibration plate, and the reprojection error of the first marker point is an error obtained by comparing the pixel coordinates of the first marker point with the position obtained by projecting the first marker position according to the currently estimated posture; According to the reprojection error size of each of the first marking points in the target image, the target image is divided into at least two image areas.

2. The method according to claim 1, characterized in that The step of dividing the target image into at least two non-overlapping image regions comprises: Acquiring distortion information of the target image; According to different distortion information, the target image is divided into at least two image areas.

3. The method according to claim 1, characterized in that The calculating the calibration parameters of the camera to be calibrated in each image area based on the parameter information of each of the marker points in all the sub-images corresponding to each of the image areas comprises: For any image region, perform the following steps: Determine each second marker point in all the sub-images of the image area, and a second marker position corresponding to each second marker point, wherein the second marker position is a marker position collected as the second marker point in the calibration plate; Based on the pixel coordinates of the second marker point and the world coordinates of the second marker position, a first initial value of the target internal and external parameters is calculated, wherein, for each of the second marker points, the parameter information of the second marker point includes the pixel coordinates of the second marker point and the world coordinates of the second marker position corresponding to the second marker point, and the target internal and external parameters are the internal and external parameters corresponding to the camera to be calibrated and the image area; According to the first initial value of the target internal and external parameters, a first target value of the target internal and external parameters and a second target value of the target distortion coefficient are calculated, wherein the target distortion coefficient is the distortion coefficient corresponding to the camera to be calibrated and the image area, and the calibration parameters are the target internal and external parameters and the target distortion coefficient.

4. The method according to claim 3, characterized in that The calculating the first initial value of the target internal and external parameters based on the pixel coordinates of the second marker point and the world coordinates of the second marker position includes: Determine a plurality of coordinate pairs corresponding to the image area, wherein each of the coordinate pairs includes a pixel coordinate of a second marker point and a world coordinate of the second marker corresponding to the second marker point; The first initial value of the internal and external parameters of the target is obtained by calculating each of the coordinate pairs based on the Zhang Zhengyou calibration method.

5. The method according to claim 3, characterized in that: Calculating the first target value of the target internal and external parameter and the second target value of the target distortion coefficient according to the first initial value of the target internal and external parameter includes: An iterative calculation is performed based on the first initial value of the target internal and external parameter and the second initial value of the target distortion coefficient to obtain the first target value of the target internal and external parameter and the second target value of the target distortion coefficient, wherein the second initial value is 0.

6. A camera-based object calibration method, characterized in that: include: Acquire a target image obtained after the target camera acquires an image of the target object, wherein the target camera is pre-calibrated by the camera calibration method according to any one of claims 1 to 5, and the target camera includes at least two calibration parameters; Determine the pixel coordinates of the points to be calibrated in the target image; Determine, according to the pixel coordinates of the to-be-calibrated point, a target area in the target image where the to-be-calibrated point is located; Acquiring target calibration parameters corresponding to the target area; The target calibration parameters are used to calibrate the points to be calibrated.

7. A camera calibration device, characterized in that: include: A first acquisition module is used to acquire a target image obtained by the camera to be calibrated acquiring an image of a calibration plate, wherein the calibration plate has at least one mark position; A division module, used for dividing the target image into regions to obtain at least two non-overlapping image regions; A second acquisition module, used for acquiring a sub-image in each image area of ​​the target image; A third acquisition module is used to acquire parameter information of each marker point in the sub-image, wherein the marker point is obtained after performing image acquisition on the marker position; A calibration parameter module, configured to calculate the calibration parameters of the camera to be calibrated in each of the image regions based on the parameter information of each of the marker points in all the sub-images corresponding to each of the image regions, and obtain at least two calibration parameters corresponding to the camera to be calibrated; The division module is specifically used for: Determine each first marker point in the target image; For each of the first marker points, the pixel coordinates of the first marker point and the world coordinates of the first marker position corresponding to the first marker point are obtained, and a reprojection error of the first marker point is calculated according to the pixel coordinates of the first marker point and the world coordinates of the first marker position, wherein the first marker position corresponding to the first marker point is the marker position collected as the first marker point in the calibration plate, and the reprojection error of the first marker point is an error obtained by comparing the pixel coordinates of the first marker point with the position obtained by projecting the first marker position according to the currently estimated posture; According to the reprojection error size of each of the first marking points in the target image, the target image is divided into at least two image areas.

8. A camera-based object calibration device, characterized in that: include: A first acquisition module, configured to acquire a target image obtained after a target camera acquires an image of a target object, wherein the target camera is pre-calibrated by the camera calibration method according to any one of claims 1 to 5, and the target camera includes at least two calibration parameters; A first determination module is used to determine the pixel coordinates of the to-be-calibrated points in the target image; A second determination module is used to determine the target area where the point to be calibrated is located in the target image according to the pixel coordinates of the point to be calibrated; A second acquisition module, used to acquire target calibration parameters corresponding to the target area; The calibration module is used to calibrate the point to be calibrated using the target calibration parameters.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

Images