Camera calibration method, device, electronic device and storage medium

Abstract

The present invention provides a camera calibration method, device, electronic device and storage medium. The camera calibration method comprises: using the camera to focus on different areas of a calibration plate respectively and photographing the image to be processed of the calibration plate after each focusing to obtain multiple images to be processed; determining a first feature point in a first image according to feature points in a clear area of ​​each image to be processed, the first image being one of the multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, and the first feature point includes feature points in the clear area of ​​the first image and at least part of feature points outside the clear area of ​​the first image; determining the camera parameters of the camera according to the first feature point. This solution can improve the accuracy of camera calibration.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a camera calibration method, a camera calibration device, an electronic device, a storage medium, and a computer program product. Background Art

[0002] Camera calibration is a basic process in the field of computer vision and image processing. It aims to perform 3D reconstruction by determining the camera's internal (such as focal length, optical center, lens distortion parameters) and external parameters (such as camera position and orientation), that is, the process of recovering the 3D coordinates of spatial points from the image. Camera calibration is usually done by establishing a relationship between the camera image pixel position and the scene point position, so as to determine the camera's internal and external parameters from the image coordinates of known feature points.

[0003] Camera calibration algorithms usually use calibration blocks or calibration plates with known structural dimensions for camera calibration, such as direct linear transformation, RAC-based camera calibration method, and planar calibration method proposed by Zhang Zhengyou. All such algorithms require a clear image of the complete calibration block or calibration plate to accurately calibrate the camera.

[0004] However, in the image captured by the camera, the clear and effective image area may only occupy a part of the image. In this case, the camera calibration cannot be accurately performed directly based on the existing camera calibration algorithm. For example, the depth of field range of a shallow depth of field camera is short, and the clear and effective image area in the captured image usually only occupies a part of the image. In this case, the camera calibration cannot be accurately performed directly based on the existing camera calibration algorithm. Summary of the invention

[0005] The present invention has been made in view of the above-mentioned problems.

[0006] According to a first aspect of the present invention, a camera calibration method is provided. The method comprises: using the camera to focus on different areas of a calibration plate respectively and taking images to be processed of the calibration plate after each focusing to obtain multiple images to be processed; determining first feature points in a first image according to feature points in a clear area of ​​each image to be processed, the first image being one of the multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, and the first feature points include feature points in the clear area of ​​the first image and at least some feature points outside the clear area of ​​the first image; and determining camera parameters of the camera according to the first feature points.

[0007] Exemplarily, a clear area of ​​any one of the multiple images to be processed has an overlapping area with clear areas of other images to be processed;

[0008] The method of determining the first feature point in the first image based on the feature points in the clear area of ​​each image to be processed includes: for any overlapping area, determining the difference in image parameters between the image to be processed related to the overlapping area based on the feature points in the overlapping area in the image to be processed related to the overlapping area, the clear area of ​​the image to be processed related to the overlapping area being one of a pair of clear areas both covering the overlapping area; mapping the feature points in the clear area of ​​each image to be processed to the first image based on the difference in image parameters; and determining the first feature point based on the feature points in the clear area of ​​the first image and the feature points mapped to the first image.

[0009] Exemplarily, the different regions are arranged along a predetermined direction of the calibration plate;

[0010] The method of mapping the feature points in the clear area of ​​each image to be processed to the first image according to the difference in the image parameters comprises: for each image to be processed that is not the first image, accumulating the differences in image parameters between the images to be processed where every two adjacent clear areas are located, so as to determine the differences in image parameters between the image to be processed and the first image, wherein the adjacent clear areas are all located between the clear area of ​​the image to be processed and the clear area of ​​the first image; and mapping the feature points in the clear area of ​​the image to be processed to the first image according to the difference in image parameters between the image to be processed and the first image.

[0011] Exemplarily, among the multiple images to be processed, the clear area of ​​the first image is closest to the center of the image.

[0012] Exemplarily, the image parameters include image magnification, image zoom center and image translation amount.

[0013] Exemplarily, there are at least 8 feature points in the overlapping area.

[0014] Exemplarily, determining the camera parameters of the camera according to the first feature point includes: determining a single mapping matrix between a point on the calibration plate and a pixel of the first image according to a pixel coordinate of the first feature point and a second feature point on the calibration plate corresponding to the first feature point;

[0015] Determine an intrinsic parameter matrix, wherein the elements of the intrinsic parameter matrix include an image zoom center and a focal length corresponding to the first image;

[0016] According to the single mapping matrix and the intrinsic parameter matrix, camera extrinsic parameters and camera intrinsic parameters of the camera are determined.

[0017] According to a second aspect of the present invention, a camera calibration device is also provided, comprising:

[0018] A receiving module, used for acquiring a plurality of images to be processed, wherein the plurality of images to be processed are obtained by focusing on different areas of a calibration plate using a camera and photographing the image to be processed of the calibration plate after each focusing;

[0019] a recognition module, configured to determine first feature points in a first image according to feature points in a clear area of ​​each image to be processed, wherein the first image is one of the multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, and the first feature points include feature points in the clear area of ​​the first image and at least part of feature points outside the clear area of ​​the first image;

[0020] A parameter determination module determines the camera parameters of the camera according to the first feature points.

[0021] According to a third aspect of the present invention, there is further provided an electronic device, comprising: a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the above-mentioned camera calibration method when the processor is executed.

[0022] According to a fourth aspect of the present invention, a storage medium is further provided, on which program instructions are stored, and the program instructions are used to execute the above-mentioned camera calibration method when running.

[0023] According to a fifth aspect of the present invention, a computer program product is further provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned camera calibration method when running.

[0024] In the above technical solution, a camera is used to focus on different areas of the calibration plate and to capture the image to be processed of the calibration plate after each focusing to obtain multiple images to be processed, and a first feature point in the first image is determined based on the feature points in the area of ​​each image to be processed where the clarity is higher than a preset threshold, and then the camera parameters of the camera are determined based on the first feature point. This solution can improve the accuracy of camera calibration when only partially accurate image information exists in the captured image of the calibration plate.

[0025] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0027] Figure 1 A schematic flow chart of a camera calibration method according to an embodiment of the present invention is shown;

[0028] Figure 2 A schematic flow chart showing determining first feature points in a first image according to feature points in a clear area of ​​each image to be processed according to an embodiment of the present invention;

[0029] Figure 3 A schematic diagram showing an image to be processed related to an overlapping area according to an embodiment of the present invention;

[0030] Figure 4 A schematic diagram showing mapping feature points in a clear area of ​​each image to be processed to a first image according to an embodiment of the present invention is shown;

[0031] Figure 5 A schematic flow chart of mapping feature points in a clear area of ​​each image to be processed to a first image according to an embodiment of the present invention is shown;

[0032] Figure 6 A schematic flow chart of determining camera parameters of a camera according to a first feature point according to an embodiment of the present invention is shown;

[0033] Figure 7 A schematic block diagram of a camera calibration device according to an embodiment of the present invention is shown; and

[0034] Figure 8 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0036] In the image captured by the camera, the clear and effective image area may only occupy a part of the image. For example, the equipment used for wafer defect detection, measurement and wafer cutting usually adopts a high-magnification microscopic imaging system. However, the objective lens of this imaging system is a shallow depth of field camera with a short depth of field range. For example, a 20x objective lens usually has a depth of field of about 1um. If a calibration block or calibration plate is used to calibrate the internal and external parameters of the camera, part of the image area in the captured image will be blurred due to the short depth of field range, which will affect the accuracy of the camera calibration.

[0037] In order to at least partially solve the above problems, a camera calibration method is proposed. The method uses a camera to focus on different areas of a calibration plate and capture images of the calibration plate to be processed after each focus. The overlapping and clear features / clear areas (these feature areas have one or more feature points) in the continuously captured images to be processed are screened out. Then, based on the feature points in the clear area of ​​each image to be processed, the first feature point in the first image is determined, and then the camera parameters of the camera are determined based on the first feature point. In this way, the camera parameters of the camera are determined based on more accurate feature points in the captured image, which can improve the accuracy of camera calibration when there is only partially accurate image information in the captured image.

[0038] Figure 1 FIG. 1 is a schematic flow chart of a camera calibration method according to an embodiment of the present invention. Figure 1 As shown, the camera calibration method may include steps S110 to S130.

[0039] In step S110, a camera is used to focus on different areas of the calibration plate and an image to be processed of the calibration plate is photographed after each focusing to obtain a plurality of images to be processed.

[0040] Different areas of the calibration plate can be different areas of the entire calibration plate, or different sub-areas in the designated area of ​​the calibration plate. Because the shooting target of multiple images to be processed is the same calibration plate, different images to be processed can be images of similar size, and each image to be processed should contain the entire calibration plate or the designated area of ​​the calibration plate to be photographed. Because the focus is performed on different areas of the calibration plate, different clear areas should exist in different images to be processed, and the clear areas of each image to be processed correspond to different areas of the calibration plate. The center of the clear area of ​​each image to be processed can be the same as the center focused when shooting the image to be processed. The image to be processed can be a color image, a grayscale image, a binary image, etc.

[0041] In step S120, based on the feature points in the clear area of ​​each image to be processed, the first feature points in the first image are determined, where the first image is one of multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, and the first feature points include the feature points in the clear area of ​​the first image and at least some of the feature points outside the clear area of ​​the first image.

[0042] The clarity of different areas in each image to be processed can be characterized according to parameters such as local contrast, structural similarity index, mean square error, etc. of each image to be processed, and the area with clarity higher than a preset threshold is determined as part of the clear area to obtain a complete clear area. The image information in the clear area can more accurately characterize the information of the partial area of ​​the calibration plate corresponding to the image information of the clear area. Different clear areas correspond to different areas of the calibration plate, so the image information in the clear area of ​​the non-first image will contain the image information of the non-clear area of ​​the first image, and the feature points in the non-clear area of ​​the first image can be determined according to the feature points in the clear area of ​​the non-first image.

[0043] The first feature point represents feature point information in a designated area of ​​the first image under the image parameters of the first image.

[0044] For example, when the image parameters in each image to be processed are consistent, the image area where the calibration plate is captured in each image to be processed is substantially consistent. Feature points can be determined in the clear area of ​​the non-first image, and then based on the position of the feature point in the non-first image, the point at the same position in the first image is determined as the feature point.

[0045] For example, the size and direction of the calibration plates in each image to be processed may be inconsistent and deflected, and the position of the feature point of the non-first image cannot be directly determined in the first image. The difference in image parameters between the non-first image and the first image can be determined first, and then the mapping relationship between the image area of ​​the calibration plate in the non-first image and the image area of ​​the calibration plate in the first image can be determined based on the difference. According to the mapping relationship, when the image parameters of the non-first image and the image parameters of the first image are the same, the position of the feature point of the clear area of ​​the non-first image can be determined, and then the point at the same position as the position in the first image can be determined as the first feature point.

[0046] The feature points in the clear area of ​​each image to be processed are points that characterize the image information of the image to be processed, such as corner points, key points, feature points extracted by features, etc. The feature points may also be manually specified points.

[0047] Optionally, only the first feature points of the unclear area of ​​the first image may be determined according to the feature points of the unclear area in the non-first image.

[0048] Optionally, an image area requiring a first feature point may be determined in the first image according to requirements. Then, based on the relationship between the image area and the clear area of ​​the image to be processed, a portion of the non-first image is selected, and the first feature point in the image area that meets the requirements in the first image is determined based only on the feature points of the clear area of ​​the non-first image. The image area corresponding to the total image area of ​​the selected clear area of ​​the non-first image should cover the image area outside the non-clear area of ​​the first image for which the first feature point needs to be determined.

[0049] In step S130 , camera parameters of the camera are determined according to the first feature point.

[0050] After the first feature point is determined, the camera parameters of the camera may be determined according to a mapping relationship between the coordinates of the first feature point in the pixel coordinate system and the coordinates of the first feature point in the world coordinate system.

[0051] In the above technical solution, a camera is used to focus on different areas of the calibration plate and to capture the image to be processed of the calibration plate after each focusing to obtain multiple images to be processed. The first feature point in the first image is determined based on the feature points in the area where the clarity of each image to be processed is higher than a preset threshold, and then the camera parameters of the camera are determined based on the first feature point. This solution can improve the accuracy of camera calibration when there is only partially accurate image information in the captured image of the calibration plate.

[0052] Exemplarily, the clear area of ​​any one of the multiple images to be processed has an overlapping area with the clear areas of other images to be processed. Figure 2 FIG. 4 is a schematic flow chart showing how to determine the first feature point in the first image based on the feature points in the clear area of ​​each image to be processed according to an embodiment of the present invention. Figure 2 As shown, the above step S120 may include steps S210 to S230.

[0053] In step S210, for any overlapping area, the difference in image parameters between the images to be processed related to the overlapping area is determined based on the feature points in the overlapping area in the images to be processed related to the overlapping area, and the clear areas of the images to be processed related to the overlapping area are respectively one of a pair of clear areas that both cover the overlapping area.

[0054] Figure 3 FIG. 4 is a schematic diagram showing an image to be processed related to an overlapping area according to an embodiment of the present invention. Figure 3As shown, for any overlapping area, a pair of clear areas that both cover the overlapping area can be used as clear areas related to the overlapping area, and then the image to be processed where the clear area related to the overlapping area is located is used as the image to be processed related to the overlapping area. In the overlapping area of ​​the image to be processed related to the overlapping area, there is at least one pair of corresponding feature points.

[0055] When the image parameters of the images to be processed related to the overlapping area are the same, because the same calibration plate is photographed, the image area occupied by the calibration plate in the images to be processed related to the overlapping area is usually the same. At this time, the positions of the feature points of the images to be processed related to the overlapping area should also be the same in the overlapping area. For example, if the images to be processed related to the overlapping area are the second image and the third image, and the coordinates of the feature point a in the second image are (1,1), then the coordinates of the feature point b corresponding to a in the third image are also (1,1).

[0056] However, due to changes in shooting angle and position, the image parameters of the images to be processed related to the overlapping area may be different. At this time, the feature points of the images to be processed related to the overlapping area that were originally at the same position may be scattered in different positions within the overlapping area. Exemplarily, the image parameters include image magnification, image zoom center, and image translation. For example, the images to be processed related to the overlapping area are the fourth image and the fifth image, the X-direction magnification of the fourth image is 1 times, and the coordinates of the feature point a in the fourth image are (1,1). The X-direction magnification of the fifth image is 2 times, and the coordinates of the feature point b corresponding to a in the fifth image are (2,1). At this time, there will be position differences between the corresponding feature points in the overlapping area, and the differences in the image parameters of the images to be processed related to the overlapping area will affect the position differences. Based on this difference, the difference in image parameters can be determined in turn. Similarly, in addition to the image magnification, the image zoom center and the image translation will also affect the positions of the corresponding feature points in the overlapping area.

[0057] Exemplarily, the images to be processed related to the overlapping area may be the sixth image and the seventh image. A mapping matrix characterizing the difference in image parameters between the sixth image and the seventh image may be determined based on the influence of the image parameters on the positions of the corresponding feature points in the overlapping area and the position difference of the corresponding feature points in the overlapping area between the clear area of ​​the sixth image and the clear area of ​​the seventh image. The mapping matrix may also be used to determine the positions where the feature points of the sixth image in the overlapping area should be located when the image parameters of the sixth image are the image parameters of the seventh image.

[0058] For example, when there is a difference in image magnification and image translation between the sixth image and the seventh image, the mapping matrix A may be determined according to the following formula 1:

[0059]

[0060] in,( u i , v i ) represents the pixel coordinates of the i-th feature point in the sixth image in the overlapping area of ​​the sixth image and the seventh image, represents the corresponding ( u i , v i )’s pixel coordinates of the i-th feature point, s u represents the difference in image magnification in the X direction between the sixth image and the seventh image, s v represents the difference in image magnification in the Y direction between the sixth image and the seventh image, ( u 0, v 0) is the image zoom center, represents the horizontal coordinate offset between the sixth image and the seventh image, Indicates the vertical coordinate offset between the sixth image and the seventh image.

[0061] For example, there are at least 8 feature points in the overlapping area. There are 4 unknown elements in the mapping matrix A of the above formula 1: s u 、s v 、s u (Δ u-u 0)+ u 0 、s v (Δ v-v 0)+ v 0. In order to determine the four unknown elements to obtain the above-mentioned mapping matrix A, it is necessary to determine at least four feature points of the sixth image in the overlapping area between the clear area of ​​the sixth image and the clear area of ​​the seventh image and feature points of the seventh image in the overlapping area corresponding to the four feature points, that is, at least eight feature points can be determined in the overlapping area.

[0062] For example, after determining at least four pairs of corresponding feature points in the overlapping area between the clear area of ​​the sixth image and the clear area of ​​the seventh image, the following formula 2 can be obtained according to the above formula 1:

[0063]

[0064] Optionally, the least square method may be used to fit the parameters in the above formula 2: su 、s v 、s u (Δ u-u 0)+ u 0 、s v (Δ v-v 0)+ v 0, and then the mapping matrix A can be determined. Similarly, for other images to be processed related to the overlapping regions, the mapping matrix A between the feature points of the images to be processed related to the overlapping regions can be determined according to the above process.

[0065] In step S220, feature points in a clear area of ​​each image to be processed are mapped to the first image according to the difference in image parameters.

[0066] For each image to be processed, when there is an overlapping area between the clear area of ​​the image to be processed and the clear area of ​​the first image, the difference in image parameters between the image to be processed and the first image can be determined according to the difference in image parameters between the images to be processed related to each overlapping area, so as to determine the mapping matrix between the image to be processed and the first image according to the above formula. Then, the feature points in the clear area of ​​the image to be processed can be determined to be mapped to the feature points of the first image according to the mapping matrix.

[0067] For each image to be processed, when there is no overlapping area between the clear area of ​​the image to be processed and the clear area of ​​the first image, the clear area between the clear area of ​​the image to be processed and the clear area of ​​the first image can be first determined as the intermediate clear area, and the difference in image parameters between the images to be processed where the intermediate clear area with overlapping areas is located can be determined to obtain the mapping matrix between the images to be processed where the intermediate clear area with overlapping areas is located. Then, according to the mapping matrix between the images to be processed where the intermediate clear areas with overlapping areas are located, the mapping matrix between the image to be processed and the first image can be determined. Then, according to the mapping matrix between the image to be processed and the first image, the feature points in the clear area of ​​the image to be processed can be determined to be mapped to the feature points of the first image. Among them, the images to be processed where the intermediate clear areas with overlapping areas can be selected as few as possible, so as to determine as few mapping matrices between the images to be processed where the intermediate clear areas with overlapping areas are located, thereby reducing the amount of calculation. It is also possible to select as many images to be processed where the intermediate clear areas with overlapping areas are located, so as to determine as many mapping matrices between the images to be processed where the intermediate clear areas with overlapping areas are located, thereby increasing the accuracy of mapping to the feature points in the first image.

[0068] Figure 4FIG. 2 shows a schematic diagram of mapping feature points in a clear area of ​​each image to be processed to a first image according to an embodiment of the present invention. Figure 4 As shown, there is an overlapping area (second overlapping area) between the clear area of ​​the sixth image and the clear area of ​​the first image, and there is an overlapping area (third overlapping area) between the clear area of ​​the seventh image and the clear area of ​​the first image. For example, according to the above formula and the corresponding feature points of the sixth image and the first image in the second overlapping area, the mapping matrix A1 between the sixth image and the first image can be directly determined, and the mapping matrix A1 can be used to determine the mapping of the feature points in the clear area of ​​the sixth image to the feature points of the first image.

[0069] Similarly, a mapping matrix A2 between the seventh image and the first image may be determined, and the mapping matrix A2 may be used to determine the mapping of feature points in the clear area of ​​the seventh image to feature points of the first image.

[0070] In step S230 , first feature points are determined according to feature points in the clear area of ​​the first image and feature points mapped to the first image.

[0071] The feature points in the clear area of ​​the first image do not need to be mapped and can be directly used as the first feature points. The first feature points represent the positions corresponding to the feature points in the clear area of ​​the different to-be-processed images in the same focal plane as the first image.

[0072] In the above technical solution, for any overlapping area, the difference in image parameters between the images to be processed related to the overlapping area is determined based on the feature points in the overlapping area in the images to be processed related to the overlapping area, and the clear areas of the images to be processed related to the overlapping area are respectively one of a pair of clear areas that both cover the overlapping area. Then, based on the difference in image parameters, the feature points in the clear area of ​​each image to be processed are mapped to the first image, and the first feature points are determined based on the feature points in the clear area of ​​the first image and the feature points mapped to the first image. In this way, in addition to the feature points in the clear area of ​​the first image, more accurate first feature points can be determined based on the feature points of the clear areas of different images to be processed, which is more conducive to improving the accuracy of camera calibration.

[0073] Exemplarily, different areas are arranged along a predetermined direction of the calibration plate. For example, different areas of the calibration plate can be focused from left to right in sequence and the image to be processed can be photographed after focusing. Shooting along one direction can facilitate the calculation of the positions between the clear areas of different images to be processed. Figure 5 FIG. 4 is a schematic flow chart showing a method of mapping feature points in a clear area of ​​each image to be processed to a first image according to an embodiment of the present invention. Figure 5As shown, the above step S220 may include steps S510 to S520.

[0074] In step S510, for each image to be processed that is not the first image, the differences in image parameters between the images to be processed where every two adjacent clear areas are located are accumulated to determine the differences in image parameters between the image to be processed and the first image, wherein the adjacent clear areas are all located between the clear area of ​​the image to be processed and the clear area of ​​the first image.

[0075] like Figure 4 As shown, the image to be processed other than the first image may be the sixth image, and the adjacent clear areas may include the clear areas of the sixth image, the clear areas of the seventh image, and the clear areas of the first image. Because different areas are arranged along the predetermined direction of the calibration plate, the clear areas of the sixth image, the clear areas of the seventh image, and the clear areas of the first image are also arranged along the predetermined direction of the calibration plate.

[0076] The overlapping area between the clear area of ​​the sixth image and the clear area of ​​the seventh image (the first overlapping area). The mapping matrix A3 between the sixth image and the seventh image can also be determined according to the above formula and the corresponding feature points of the sixth image and the seventh image in the first overlapping area, and the feature points in the clear area of ​​the sixth image are determined to be mapped to the feature points of the seventh image as the third feature points by using the mapping matrix A3. Then the third feature points in the clear area of ​​the seventh image are determined to be mapped to the feature points of the first image by using the mapping matrix A2. In this way, the feature points of the clear area of ​​the sixth image can also be mapped to the first image, and at this time, the mapping matrix A1 between the sixth image and the first image is A2A3. In this way, even if there is no overlapping area between the clear area of ​​the sixth image and the clear area of ​​the first image, the difference in image parameters between the sixth image and the first image can be determined according to the corresponding feature points of the sixth image and the seventh image in the first overlapping area, and the corresponding feature points of the seventh image and the first image in the third overlapping area.

[0077] In step S520, feature points in a clear area of ​​the image to be processed are mapped to the first image according to differences in image parameters between the image to be processed and the first image.

[0078] After calculating the difference in image parameters between the image to be processed and the first image, a matrix representing the mapping relationship between the feature points in the clear area of ​​the image to be processed and the feature points in the clear area of ​​the first image can be obtained, and then the mapping matrix can be used to map the feature points in the clear area of ​​the image to be processed to the first image according to the above formula.

[0079] Exemplarily, among the multiple images to be processed, the clear area of ​​the first image is closest to the center of the image.

[0080] The image information in the clear area close to the edge of the image is more susceptible to image distortion. Therefore, the feature points in the clear area close to the edge of the image to be processed can be mapped to the image center of the image to be processed as much as possible to weaken the influence of image distortion.

[0081] In the above technical solution, when different areas are arranged along a predetermined direction of the calibration plate, for each image to be processed other than the first image, the difference in image parameters between the images to be processed where each two adjacent clear areas are located is accumulated to determine the difference in image parameters between the image to be processed and the first image. Then, based on the difference in image parameters between the image to be processed and the first image, the feature points in the clear area of ​​the image to be processed are mapped to the first image. In this way, the image to be processed for which the difference in image parameters needs to be determined can be selected relatively simply according to the arrangement order, which can reduce the computational complexity of camera calibration.

[0082] Figure 6 FIG. 4 is a schematic flow chart showing a method of determining camera parameters of a camera according to a first feature point according to an embodiment of the present invention. Figure 6 As shown, the above step S130 may include steps S610 to S630.

[0083] In step S610, a single mapping matrix between a point on the calibration plate and a pixel of the first image is determined according to the pixel coordinates of the first feature point and a second feature point on the calibration plate corresponding to the first feature point.

[0084] The single mapping matrix represents the mapping relationship between the pixel coordinates of the first feature point and the second feature point on the calibration plate corresponding to the first feature point.

[0085] For example, the world coordinate system can be located on the calibration plate plane, and the position of the second feature point on the calibration plate in the second world coordinate system can be determined. Then, the single mapping matrix can be determined based on the transformation relationship between the pixel coordinate system and the world coordinate system.

[0086] For example, let z c =1, the single mapping matrix H can be determined according to the following formula 3 and formula 4:

[0087]

[0088] in,( U k , V k ) represents the pixel coordinates of the kth feature point of the image to be processed (the eighth image) other than the first image, express( U k , Vk ) is mapped to the pixel coordinates of the first feature point of the first image, ( X Wk , Y Wk )express( U k , V k ) is the coordinate of the second feature point on the calibration plate corresponding to in the world coordinate system. H 11 , H 12 , H 13 , H 21 , H 22 , H 23 , H 31 , H 32 , H 33 is the unknown element of the single mapping matrix H. s u represents the image magnification ratio in the X direction between the eighth image and the first image, s v represents the image magnification ratio in the Y direction between the eighth image and the first image, Δ u represents the horizontal coordinate offset between the eighth image and the first image, Δ v Indicates the vertical coordinate offset between the eighth image and the first image.

[0089] After transforming the above formula 3, it can be seen that each second feature point on the calibration plate can form two equations to be solved separately and , ( X Wk , Y Wk ) The physical position coordinates can be obtained in advance. However, there are 8 unknown elements in the single mapping matrix H, so it is necessary to obtain the pixel coordinates of at least 4 feature points from the image to be processed to determine the pixel coordinates of the 4 first feature points, so that each unknown element in the single mapping matrix H can be determined.

[0090] For example, for the k feature points of the eighth image [( U 1, V 1),…, ( U k , V k )] mapped to the pixel coordinates of the first feature point of the first image[ ], the single mapping matrix H can be determined according to the following formula 5:

[0091]

[0092] According to the above formula, the least squares method can be used to fit H 11 , H 12 , H 13 , H 21 , H 22 , H 23 , H 31 , H 32 , H 33 , to obtain the single mapping matrix H.

[0093] In step S620, an internal parameter matrix is ​​determined, wherein the elements of the internal parameter matrix include the image zoom center and focal length corresponding to the first image.

[0094] The intrinsic parameter matrix represents the influence of the image zoom center and focal length corresponding to the first image on the single mapping matrix.

[0095] For example, the internal parameter matrix I can be determined according to the following formula 6:

[0096]

[0097] in, f x represents the horizontal focal length, f y represents the vertical focal length, ( u 0, v 0) is the image scaling center coordinate. f x , f y , u 0. v 0 is an unknown element.

[0098] In step S630, the camera extrinsic parameters and the camera intrinsic parameters of the camera are determined according to the single mapping matrix and the intrinsic parameter matrix.

[0099] For example, there is a relationship between the single mapping matrix H and the internal parameter matrix I as shown in the following formula 7:

[0100]

[0101] in,R 1 and R 2 is the rotation vector of the camera relative to the calibration plate plane, is the translation vector of the camera relative to the calibration plate plane, h 1 、h 2 、h 3 represent the first, second and third column vectors of the matrix respectively.

[0102] We can let the matrix B = (I -1 ) T I -1 , and then determine the matrix B according to the following formula 8 and formula 9:

[0103]

[0104] in, B 11 、B 12 、B 13 、B 21 、B 22 、B 23 、B 31 、B 32 、B 33 are the unknown elements of matrix B. is an unknown constant.

[0105] Let r = B 33 , the following formula 10 can be obtained from the above formula 9:

[0106]

[0107] Among them, vector b represents the relevant information of the camera's internal parameters. No matter how many sets of calibration plate images placed at different angles are taken, it is a fixed value. When taking an image to be processed of a calibration plate placed at a pitch angle, two equations about b can be obtained (which can be determined by formula 10). Because vector b has at least 4 unknown quantities: B 11 、B 13 、B 22 、B 23Therefore, at least two images of the calibration plate placed at different pitch angles need to be taken to obtain at least four equations to solve the four unknowns of vector b. Singular value decomposition or least squares method can then be used to solve b.

[0108] The matrix V in Formula 10 can be decomposed into the product of three matrices: V=U ∑ V T . Among them, U and V are orthogonal matrices, ∑ is a diagonal matrix, and the elements on its diagonal are singular values, arranged from large to small. Next, we can analyze the singular value, which reflects the linear independence of the column vectors of the matrix V. If there is a zero singular value in ∑, the corresponding column vectors are linearly dependent. Next, extract the column vector corresponding to the minimum singular value (ideally it should be zero) from V (the right singular matrix of the singular value decomposition). This vector is the above formula Vb=0 For example, after the singular value decomposition of matrix V, its last two singular values ​​are zero, and the corresponding V T The last two columns are v 4 and v 5. Vb=0 The solution is: b = c 4 v 4+ c 5 v 5, among which c 4 and c 5 is a constant. From this we can determine the B 11 、B 13 、B 22 、B 23 .

[0109] As shown in the following table, the camera internal parameters can be determined based on the elements in the obtained vector b:

[0110]

[0111] according to f x 、f y 、u 0 、v 0 The elements in the intrinsic parameter matrix I can be determined, and then the camera external parameters shown in Table 2 can be determined:

[0112]

[0113] Furthermore, the orientation angle of the camera relative to the plane of the calibration plate and the coordinate system translation can be calculated through Euler angle conversion.

[0114] In the above technical solution, a single mapping matrix between the point on the calibration plate and the pixel of the first image is determined according to the pixel coordinates of the first feature point and the second feature point on the calibration plate corresponding to the first feature point, an intrinsic parameter matrix including the image zoom center and focal length corresponding to the first image is determined, and the camera extrinsic parameters and camera intrinsic parameters of the camera are determined according to the single mapping matrix and the intrinsic parameter matrix. The camera extrinsic parameters and camera intrinsic parameters of the camera can be determined more accurately by using the characteristics between the single mapping matrix and the intrinsic parameter matrix.

[0115] Figure 7 FIG. 4 is a schematic block diagram of a camera calibration device according to an embodiment of the present invention. Figure 7 As shown, the camera calibration device includes a receiving module 710, an identification module 720, and a parameter determination module 730.

[0116] The receiving module 710 is used to obtain a plurality of images to be processed. The plurality of images to be processed are obtained by focusing on different areas of the calibration plate using a camera and photographing the image to be processed of the calibration plate after each focusing.

[0117] The recognition module 720 is used to determine the first feature points in the first image based on the feature points in the clear area of ​​each image to be processed, where the first image is one of multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, and the first feature points include the feature points in the clear area of ​​the first image and at least some of the feature points outside the clear area of ​​the first image.

[0118] The parameter determination module 730 is used to determine the camera parameters of the camera according to the first feature point.

[0119] Exemplarily, the clear area of ​​any one of the multiple images to be processed has an overlapping area with the clear areas of other images to be processed. The recognition module 720 includes a difference determination submodule, a mapping submodule and a first determination submodule. The difference determination submodule is used to determine the difference in image parameters between the images to be processed related to the overlapping area according to the feature points in the overlapping area in the image to be processed related to the overlapping area, and the clear area of ​​the image to be processed related to the overlapping area is one of a pair of clear areas that both cover the overlapping area. The mapping submodule is used to map the feature points in the clear area of ​​each image to be processed to the first image according to the difference in image parameters. The first determination submodule is used to determine the first feature point according to the feature points in the clear area of ​​the first image and the feature points mapped to the first image.

[0120] Exemplarily, different areas are arranged along a predetermined direction of the calibration plate. The mapping submodule includes a difference accumulation submodule and a first mapping submodule. The difference accumulation submodule is used to accumulate the differences in image parameters between the images to be processed where each two adjacent clear areas are located for each image to be processed other than the first image, so as to determine the differences in image parameters between the image to be processed and the first image, wherein the adjacent clear areas are located between the clear area of ​​the image to be processed and the clear area of ​​the first image. The first mapping submodule is used to map feature points in the clear area of ​​the image to be processed to the first image according to the differences in image parameters between the image to be processed and the first image.

[0121] Exemplarily, among the multiple images to be processed, the clear area of ​​the first image is closest to the center of the image.

[0122] Exemplarily, the image parameters include image magnification, image zoom center, and image translation amount.

[0123] Exemplarily, there are at least 8 feature points in the overlapping area.

[0124] Exemplarily, the parameter determination module 730 includes a second determination submodule, a third determination submodule and a fourth determination submodule.

[0125] The second determination submodule is used to determine a single mapping matrix between a point on the calibration plate and a pixel of the first image based on the pixel coordinates of the first feature point and a second feature point on the calibration plate corresponding to the first feature point. The third determination submodule is used to determine an intrinsic parameter matrix, wherein the elements of the intrinsic parameter matrix include an image zoom center and a focal length corresponding to the first image. The fourth determination submodule is used to determine camera extrinsic parameters and camera intrinsic parameters of the camera based on the single mapping matrix and the intrinsic parameter matrix.

[0126] According to another aspect of the present invention, an electronic device is provided. Figure 8 FIG. 1 shows a schematic block diagram of an electronic device according to an embodiment of the present invention. Figure 8 As shown, the electronic device includes a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute the camera calibration method described above when the processor is executed.

[0127] In addition, according to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the camera calibration method according to the embodiment of the present invention, and is used to implement the corresponding modules in the camera calibration device according to the embodiment of the present invention. The storage medium may include, for example, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0128] According to another aspect of the present invention, a computer program product is further provided, comprising computer program instructions, wherein the computer program instructions are used to execute the above-mentioned camera calibration method when running.

[0129] A person skilled in the art can understand the specific implementation and beneficial effects of the above-mentioned camera calibration device, electronic device, storage medium and computer program product by reading the above-mentioned detailed description of the camera calibration method. For the sake of brevity, they will not be described in detail here.

[0130] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0131] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0133] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0134] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0135] Those skilled in the art will understand that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0136] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0137] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in a camera calibration device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0138] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0139] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A camera calibration method, characterized in that: The method comprises: Using the camera, focusing on different areas of the calibration plate respectively and photographing the image to be processed of the calibration plate after each focusing, so as to obtain a plurality of images to be processed; Determine, according to feature points in an overlapping area of ​​clear areas of each image to be processed, a first feature point in a first image, wherein the first image is one of the multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, the first feature point includes feature points in the clear area of ​​the first image and at least part of feature points outside the clear area of ​​the first image, and the clear area of ​​any image to be processed among the multiple images to be processed has an overlapping area with the clear areas of other images to be processed; A camera parameter of the camera is determined according to the first feature point.

2. The method according to claim 1, characterized in that The step of determining the first feature point in the first image according to the feature points in the overlapping area of ​​the clear area of ​​each image to be processed comprises: For any overlapping area, according to the feature points in the overlapping area in the image to be processed related to the overlapping area, the difference in image parameters between the images to be processed related to the overlapping area is determined, and the clear areas of the images to be processed related to the overlapping area are respectively one of a pair of clear areas both covering the overlapping area; According to the difference of the image parameters, mapping the feature points in the clear area of ​​each image to be processed to the first image; The first feature points are determined according to the feature points in the clear area of ​​the first image and the feature points mapped to the first image.

3. The method according to claim 2, characterized in that The different areas are arranged along a predetermined direction of the calibration plate; Mapping the feature points in the clear area of ​​each image to be processed to the first image according to the difference in the image parameters includes: For each image to be processed other than the first image, Accumulating the differences in image parameters between the images to be processed where every two adjacent clear areas are located, to determine the differences in image parameters between the image to be processed and the first image, wherein the adjacent clear areas are all located between the clear area of ​​the image to be processed and the clear area of ​​the first image; According to the difference in image parameters between the image to be processed and the first image, feature points in the clear area of ​​the image to be processed are mapped to the first image.

4. The method according to claim 3, characterized in that Among the multiple images to be processed, the clear area of ​​the first image is closest to the center of the image.

5. The method according to claim 2, characterized in that: The image parameters include image magnification, image zoom center and image translation amount.

6. The method according to claim 2, characterized in that There are at least 8 feature points in the overlapping area.

7. The method according to claim 1, characterized in that The step of determining the camera parameters of the camera according to the first feature point includes: Determine a single mapping matrix between a point on the calibration plate and a pixel of the first image according to a pixel coordinate of the first feature point and a second feature point on the calibration plate corresponding to the first feature point; Determine an intrinsic parameter matrix, wherein the elements of the intrinsic parameter matrix include an image zoom center and a focal length corresponding to the first image; According to the single mapping matrix and the intrinsic parameter matrix, camera extrinsic parameters and camera intrinsic parameters of the camera are determined.

8. A camera calibration device, characterized in that: include: A receiving module, used for acquiring a plurality of images to be processed, wherein the plurality of images to be processed are obtained by focusing on different areas of a calibration plate using a camera and photographing the image to be processed of the calibration plate after each focusing; an identification module, configured to determine first feature points in a first image according to feature points in an overlapping area of ​​clear areas of each image to be processed, wherein the first image is one of the multiple images to be processed, wherein the clear area is an area in the image to be processed whose clarity is higher than a preset threshold, the first feature points include feature points in the clear area of ​​the first image and at least part of feature points outside the clear area of ​​the first image, and the clear area of ​​any image to be processed among the multiple images to be processed has an overlapping area with the clear areas of other images to be processed; A parameter determination module determines the camera parameters of the camera according to the first feature points.

9. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions, which are used by the processor to execute the camera calibration method according to any one of claims 1 to 7 when executed.

10. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the camera calibration method according to any one of claims 1 to 7 when the processor is running.

11. A computer program product comprising computer program instructions, characterized in that The computer program instructions are used to execute the camera calibration method according to any one of claims 1 to 7 when the processor is running.

Images