Camera calibration method and device, electronic equipment and medium

Abstract

This disclosure proposes a camera calibration method, apparatus, electronic device, and medium. The method includes: acquiring a target image captured by a camera to be calibrated; wherein the target image displays multiple marker patterns; extracting edge curves belonging to the same marker pattern from the target image; determining the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system based on the image positions of the edge curves of the multiple marker patterns in the target image; and determining the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of the multiple marker patterns. Therefore, the extrinsic parameter calibration of the camera to be calibrated can be completed from a single image captured by the camera, simplifying user operation and improving the user experience.

Description

Technical Field

[0001] This disclosure relates to the field of camera calibration technology, and in particular to a camera calibration method, apparatus, electronic device and medium. Background Technology

[0002] Camera calibration is a crucial technique in image processing. In image measurement and machine vision applications, to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of the camera's imaging must be established. These geometric model parameters are the camera parameters. These camera parameters can be obtained through experimentation and calculation; the process of solving for the camera parameters is called camera calibration. The accuracy of the calibration determines whether the machine vision system can effectively perform operations such as locating, detecting, segmenting, and ranging regions of interest in an image.

[0003] Camera calibration can be divided into two sub-processes: intrinsic parameter calibration and extrinsic parameter calibration. The camera extrinsic parameters, also known as camera pose, are generally represented in matrix form and can be decomposed into a translation matrix (position) and a rotation matrix (pose). Camera extrinsic parameter calibration refers to determining a rotation matrix and a translation matrix using a certain method, thereby calculating the complete extrinsic parameter matrix. A point in the world coordinate system is multiplied by the extrinsic parameter matrix, and thus enters the camera coordinate system; conversely, a point in the camera coordinate system is multiplied by the inverse of the extrinsic parameter matrix, and thus enters the world coordinate system. In other words, the camera extrinsic parameters indicate the mapping relationship between the camera coordinate system and the world coordinate system.

[0004] Therefore, it is very important to calibrate the camera to determine its extrinsic parameters. Summary of the Invention

[0005] This disclosure aims to at least partially address one of the technical problems in the related art.

[0006] This disclosure proposes a camera calibration method, apparatus, electronic device, and medium to calibrate the external parameters of a camera based on an image captured by the camera to be calibrated, thereby simplifying user operation and improving user experience.

[0007] The first aspect of this disclosure provides a camera calibration method, including:

[0008] Acquire a target image captured by the camera to be calibrated; wherein the target image displays multiple marker patterns;

[0009] Extract edge curves belonging to the same marker pattern from the target image;

[0010] Based on the image position of the edge curves of the multiple marker patterns in the target image, determine the target coordinates and normal vector of the center point of the multiple marker patterns in the camera coordinate system;

[0011] The extrinsic parameters of the camera to be calibrated are determined based on the target coordinates and normal vectors of the center points of the multiple marked patterns.

[0012] A second aspect of this disclosure provides another camera calibration apparatus, comprising:

[0013] The acquisition module is used to acquire a target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns;

[0014] The extraction module is used to extract edge curves belonging to the same marker pattern from the target image;

[0015] The first determining module is used to determine the target coordinates and normal vector of the center point of the multiple marking patterns in the camera coordinate system based on the image position of the edge curves of the multiple marking patterns in the target image;

[0016] The second determining module is used to determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of the multiple marking patterns.

[0017] A third aspect of this disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a camera calibration method proposed in a first aspect of this disclosure.

[0018] A fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing the computer to perform a camera calibration method according to a first aspect of this disclosure.

[0019] A fifth aspect of this disclosure provides a computer program that, when executed by a processor, implements the camera calibration method described in the first aspect of this disclosure.

[0020] One embodiment of the present disclosure described above has at least the following advantages or beneficial effects:

[0021] By acquiring a target image captured by the camera to be calibrated, which displays multiple marker patterns, edge curves belonging to the same marker pattern are extracted from the target image. Based on the image positions of the edge curves of the multiple marker patterns in the target image, the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system are determined. Based on the target coordinates and normal vectors of the center points of the multiple marker patterns, the extrinsic parameters of the camera to be calibrated are determined. Therefore, the extrinsic parameter calibration of the camera to be calibrated can be completed from a single image captured by the camera, simplifying user operation and improving the user experience.

[0022] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0023] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0024] Figure 1 This is a schematic flowchart of the camera calibration method provided in Embodiment 1 of this disclosure;

[0025] Figure 2 This is a schematic flowchart of the camera calibration method provided in Embodiment 2 of this disclosure;

[0026] Figure 3 This is a schematic flowchart of the camera calibration method provided in Embodiment 3 of this disclosure;

[0027] Figure 4 This is a schematic flowchart of the camera calibration method provided in Embodiment 4 of this disclosure;

[0028] Figure 5 This is a schematic diagram of the camera calibration device provided in Embodiment 5 of this disclosure;

[0029] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0030] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0031] A TV top-mounted camera refers to a camera installed on a television set. Besides some televisions having a built-in camera, consumers can also purchase aftermarket cameras and install them themselves. Currently, there are two typical uses for self-installed cameras: the first is for video interaction, such as video conferencing and interactive games, in which case the camera is capturing the television viewer; the second is for analyzing the content of the television screen, such as in viewing atmosphere enhancement systems, in which case the camera is capturing the television screen itself.

[0032] For the second application, the existing method for calibrating camera extrinsic parameters involves requiring consumers to affix seven square markers along the frame of the television: three markers on the left and right sides, and one marker in the middle of the bottom frame. Using these markers, the calibration software can establish the external contour and dimensional relationships of the television, thereby calculating the extrinsic parameters between the camera coordinate system and the television coordinate system (the coordinate system established based on the seven markers).

[0033] However, the above-mentioned camera extrinsic parameter calibration method has the following drawbacks:

[0034] First, a large number of markers are required. Since the markers are affixed by ordinary consumers without any training, some markers may have large positional errors due to a lack of reliable positioning methods, and the accuracy of the calibration cannot be guaranteed.

[0035] Secondly, the material, shape, and color of the markers are specially designed and can only be provided by the manufacturer. When the camera position changes for any reason and the consumer has to recalibrate, if some markers have been lost, the consumer must seek technical support from the manufacturer, which increases the consumer's time cost.

[0036] Third, the markers need to be affixed to the television. When the calibration is completed and the markers need to be removed, each operation increases the risk of physical damage and surface contamination to the television.

[0037] Fourth, the calibration process is time-consuming and labor-intensive. To ensure the calibration effect, consumers may need to adjust the position of the markers multiple times, while also taking care to prevent accidental damage to the television. Therefore, consumers need to be careful when pasting and removing the markers, resulting in high time costs and a poor user experience.

[0038] In response to at least one of the aforementioned problems, this disclosure proposes a camera calibration method, apparatus, electronic device, and storage medium.

[0039] The camera calibration method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0040] Figure 1This is a schematic flowchart of the camera calibration method provided in Embodiment 1 of this disclosure.

[0041] The camera calibration method of this disclosure can be applied to any electronic device to enable the electronic device to perform camera calibration functions.

[0042] Among them, electronic devices can be any device with computing capabilities, such as personal computers, mobile terminals, servers, etc. Mobile terminals can be hardware devices with various operating systems, touch screens and / or displays, such as mobile phones, tablets, personal digital assistants, wearable devices, etc.

[0043] like Figure 1 As shown, the camera calibration method may include the following steps:

[0044] Step 101: Obtain the target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns.

[0045] In this embodiment of the disclosure, the camera to be calibrated refers to the camera that needs to undergo extrinsic parameter calibration.

[0046] In this embodiment of the disclosure, the number of marking patterns may include, but is not limited to, two.

[0047] In this embodiment of the disclosure, there are no restrictions on the shape and color of the marking pattern. For example, the shape of the marking pattern can be a circle, an ellipse, a square, a rectangle, etc., and the color of the marking pattern can be black, white, red, etc.

[0048] In this embodiment of the disclosure, the camera to be calibrated can capture multiple marker patterns to obtain a target image, thereby enabling the acquisition of the target image captured by the camera to be calibrated in this disclosure.

[0049] Step 102: Extract edge curves belonging to the same marker pattern from the target image.

[0050] In this embodiment of the disclosure, edge curves belonging to the same marker image are extracted from the target image based on image recognition technology. For example, edge detection can be performed on the target image to obtain edge curves belonging to the same marker pattern.

[0051] Step 103: Based on the image positions of the edge curves of the multiple marker patterns in the target image, determine the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system.

[0052] In this embodiment, the origin O of the camera coordinate system is the optical center (or simply optical center) of the camera to be calibrated. The vertical axis (z-axis) is the optical axis of the camera to be calibrated. The plane formed by the horizontal axis (x-axis) and the vertical axis (y-axis) is perpendicular to the z-axis. The horizontal direction is taken as the x-axis direction, and the y-axis is perpendicular to the x-axis. That is, the origin of the camera coordinate system is the optical center of the camera to be calibrated, the x-axis of the camera coordinate system is parallel to the x-axis of the image coordinate system of the target image, the y-axis of the camera coordinate system is parallel to the y-axis of the image coordinate system of the target image, and the z-axis is the optical axis of the camera to be calibrated. The intersection of the optical axis and the target image plane is the origin of the image coordinate system.

[0053] The image coordinate system is a two-dimensional rectangular coordinate system. The origin of the image coordinate system is the center of the target image, and the x-axis and y-axis are parallel to the two sides of the target image, with the x-axis pointing horizontally to the right and the y-axis pointing vertically downwards.

[0054] In this embodiment of the disclosure, for any one of the multiple marking patterns, the coordinates of the center point of the multiple marking patterns in the camera coordinate system (denoted as the target coordinates in this disclosure) and the normal vector of the center point of the multiple marking patterns in the camera coordinate system can be determined based on the image position of the edge curve of the marking pattern in the target image.

[0055] Step 104: Determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marker patterns.

[0056] In this embodiment of the disclosure, the extrinsic parameters of the camera to be calibrated can be determined based on the target coordinates and normal vectors of the center points of multiple marked images.

[0057] The camera calibration method of this disclosure involves acquiring a target image captured by a camera to be calibrated, wherein the target image displays multiple marker patterns; extracting edge curves belonging to the same marker pattern from the target image; determining the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system based on the image positions of the edge curves of the multiple marker patterns in the target image; and determining the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of the multiple marker patterns. Therefore, the extrinsic parameter calibration of the camera to be calibrated can be completed based on a single image captured by the camera, simplifying user operation and improving the user experience.

[0058] To clearly illustrate how the above embodiments extract edge curves belonging to the same marker pattern from a target image, this disclosure also proposes a camera calibration method.

[0059] Figure 2 This is a schematic flowchart of the camera calibration method provided in Embodiment 2 of this disclosure.

[0060] like Figure 2As shown, the camera calibration method may include the following steps:

[0061] Step 201: Obtain the target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns.

[0062] The explanation of step 201 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0063] In one possible implementation of the embodiments of this disclosure, in order to improve the accuracy and reliability of the external parameter calculation results, the target image can also be preprocessed, wherein the preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing and erosion processing.

[0064] As an example, when preprocessing includes color space transformation, the target image can be transformed from a three-channel (R (red), G (green), and B (blue)) color image into a single-channel grayscale image. For instance, when performing color space transformation on the target image, assuming that both input and output are normalized to the [0,1] interval, the transformation formula used can be:

[0065] Y=CLIP(0.299*R+0.587*G+0.114*B); (1)

[0066] Where R, G, and B are the values ​​of each color channel before pixel transformation, Y is the pixel value after pixel transformation, and CLIP() function restricts the output pixel value Y to the range of [0,1].

[0067] As an example, when preprocessing includes noise reduction and smoothing, the noise reduction and smoothing process includes, but is not limited to, filtering processes such as Gaussian filtering and median filtering.

[0068] As an example, when preprocessing includes binarization, a binarization algorithm can be used to binarize the target image.

[0069] As an example, when preprocessing includes erosion, an erosion algorithm can be used to erode the target image.

[0070] Step 202: Perform edge detection on the target image to obtain a set of edge curves, wherein the set of edge curves includes at least one edge curve.

[0071] In this embodiment of the disclosure, edge detection can be performed on the target image based on an edge detection algorithm to obtain a set of edge curves, wherein the set of edge curves includes at least one edge curve.

[0072] Step 203: For any edge curve in the set of edge curves, determine the length of that edge curve.

[0073] In this embodiment of the disclosure, the length of any edge curve in the set of edge curves can be determined. For example, the length of the edge curve can be determined based on the coordinates of each pixel on the edge curve.

[0074] Step 204: If the length is greater than the first length threshold, the arbitrary edge curve is divided into multiple sub-edge curves, and the arbitrary edge curve is deleted from the edge curve set, and the multiple sub-edge curves are added to the edge curve set.

[0075] The first length threshold is a pre-set length threshold.

[0076] In this embodiment of the disclosure, if the length of a certain edge curve is greater than a first length threshold, the edge curve can be segmented to obtain multiple sub-edge curves, wherein the length of each sub-edge curve is less than or equal to the first length threshold. Then, the edge curve can be deleted from the edge curve set, and the multiple sub-edge curves can be added to the edge curve set.

[0077] In one possible implementation of this disclosure, to reduce the impact of shorter curves on subsequent calculation results, if the length of an edge curve is less than a second length threshold, the edge curve can be deleted. The second length threshold is a pre-set length threshold, and it is less than a first length threshold.

[0078] Step 205: Cluster the curves in the updated edge curve set to obtain multiple clusters.

[0079] In this embodiment of the disclosure, the curves in the updated set of edge curves can be clustered based on the boundary clustering algorithm to obtain multiple clusters.

[0080] Step 206: Based on the size of the multiple marker patterns and the distance between the multiple marker patterns, determine the target cluster to which each marker pattern belongs from the multiple clusters, wherein the target cluster includes edge curves belonging to the same marker pattern.

[0081] It should be noted that the multiple marker patterns are pre-set, and the size of each marker pattern (for example, assuming that the size and shape of each marker pattern are the same, all are circles with a radius of r) and the distance between each marker pattern can be obtained.

[0082] In this embodiment of the disclosure, the target cluster to which each marker pattern belongs can be determined from multiple clusters based on the size of the multiple marker patterns and the distance between the multiple marker patterns.

[0083] For example, suppose there are two marker patterns, both of which are circles with a radius of r. If there are three clusters, namely cluster 1, cluster 2 and cluster 3, and the radius of the circle indicated by each edge curve in cluster 1 is equal to the radius of the circle indicated by each edge curve in cluster 2, then cluster 1 and cluster 2 can be regarded as the target clusters.

[0084] Step 207: Based on the image positions of the edge curves of the multiple marker patterns in the target image, determine the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system.

[0085] In this embodiment of the disclosure, the target coordinates and normal vector of the center point of the marker pattern corresponding to each target cluster in the camera coordinate system can be determined based on the image position of the edge curve in each target cluster in the target image.

[0086] Step 208: Determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marker patterns.

[0087] The explanation of step 208 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0088] The camera calibration method of this disclosure can effectively extract edge curves belonging to the same marker pattern from a target image through edge detection and clustering.

[0089] To clearly illustrate how the target coordinates and normal vectors of the center points of multiple marker patterns in the camera coordinate system are determined in any embodiment of this disclosure, this disclosure also proposes a camera calibration method.

[0090] Figure 3 This is a schematic flowchart of the camera calibration method provided in Embodiment 3 of this disclosure.

[0091] like Figure 3 As shown, the camera calibration method may include the following steps:

[0092] Step 301: Obtain the target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns.

[0093] Step 302: Perform edge detection on the target image to obtain a set of edge curves, wherein the set of edge curves includes at least one edge curve.

[0094] Step 303: For any edge curve in the set of edge curves, determine the length of any edge curve.

[0095] Step 304: If the length is greater than the first length threshold, any edge curve is divided into multiple sub-edge curves, and any edge curve is deleted from the edge curve set, and multiple sub-edge curves are added to the edge curve set.

[0096] Step 305: Cluster the curves in the updated edge curve set to obtain multiple clusters.

[0097] Step 306: Based on the size of the multiple marker patterns and the distance between the multiple marker patterns, determine the target cluster to which each marker pattern belongs from the multiple clusters.

[0098] The explanation of steps 301 to 306 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0099] Step 307: For any one of the multiple marker patterns, extract multiple target edge points from the target cluster to which that marker pattern belongs.

[0100] In this embodiment of the disclosure, for any one of the multiple marker patterns, multiple target edge points can be extracted from the target cluster to which the marker image belongs.

[0101] As one possible approach, in order to balance computational speed and accuracy, at least one target edge point can be extracted from each edge curve in the target cluster.

[0102] As an example, for any curve in the target cluster to which the marking pattern belongs, at least one candidate edge point can be extracted from the curve, and the image position of the candidate edge points on each curve in the target cluster in the target image can be determined. Thus, the distance between each candidate edge point can be determined based on the image position of each candidate edge point in the target image, so as to determine the target edge point from each candidate edge point based on the distance between each candidate edge point.

[0103] For example, candidate edge points whose distance is greater than a set distance threshold can be used as target edge points.

[0104] Step 308: Based on the image positions of multiple target edge points in the target image, determine the target coordinates and normal vector of the center point of any marker pattern in the camera coordinate system.

[0105] In this embodiment of the disclosure, the target coordinates and normal vector of the center point of the above-mentioned marking pattern in the camera coordinate system can be determined based on the image positions of multiple target edge points in the target image (such as the position coordinates of multiple target edge points in the image coordinate system).

[0106] As one possible implementation, the target coefficient matrix can be determined based on the image positions of multiple target edge points in the target image (e.g., the position coordinates of multiple target edge points in the image coordinate system), where the target coefficient matrix is ​​used to indicate the shape of the aforementioned marking pattern.

[0107] As an example, taking a circular shape as an example, due to the principle of perspective, the marker pattern appears as an ellipse in the target image. The position coordinates (x, y) of multiple target edge points in the image coordinate system can be substituted into the following ellipse matrix equation:

[0108] GX = b; (2)

[0109] in,

[0110] in,

[0111] Where m represents the number of target edge points, (x0, y0), (x1, y1), ..., (x m-1 ,y m-1 ) represent the position coordinates of m target edge points in the image coordinate system.

[0112] Then, the coefficients of the ellipse equation can be estimated using the least squares method, yielding: X = (G T G) -1 G T b. Then, the corresponding target coefficient matrix Q can be constructed using the coefficients X as the marking pattern. For example, Q can be a 3×3 symmetric matrix containing 6 elements with different values, of which 5 elements are B, C, D, E, and F in X, and the other element has a fixed value (such as 1).

[0113] In this disclosure, the target coefficient matrix can also be decomposed to obtain a diagonal matrix and an intermediate matrix. For example, if the diagonal matrix is ​​denoted as Λ and the intermediate matrix as V, then:

[0114] Q = VΛV T (3)

[0115] in,

[0116] Then, based on the values ​​of each diagonal element in the diagonal matrix, an intermediate vector can be generated. For example, the intermediate vector could be... Based on the values ​​of each diagonal element and the dimensions of the marked pattern, intermediate coefficients are generated. For example, assuming the marked pattern is circular with radius r, the intermediate coefficient σ can be: Therefore, in this disclosure, the target coordinates of the center point of the above-mentioned marking pattern in the camera coordinate system can be determined based on the intermediate coefficients, intermediate vectors, and intermediate matrices. For example, if the target coordinates of the center point are C, then:

[0117]

[0118] Among them, s1 to s3 represent positive and negative signs, and the correct sign can be retained according to the actual situation.

[0119] Furthermore, the normal vector of the center point of the above-mentioned marking pattern in the camera coordinate system can be determined based on the intermediate vector and the intermediate matrix. For example, if the normal vector of the center point is N, then:

[0120]

[0121] Step 309: Determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marker patterns.

[0122] The explanation of step 309 can be found in the relevant description in any embodiment of this disclosure, and will not be repeated here.

[0123] The camera calibration method of this disclosure can effectively calculate the coordinates and normal vector of the center point of the marker pattern in the camera coordinate system based on the image position of the target edge point on the edge curve of each marker pattern in the target image.

[0124] To clearly illustrate how the extrinsic parameters of the camera to be calibrated are determined based on the target coordinates and normal vectors of the center points of multiple marker patterns in any embodiment of this disclosure, this disclosure also proposes a camera calibration method.

[0125] Figure 4 This is a schematic flowchart of the camera calibration method provided in Embodiment 4 of this disclosure.

[0126] like Figure 4 As shown, the camera calibration method may include the following steps:

[0127] Step 401: Obtain the target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns.

[0128] Step 402: Extract edge curves belonging to the same marker pattern from the target image.

[0129] Step 403: Based on the image positions of the edge curves of the multiple marker patterns in the target image, determine the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system.

[0130] The explanations of steps 401 to 403 can be found in the relevant descriptions in any embodiment of this disclosure, and will not be repeated here.

[0131] Step 404: Determine the first unit vector of the horizontal axis of the world coordinate system based on the difference between the target coordinates of the center points of each marked pattern.

[0132] In this embodiment of the disclosure, the world coordinate system can be a coordinate system pre-established based on multiple marking patterns. For example, taking two marking patterns, each of which is circular, as an example, the midpoint of the line connecting the centers of the two marking patterns can be used as the origin of the world coordinate system, the direction of the line connecting the centers can be used as the x-axis direction, the direction perpendicular to the x-axis on the marking pattern plane can be used as the y-axis direction, and the z-axis can be perpendicular to the marking pattern plane.

[0133] In this embodiment of the disclosure, the first unit vector of the horizontal axis (x-axis) of the world coordinate system can be determined based on the difference between the target coordinates of the center points of each marking pattern.

[0134] Taking a case with two marker patterns as an example, and the target coordinates of the center points of the two marker patterns being C1 and C2 respectively, then the first unit vector i of the horizontal axis (x-axis) of the world coordinate system is... w It can be:

[0135] i w =(C2-C1) / |C2-C1|; (6)

[0136] Among them, i w The x-axis unit vector in the world coordinate system is projected onto the camera coordinate system.

[0137] Step 405: Determine the second unit vector of the vertical axis of the world coordinate system based on the normal vector.

[0138] In this embodiment of the disclosure, the second unit vector of the vertical axis (z-axis) of the world coordinate system can be determined based on the normal vector.

[0139] Taking a case where there are two marker patterns as an example, the second unit vector k of the vertical axis (z-axis) of the world coordinate system is... w Possible forms:

[0140] k w =N; (7)

[0141] Where, k w The z-axis unit vector in the world coordinate system is projected onto the camera coordinate system.

[0142] Step 406: Determine the third unit vector of the vertical axis of the world coordinate system based on the first unit vector and the second unit vector.

[0143] In this embodiment of the disclosure, the third unit vector of the vertical axis (y-axis) can be determined based on the first unit vector of the horizontal axis (x-axis) and the second unit vector of the vertical axis (z-axis) of the world coordinate system.

[0144] Using the example above, the third unit vector j of the world coordinate system's vertical axis (y-axis) is... w Possible forms:

[0145] j w =k w ×i w (8)

[0146] Where, j w The projection coordinates of the y-axis unit vector in the world coordinate system onto the camera coordinate system.

[0147] Step 407: Determine the extrinsic parameters of the camera to be calibrated based on the first unit vector, the second unit vector, and the third unit vector.

[0148] In this embodiment of the disclosure, the extrinsic parameters of the camera to be calibrated can be determined based on the first unit vector, the second unit vector, and the third unit vector.

[0149] For example, if the extrinsic parameters of the camera to be calibrated are matrix M, then:

[0150] M = (i w j w k w (9)

[0151] The camera calibration method of this disclosure can effectively calculate the extrinsic parameters of the camera to be calibrated based on the unit vectors of each axis in the world coordinate system.

[0152] As an example of an application scenario, let's take the camera to be calibrated as a top-mounted camera on a television, and the purpose of calibrating the camera is to analyze the content of the television screen. Assuming that there are two marker patterns and the marker images are drawn on calibration cards, the two calibration cards can be attached to the television screen so that the calibration cards are coplanar with the television plane, and a world coordinate system can be established based on the two calibration cards.

[0153] Adjust the position of the camera lens of the camera to be calibrated so that the two calibration cards appear completely and symmetrically in the shooting frame, and then lock the camera to prevent its position from changing. Establish a camera coordinate system with the optical center and optical axis of the camera as the reference.

[0154] The camera captures a target image. The calibration card information is extracted from the target image to obtain the rotation matrix of the two calibration cards and the position coordinates of the center point of the two calibration cards in the camera coordinate system. A set of reference points is selected in the world coordinate system (e.g., the position points of the unit vectors on each coordinate axis of the world coordinate system). The world coordinates of the reference points are used as the source values, and the camera coordinates of the reference points calculated in the target image are used as the target values. The transformation matrix from the source values ​​to the target values ​​is solved. This transformation matrix is ​​the extrinsic parameter of the camera.

[0155] Alternatively, the two calibration cards can be identical in design, both being square cards.

[0156] Optionally, the calibration card has suitable hardness, thickness, and flatness. When the bottom of the calibration card contacts the lower bezel of the TV, the calibration card naturally enters a state of force equilibrium under its own weight. The lower surface of the calibration card is attached to the lower bezel of the TV, the side of the calibration card is attached to the side bezel of the TV, and the back of the calibration card is attached to the TV screen. It will not fall off automatically without external force. That is, a calibration card is placed in the lower left corner and the lower right corner of the TV screen.

[0157] Optionally, the calibration card has an adsorption design. After the calibration card has been properly positioned, gently press the front of the calibration card with your hand or a tool to create a vacuum adsorption effect between the calibration card and the TV screen.

[0158] Optionally, the calibration card has a pure white background with a black circle (referred to as a marking pattern in this disclosure) printed on it, the diameter of which is the same as or close to the side length of the calibration card. Alternatively, the calibration card may have a pure black background with a white circle printed on it, the diameter of which is the same as or close to the side length of the calibration card.

[0159] Optionally, the calibration card adopts a symmetrical design, with the front and back sides having the same pattern, and the calibration card can be used after being rotated 90°, 180°, or 270°, with the same effect.

[0160] Optionally, the world coordinate system uses the midpoint of the line connecting the centers of the two calibration cards as the origin, the direction of the line connecting the centers as the x-axis (the right-hand direction is positive when facing the television), the direction perpendicular to the x-axis on the calibration card plane as the y-axis (upward is positive), and the z-axis is perpendicular to the calibration card plane and points in the direction of the television viewer.

[0161] Optionally, the camera coordinate system is based on the optical center of the camera (referred to as the optical center) as the origin O, the optical axis of the camera as the z-axis (positive in the direction away from the TV), the plane formed by the x-axis and y-axis is perpendicular to the z-axis, the horizontal direction is the x-axis direction (positive in the right-hand direction when facing the TV), and the y-axis is perpendicular to the x-axis (positive upwards).

[0162] Optionally, a target image captured by the camera is called image I0, where I0 contains two calibration cards. Due to the principle of perspective, the perfect circle on each calibration card appears as an ellipse in image I0; the one on the left is called E1, and the one on the right is called E2.

[0163] Optionally, before using image I0 to calculate E1 and E2 information, image I0 can be preprocessed, and the preprocessed image is used when calculating ellipse information.

[0164] As an example, the preprocessing of image I0 includes a color space transformation, which converts image I0 from a three-channel color image into a single-channel grayscale image, the resulting grayscale image being called I1.

[0165] For example, when performing color space transformation on image I0, assuming that both input and output are normalized to the [0,1] interval, the transformation formula used is: Y=CLIP(0.299*R+0.587*G+0.114*B), where the CLIP() function restricts the output pixel value Y to the [0,1] interval.

[0166] As an example, the preprocessing of image I0 includes noise reduction and smoothing, with grayscale image I1 as input and grayscale image I2 as output.

[0167] For example, noise reduction and smoothing of the grayscale image I1 can be achieved using filtering algorithms such as Gaussian filtering and median filtering.

[0168] As an example, the preprocessing of image I0 includes edge detection and filtering steps. The input to this operation is a grayscale image I2, and the output is an edge image I3. The pixel value of image I3 represents the edge intensity value at the location of that pixel. Assume that there are n qualified edge curves in the filtered image I3, and define the set Ω = {λ...} i , i∈[1,n]}, where λ i Let l represent the i-th edge curve, and let l be its length. i express.

[0169] For example, the method for edge detection and filtering of image I2 is to use the Canny edge detection operator to obtain an initial edge image, and then use the connected component labeling algorithm to filter and process the edges present in the initial edge image. Overly long continuous edges are split into multiple shorter continuous edges, and overly short edges are deleted, so that the output image I3 contains only continuous edges of acceptable length.

[0170] As an example, the preprocessing of image I0 includes binarization and erosion operations. The input to this operation is the edge map I3, and the output is the preprocessed image I4.

[0171] Optionally, the preprocessed image I4 can be used to extract possible elliptical points (referred to as candidate edge points in this disclosure). During the extraction process, it is determined how many different ellipses exist, with each ellipse retaining no more than Max1 points, and a total of no more than Max2 points. For example, a boundary clustering algorithm can be used to extract edge curves belonging to the same ellipse from the preprocessed image I4, and elliptical points can be extracted from each edge curve. Here, Max1 can be 100, and Max2 can be 2000.

[0172] Optionally, the extracted elliptical points can be paired for analysis. Based on preset constraints (such as the distance between adjacent candidate edge points), points that meet the calibration elliptical features (referred to as target edge points in this disclosure) can be selected, and other irrelevant elliptical points can be deleted.

[0173] After identifying the elliptical points corresponding to E1 and E2, the coefficients of the elliptical equations for E1 and E2 are solved in two steps, one ellipse at a time. Specifically, the known coordinates (x, y) of all elliptical points in the image coordinate system are substituted into the matrix equation GX = b, where the subscript m represents the number of valid elliptical points (i.e., the number of target edge points).

[0174]

[0175]

[0176] Then, the coefficients X of the elliptic equation are estimated using the least squares method, resulting in: X = (G T G) -1 G T b;

[0177] After extracting the coefficients X of the calibration ellipses E1 and E2 from the image, a corresponding coefficient matrix is ​​constructed for each ellipse using X, resulting in two matrices Q1 and Q2. Then, each matrix (Q1 and Q2) is decomposed into the form of the product of three independent matrices: Q = VΛV T ,in,

[0178]

[0179] After calculating matrices Q1 and Q2, the coordinates C of the center points of the calibration ellipses E1 and E2 in the camera coordinate system and the normal vector N are calculated using the known parameters of their decomposition matrices, as follows:

[0180]

[0181]

[0182] in, s1~s3 represent positive and negative signs. The correct sign should be retained according to the actual situation. r represents the radius of the circle on the calibration card, which is a known constant.

[0183] After obtaining the coordinates C1 and C2 of the center point of the calibration ellipse, we can obtain the coordinates (C1+C2) / 2 of the origin of the world coordinate system in the camera coordinate system, as well as the unit vectors of the three coordinate axes of the world coordinate system.

[0184] Among them, i w = (C2-C1) / |C2-C1|, which is the projected coordinate of the unit vector of the x-axis in the world coordinate system in the camera coordinate system;

[0185] j w =k w ×i w , where is the projected coordinate of the world coordinate system y-axis unit vector in the camera coordinate system;

[0186] k w =N, which is the projected coordinate of the z-axis unit vector in the world coordinate system onto the camera coordinate system.

[0187] The extrinsic parameter matrix is ​​M = (i w j w k w ).

[0188] In summary, this disclosure allows for the calibration of camera extrinsic parameters using only a single image. Specifically, two calibration cards of identical specifications are used, each with a calibration circle printed on it. During calibration, a target image is captured using a camera. The target image is preprocessed to obtain an edge image. The edge image is then analyzed using an algorithm to select elliptical points (referred to as target edge points in this disclosure) that conform to the characteristics of the calibration circle. The coefficient matrix of the ellipse equation is solved using the selected elliptical points, and eigenvalue decomposition is performed on the coefficient matrix to calculate the coordinates of the center of the calibration circle (coordinates in the camera coordinate system) and the normal vector.

[0189] The extrinsic parameter matrix of the camera is solved using the coordinates of the center of the calibration circle and the normal vector.

[0190] Optionally, the calibration card is square, with an identical circle on each side; the calibration card has structural rigidity and can maintain its stability when there are three support points; the calibration card is planar and can form a vacuum adsorption effect with another plane.

[0191] Optionally, during calibration, the two calibration cards are located on the same plane, and each calibration circle must have at least 1 / 4 of its arc visible in the target image.

[0192] Optionally, the preprocessing of the target image is as follows: color space transformation, using the formula Y = CLIP(0.299*R + 0.587*G + 0.114*B), to convert the three-channel color image into a single-channel grayscale image; noise reduction filtering is applied to the grayscale image; edge detection, edge processing, and edge filtering are performed on the noise-reduced and filtered image; and binarization and erosion operations are performed on the filtered edges.

[0193] Optionally, the filtered edge images are analyzed to retain edges that are likely arcs and delete non-arc edges; a boundary clustering algorithm is used to analyze the arcs to determine how many independent ellipses exist in the image, which arcs each independent ellipse contains, and a representative coordinate point is selected for each arc; the filtered ellipses are analyzed to retain ellipses related to the calibration circle and delete other interfering ellipses. For example, based on the radius of the calibration circle and the distance between the calibration circles, a target cluster matching the number of calibration circles can be selected from multiple clusters; the ellipses in the selected target clusters related to the calibration circle are optimized, and each ellipse retains only a certain number of high-quality ellipse points (i.e., target edge points), such as deleting ellipse points with small distances.

[0194] Optionally, the coefficient matrix of the elliptic equation is calculated using the high-quality elliptic points obtained by the boundary clustering algorithm; eigenvalue decomposition is performed on the coefficient matrix of the elliptic equation; the eigenvalues ​​and eigenvectors obtained by the decomposition are used to calculate the center and normal vector of the calibration circle; and the extrinsic parameter matrix of the camera is calculated using the center and normal vector of the calibration circle.

[0195] Optionally, the number of edge curves in the target image may be no less than 100 and no more than 10,000.

[0196] Optionally, the number of continuous edge curves that conform to the arc characteristics selected from the edge curves can be no less than 20 and no more than 2000.

[0197] Optionally, continuous edge curves that conform to the characteristics of circular arcs can be classified or clustered, and the number of clusters obtained can be no less than 2 and no more than 20.

[0198] Optionally, clusters are selected based on prior knowledge, and two target clusters that conform to the characteristics of the calibration circle are output. Each target cluster outputs no less than 10 and no more than 100 elliptical points (referred to as target edge points in this disclosure).

[0199] Optionally, the camera's extrinsic matrix can be calculated based on information about the elliptical points (i.e., target edge points) in the two target clusters and prior knowledge about the calibration circle.

[0200] In summary, the camera calibration method provided in this disclosure has the following advantages:

[0201] First, camera extrinsic parameters can be calibrated using only a single image;

[0202] Secondly, the camera's extrinsic parameters can be calibrated using only two calibration cards;

[0203] Third, it allows consumers to create their own calibration cards to calibrate camera external parameters;

[0204] Fourth, camera extrinsic parameters can be calibrated without manually measuring the camera position.

[0205] The camera calibration method provided in this disclosure can simplify the operation steps for consumers and reduce the risk of physical damage and surface contamination to the television set.

[0206] With the above Figures 1 to 4 Corresponding to the camera calibration method provided in the embodiments, this disclosure also provides a camera calibration device. Since the camera calibration device provided in the embodiments of this disclosure is similar to the one described above... Figures 1 to 4 The camera calibration method provided in the embodiments corresponds to the camera calibration device provided in the embodiments of this disclosure, and will not be described in detail in the embodiments of this disclosure.

[0207] Figure 5 This is a schematic diagram of the camera calibration device provided in Embodiment 5 of this disclosure.

[0208] like Figure 5 As shown, the camera calibration device 500 may include: an acquisition module 501, an extraction module 502, a first determination module 503, and a second determination module 504.

[0209] The acquisition module 501 is used to acquire the target image captured by the camera to be calibrated; the target image displays multiple marker patterns.

[0210] Extraction module 502 is used to extract edge curves belonging to the same marker pattern from the target image.

[0211] The first determining module 503 is used to determine the target coordinates and normal vectors of the center points of the multiple marking patterns in the camera coordinate system based on the image positions of the edge curves of the multiple marking patterns in the target image.

[0212] The second determining module 504 is used to determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marking patterns.

[0213] In one possible implementation of this disclosure, the extraction module 502 is configured to: perform edge detection on the target image to obtain an edge curve set, wherein the edge curve set includes at least one edge curve; determine the length of any edge curve in the edge curve set; if the length is greater than a first length threshold, segment the edge curve to obtain multiple sub-edge curves, delete the edge curve from the edge curve set, and add the multiple sub-edge curves to the edge curve set; cluster the curves in the updated edge curve set to obtain multiple clusters; and determine the target cluster to which each marker pattern belongs from the multiple clusters based on the size of the multiple marker patterns and the distance between the multiple marker patterns.

[0214] In one possible implementation of this disclosure, the camera calibration device 500 may further include:

[0215] The deletion module is used to delete any edge curve from the set of edge curves if its length is less than a second length threshold; wherein the second length threshold is less than a first length threshold.

[0216] In one possible implementation of this disclosure, the first determining module 503 is configured to: extract multiple target edge points from the target cluster to which any of the multiple marking patterns belongs for any one of the multiple marking patterns; and determine the target coordinates and normal vector of the center point of any one of the marking patterns in the camera coordinate system based on the image positions of the multiple target edge points in the target image.

[0217] In one possible implementation of this disclosure, the first determining module 503 is configured to: extract at least one candidate edge point from any curve in any target cluster to which any marking pattern belongs; determine the distance between each candidate edge point based on the image position of the candidate edge points on each curve in the target cluster in the target image; and determine the target edge point from each candidate edge point based on the distance between each candidate edge point.

[0218] In one possible implementation of this disclosure, the first determining module 503 is configured to: determine a target coefficient matrix based on the image positions of multiple target edge points in a target image, wherein the target coefficient matrix is ​​used to indicate the shape of any marking pattern; decompose the target coefficient matrix to obtain a diagonal matrix and an intermediate matrix; generate an intermediate vector based on the values ​​of each diagonal element in the diagonal matrix; generate intermediate coefficients based on the values ​​of each diagonal element in the diagonal matrix and the size of any marking pattern; determine the target coordinates of the center point of any marking pattern in the camera coordinate system based on the intermediate coefficients, intermediate vectors, and intermediate matrix; and determine the normal vector of the center point of any marking pattern in the camera coordinate system based on the intermediate vectors and intermediate matrix.

[0219] In one possible implementation of this disclosure, the second determining module 504 is configured to: determine a first unit vector of the horizontal axis of the world coordinate system based on the difference between the target coordinates of the center points of each marking pattern; determine a second unit vector of the vertical axis of the world coordinate system based on the normal vector; determine a third unit vector of the vertical axis of the world coordinate system based on the first unit vector and the second unit vector; and determine the extrinsic parameters of the camera to be calibrated based on the first unit vector, the second unit vector, and the third unit vector.

[0220] In one possible implementation of this disclosure, the camera calibration device 500 may further include:

[0221] The processing module is used to preprocess the target image; wherein the preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing and erosion processing.

[0222] The camera calibration apparatus of this disclosure acquires a target image captured by a camera to be calibrated, wherein the target image displays multiple marker patterns; edge curves belonging to the same marker pattern are extracted from the target image; based on the image positions of the edge curves of the multiple marker patterns in the target image, the target coordinates and normal vectors of the center points of the multiple marker patterns in the camera coordinate system are determined; and based on the target coordinates and normal vectors of the center points of the multiple marker patterns, the extrinsic parameters of the camera to be calibrated are determined. Therefore, the extrinsic parameter calibration of the camera to be calibrated can be completed based on a single image captured by the camera, simplifying user operation and improving the user experience.

[0223] To implement the above embodiments, this disclosure also proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the camera calibration method proposed in any of the foregoing embodiments of this disclosure.

[0224] To implement the above embodiments, this disclosure also proposes a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the camera calibration method as proposed in any of the foregoing embodiments of this disclosure.

[0225] To implement the above embodiments, this disclosure also proposes a computer program product that, when the instructions in the computer program product are executed by a processor, performs a camera calibration method as proposed in any of the foregoing embodiments of this disclosure.

[0226] Figure 6 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. Figure 6 The electronic device 12 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0227] like Figure 6 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0228] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0229] Electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0230] Memory 28 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 6 Not shown; usually referred to as a "hard drive"). Although Figure 6Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disc drive for reading and writing to a removable non-volatile optical disc (e.g., a compact disc read-only memory (CD-ROM), a digital video disc read-only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. Memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0231] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0232] Electronic device 12 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 12, and / or with any device that enables electronic device 12 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 12 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 6 As shown, network adapter 20 communicates with other modules of electronic device 12 via bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0233] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the methods mentioned in the foregoing embodiments.

[0234] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0235] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0236] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0237] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0238] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0239] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0240] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0241] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. A camera calibration method, characterized in that, The method includes: Acquire a target image captured by the camera to be calibrated; wherein the target image displays multiple marker patterns; The edge curves belonging to the same marking pattern are extracted from the target image, wherein each edge curve of the target image forms multiple clusters, each marking image has a target cluster to which it belongs, and the target cluster includes edge curves belonging to the same marking pattern; For each marker pattern, the target coordinates and normal vector of the center point of the marker pattern in the camera coordinate system are determined based on the image positions of multiple target edge points in the target cluster to which it belongs in the target image. The extrinsic parameters of the camera to be calibrated are determined based on the target coordinates and normal vectors of the center points of multiple marked patterns.

2. The method according to claim 1, characterized in that, Extracting edge curves belonging to the same marker pattern from the target image includes: Edge detection is performed on the target image to obtain a set of edge curves, wherein the set of edge curves includes at least one edge curve; For any edge curve in the set of edge curves, determine the length of that edge curve; If the length is greater than a first length threshold, any edge curve is segmented to obtain multiple sub-edge curves, and any edge curve is deleted from the edge curve set, and the multiple sub-edge curves are added to the edge curve set. The curves in the updated set of edge curves are clustered to obtain multiple clusters; Based on the size of the plurality of marker patterns and the distance between the plurality of marker patterns, the target cluster to which each marker pattern belongs is determined from the plurality of clusters.

3. The method according to claim 2, characterized in that, After determining the length of any edge curve, the method further includes: If the length is less than the second length threshold, then any of the edge curves is removed from the set of edge curves; Wherein, the second length threshold is less than the first length threshold.

4. The method according to claim 2, characterized in that, For each marker pattern, based on the image positions of multiple target edge points in the target image within its target cluster, the target coordinates and normal vector of the center point of the marker pattern in the camera coordinate system are determined, including: For any one of the multiple marking patterns, extract multiple target edge points from the target cluster to which the any one marking pattern belongs; Based on the image positions of the multiple target edge points in the target image, determine the target coordinates and normal vector of the center point of any marker pattern in the camera coordinate system.

5. The method according to claim 4, characterized in that, The step of extracting multiple target edge points from the target cluster to which any of the marked patterns belongs includes: For any curve in the target cluster to which any of the marked patterns belong, extract at least one candidate edge point from the curve. The distance between each candidate edge point is determined based on the image position of the candidate edge points on each curve in the target cluster in the target image; The target edge point is determined from the candidate edge points based on the distance between them.

6. The method according to claim 4, characterized in that, The step of determining the target coordinates and normal vector of the center point of any marker pattern in the camera coordinate system based on the image positions of the plurality of target edge points in the target image includes: Based on the image positions of the plurality of target edge points in the target image, a target coefficient matrix is ​​determined, wherein the target coefficient matrix is ​​used to indicate the shape of any of the marker patterns; The target coefficient matrix is ​​decomposed to obtain a diagonal matrix and an intermediate matrix; Based on the values ​​of each diagonal element in the diagonal matrix, an intermediate vector is generated; Based on the values ​​of each diagonal element in the diagonal matrix and the size of any marked pattern, intermediate coefficients are generated; Based on the intermediate coefficients, intermediate vectors, and intermediate matrices, determine the target coordinates of the center point of any marker pattern in the camera coordinate system; Based on the intermediate vector and intermediate matrix, determine the normal vector of the center point of any marked pattern in the camera coordinate system.

7. The method according to any one of claims 1-5, characterized in that, The step of determining the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marked patterns includes: Based on the difference between the target coordinates of the center points of each of the marked patterns, determine the first unit vector of the horizontal axis of the world coordinate system; Based on the normal vector, determine the second unit vector of the vertical axis of the world coordinate system; Based on the first unit vector and the second unit vector, determine the third unit vector of the vertical axis of the world coordinate system; The extrinsic parameters of the camera to be calibrated are determined based on the first unit vector, the second unit vector, and the third unit vector.

8. The method according to any one of claims 1-5, characterized in that, After acquiring the target image captured by the camera to be calibrated, the method further includes: The target image is preprocessed; The preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing, and erosion processing.

9. A camera calibration device, characterized in that, The device includes: The acquisition module is used to acquire a target image captured by the camera to be calibrated; wherein, the target image displays multiple marker patterns; An extraction module is used to extract edge curves belonging to the same marker pattern from the target image, wherein each edge curve of the target image forms multiple clusters, each marker image has a target cluster to which it belongs, and the target cluster includes edge curves belonging to the same marker pattern; The first determining module is used to determine, for each marking pattern, the target coordinates and normal vector of the center point of the marking pattern in the camera coordinate system based on the image position of multiple target edge points in the target cluster to which it belongs in the target image. The second determining module is used to determine the extrinsic parameters of the camera to be calibrated based on the target coordinates and normal vectors of the center points of multiple marking patterns.

10. The apparatus according to claim 9, characterized in that, The extraction module is used for: Edge detection is performed on the target image to obtain a set of edge curves, wherein the set of edge curves includes at least one edge curve; For any edge curve in the set of edge curves, determine the length of that edge curve; If the length is greater than a first length threshold, any edge curve is segmented to obtain multiple sub-edge curves, and any edge curve is deleted from the edge curve set, and the multiple sub-edge curves are added to the edge curve set. The curves in the updated set of edge curves are clustered to obtain multiple clusters; Based on the size of the plurality of marker patterns and the distance between the plurality of marker patterns, the target cluster to which each marker pattern belongs is determined from the plurality of clusters.

11. The apparatus according to claim 10, characterized in that, The device further includes: A deletion module is used to delete any edge curve from the set of edge curves if the length is less than a second length threshold. Wherein, the second length threshold is less than the first length threshold.

12. The apparatus according to claim 10, characterized in that, The first determining module is used for: For any one of the multiple marking patterns, extract multiple target edge points from the target cluster to which the any one marking pattern belongs; Based on the image positions of the multiple target edge points in the target image, determine the target coordinates and normal vector of the center point of any marker pattern in the camera coordinate system.

13. The apparatus according to claim 12, characterized in that, The first determining module is used for: For any curve in the target cluster to which any of the marked patterns belong, extract at least one candidate edge point from the curve. The distance between each candidate edge point is determined based on the image position of the candidate edge points on each curve in the target cluster in the target image; The target edge point is determined from the candidate edge points based on the distance between them.

14. The apparatus according to claim 12, characterized in that, The first determining module is used for: Based on the image positions of the plurality of target edge points in the target image, a target coefficient matrix is ​​determined, wherein the target coefficient matrix is ​​used to indicate the shape of any of the marker patterns; The target coefficient matrix is ​​decomposed to obtain a diagonal matrix and an intermediate matrix; Based on the values ​​of each diagonal element in the diagonal matrix, an intermediate vector is generated; Based on the values ​​of each diagonal element in the diagonal matrix and the size of any marked pattern, intermediate coefficients are generated; Based on the intermediate coefficients, intermediate vectors, and intermediate matrices, determine the target coordinates of the center point of any marker pattern in the camera coordinate system; Based on the intermediate vector and intermediate matrix, determine the normal vector of the center point of any marked pattern in the camera coordinate system.

15. The apparatus according to any one of claims 9-13, characterized in that, The second determining module is used for: Based on the difference between the target coordinates of the center points of each of the marked patterns, determine the first unit vector of the horizontal axis of the world coordinate system; Based on the normal vector, determine the second unit vector of the vertical axis of the world coordinate system; Based on the first unit vector and the second unit vector, determine the third unit vector of the vertical axis of the world coordinate system; The extrinsic parameters of the camera to be calibrated are determined based on the first unit vector, the second unit vector, and the third unit vector.

16. The apparatus according to any one of claims 9-13, characterized in that, The device further includes: The processing module is used to preprocess the target image; The preprocessing includes at least one of color space transformation processing, noise reduction and smoothing processing, binarization processing, and erosion processing.

17. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images