Calibration plate, calibration method, device, equipment and storage medium

Abstract

The application provides a calibration board, a calibration method, a device, equipment and a storage medium. The calibration board comprises a substrate, at least one planar identification pattern and at least one three-dimensional identification block. The planar identification pattern and the three-dimensional identification block are arranged on the substrate. The arrangement area of the planar identification pattern and the arrangement area of the three-dimensional identification block are adjacent. In the application, a depth camera can simultaneously detect the planar identification pattern and the three-dimensional identification block on the calibration board, and the pose information in a two-dimensional space and the pose information in a three-dimensional space are respectively calculated based on the planar identification pattern and the three-dimensional identification block. The calculation error of the pose information in the two-dimensional space is quantitatively evaluated by using the information of the three-dimensional identification block, and the calculation error of the pose information in the three-dimensional space is quantitatively evaluated by using the information of the planar identification pattern. The final external parameter calibration result is determined based on the two calculated errors. The two calculated pose information is mutually compensated and checked, so that the final external parameter calibration result is more stable and accurate.

Description

Technical Field

[0001] This application belongs to the field of machine vision technology, specifically relating to a calibration board, calibration method, device, equipment, and storage medium. Background Technology

[0002] Calibration boards have wide applications in machine vision or image measurement. By taking images of the calibration board with a depth camera and calculating them with a calibration algorithm, the pose information between the depth camera and the calibration board can be obtained, thus achieving the external parameter calibration of the depth camera.

[0003] A extrinsic parameter calibration scheme for depth cameras is proposed in related technologies. This scheme is based on a calibration board containing a specific planar pattern. The depth camera detects the corner points of the specific planar pattern. Based on the known geometric relationship between the size of the specific planar pattern and the detected corner points, the pose information between the depth camera and the calibration board is calculated using the PNP (Perspective-n-Point) algorithm.

[0004] In the aforementioned related technologies, corner points are generally detected in the grayscale or amplitude image of the calibration board. Errors in corner point detection and distortions in the grayscale or amplitude image can lead to significant errors in the final calibration results. Summary of the Invention

[0005] This application proposes a calibration plate, calibration method, apparatus, device, and storage medium. The depth camera can simultaneously detect planar marker patterns and three-dimensional marker blocks from the calibration plate. Based on the planar marker patterns and three-dimensional marker blocks, pose information in two-dimensional space and pose information in three-dimensional space are calculated respectively. The calculated pose information is used to compensate and verify each other, making the final external parameter calibration result more stable and accurate.

[0006] A first aspect of this application provides a calibration board, comprising: a substrate, at least one planar marking pattern, and at least one three-dimensional marking block;

[0007] Both the planar logo pattern and the three-dimensional logo block are disposed on the substrate;

[0008] The area where the planar logo pattern is arranged is adjacent to the area where the three-dimensional logo block is arranged.

[0009] A second aspect of this application provides a calibration method based on the calibration board described in the first aspect, comprising:

[0010] The calibration board is photographed using a depth camera to obtain a depth image and a planar image of the calibration board;

[0011] Based on the image of the stereo marker block in the depth image, calculate the first pose information between the depth camera and the calibration plate;

[0012] Based on the image of the planar marker pattern in the planar image, calculate the second pose information between the depth camera and the calibration plate;

[0013] Based on the first pose information and the second pose information, the extrinsic parameter calibration result of the depth camera is determined.

[0014] A third aspect of this application provides a calibration apparatus based on the calibration plate described in the first aspect above, the apparatus comprising:

[0015] The imaging module is used to capture images of the calibration board using a depth camera, thereby obtaining depth images and planar images of the calibration board.

[0016] The calculation module is used to calculate the first pose information between the depth camera and the calibration plate based on the image of the stereo marker block in the depth image; and to calculate the second pose information between the depth camera and the calibration plate based on the image of the planar marker pattern in the planar image.

[0017] The determination module is used to determine the extrinsic parameter calibration result of the depth camera based on the first pose information and the second pose information.

[0018] The fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the second aspect of this application.

[0019] A fifth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method described in the second aspect of this application.

[0020] The technical solutions provided in this application embodiment have at least the following technical effects or advantages:

[0021] In this embodiment, the calibration plate has both planar marker patterns and 3D marker blocks. The depth camera can simultaneously detect both the planar marker patterns and 3D marker blocks on the calibration plate, and calculate pose information in two-dimensional space and three-dimensional space based on the planar marker patterns and 3D marker blocks, respectively. The calculation error of the pose information in two-dimensional space is quantitatively evaluated using the information from the 3D marker blocks, and the calculation error of the pose information in three-dimensional space is quantitatively evaluated using the information from the planar marker patterns. The final extrinsic parameter calibration result is determined based on the two evaluated calculation errors. This achieves mutual compensation and verification using the two calculated pose information, making the final extrinsic parameter calibration result more stable and accurate.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0024] In the attached diagram:

[0025] Figure 1 A schematic diagram of a calibration plate provided in one embodiment of this application is shown;

[0026] Figure 2 A schematic flowchart of a calibration method provided in an embodiment of this application is shown;

[0027] Figure 3 It shows Figure 1 A schematic diagram of the grayscale or amplitude map of the calibration plate shown;

[0028] Figure 4 It shows Figure 1 A schematic diagram of the depth image of the calibration board after grayscale conversion;

[0029] Figure 5 It shows the basis Figure 1 The diagram shows the process of using the calibration board to calibrate the extrinsic parameters of the depth camera.

[0030] Figure 6 This paper shows a schematic diagram of the structure of a calibration device provided in one embodiment of the present application;

[0031] Figure 7 This illustration shows a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0032] Figure 8 A schematic diagram of a storage medium provided in one embodiment of this application is shown.

[0033] The above appendix Figure 1 The meanings of the symbols in the text are as follows:

[0034] 1: Base plate, 2: Flat logo pattern, 3: Three-dimensional logo block, 4: Logo frame. Detailed Implementation

[0035] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0036] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0037] The calibration board, calibration method, apparatus, device, and storage medium according to embodiments of this application are described below with reference to the accompanying drawings.

[0038] Currently, two common approaches are used for extrinsic parameter calibration of depth cameras. The first approach uses a calibration board with a specific planar pattern. The depth camera detects the corner points of the planar pattern, and the pose information between the depth camera and the calibration board is calculated using the PNP algorithm based on the known geometric relationship between the dimensions of the planar pattern and the detected corner points. The second approach uses a 3D calibration board with a specific stereo marker pattern. The depth camera detects the 3D coordinates of the stereo marker pattern on the 3D calibration board, and the pose information between the depth camera and the calibration board is calculated using ICP (Iterative Closest Point) based on the known 3D coordinates of the stereo marker pattern and the detected 3D coordinates.

[0039] In the first approach described above, since corner points are detected using the grayscale or amplitude image of the calibration board, errors in corner point detection and distortions in the grayscale or amplitude image can lead to significant errors in the final calibration result. In the second approach, the calibration accuracy depends on the precision of the depth camera's measurement of the specific stereoscopic marker pattern. Measurement errors by the depth camera and error points generated near the specific stereoscopic marker pattern can both result in significant errors in the final calibration result.

[0040] Based on this, embodiments of this application propose a novel calibration board, such as... Figure 1 As shown, the calibration plate includes a substrate 1, at least one planar marking pattern 2, and at least one three-dimensional marking block 3.

[0041] The planar logo pattern 2 and the three-dimensional logo block 3 are both disposed on the substrate 1, and the arrangement area of ​​the planar logo pattern 2 and the arrangement area of ​​the three-dimensional logo block 3 are adjacent.

[0042] The calibration board features planar marker patterns 2 and stereo marker blocks 3, enabling simultaneous detection of both planar and stereo calibration marks during depth camera calibration. This results in two extrinsic parameter calibration results, facilitating mutual compensation and verification, leading to more stable and accurate final calibration. Furthermore, the adjacent arrangement of planar marker patterns 2 and stereo marker blocks 3 ensures detection within similar image regions, mitigating detection errors introduced by markers in significantly different image regions and further improving the accuracy of camera calibration based on this calibration board.

[0043] The shape of the aforementioned planar marking pattern 2 includes at least one or more of the following: shapes with corners or shapes without corners and axially symmetrical. Shapes with corners may include triangles, rectangles, squares, rhombuses, trapezoids, pentagons, hexagons, stars, etc. Shapes without corners and axially symmetrical may include circles, ellipses, or any shape whose center point can be identified through image processing.

[0044] It should be noted that when multiple planar marking patterns 2 are set in the calibration plate, the shapes of these multiple planar marking patterns 2 can be the same or different. For example, suppose there are 4 planar marking patterns 2 in the calibration plate, and all 4 planar marking patterns 2 are rectangles. Or, suppose there are 2 of these 4 planar marking patterns 2 that are rectangles and the other 2 that are triangles. Figure 1 Eight planar logo patterns 2 are schematically drawn in the diagram, and all eight planar logo patterns 2 are square in shape.

[0045] The flat marking pattern 2 has a variety of possible shapes, which can enrich the product diversity of the calibration board and make the calibration board applicable to a wider range of application scenarios.

[0046] Since the algorithm used for extrinsic parameter calibration based on planar marking patterns 2 is typically the PNP algorithm, and the PNP algorithm requires at least three coordinate points to perform the relevant calculations for extrinsic parameter calibration, in some embodiments of this application, the total number of corner points and / or center points of all planar marking patterns 2 on the substrate 1 is greater than or equal to 3. In some embodiments, if there is only one planar marking pattern 2 on the calibration board, then the planar marking pattern 2 must have at least 3 corner points, and the planar marking pattern 2 can be a triangle, quadrilateral, pentagon, or other shape with more sides. In other embodiments, if all planar marking patterns on the calibration board are cornerless and axially symmetric, then there should be at least three cornerless and axially symmetric planar marking patterns, so that extrinsic parameter calibration can be achieved by detecting three center points. In still other embodiments, there can be multiple planar marking patterns 2 on the calibration board, and the sum of the corner points and / or center points of these planar marking patterns 2 is greater than or equal to 3.

[0047] By limiting the sum of the corner points and / or center points of all planar marking patterns 2 on the calibration board to be greater than or equal to 3, the planar marking patterns 2 on the calibration board meet the characteristics of the PNP algorithm required for extrinsic parameter calibration, ensuring that the corresponding algorithm can be used for extrinsic parameter calibration based on the calibration board.

[0048] In application scenarios where the PNP algorithm is not used for extrinsic parameter calibration, other algorithms may not limit the number of corner points or center points. In this case, the planar marking pattern 2 in the calibration board can be one or more, and the shape of the planar marking pattern 2 can be any shape that has corner points or a shape without corner points and is axially symmetric.

[0049] In some embodiments of this application, the three-dimensional marker block 3 includes a marker block with a cutout on the substrate 1, and the shape of the cutout marker block is centrally symmetrical. The centrally symmetrical shape may include a circle, a square, a rhombus, etc.

[0050] The 3D marker block 3 is designed with a cutout shape, allowing its detection based on the depth image of the calibration plate. During camera calibration, a depth image of the calibration plate is captured using a depth camera, and the 3D marker block 3 is detected from this image for extrinsic parameter calibration. When the 3D marker block 3 is a cutout, only the 3D coordinates of a ring around its edge can be detected from the depth image. The center point coordinates of the cutout 3D marker block 3 are obtained by calculating the average value of these rings. This center point coordinate is used to identify the position of the 3D marker block 3. For this calculation method, if the cutout 3D marker block 3 has a centrally symmetrical shape, the calculated center point coordinates are more accurate, thus improving the precision of the extrinsic parameter calibration.

[0051] In other embodiments, the three-dimensional marking block 3 may not be hollowed out, but rather protrude from the surface of the substrate 1 and / or be recessed into the surface of the substrate 1. Here, the surface of the substrate 1 refers to the side of the substrate 1 on which the planar marking pattern 2 is provided.

[0052] Since the stereo marker block 3 needs to be detected from the depth image of the calibration board during the camera calibration process, the stereo marker block 3 can be set on the calibration board by raising or lowering it. This allows the depth image to contain the point cloud of the stereo marker block 3, which helps to directly use the point cloud of the stereo marker block 3 to calculate the pose information between the depth camera and the calibration board. There is no need to further calculate the position or size of the stereo marker block 3 based on the point cloud, which can save computation and improve the efficiency of camera calibration.

[0053] Since the depth camera uses lasers to photograph the calibration board, it also captures a planar image of the calibration board simultaneously with the depth image. This planar image is either an amplitude map or a grayscale image. Different colors have different reflectivities to the laser, resulting in different amplitude values ​​for pixels corresponding to different colors in the amplitude map. Therefore, the colors of the substrate 1, the planar marking pattern 2, and / or the three-dimensional marking block 3 all affect the camera calibration accuracy. In some embodiments of this application, the background color of the substrate 1 has a reflectivity greater than a preset reflectivity at the operating frequency of the depth camera. The background color is the color of the side of the substrate 1 where the planar marking pattern 2 and the three-dimensional marking block 3 are located. For example, the background color of the substrate 1 can be gray, and the grayscale value can be 177.

[0054] The operating frequency of the depth camera is the same as its laser emission frequency. The preset reflectivity can be 80% or 90%, etc. This embodiment does not limit the specific value of the preset reflectivity; it can be set according to requirements in practical applications.

[0055] The background color of substrate 1 has a reflectivity greater than the preset reflectivity at the operating frequency of the depth camera. This means that substrate 1 absorbs only a small portion of the laser emitted by the depth camera, while most of it is reflected back to the depth camera. This results in a more accurate amplitude map for the depth camera. Camera calibration based on a more accurate amplitude map helps improve the accuracy of camera calibration.

[0056] In this embodiment of the application, the grayscale difference between the color of the planar marking pattern 2 and the background color after grayscale conversion is greater than a first preset threshold, and the difference between the reflectance of the color of the planar marking pattern 2 and the background color to the working frequency of the depth camera is greater than a second preset threshold.

[0057] The first preset threshold can be 127, 130, 150, etc. The second preset threshold can be 50%, 55%, or 60%, etc. This application embodiment does not limit the specific values ​​of the first and second preset thresholds; they can be set according to requirements in practical applications.

[0058] After converting the color of the planar marking pattern 2 and the background color of the substrate 1 to grayscale, the grayscale difference between the two is calculated, and the grayscale difference is limited to be greater than a first preset threshold. This ensures that the grayscale values ​​of the planar marking pattern 2 and the background of the substrate 1 in the grayscale image obtained by the depth camera from the calibration board are significantly different, making it easier to identify the planar marking pattern 2 from the grayscale image, improving the accuracy of identifying the planar marking pattern 2, and thus improving the accuracy of camera calibration.

[0059] The difference between the reflectance of the color of the planar marking pattern 2 and the background color at the working frequency of the depth camera is greater than the second preset threshold. This ensures that the amplitude of the pixels of the planar marking pattern 2 in the amplitude map obtained by the depth camera from the calibration board is significantly different from the amplitude of the background pixels of the substrate 1. This makes it easier to identify the planar marking pattern 2 from the amplitude map, improves the accuracy of identifying the planar marking pattern 2, and thus improves the accuracy of camera calibration.

[0060] In this embodiment, planar marking patterns 2 and three-dimensional marking blocks 3 are evenly arranged on the substrate 1 according to a certain cross-arrangement rule. The cross-arrangement rule can be that two rows of planar marking patterns 2 are sandwiched between two rows of three-dimensional marking blocks 3, such as... Figure 1 As shown. Alternatively, the cross arrangement rule is that one row of planar sign patterns 2 and one row of three-dimensional sign blocks 3 are arranged alternately. Alternatively, the cross arrangement rule is that one column of planar sign patterns 2 and one column of three-dimensional sign patterns are arranged alternately. Alternatively, the cross arrangement rule is that each row consists of one planar sign pattern 2 and one three-dimensional sign block 3 arranged alternately. The embodiments of this application do not limit the specific rules of the cross arrangement rule, and in practical applications, the planar sign patterns 2 and three-dimensional sign blocks 3 can be arranged arbitrarily.

[0061] The distance between adjacent planar marker patterns 2 and / or three-dimensional marker blocks 3 on the calibration plate is equal, so that they are evenly distributed on the substrate 1. This ensures that the planar marker patterns 2 and three-dimensional marker blocks 3 can be detected in the same image area, avoiding detection errors introduced by the planar marker patterns 2 and three-dimensional marker blocks 3 in different image areas with large differences, and further improving the accuracy of camera calibration based on the calibration plate.

[0062] The calibration plate also includes a marker frame 4 disposed on the base plate 1, with the planar marker pattern 2 and the three-dimensional marker block 3 both disposed within the area enclosed by the marker frame 4. After the depth camera captures depth and planar images of the calibration frame, it can quickly locate the planar marker pattern 2 and the three-dimensional marker block 3 within the smaller image area by recognizing the marker frame 4, thereby improving the efficiency of camera calibration.

[0063] In this embodiment, the calibration plate has both planar marker patterns and stereo marker blocks, enabling simultaneous detection of both planar and stereo calibration marks on a single calibration plate during depth camera calibration. This results in two extrinsic parameter calibration results, facilitating mutual compensation and verification, thus making the final extrinsic parameter calibration result more stable and accurate. The planar marker patterns and stereo marker blocks are arranged adjacently on the calibration plate, ensuring they are detected in similar image regions. This avoids detection errors introduced by planar marker patterns and stereo marker blocks in significantly different image regions, further improving the accuracy of camera calibration based on this calibration plate.

[0064] Based on the calibration board provided in the above embodiments, some other embodiments of this application provide a calibration method. The method involves taking pictures of the calibration board with a depth camera, calculating two pose information based on the obtained depth image and planar image, and performing mutual verification and error compensation on the two pose information to obtain a more accurate extrinsic parameter calibration result.

[0065] See Figure 2 The method specifically includes the following steps:

[0066] Step 101: Take a picture of the calibration board with a depth camera to obtain the depth image and planar image of the calibration board.

[0067] The calibration plate is placed at a preset distance from the depth camera. The depth camera then photographs the calibration plate to obtain a depth image and a planar image of the calibration plate. The preset distance can be 10cm, 20cm, or 1m, etc. The planar image can be a grayscale image or an amplitude image. If the depth camera has the function of capturing grayscale images, the planar image can be a grayscale image. If the depth camera does not have the function of capturing grayscale images, an amplitude image can be obtained by reflecting the laser emitted by the depth camera using the calibration plate.

[0068] in, Figure 3 A schematic diagram of the amplitude map or grayscale image is shown. Figure 4 This is a schematic diagram of the depth image after grayscale conversion.

[0069] Step 102: Calculate the first pose information between the depth camera and the calibration plate based on the image of the stereo marker block in the depth image.

[0070] Specifically, the first pose information is calculated through the following steps S1-S3:

[0071] S1: Determine the first target image region where the calibration board is located in the depth image.

[0072] In the depth image, the pixel value of each pixel represents the distance between the corresponding point on the calibration board and the depth image. In step S101, the depth image is captured at preset distances on the calibration board. This preset distance is pre-configured in a device used to calibrate the depth camera; this device can be a terminal or a server, etc. After obtaining the depth image captured by the depth camera, the device determines a distance interval based on the pre-configured preset distance. The lower limit of this distance interval is the difference between the preset distance and a preset step value, and the upper limit is the sum of the preset distance and the preset step value. The preset step value can be 2cm, 3cm, or 5cm, etc.

[0073] By setting an appropriate preset step value, the pixel values ​​of all pixels corresponding to the calibration board can be ensured to fall within the determined distance range. All pixels whose pixel values ​​belong to this distance range are identified from the depth map. The image region formed by all these identified pixels will necessarily contain the image of the calibration board. Therefore, the image region formed by all identified pixels can be directly defined as the first target image region where the calibration board is located.

[0074] In another implementation, the bounding boxes on the calibration board can be detected first in the planar image. If the planar image is a grayscale image, the grayscale value of each pixel in the bounding box is the same. By detecting pixels with the same grayscale value, a bounding box forming a closed loop can be detected. If the planar image is an amplitude image, the amplitude value of each pixel in the bounding box is the same. By detecting pixels with the same amplitude value, a bounding box forming a closed loop can also be detected. After the bounding box is detected, the image region within the bounding box is the image region where the calibration board is located. Mapping the position of the bounding box in the planar image to the depth image allows the bounding box to be determined in the depth image, and the image region within the bounding box is determined as the first target image region where the calibration board is located.

[0075] After determining the first image region where the calibration board is located using the above method, the first pose information between the depth camera and the calibration board is calculated based on the stereo marker block through the following steps S2 and S3.

[0076] S2: Extract the point cloud corresponding to the stereoscopic marker block from the first target image region.

[0077] The calibration plate includes one or more stereo marker blocks. If the stereo marker block is protruding from or recessed from the surface of the calibration plate, the pixel value of each pixel of the stereo marker block will not be equal to the preset distance between the calibration plate and the depth camera. Therefore, the point cloud of each stereo marker block can be obtained by extracting the pixel points whose pixel values ​​are not equal to the preset distance from the first target image area.

[0078] For the 3D marker block with a cutout on the surface of the calibration plate, the pixel values ​​of the cutout portion represent a large distance; these pixels actually represent the pixels of objects behind the calibration plate. Therefore, only the pixels around the cutout ring of the 3D marker block are captured. If the 3D marker block is a cutout circle, only the pixels on the edge of that circle are captured. The captured pixels are then used as the point cloud of the 3D marker block.

[0079] S3: Calculate the first pose information between the depth camera and the calibration board based on the point cloud corresponding to the stereo marker block.

[0080] Before using the calibration board to calibrate the extrinsic parameters of the depth camera, the actual three-dimensional information of each stereo marker block on the calibration board is pre-configured in the device. This actual three-dimensional information may include the size of each stereo marker block, the distance between each point on the stereo marker block and the depth camera, and so on.

[0081] For stereo marker blocks that protrude or are recessed on the surface of the calibration plate, a mapping relationship between the three-dimensional information in the depth image and the actual three-dimensional information is constructed based on the point cloud of the stereo marker block detected from the depth image and the pre-configured actual three-dimensional information of the stereo marker block.

[0082] For the 3D marker block with a cutout on the calibration plate, the 3D coordinates of the center point of the 3D marker block are calculated based on the coordinates of the pixels around the edge of the cutout 3D marker block obtained in step S1. Specifically, the average value of the 3D coordinates of the pixels around the edge can be calculated to obtain the 3D coordinates of the center point of the 3D marker block. The coordinates of the center point of the cutout 3D marker block can also be detected in the grayscale image of the calibration plate. Then, based on the calculated 3D coordinates of the center point and the coordinates of the center point of the 3D marker block in the pre-configured actual 3D information in the device, a mapping relationship between the 3D information in the depth image and the actual 3D information is constructed.

[0083] Based on the 3D-3D mapping relationship constructed above, the first pose information between the depth camera and the calibration plate is calculated using the ICP algorithm. This first pose information includes the rotation matrix and translation matrix in three-dimensional space.

[0084] Step 103: Calculate the second pose information between the depth camera and the calibration board based on the image of the planar marker pattern in the planar image.

[0085] Specifically, the second pose information is calculated through the following steps S4-S6:

[0086] S4: Identify the second target image region where the calibration board is located in the planar image.

[0087] The marker frame on the calibration board is identified in the planar image, and the image area within the marker frame is determined as the second target image area where the calibration board is located. The specific process of identifying the second target image area has been described in detail in step S1 and will not be repeated here.

[0088] S5: Determine the geometric feature information of the planar identification pattern from the second target image region.

[0089] Since a planar marker pattern uses only one color, all pixels within the same planar marker pattern have the same grayscale value in the grayscale image of the calibration board. Similarly, all pixels within the same planar marker pattern have the same amplitude value in the amplitude image of the calibration board. Therefore, regardless of whether the planar image is a grayscale image or an amplitude image, each planar marker pattern can be detected by detecting pixels with the same pixel value in the second target image region. If the planar marker pattern has corner points, the coordinates of each corner point can be identified. If the planar marker pattern has a circular or elliptical shape, the coordinates of its center point can be identified.

[0090] The identified corner coordinates and / or center coordinates can be directly used as the geometric feature information of the planar sign pattern. Alternatively, the dimensions of the planar sign pattern can be determined based on the identified corner coordinates and / or center coordinates, and the determined dimensions can be used as the geometric feature information of the planar sign pattern.

[0091] S6: Calculate the second pose information between the depth camera and the calibration plate based on the geometric feature information of the planar marking pattern.

[0092] Before using the calibration board to calibrate the extrinsic parameters of the depth camera, the actual three-dimensional information of each planar marker pattern on the calibration board is pre-configured in the device. This actual three-dimensional information may include the size of each planar marker pattern, its position coordinates on the calibration board, and its distance from the depth camera.

[0093] Based on the geometric feature information of the planar marker pattern obtained in step S5, a 2D-3D mapping relationship is constructed between the geometric feature information and the actual 3D information of the planar marker pattern. Based on this mapping relationship, the second pose information between the depth camera and the calibration board is calculated using the PNP algorithm. The second pose information includes rotation and translation matrices in two-dimensional space.

[0094] The embodiments of this application do not limit the execution order of the above steps 102 and 103; they can be executed synchronously or in any order. Figure 2 The example shown is based on the simultaneous execution of steps 102 and 103.

[0095] Step 104: Determine the extrinsic parameter calibration results of the depth camera based on the first pose information and the second pose information.

[0096] After calculating the first pose information based on the 3D marker block in step 102 and the second pose information based on the planar marker pattern in step 103, the first pose information and the second pose information are cross-checked and error compensated. Specifically, this is performed through the following steps S7-S9, including:

[0097] S7: Determine the first calculation error of the first pose information based on the first pose information, the planar image, and the preset actual pose.

[0098] The first pose information is the pose information of the depth camera and the calibration board in three-dimensional space, including the rotation and translation matrices in three-dimensional space. Assuming the first pose information is accurate, it should also be accurate after being converted to two-dimensional space. Since the calibration board contains both 3D marker blocks and planar marker patterns, the planar marker patterns can be used to evaluate the calculation error of the first pose information in three-dimensional space.

[0099] Specifically, the two-dimensional coordinates of the planar marker pattern are obtained from the planar image. The rotation and translation matrices included in the first pose information are multiplied by the two-dimensional coordinates of the planar marker pattern to obtain the first verification pose information. The first verification pose information is the pose information transformed from the first pose information in three-dimensional space to two-dimensional space.

[0100] A preset actual pose is pre-configured in the device. This preset actual pose is the actual pose information of the depth camera and the calibration board in two-dimensional space. The difference between the first verification pose information and the preset actual pose is calculated. Specifically, the difference between the rotation matrix in the first verification pose information and the rotation matrix in the preset actual pose is calculated, as well as the difference between the translation matrix in the first verification pose information and the translation matrix in the preset actual pose is calculated, to obtain the first calculation error corresponding to the first pose information.

[0101] S8: Determine the second calculation error of the second pose information based on the second pose information, the depth image and the preset actual pose.

[0102] The second pose information is the pose information of the depth camera and the calibration board in two-dimensional space, including the rotation and translation matrices in two-dimensional space. Assuming the second pose information is accurate, it should also be accurate after being converted to three-dimensional space. Since the calibration board contains both 3D marker blocks and planar marker patterns, the 3D marker blocks can be used to evaluate the calculation error of the first pose information in two-dimensional space.

[0103] Specifically, the point cloud of the stereo marker block is obtained from the depth image. The rotation and translation matrices included in the second pose information are multiplied by the point cloud to obtain the second verification pose information. The second verification pose information is the pose information transformed from the second pose information in two-dimensional space to the pose information in three-dimensional space.

[0104] Another preset actual pose is pre-configured in the device. This preset actual pose is the actual pose information of the depth camera and the calibration board in three-dimensional space. The difference between the second verification pose information and the preset actual pose is calculated. Specifically, the difference between the rotation matrix in the second verification pose information and the rotation matrix in the preset actual pose is calculated, as well as the difference between the translation matrix in the second verification pose information and the translation matrix in the preset actual pose is calculated, to obtain the second calculation error corresponding to the second pose information.

[0105] S9: Determine the extrinsic parameter calibration result of the depth camera based on the first calculation error, the second calculation error, the first pose information, and the second pose information.

[0106] After obtaining the first calculation error corresponding to the first pose information and the second calculation error corresponding to the second pose information through the above steps, the final external parameter calibration result is determined based on the first calculation error and the second calculation error.

[0107] In this embodiment, the device is pre-configured with a preset error threshold, which is the maximum allowable error for the calculated pose information. The first calculation error and the second calculation error are compared with the preset error threshold. If both the first and second calculation errors are less than the preset error threshold, then the weight values ​​for the first pose information and the second pose information are determined based on the first and second calculation errors, respectively. The magnitude of the calculation error is inversely proportional to the magnitude of the weight value; the larger the calculation error, the smaller the weight value. For example, if the first calculation error is 5% and the second calculation error is 10%, then the weight value for the first pose information can be 0.7, and the weight value for the second pose information can be 0.3.

[0108] After determining the weight values, the first pose information and the second pose information are weighted and averaged according to their respective weight values ​​to obtain the extrinsic parameter calibration results of the depth camera.

[0109] When both the first and second calculation errors are less than the preset error threshold, a weighted average is performed on the first pose information and the second pose information to make the final external parameter calibration result more accurate.

[0110] In other embodiments, if either the first calculation error or the second calculation error is greater than or equal to a preset error threshold, the pose information with a calculation error greater than or equal to the preset error threshold is discarded, and the pose information with a calculation error less than the preset error threshold in the first pose information and the second pose information is determined as the extrinsic parameter calibration result of the depth camera. If both the first calculation error and the second calculation error are greater than or equal to the preset error threshold, the depth camera is recalibrated.

[0111] To facilitate understanding of the calibration method provided in the embodiments of this application, Figure 1 The calibration process will be explained using the calibration board shown as an example, along with the accompanying diagram. Figure 5 As shown, the depth camera first acquires a depth image and a grayscale or amplitude image of the calibration board. The image region containing the calibration board is identified in both the depth image and the grayscale or amplitude image. Corner points of the square planar marker pattern are detected in the grayscale or amplitude image, and the centers of the circular 3D marker blocks are detected in the depth image. Based on the detected corner points, the PNP algorithm is used to calculate pose information. Based on the detected center points, the ICP algorithm is used to calculate pose information. The two pose information sets are then used for mutual compensation and verification, and the final extrinsic parameter calibration result is output.

[0112] In this embodiment, the calibration plate has both planar marker patterns and 3D marker blocks. The depth camera can simultaneously detect both the planar marker patterns and 3D marker blocks on the calibration plate, and calculate pose information in two-dimensional space and three-dimensional space based on the planar marker patterns and 3D marker blocks, respectively. The calculation error of the pose information in two-dimensional space is quantitatively evaluated using the information from the 3D marker blocks, and the calculation error of the pose information in three-dimensional space is quantitatively evaluated using the information from the planar marker patterns. The final extrinsic parameter calibration result is determined based on the two evaluated calculation errors. This achieves mutual compensation and verification using the two calculated pose information, making the final extrinsic parameter calibration result more stable and accurate.

[0113] This application also provides a calibration apparatus for performing the calibration method provided in any of the above embodiments. For example... Figure 6 As shown, the device includes:

[0114] The imaging module 201 is used to capture images of the calibration board using a depth camera to obtain depth and planar images of the calibration board.

[0115] The calculation module 202 is used to calculate the first pose information between the depth camera and the calibration plate based on the image of the stereo marker block in the depth image; and to calculate the second pose information between the depth camera and the calibration plate based on the image of the planar marker pattern in the planar image.

[0116] The determination module 203 is used to determine the extrinsic parameter calibration results of the depth camera based on the first pose information and the second pose information.

[0117] The calculation module 202 is used to determine the first target image region where the calibration board is located in the depth image; extract the point cloud corresponding to the stereo marker block from the first target image region; and calculate the first pose information between the depth camera and the calibration board based on the point cloud corresponding to the stereo marker block.

[0118] The calculation module 202 is used to identify the second target image region where the calibration board is located in the planar image; determine the geometric feature information of the planar marker pattern from the second target image region; and calculate the second pose information between the depth camera and the calibration board based on the geometric feature information of the planar marker pattern.

[0119] The determination module 203 is used to determine the first calculation error of the first pose information based on the first pose information, the planar image and the preset actual pose; determine the second calculation error of the second pose information based on the second pose information, the depth image and the preset actual pose; and determine the extrinsic parameter calibration result of the depth camera based on the first calculation error, the second calculation error, the first pose information and the second pose information.

[0120] The determination module 203 is used to obtain the two-dimensional coordinates of the planar marker pattern from the planar image; multiply the rotation matrix and translation matrix included in the first pose information with the two-dimensional coordinates to obtain the first verification pose information; calculate the difference between the first verification pose information and the preset actual pose to obtain the first calculation error corresponding to the first pose information.

[0121] The determination module 203 is used to obtain the point cloud of the stereo marker block from the depth image; multiply the rotation matrix and translation matrix included in the second pose information with the point cloud to obtain the second verification pose information; calculate the difference between the second verification pose information and the preset actual pose to obtain the second calculation error corresponding to the second pose information.

[0122] The determination module 203 is used to determine the weight values ​​of the first pose information and the second pose information respectively based on the first calculation error and the second calculation error if both the first calculation error and the second calculation error are less than the preset error threshold; and to perform a weighted average of the first pose information and the second pose information based on their respective weight values ​​to obtain the extrinsic parameter calibration result of the depth camera.

[0123] The determination module 203 is used to determine the pose information in the first pose information and the second pose information whose calculation error is less than the preset error threshold as the extrinsic parameter calibration result of the depth camera if either the first calculation error or the second calculation error is greater than or equal to a preset error threshold; if both the first calculation error and the second calculation error are greater than or equal to the preset error threshold, the depth camera is recalibrated.

[0124] The calibration device and calibration method provided in the above embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application stored therein.

[0125] This application also provides an electronic device for performing the above-described calibration method. Please refer to... Figure 7 This illustrates a schematic diagram of an electronic device provided by some embodiments of this application. For example... Figure 7 As shown, the electronic device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the calibration method provided in any of the foregoing embodiments of this application.

[0126] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0127] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The calibration method disclosed in any of the foregoing embodiments of this application can be applied to the processor 300, or implemented by the processor 300.

[0128] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0129] The electronic device and the calibration method provided in this application are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0130] This application also provides a computer-readable storage medium corresponding to the calibration method provided in the foregoing embodiments. Please refer to... Figure 8 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the calibration method provided in any of the foregoing embodiments.

[0131] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0132] The computer-readable storage medium provided in the above embodiments of this application and the calibration method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0133] It should be noted that:

[0134] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0135] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0136] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0137] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A calibration method for a calibration plate, characterized in that, The method includes: The calibration board is photographed using a depth camera to obtain a depth image and a planar image of the calibration board; Based on the image of the stereo marker block in the depth image, calculate the first pose information between the depth camera and the calibration plate; Based on the image of the planar marker pattern in the planar image, calculate the second pose information between the depth camera and the calibration plate; Based on the first pose information, the planar image, and the preset actual pose, determine the first calculation error of the first pose information; Based on the second pose information, the depth image, and the preset actual pose, a second calculation error of the second pose information is determined; Based on the first calculation error, the second calculation error, the first pose information, and the second pose information, the extrinsic parameter calibration result of the depth camera is determined; The determination of the extrinsic parameter calibration result of the depth camera based on the first calculation error, the second calculation error, the first pose information, and the second pose information includes: If both the first calculation error and the second calculation error are less than a preset error threshold, then the weight values ​​of the first pose information and the second pose information are determined according to the first calculation error and the second calculation error, respectively; and the first pose information and the second pose information are weighted and averaged according to their respective weight values ​​to obtain the extrinsic parameter calibration result of the depth camera.

2. The method according to claim 1, characterized in that, Based on the image of the stereo marker block in the depth image, the first pose information between the depth camera and the calibration plate is calculated, including: The first target image region where the calibration plate is located is determined in the depth image; Extract the point cloud corresponding to the stereoscopic marker block from the first target image region; Based on the point cloud corresponding to the stereo marker block, calculate the first pose information between the depth camera and the calibration plate.

3. The method according to claim 1, characterized in that, Based on the image of the planar marker pattern in the planar image, the second pose information between the depth camera and the calibration plate is calculated, including: Identify the second target image region where the calibration plate is located in the planar image; The geometric feature information of the planar identification pattern is determined from the second target image region; The second pose information between the depth camera and the calibration plate is calculated based on the geometric feature information of the planar marking pattern.

4. The method according to claim 1, characterized in that, Based on the first pose information, the planar image, and the preset actual pose, a first calculation error of the first pose information is determined, including: Obtain the two-dimensional coordinates of the planar logo pattern from the planar image; Multiply the rotation matrix and translation matrix included in the first pose information by the two-dimensional coordinates to obtain the first verified pose information; The difference between the first verified pose information and the preset actual pose is calculated to obtain the first calculation error corresponding to the first pose information.

5. The method according to claim 1, characterized in that, Based on the second pose information, the depth image, and the preset actual pose, a second calculation error of the second pose information is determined, including: Obtain the point cloud of the stereo marker block from the depth image; Multiply the rotation matrix and translation matrix included in the second pose information with the point cloud to obtain the second verification pose information; The difference between the second verified pose information and the preset actual pose is calculated to obtain the second calculation error corresponding to the second pose information.

6. The method according to claim 1, characterized in that, Based on the first calculation error, the second calculation error, the first pose information, and the second pose information, the extrinsic parameter calibration result of the depth camera is determined, including: If either the first calculation error or the second calculation error is greater than or equal to a preset error threshold, then the pose information in the first pose information and the second pose information whose calculation error is less than the preset error threshold is determined as the extrinsic parameter calibration result of the depth camera. If both the first calculation error and the second calculation error are greater than or equal to the preset error threshold, the depth camera is recalibrated.

7. A calibration device for a calibration plate, characterized in that, The device includes: The imaging module is used to capture images of the calibration board using a depth camera, thereby obtaining depth images and planar images of the calibration board. The calculation module is used to calculate the first pose information between the depth camera and the calibration plate based on the image of the stereo marker block in the depth image; and to calculate the second pose information between the depth camera and the calibration plate based on the image of the planar marker pattern in the planar image. The determination module is configured to determine a first calculation error of the first pose information based on the first pose information, the planar image, and the preset actual pose; determine a second calculation error of the second pose information based on the second pose information, the depth image, and the preset actual pose; and determine the extrinsic parameter calibration result of the depth camera based on the first calculation error, the second calculation error, the first pose information, and the second pose information. The determination of the extrinsic parameter calibration result of the depth camera based on the first calculation error, the second calculation error, the first pose information, and the second pose information includes: If both the first calculation error and the second calculation error are less than a preset error threshold, then the weight values ​​of the first pose information and the second pose information are determined according to the first calculation error and the second calculation error, respectively; and the first pose information and the second pose information are weighted and averaged according to their respective weight values ​​to obtain the extrinsic parameter calibration result of the depth camera.

8. A calibration plate, characterized in that, A calibration method for a calibration plate according to any one of claims 1-6, wherein the calibration plate comprises: a substrate, at least one planar marking pattern, and at least one three-dimensional marking block; Both the planar logo pattern and the three-dimensional logo block are disposed on the substrate; The area where the planar logo pattern is arranged is adjacent to the area where the three-dimensional logo block is arranged.

9. The calibration plate according to claim 8, characterized in that, The shape of the planar logo pattern includes at least one or more of the following: a shape with corners or a shape without corners and axially symmetric.

10. The calibration plate according to claim 8, characterized in that, The three-dimensional marker block includes a marker block with a cutout on the substrate, and the cutout marker block has a centrally symmetrical shape.

11. The calibration plate according to claim 8 or 10, characterized in that, The three-dimensional marking block includes marking blocks that protrude from the surface of the substrate and / or marking blocks that are recessed into the surface of the substrate, wherein the surface of the substrate is the side on which the planar marking pattern is provided.

12. The calibration plate according to any one of claims 8-10, characterized in that, The background color of the substrate has a reflectivity greater than a preset reflectivity at the operating frequency of the depth camera. The background color is the color of the side of the substrate on which the planar marking pattern and the three-dimensional marking block are set.

13. The calibration plate according to claim 12, characterized in that, The difference in grayscale between the color of the planar marker pattern and the background color after grayscale conversion is greater than a first preset threshold, and the difference in reflectivity between the color of the planar marker pattern and the background color at the working frequency of the depth camera is greater than a second preset threshold.

14. The calibration plate according to any one of claims 8-10, characterized in that, The planar logo pattern and the three-dimensional logo block are evenly arranged on the substrate according to a certain cross-arrangement rule.

15. The calibration plate according to any one of claims 8-10, characterized in that, The calibration plate also includes an identification frame disposed on the substrate, wherein the planar identification pattern and the three-dimensional identification block are both disposed within the area enclosed by the identification frame.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-6.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by a processor to implement the method as described in any one of claims 1-6.

Images