Calibration method, calibration device, computer program product, equipment and storage medium

Abstract

The embodiment of the invention provides a calibration method and calibration device for three-dimensional scanning equipment, a computer program product, equipment and a storage medium. A marker and at least two calibration plates can be combined to obtain a large calibration plate, the marker comprises a plurality of mark points with known relative position relations, the calibration plates comprise a plurality of mark points with the relative position relations, and then the large calibration plate is used for calibrating the three-dimensional scanning equipment. According to the method for obtaining the large calibration plate through the combination of the marker and the calibration plate, only a small number of mark points with known positions need to be arranged when the marker is manufactured, the positions of the mark points in the calibration plate do not need to be determined in advance, and therefore the combined large calibration plate can be manufactured and used more conveniently while the calibration precision is guaranteed.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional scanning technology, and in particular to a calibration method, a calibration device, a computer program product, a device and a storage medium for a three-dimensional scanning device. Background Art

[0002] Before using the three-dimensional scanning device, a calibration plate is usually used to calibrate the internal and external parameters of the camera in the three-dimensional scanning device. When using the calibration plate to calibrate the three-dimensional scanning device, it is usually required that the calibration plate can cover the scanning angle of the three-dimensional scanning device. Therefore, for some three-dimensional scanning devices with a large scanning angle, a large-sized calibration plate is usually used to calibrate them. Considering that the large-sized calibration plate made of lightweight materials is easy to deform, the accuracy of the calibrated internal and external parameters cannot be guaranteed. Therefore, for large-sized calibration plates, marble and other materials that are not easy to deform are usually used. The weight is large and inconvenient to use, and the large-sized calibration plate is also inconvenient to transport. Of course, a small-sized calibration plate can also be used to calibrate the three-dimensional scanning device with a large scanning angle. However, when using a small-sized calibration plate, in order to ensure that the calibration plate covers the scanning angle of the three-dimensional scanning device, the three-dimensional scanning device needs to collect calibration images at more positions. The number of calibration images collected is usually about 4 times the number of calibration images collected when using a large calibration plate. Obviously, after the number of calibration images collected is greatly increased, the calculation amount of the subsequent image processing stage will be greatly increased, reducing the calibration efficiency.

[0003] Therefore, it is necessary to provide a calibration solution that is convenient for the transportation and use of the calibration plate and can reduce the number of calibration images collected during the calibration process and reduce the amount of calculation. Summary of the invention

[0004] The present disclosure provides a calibration method, a calibration device, a computer program product, a device and a storage medium for a three-dimensional scanning device.

[0005] According to a first aspect of an embodiment of the present disclosure, a calibration method for a three-dimensional scanning device is provided, the method comprising: Using at least two cameras to be calibrated in the three-dimensional scanning device to respectively acquire images of the calibration device to obtain multiple frames of calibration images; wherein the calibration device includes a marker and at least two calibration plates placed near the marker, the marker includes a plurality of first marker points whose relative position relationship is known, and each of the calibration plates includes a plurality of second marker points whose relative position relationship is unknown; Performing three-dimensional reconstruction on the calibration device based on initial intrinsic parameters of each of the at least two cameras, initial extrinsic parameters between the at least two cameras, and the multiple frames of calibration images to obtain reconstructed three-dimensional coordinates of each of the first marker point and the second marker point in the multiple frames of calibration images; Based on the reconstructed three-dimensional coordinates, the reprojection errors of the first marker point and the second marker point in the multiple frames of calibration images are determined, and the initial extrinsic parameters are optimized based on the reprojection errors and the relative position relationship between the first marker points to obtain the calibration extrinsic parameters between the at least two cameras.

[0006] According to a second aspect of an embodiment of the present disclosure, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed, the method mentioned in the first aspect is implemented.

[0007] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a processor, a memory, and computer instructions stored in the memory and executable by the processor, wherein when the processor executes the computer instructions, the method mentioned in the first aspect above can be implemented.

[0008] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored. When the computer instructions are executed, the method mentioned in the first aspect is implemented.

[0009] According to a fifth aspect of an embodiment of the present disclosure, a calibration device is provided, comprising a marker and at least two calibration plates, wherein the marker comprises a plurality of first marker points whose relative position relationships are known, and each of the calibration plates comprises a plurality of second marker points whose relative position relationships are unknown; wherein, when the calibration device is used to calibrate a device, the at least two calibration plates are placed near the marker to combine to obtain a large-sized combined calibration plate.

[0010] In the disclosed embodiment, a large calibration plate can be obtained by combining a marker and at least two calibration plates, and the large calibration plate obtained by the combination is used to calibrate the three-dimensional scanning device. Among them, some marker points with known positions can be pre-set in the marker, and some marker points can be set arbitrarily in the calibration plate, and the positions of these marker points do not need to be known. Since the positions of the marker points in the calibration plate do not need to be determined in advance, it is convenient to make, and thus a large number of feature points can be provided for stereo matching in the subsequent calibration process by using a combination of multiple calibration plates to improve the matching accuracy. A small number of marker points with known positions can be set in the marker to further calibrate the external parameters and improve the calibration accuracy. By combining the large calibration plate with this "marker + calibration plate", only a small number of marker points with known positions need to be set when making the marker, and the positions of the marker points in the calibration plate do not need to be determined in advance. Therefore, while ensuring the calibration accuracy, the production of the combined large calibration plate is made simpler and more convenient.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0013] Figure 1 It is a flow chart of a three-dimensional scanning device calibration method according to an embodiment of the present disclosure.

[0014] Figure 2 It is a schematic diagram of a calibration device according to an embodiment of the present disclosure.

[0015] Figure 3 It is a schematic diagram of a marker according to an embodiment of the present disclosure.

[0016] Figure 4 It is a schematic diagram of another calibration device according to an embodiment of the present disclosure.

[0017] Figure 5 It is a schematic diagram of collecting calibration images using a three-dimensional scanning device according to an embodiment of the present disclosure.

[0018] Figure 6 It is a schematic diagram of the logical structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0020] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0021] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0022] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0023] Some 3D scanning devices include a binocular vision system, which uses two cameras in the binocular vision system to obtain two images of the same scene from different perspectives. The depth information of the scene can be restored through the parallax of the two images and the internal and external parameters of the two cameras to achieve 3D reconstruction. Although the internal and external parameters of the two cameras are pre-calibrated when the 3D scanning device leaves the factory. However, during the use of the 3D scanning device, various factors such as temperature and external force may cause the structural parts that fix the camera to deform or shift in position, causing the external parameters of the two cameras to change. In order to obtain more accurate 3D reconstruction results, before using the 3D scanning device to perform 3D reconstruction of the target to be reconstructed, it is usually necessary to recalibrate the external parameters of the two cameras in the 3D scanning device, and use the calibrated external parameters to obtain the 3D reconstruction results.

[0024] Usually, the external parameters of the two cameras in the three-dimensional scanning device can be calibrated using a pre-set calibration plate. The calibration plate is usually composed of multiple feature points or targets (such as black and white grids, spherical or circular marks, chessboards, etc.) with known positions and sizes. The size and shape of the calibration plate must be accurate and stable. During the calibration process, the calibration plate can be scanned multiple times from different angles and positions using a three-dimensional scanner to obtain multiple frames of calibration images. Since the three-dimensional coordinates of the feature points in the calibration plate (such as the corner points of the black and white grid, the center of the circular mark, etc.) are known, the internal and external parameters of the two cameras can be determined based on the two-dimensional coordinates of these feature points in the calibration image and the three-dimensional coordinates of the feature points.

[0025] When using a calibration plate to calibrate a 3D scanning device, it is usually required that the calibration plate can cover the scanning angle of view of the 3D scanning device (i.e., the FOV of the two cameras). For example, in order to obtain accurate calibration results, the calibration plate should account for 70% to 80% of the calibration image captured by the camera in the 3D scanning device. Therefore, for some 3D scanning devices with a large scanning angle, a large-sized calibration plate is usually used for calibration. Considering that a large-sized calibration plate made of lightweight materials is easy to deform, the accuracy of the calibrated internal and external parameters cannot be guaranteed. Therefore, large-sized calibration plates are usually made of materials that are not easy to deform, such as marble, which are heavy and inconvenient to use. In addition, large-sized calibration plates are also inconvenient to transport. Of course, a small-sized calibration plate can also be used to calibrate a 3D scanning device with a large scanning angle. However, when using a small-sized calibration plate, in order to ensure that the calibration plate covers the scanning angle of the 3D scanning device, the 3D scanning device needs to be moved to more positions to capture images of the small-sized calibration plate. Usually, the number of calibration images captured is about 4 times the number of calibration images captured when using a large calibration plate. Obviously, when the number of captured calibration images is greatly increased, the amount of calculation in the subsequent image processing stage will be greatly increased, thereby reducing the calibration efficiency.

[0026] Therefore, it is necessary to provide a calibration solution that is convenient for the transportation and use of the calibration plate and can reduce the number of images collected during the calibration process and reduce the amount of calculation.

[0027] Based on this, the embodiment of the present application thinks that a large-size calibration plate can be obtained by combining multiple small-size calibration plates, which is convenient for the transportation of the calibration plate and can also reduce the number of calibration images to be collected. Further, considering that for each small-size calibration plate, if feature points need to be set in advance in each small-size calibration plate and the positions of these feature points are determined, the production of the calibration plate will be very cumbersome due to the large number of feature points. Therefore, in order to further simplify the production of the calibration plate, the present application thinks that a large calibration plate can be obtained by combining a marker and at least two calibration plates, and the large calibration plate obtained by the combination is used to calibrate the three-dimensional scanning device. Among them, some marker points with known positions can be pre-set in the marker, and some marker points can be set arbitrarily in the calibration plate, and the positions of these marker points do not need to be known. Since the positions of the marker points in the calibration plate do not need to be determined in advance, it is convenient to make, so a large-scale calibration plate can be obtained by combining multiple calibration plates, so that there are more feature points distributed in the image collected by the camera to be calibrated, providing a large number of feature points for stereo matching in the subsequent calibration process to improve the matching accuracy, and then more accurate internal and external parameters can be obtained. A small number of landmark points with known positions can be set in the markers to further calibrate the external parameters and improve the calibration accuracy.

[0028] By combining the "marker + calibration plate" to obtain a large calibration plate, only a small number of marker points with known positions need to be set when making the marker, and the positions of the marker points in the calibration plate do not need to be determined in advance. Therefore, while ensuring the calibration accuracy, the production of the combined large calibration plate is made simpler and more convenient. In addition, when providing the user with a calibration device for calibrating a three-dimensional scanning device, only the "marker" can be provided, and the calibration plate can be completely made by the user, making the transportation of the calibration plate more convenient.

[0029] Based on the above invention concept, the embodiment of the present application provides a calibration method for a three-dimensional scanning device, wherein the calibration method can be executed by the three-dimensional scanning device, for example, the three-dimensional scanning device can determine the external parameters of the two cameras based on the collected calibration images. Of course, it can also be executed by other electronic devices (such as mobile phones, tablets, computers, etc.) that are connected to the three-dimensional scanning device for communication. For example, the three-dimensional scanning device can send the collected calibration images to these devices, and these devices determine the external parameters of the two cameras based on the calibration images.

[0030] like Figure 1 As shown, the method may include the following steps: S102, using at least two cameras to be calibrated in the three-dimensional scanning device to respectively acquire images of the calibration device to obtain multiple frames of calibration images; wherein the calibration device includes a marker and at least two calibration plates placed near the marker, the marker includes a plurality of first marker points whose relative position relationship is known, and each of the calibration plates includes a plurality of second marker points whose relative position relationship is unknown; Before calibrating the 3D scanning device, the user can first use the combination of "marker + calibration plate" to obtain a calibration device for calibrating the 3D scanning device, such as Figure 2As shown, it is a schematic diagram of a calibration device in an embodiment of the present application, wherein the shape of the marker, the number of calibration plates, and the placement of the calibration plates are not limited to the method shown in the figure. The marker may include multiple pre-set marker points, hereinafter referred to as first marker points, wherein the relative position relationship between the multiple first marker points is known. The shape of the marker may be various shapes such as square, long strip, circle, etc., which are not limited in the embodiment of the present application. The number, size, and arrangement of the first marker points can also be set based on actual needs. For example, they can be evenly arranged in the marker or unevenly arranged. The sizes of multiple first marker points can be consistent or inconsistent, and can be specifically set based on actual needs. Since in the process of calibrating the external parameters of the two cameras, the marker needs to be used as a "ruler" to calibrate the external parameters, the relative position relationship between the multiple first marker points in the marker needs to be predetermined. For example, in actual use, the real three-dimensional coordinates of each first marker point can be stored. Of course, if the first marker points are arranged in an array, only the distance between the first marker points can be stored, and the real three-dimensional coordinates of each first marker point can be obtained based on the distance.

[0031] Multiple marking points can also be pre-set in the calibration plate, hereinafter referred to as second marking points. Among them, the relative position relationship between the second marking points does not need to be determined in advance, and therefore, the setting of the second marking points can be relatively arbitrary. The distribution mode of the second marking points in the calibration plate, the size and number of the second marking points can also be set based on actual needs. For example, the distribution mode can be random distribution or array distribution. The sizes of the second marking points can be consistent or inconsistent, and this application does not impose any restrictions. In addition, the distribution mode, size and number of the second marking points in different calibration plates can be the same or different.

[0032] Among them, the position of the second marker points in the calibration plate does not need to be determined in advance, and the production is relatively simple, so the number of them can be larger, so as to provide more feature points for stereo matching in the subsequent calibration process and improve the matching accuracy. The position of each first marker point in the marker needs to be determined in advance, and the production process is relatively cumbersome. Therefore, the number of first marker points can be smaller, and the external parameters are further calibrated through a small number of marker points with known positions to improve the calibration accuracy.

[0033] The calibration plates can be placed on both sides of the marker, that is, the marker should be placed at the center of the calibration device as much as possible, so that each frame of the calibration image collected includes the first marker point.

[0034] After obtaining the calibration device using the combination of "marker + calibration plate", the calibration device can be used to calibrate two or more cameras to be calibrated in the 3D scanning device to determine the external parameters between any two cameras. Among them, for scenes with 3 or more cameras to be calibrated, they can be calibrated in pairs first, and then the relative external parameters between any two cameras can be determined. Of course, the 3 or more cameras can also be calibrated at the same time, which can be flexibly set based on actual needs.

[0035] During the calibration process, at least two cameras to be calibrated can be used to collect images of the calibration device respectively to obtain multiple frames of calibration images. Among them, a three-dimensional scanning device can be used to collect images of the calibration device at different positions and angles to obtain calibration images at different positions and angles, so that these calibration images cover various areas of the calibration device as much as possible.

[0036] S104, based on the initial internal parameters of each of the at least two cameras, the initial external parameters between the at least two cameras, and the multiple frames of calibration images, the calibration device is three-dimensionally reconstructed to obtain the reconstructed three-dimensional coordinates of the first mark point and the second mark point in the multiple frames of calibration images; in step S104, since the internal parameters and external parameters of any two cameras in the three-dimensional scanning device are calibrated before leaving the factory, or they are pre-calibrated before each use. Therefore, after acquiring multiple frames of calibration images, the calibration device can be three-dimensionally reconstructed based on the initial internal parameters of each of the at least two cameras, the initial external parameters between the at least two cameras, and the multiple frames of calibration images, to obtain the reconstructed three-dimensional coordinates of the first mark point and the second mark point in the multiple frames of calibration images. Among them, the initial internal parameters and the initial external parameters can be the default internal parameters and external parameters at the time of leaving the factory, or the internal parameters and external parameters determined by the last calibration, and the embodiments of the present application are not limited.

[0037] Since at least two cameras simultaneously capture images of the calibration device, stereo matching can be performed on two frames of calibration images captured by at least two cameras at the same time to determine the depth information of the first marker point and the second marker point in the calibration image. At the same time, for calibration images captured by the same camera at different times, inter-frame matching can also be performed based on the overlapping areas in these calibration images, and then the reconstructed three-dimensional coordinates of the first marker point and the second marker point in these multiple frames of calibration images can be obtained.

[0038] S106. Determine the reprojection errors of the first marker point and the second marker point in the multiple calibration image frames based on the reconstructed three-dimensional coordinates, optimize the initial extrinsic parameters based on the reprojection errors and the relative position relationship between the first marker points, and obtain the calibration extrinsic parameters between the at least two cameras.

[0039] In step S106, after obtaining the reconstructed three-dimensional coordinates of the first marker point and the second marker point in the multiple frames of calibration images, the reprojection errors of the first marker point and the second marker point can be determined based on the reconstructed three-dimensional coordinates, and then the initial extrinsic parameters are optimized using the reprojection errors and the relative position relationship between the first marker points as constraints to obtain the calibration extrinsic parameters between at least two cameras.

[0040] For example, in some embodiments, when the initial external parameters are optimized based on the reprojection error and the relative position relationship between the first marker points to obtain the calibration external parameters between at least two cameras, the initial external parameters can be optimized to obtain the optimized external parameters by minimizing the reprojection error of the first marker point and the second marker point as the optimization purpose. For example, on the basis of the initial external parameters, the external parameters can be continuously adjusted (increasing or decreasing the initial external parameters with a certain gradient), and then the reconstructed three-dimensional coordinates can be reprojected into the calibration image based on the adjusted external parameters, and the reprojection errors of the first marker points and the second marker points can be determined, and the external parameters with the minimum reprojection error can be used as the optimized external parameters after optimization. Of course, considering the errors in stereo matching and inter-frame matching during the three-dimensional reconstruction process, the optimized external parameters obtained in this way are not accurate enough. Therefore, the optimized external parameters can be further calibrated and constrained based on the relative position relationship between any two first marker points, so that the calibration external parameters finally determined have higher accuracy. Specifically, the three-dimensional coordinates of the first marker points in the multiple calibration images can be determined based on the optimized external parameters, which are referred to as optimized three-dimensional coordinates below. Since the relative positional relationship between the first marker points is known, the real three-dimensional coordinates of each first marker point can be obtained. Then, the optimized external parameters can be optimized again based on the difference between the optimized three-dimensional coordinates and the real three-dimensional coordinates of the first marker points in the multiple calibration images to obtain the calibration external parameters between the two cameras. For example, the optimization target can be to minimize the sum of the deviations between the optimized three-dimensional coordinates and the real three-dimensional coordinates of the multiple first marker points, and the optimization parameters can be optimized to obtain the calibration parameters.

[0041] First, the initial extrinsic parameters are optimized using the reprojection errors of a large number of feature points (i.e., the first landmark point and the second landmark point), so that relatively accurate optimized extrinsic parameters can be obtained. Then, the optimized extrinsic parameters can be further calibrated based on the real three-dimensional coordinates of the first landmark point, so that highly accurate calibration extrinsic parameters can be obtained.

[0042] Of course, in some embodiments, the initial extrinsic parameters can also be calibrated based on the relative position relationship between the first marker points. For example, the real three-dimensional coordinates of the first marker points can be determined based on the above relative position relationship. After obtaining the reconstructed three-dimensional coordinates of the first marker points in multiple frames of calibration images, the initial extrinsic parameters are calibrated based on the difference between the reconstructed three-dimensional coordinates of the first marker points and the real three-dimensional coordinates to obtain optimized extrinsic parameters. Then, the optimized extrinsic parameters and the reconstructed three-dimensional coordinates of the first marker points and the second marker points are used to determine the reprojection errors of the first marker points and the second marker points in the multiple frames of calibration images. The optimized extrinsic parameters are further optimized to obtain calibration extrinsic parameters.

[0043] In some embodiments, an optimization function can also be constructed based on the reprojection errors of the first marker point and the second marker point, and the difference between the optimized three-dimensional coordinates and the true three-dimensional coordinates of the first marker point. For example, the sum of the two is used as the optimization target, and then the initial extrinsic parameters are continuously adjusted to minimize the sum of the two, and the extrinsic parameters at this time are used as calibration extrinsic parameters.

[0044] Among them, there are many specific ways to use the reprojection error and the relative position relationship between any two first landmark points as the initial external parameters for constraint optimization, which can be set based on actual needs and are not limited in the embodiments of the present application.

[0045] In some instances, each first marker point in the marker corresponds to an identifier, which can be used to uniquely identify the first marker point. For example, the identifier can be the number of the marker point, etc. The identifier of each first marker point is stored in association with the real three-dimensional coordinates of the first marker point. The real three-dimensional coordinates of each first marker point are determined based on the relative position relationship between any two first marker points. In addition, in order to distinguish the first marker points in the calibration image, the real three-dimensional coordinates corresponding to the first marker points in the calibration image can be determined. When setting the first marker points in the marker, the multiple first marker points can be distinguished by visual information.

[0046] In some embodiments, the visual information may include one or more of the following: the size of the first marker point, the shape of the first marker point, the color of the first marker point, the arrangement of the first marker point, and the marking information around the first marker point. For example, the size, shape, and color of different first marker points may be different, and this information may be stored in association with the identifier of the first marker point, so that after acquiring the calibration image, the size, shape, color, etc. of the first marker point in the calibration image may be identified to determine the identifier of the first marker point, and then determine the true three-dimensional coordinates of the first marker point. Of course, some marks may also be made around the first marker point, such as marking the number of the first marker point, so that after acquiring the calibration image, the number of the first marker point in the calibration image may be identified to determine the true three-dimensional coordinates of the first marker point.

[0047] In some embodiments, in order to make the setting of the first marker point in the marker easier, the multiple first marker points in the marker can be divided into multiple groups, and the arrangement of the first marker points in the multiple groups is different. That is, by setting the arrangement of the first marker points in each group to a different arrangement, after the calibration image is acquired, the arrangement of the current first marker point can be identified from the calibration image to determine which group it belongs to, and then the first marker points in each group can be further distinguished based on the position, size, color, etc. of the first marker points in each group.

[0048] In some embodiments, the marker may be a long strip marker, and the first marker points of the multiple groups may be equally spaced along the long side of the marker. That is, the groups are equally spaced in the marker, so that when the three-dimensional scanning device moves to different positions, the first marker points are included in the collected calibration images as much as possible.

[0049] For example, Figure 3 As shown, it is a schematic diagram of a marker in an embodiment of the present application, the marker is a long strip, and there are three groups of first marker points distributed at equal intervals in the marker, wherein each group includes a large-sized first marker point and a plurality of small-sized first marker points, and the arrangement of the large-sized first marker points and the plurality of small-sized first marker points in different groups is different. After acquiring the calibration image, the group to which the first marker point belongs can be determined based on the arrangement of the first marker points, and then the first marker points in the group can be further distinguished based on the size and relative position relationship of the first marker points to determine their numbers.

[0050] In some embodiments, the marker is a long strip marker, and the at least two calibration plates can be symmetrically placed along the long side of the marker. That is, the marker is set to a long strip, and the calibration plates are symmetrically placed along the long side of the long strip marker, so that the combined calibration device is more compact and occupies less space. And the marker is located at the center of the entire calibration device, so that the calibration image collected subsequently can include the first marker point, so as to use the relative position relationship of the first marker point to constrain and calibrate the reconstructed three-dimensional points in the calibration image.

[0051] Considering that if the number of calibration plates in the calibration device is too small, the scanning angle of the three-dimensional scanning device cannot be covered, and more calibration images need to be collected later, which increases the amount of calculation. If the number of calibration plates is too large and exceeds the actual number required, the user operations in the calibration process will be increased, making the calibration process more cumbersome. Therefore, in some embodiments, in order to make the number of calibration plates in the calibration device reasonable, the number of calibration plates can be determined based on the scanning angle of the three-dimensional scanning device. The larger the scanning angle, the more calibration plates, that is, the number of calibration plates is positively correlated with the scanning angle of the three-dimensional scanning device. Of course, in some embodiments, under the premise that the size of the calibration plate is fixed, the mapping relationship between the scanning angle of the three-dimensional scanning device and the number of calibration plates can be set in advance, or the mapping relationship between the type of the three-dimensional scanning device and the number of calibration plates can be set, so that the user can place an appropriate number of calibration plates directly based on the mapping relationship during calibration.

[0052] In some embodiments, Figure 4 As shown, considering the scanning angle of the existing three-dimensional scanning equipment, the number of the calibration plates can be 6, and 3 calibration plates can be placed on each side of the marker, and the 3 calibration plates can be symmetrically placed on both sides of the marker.

[0053] In some embodiments, in order to obtain accurate calibration external parameters based on the captured calibration images, the multiple frames of calibration images may be images captured when the 3D scanning device is located at different positions and at different angles to the calibration device. Figure 5 As shown, the three-dimensional scanning device can be aimed at the center of the marker, moved up and down to several positions along the direction perpendicular to the plane where the calibration plate is located, and the image obtained at each position is collected. Then, the three-dimensional scanning device can be aimed at the calibration plates placed on both sides of the marker, moved to several positions along the direction parallel to the plane where the calibration plate is located, and at each position, the three-dimensional scanning device is controlled to form a certain angle with the four directions of the front, back, left, and right of the plane where the calibration plate is located to collect the image. Among them, each frame of the calibration image collected includes the first marker point in the marker.

[0054] For example, Figure 4As shown in the figure, assuming that the marker is a long strip marker, and three calibration plates are placed on both sides of the marker, when collecting the calibration image, the collection process is as follows: (1) Align the 3D scanning device with the marker and calibrate the device plane vertically, move the 3D scanning device up and down to 5 different positions, and collect a calibration image at each position; (2) Aim the 3D scanning device at the three calibration plates on one side of the landmark, maintaining a 20-degree angle with the plane of the calibration plates. Collect 5 positions in each tilt direction, for a total of 20 sets of images.

[0055] (3) Aim the 3D scanning device at the three calibration plates on the other side of the marker, maintaining a 20-degree angle with the plane of the calibration plates. Collect 5 positions in each tilt direction, for a total of 20 sets of images.

[0056] In some embodiments, at least two cameras to be calibrated may be two black-and-white cameras in a three-dimensional scanning device, wherein the three-dimensional scanning device further includes a texture camera for collecting texture information of the target to be reconstructed. After obtaining the calibration extrinsic parameters between the two black-and-white cameras, since the calibration extrinsic parameters are relatively accurate, the three-dimensional coordinates of the first marker point and the second marker point in the calibration device can be further determined based on the calibration extrinsic parameters. This is equivalent to currently obtaining a large calibration plate with known coordinates of each marker point, and then the large calibration plate with known coordinates of each marker point can be used to calibrate the extrinsic parameters between the black-and-white camera and the texture camera. Specifically, for any of the two black-and-white cameras, the black-and-white camera and the texture camera can be used to respectively perform image acquisition on the calibration device to obtain multiple frames of calibration images, wherein the three-dimensional coordinates of each first marker point and the second marker point in the calibration image are known, and then the calibration extrinsic parameters of the black-and-white camera and the texture camera can be determined based on the two-dimensional coordinates of the first marker point and the second marker in the calibration image, and the three-dimensional coordinates.

[0057] In addition, an embodiment of the present application further provides a calibration device, which includes a marker and at least two calibration plates, wherein the marker includes a plurality of first marker points with known relative position relationships, and each calibration plate includes a plurality of second marker points with unknown relative position relationships; wherein, when the calibration device is used to calibrate a device, the at least two calibration plates are placed near the marker to combine to obtain a large-sized combined calibration plate. By combining a marker and a plurality of calibration plates to obtain a large calibration, the size and weight of each calibration plate can be reduced, making it easier to transport and use the calibration device.

[0058] In some embodiments, multiple first marker points in the marker can be distinguished by visual information, each first marker point corresponds to an identifier, and the identifier of each first marker point is stored in association with the real three-dimensional coordinates of the first marker point, wherein the real three-dimensional coordinates of each first marker point are determined based on the relative position relationship between the first marker points.

[0059] In some embodiments, the visual information includes one or more of the following: the size of the first marker point, the shape of the first marker point, the color of the first marker point, the arrangement of the first marker point, and marking information located around the first marker point.

[0060] In some embodiments, the plurality of first marker points in the marker are divided into a plurality of groups, and the first marker points in the plurality of groups are arranged in different ways.

[0061] In some embodiments, the marker is a long strip-shaped marker, and when the calibration device is used to calibrate the equipment, the at least two calibration plates are symmetrically placed along the long side of the marker.

[0062] In some embodiments, the number of the calibration plates is 6, with 3 calibration plates placed on each side of the marker.

[0063] The specific structure of the calibration device can refer to the description in the above embodiment and will not be repeated here.

[0064] It is not difficult to understand that the solutions described in the above embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space reasons, they are not listed one by one in the embodiments of this disclosure.

[0065] Accordingly, an embodiment of the present disclosure further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed, the method described in any of the above embodiments is implemented.

[0066] Furthermore, the present disclosure also provides an electronic device, such as Figure 6 As shown, the device includes a processor 61, a memory 62, and computer instructions stored in the memory 62 and executable by the processor 61. When the processor 61 executes the computer instructions, the method described in any one of the above embodiments is implemented.

[0067] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0068] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0069] It can be known from the above description of the implementation mode that a person skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure or some parts of the embodiments.

[0070] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0071] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment of the present disclosure. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0072] The above is only a specific implementation of the embodiment of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present disclosure. These improvements and modifications should also be regarded as the protection scope of the embodiment of the present disclosure.

Claims

1. A calibration method for a three-dimensional scanning device, characterized in that: The method comprises: Using at least two cameras to be calibrated in the three-dimensional scanning device to respectively acquire images of the calibration device to obtain multiple frames of calibration images; wherein the calibration device includes a marker and at least two calibration plates placed near the marker, the marker includes a plurality of first marker points whose relative position relationship is known, and each of the calibration plates includes a plurality of second marker points whose relative position relationship is unknown; Performing three-dimensional reconstruction on the calibration device based on initial intrinsic parameters of each of the at least two cameras, initial extrinsic parameters between the at least two cameras, and the multiple frames of calibration images to obtain reconstructed three-dimensional coordinates of each of the first marker point and the second marker point in the multiple frames of calibration images; Based on the reconstructed three-dimensional coordinates, the reprojection errors of the first marker point and the second marker point in the multiple frames of calibration images are determined, and the initial extrinsic parameters are optimized based on the reprojection errors and the relative position relationship between the first marker points to obtain the calibration extrinsic parameters between the at least two cameras.

2. The method according to claim 1, characterized in that Optimizing the initial extrinsic parameter based on the reprojection error and the relative position relationship between the first marker points to obtain a calibration extrinsic parameter between the at least two cameras includes: Taking minimizing the reprojection error as the optimization goal, the initial extrinsic parameter is optimized to obtain the optimized extrinsic parameter; based on the optimized extrinsic parameter, the optimized three-dimensional coordinates of the first marker point in the multiple calibration images are determined; based on the difference between the optimized three-dimensional coordinates of the first marker point in the multiple calibration images and the real three-dimensional coordinates of the first marker point, the optimized extrinsic parameter is optimized again to obtain the calibration extrinsic parameter between the at least two cameras; or The initial extrinsic parameters are optimized based on the difference between the reconstructed three-dimensional coordinates of the first marker point in the multiple frames of calibration images and the real three-dimensional coordinates of the first marker point to obtain optimized extrinsic parameters; based on the optimized extrinsic parameters and the reconstructed three-dimensional coordinates of the first marker point and the second marker point in the multiple frames of calibration images, the reprojection errors of the first marker point and the second marker point in the multiple frames of calibration images are determined; the optimized extrinsic parameters are optimized again with minimizing the reprojection error as the optimization goal to obtain the calibration extrinsic parameters between the at least two cameras; The real three-dimensional coordinates are determined based on the relative position relationship between the first marker points.

3. The method according to claim 1, characterized in that The multiple first marker points in the marker can be distinguished by visual information, each first marker point corresponds to an identifier, and the identifier of each first marker point is stored in association with the real three-dimensional coordinates of the first marker point, wherein the real three-dimensional coordinates of each first marker point are determined based on the relative position relationship between the first marker points.

4. The method according to claim 3, characterized in that The visual information includes one or more of the following: the size of the first marker point, the shape of the first marker point, the color of the first marker point, the arrangement of the first marker point, and marking information around the first marker point; and / or The plurality of first marking points in the marker are divided into a plurality of groups, and the arrangement modes of the first marking points in the plurality of groups are different.

5. The method according to claim 4, characterized in that The marker is a long strip-shaped marker, and the multiple groups are distributed at equal intervals along the long side of the marker.

6. The method according to claim 1, characterized in that The marker is a long strip-shaped marker, and the at least two calibration plates are symmetrically placed along the long side of the marker; and / or The number of the calibration plates is positively correlated with the scanning angle of the three-dimensional scanning device.

7. The method according to claim 6, characterized in that The number of the calibration plates is 6, and 3 calibration plates are placed on each side of the marker.

8. The method according to claim 1, characterized in that The multiple frames of calibration images include: Aligning the three-dimensional scanning device with the center position of the marker, moving it to several positions along a direction perpendicular to the plane where the calibration plate is located, and acquiring an image at each position; and The three-dimensional scanning device is respectively aimed at the calibration plates placed on both sides of the marker, and is moved to several positions along the direction parallel to the plane where the calibration plates are located, and at each position, the three-dimensional scanning device is controlled to form a certain angle with the plane where the calibration plates are located in the four directions of front, back, left, and right to acquire the image; Wherein, each frame of the calibration image includes one or more of the first marking points.

9. The method according to claim 1, characterized in that: The at least two cameras include two black-and-white cameras, and the three-dimensional scanning device also includes a texture camera. After obtaining the calibration extrinsic parameters between the two black-and-white cameras, the method further includes: Determine the three-dimensional coordinates of the first marking point and the second marking point in the calibration device based on the calibration external parameters; For any one of the two black-and-white cameras, using the black-and-white camera and the texture camera to respectively collect images of the calibration device; The calibration extrinsic parameters of the black-and-white camera and the texture camera are determined based on the collected multiple frames of calibration images and the three-dimensional coordinates of the first marker point and the second marker point.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed, the method according to any one of claims 1 to 9 is implemented.

11. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and computer instructions stored in the memory and executable by the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A calibration device, characterized in that: The calibration device includes a marker and at least two calibration plates, the marker includes a plurality of first marker points whose relative position relationship is known, and each of the calibration plates includes a plurality of second marker points whose relative position relationship is unknown; wherein, when the calibration device is used to calibrate the equipment, the at least two calibration plates are placed near the marker to combine to obtain a large-sized combined calibration plate.

14. The calibration device according to claim 13, characterized in that: The plurality of first marking points in the marker can be distinguished by visual information, each first marking point corresponds to an identifier, and the identifier of each first marking point is stored in association with the real three-dimensional coordinates of the first marking point, wherein the real three-dimensional coordinates of each first marking point are determined based on the relative position relationship between the first marking points; and / or The marker is a long strip-shaped marker. When the calibration device is used to calibrate the equipment, the at least two calibration plates are symmetrically placed along the long side direction of the marker.

15. The calibration device according to claim 14, characterized in that: The number of the calibration plates is 6, and 3 calibration plates are placed on each side of the marker.

16. The calibration device according to claim 14, characterized in that: The visual information includes one or more of the following: the size of the first marker point, the shape of the first marker point, the color of the first marker point, the arrangement of the first marker point, and marking information around the first marker point; and / or The plurality of first marking points in the marker are divided into a plurality of groups, and the arrangement modes of the first marking points in the plurality of groups are different.

Images