Three-dimensional stereoscopic space vision calibration device, method, system and readable storage medium

Abstract

The application discloses a three-dimensional space visual calibration device, method, system and readable storage medium, wherein the method comprises the following steps: a cubic structure module is provided with different types of visual coding target surfaces on the surface of the cubic structure module, and is used for calibration at multiple space angles; at least one inertial measurement unit is embedded in the cubic structure module, and is used for detecting current attitude information of the device in real time; a wireless communication module is in communication connection with the inertial measurement unit, and is used for sending the attitude data to a preset image processing system; and at least one identification surface module is integrated on the surface of the cubic structure module, and is used for identifying the current visible target surface number or type. Through the innovative three-dimensional calibration device design, multi-sensor fusion and optimization algorithm, the application realizes automatic, high-precision and high-robustness three-dimensional visual calibration, and is suitable for complex scenes such as a multi-camera system, a mobile robot and industrial automation.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and more specifically, to a three-dimensional spatial vision calibration device, method, system, and readable storage medium. Background Technology

[0002] Current machine vision systems mainly rely on traditional two-dimensional calibration boards, such as checkerboards and dot matrices, for camera parameter calibration. These methods have the following shortcomings:

[0003] 1. Simple structure and lack of spatial information: Two-dimensional calibration boards only provide planar information, which is difficult to meet the needs of spatial point redundancy and depth perception in stereo reconstruction and multi-view systems;

[0004] 2. The calibration process is complex and requires repeated adjustments at multiple angles: manual rotation or switching of the plate surface increases labor costs;

[0005] 3. Easily affected by occlusion, lighting, and angle changes: This leads to increased feature point detection errors and a higher probability of calibration failure;

[0006] 4. Not suitable for mobile robots or automated deployment requirements in industrial production lines: Cannot achieve online self-calibration and equipment collaboration. Summary of the Invention

[0007] The purpose of this invention is to provide a three-dimensional spatial visual calibration device, method, system, and readable storage medium. It proposes a novel multi-faceted coded spatial calibration device, which, in conjunction with an inertial measurement unit and algorithm optimization methods, achieves automated and highly robust three-dimensional visual calibration.

[0008] The first aspect of the present invention provides a three-dimensional spatial visual calibration device, comprising:

[0009] A cubic structure module, the surface of which is provided with different types of visual coding target surfaces for calibration at multiple spatial angles;

[0010] At least one inertial measurement unit is embedded inside the cube structure module for real-time detection of the device's current attitude information;

[0011] A wireless communication module, communicatively connected to the inertial measurement unit, is used to send attitude data to a preset image processing system;

[0012] At least one identification surface module is integrated on the surface of the cube structure module for identifying the currently visible target surface number or type.

[0013] This solution also includes an adjustable base, which is installed at the bottom of the cubic structure module and is used to adjust the height and pitch angle of the device.

[0014] In this scheme, the encoded target surface includes a checkerboard pattern, a dot matrix, an ArUco code, or a custom pattern encoding.

[0015] A second aspect of the present invention provides a three-dimensional spatial visual calibration method, applied to any of the three-dimensional spatial visual calibration devices described in the present invention, comprising the following steps:

[0016] Acquire image data from the camera acquisition and calibration device, and simultaneously record attitude data measured by the inertial measurement unit within the device;

[0017] Based on the image data, feature extraction and matching are performed to obtain image point sets and target surface number data;

[0018] A projection model is constructed based on the image point set to obtain projection model parameters. An objective function is defined based on the projection model parameters, combined with the image point set and the target surface numbering data.

[0019] The camera parameters are obtained by nonlinear iterative optimization based on the objective function, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

[0020] In this solution, the step of extracting and matching features based on the image data to obtain the image point set and target surface number data specifically includes:

[0021] The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code;

[0022] Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code;

[0023] An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

[0024] In this scheme, the step of constructing a projection model based on the image point set to obtain projection model parameters, and defining an objective function based on the projection model parameters in combination with the image point set and the target surface numbering data, specifically includes:

[0025] Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is:

[0026] ;

[0027] in, For 2D image points, For 3D image points, For the camera intrinsic parameter matrix, For camera external parameters, For rotation matrix, It is a translation vector;

[0028] Visual constraints are applied based on the projection model to calculate the reprojection error;

[0029] The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints.

[0030] Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors;

[0031] The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is:

[0032] ;

[0033] in, Let be the objective function. For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface.

[0034] A third aspect of the present invention also provides a three-dimensional spatial vision calibration system, including a memory and a processor. The memory includes a three-dimensional spatial vision calibration method program, which, when executed by the processor, performs the following steps:

[0035] Acquire image data from the camera acquisition and calibration device, and simultaneously record attitude data measured by the inertial measurement unit within the device;

[0036] Based on the image data, feature extraction and matching are performed to obtain image point sets and target surface number data;

[0037] A projection model is constructed based on the image point set to obtain projection model parameters. An objective function is defined based on the projection model parameters, combined with the image point set and the target surface numbering data.

[0038] The camera parameters are obtained by nonlinear iterative optimization based on the objective function, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

[0039] In this solution, the step of extracting and matching features based on the image data to obtain the image point set and target surface number data specifically includes:

[0040] The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code;

[0041] Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code;

[0042] An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

[0043] In this scheme, the step of constructing a projection model based on the image point set to obtain projection model parameters, and defining an objective function based on the projection model parameters in combination with the image point set and the target surface numbering data, specifically includes:

[0044] Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is:

[0045] ;

[0046] in, For 2D image points, For 3D image points, For the camera intrinsic parameter matrix, For camera external parameters, For rotation matrix, It is a translation vector;

[0047] Visual constraints are applied based on the projection model to calculate the reprojection error;

[0048] The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints.

[0049] Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors;

[0050] The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is:

[0051] ;

[0052] in, Let be the objective function. For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface.

[0053] A fourth aspect of the present invention provides a computer-readable storage medium comprising a machine program for a three-dimensional spatial visual calibration method, wherein when executed by a processor, the three-dimensional spatial visual calibration method program implements the steps of a three-dimensional spatial visual calibration method as described in any of the preceding claims.

[0054] This invention discloses a three-dimensional spatial vision calibration device, method, system, and readable storage medium. Through innovative stereo calibration device design, multi-sensor fusion, and optimization algorithms, it effectively solves the limitations of traditional two-dimensional calibration boards, achieving automated, high-precision, and highly robust three-dimensional vision calibration, and is suitable for complex scenarios such as multi-camera systems, mobile robots, and industrial automation. Attached Figure Description

[0055] Figure 1 A schematic diagram of the structure of a three-dimensional spatial vision calibration device according to the present invention is shown;

[0056] Figure 2 A flowchart of a three-dimensional spatial visual calibration method according to the present invention is shown;

[0057] Figure 3 This diagram illustrates the target surface numbering for a three-dimensional spatial visual calibration method according to the present invention.

[0058] Figure 4 A block diagram of a three-dimensional spatial vision calibration system according to the present invention is shown;

[0059] Component labeling: 1. Cube structure module; 2. Wireless communication module; 3. Identification panel module; 4. Adjustable base. Detailed Implementation

[0060] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0061] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0062] Figure 1 A schematic diagram of the structure of a three-dimensional spatial visual calibration device according to this application is shown.

[0063] like Figure 1 As shown, this application discloses a three-dimensional spatial visual calibration device, comprising:

[0064] A cubic structure module, the surface of which is provided with different types of visual coding target surfaces for calibration at multiple spatial angles;

[0065] At least one inertial measurement unit is embedded inside the cube structure module for real-time detection of the device's current attitude information;

[0066] A wireless communication module, communicatively connected to the inertial measurement unit, is used to send attitude data to a preset image processing system;

[0067] At least one identification surface module is integrated on the surface of the cube structure module for identifying the currently visible target surface number or type.

[0068] It should be noted that, in this embodiment, as Figure 1 As shown, the three-dimensional spatial visual calibration device described in this invention includes a cube structure module 1, at least one inertial measurement unit, a wireless communication module 2, and at least one marking surface module 3. The surface 1 of the cube structure module is provided with different types of visual coding target surfaces for calibration at multiple spatial angles. Accordingly, the coding target surfaces include checkerboard, dot matrix, ArUco code, or custom pattern coding.

[0069] Furthermore, the inertial measurement unit is embedded inside the cube structure module, therefore... Figure 1 Not illustrated, it is specifically used to detect the current attitude information of the device in real time. A wireless communication module 2 is connected to the inertial measurement unit to send attitude data to a preset image processing system, and at least one marking surface module 3 is integrated on the surface of the cube structure module to mark the currently visible target surface number or type.

[0070] Furthermore, in this embodiment, as Figure 1 As shown, the device also includes an adjustable base 4, which is installed at the bottom of the cubic structure module 1 and is used to adjust the height and pitch angle of the device.

[0071] Figure 2 A flowchart of a three-dimensional spatial visual calibration method according to this application is shown.

[0072] like Figure 2 As shown, this application discloses a three-dimensional spatial visual calibration method, including the following steps:

[0073] S202, acquire image data from the camera acquisition calibration device, and simultaneously record attitude data measured by the inertial measurement unit inside the device;

[0074] S204, Based on the image data, feature extraction and matching are performed to obtain image point set and target surface number data;

[0075] S206, construct a projection model based on the image point set to obtain projection model parameters, and define an objective function based on the projection model parameters in combination with the image point set and the target surface numbering data;

[0076] S208. Based on the objective function, perform nonlinear iterative optimization to obtain the camera parameters, verify the calibration accuracy based on the camera parameters, and output a calibration report based on the accuracy index.

[0077] It should be noted that, in this embodiment, during calibration, images of the calibration device are acquired from multiple perspectives, and the attitude data output by the IMU (Inertial Measurement Unit) is recorded simultaneously, thereby forming an image-attitude data pair. That is, the image data of the calibration device is acquired by the camera, and the attitude data measured by the inertial measurement unit in the device is recorded simultaneously. Then, based on the image data, feature extraction and matching are performed to obtain the image point set and target surface number data, that is, the feature points (such as corner points or center points) of the coded target surface in the image are extracted, and the image point set with a two-dimensional-three-dimensional correspondence is established.

[0078] Further, a projection model is constructed, specifically based on the image point set to obtain projection model parameters. Then, based on the projection model parameters, combined with the image point set and the target surface numbering data, an objective function is defined. The objective function includes reprojection error (image error), IMU attitude prior error, and spatial plane geometric constraint terms. The objective function is then solved by nonlinear iterative optimization to obtain camera parameters. Specifically, the Levenberg-Marquardt Optimization Algorithm (LM algorithm) can be used to jointly optimize the objective function, solving for the camera intrinsic parameter matrix and extrinsic parameter transformation matrix (rotation matrix and translation vector). The calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index. The report includes calibration parameters such as the camera intrinsic parameter matrix and extrinsic parameter transformation matrix.

[0079] Furthermore, the camera parameters are obtained by nonlinear iterative optimization based on the objective function, wherein the optimized camera parameters include an intrinsic parameter matrix K and an extrinsic parameter matrix. Where R is the rotation matrix and t is the translation vector, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

[0080] According to an embodiment of the present invention, the step of obtaining image point sets and target surface number data by feature extraction and matching based on the image data specifically includes:

[0081] The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code;

[0082] Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code;

[0083] An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

[0084] It should be noted that, in this embodiment, as Figure 3As shown, this diagram illustrates the acquisition of target surface numbers. A preset encoding and recognition algorithm (e.g., ArUco detection) is used to determine the target surface numbers in the image data, resulting in target surface number data. ArUco detection corresponds to Augmented Reality University of Cordoba, specifically used to detect and recognize specific binary square markers (i.e., ArUco codes). Accordingly, the identified target surfaces include checkerboard target surfaces, dot matrix target surfaces, and ArUco codes. The target surface number data is used for subsequent spatial constraints, which will be explained in detail in the following description.

[0085] Furthermore, in this embodiment, feature point extraction is performed, specifically extracting the corner points of the checkerboard target surface, the center of the dot matrix target surface, the marked corner points and ID of the ArUco code to obtain feature points. Then, the extracted feature points (2D image points) are matched with preset 3D target surface coordinates (e.g., the CAD model of the preset calibration device) to obtain an image point set, which corresponds to a two-dimensional-three-dimensional corresponding point set.

[0086] According to an embodiment of the present invention, the step of constructing a projection model based on the image point set to obtain projection model parameters, and defining an objective function based on the projection model parameters in combination with the image point set and the target surface numbering data, specifically includes:

[0087] Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is:

[0088] ;

[0089] in, For 2D image points, Here, K represents the 3D image points, and K is the camera intrinsic parameter matrix. Let R be the camera extrinsic parameter, R be the rotation matrix, and t be the translation vector;

[0090] Visual constraints are applied based on the projection model to calculate the reprojection error;

[0091] The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints.

[0092] Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors;

[0093] The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is:

[0094] ;

[0095] in, Let be the objective function. For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface.

[0096] It should be noted that, in this embodiment, the construction of the projection model and the definition of the objective function are described. The projection model parameters are obtained by establishing a projection equation based on the image point set. The projection equation is as follows:

[0097] ;

[0098] in, For 2D image points, For 3D image points, For the camera intrinsic parameter matrix, For camera external parameters, For rotation matrix, Let be the translation vector, where, if IMU attitude data is used, the IMU attitude data can be converted into a rotation matrix as an initial estimate of the extrinsic parameter R.

[0099] Furthermore, in this embodiment, visual constraints are applied based on the projection model to calculate the reprojection error, prior constraints are applied based on the attitude data and the attitude matrix to be optimized to obtain the attitude error, and the coplanarity of 3D points on the same target surface is forced to complete the spatial geometric constraints to obtain the spatial structure error. Finally, the objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error.

[0100] Figure 4 A block diagram of a three-dimensional spatial visual calibration system according to the present invention is shown.

[0101] like Figure 4 As shown, this invention discloses a three-dimensional spatial vision calibration system, including a memory and a processor. The memory includes a three-dimensional spatial vision calibration method program, which, when executed by the processor, performs the following steps:

[0102] Acquire image data from the camera acquisition and calibration device, and simultaneously record attitude data measured by the inertial measurement unit within the device;

[0103] Based on the image data, feature extraction and matching are performed to obtain image point sets and target surface number data;

[0104] A projection model is constructed based on the image point set to obtain projection model parameters. An objective function is defined based on the projection model parameters, combined with the image point set and the target surface numbering data.

[0105] The camera parameters are obtained by nonlinear iterative optimization based on the objective function, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

[0106] It should be noted that, in this embodiment, during calibration, images of the calibration device are acquired from multiple perspectives, and the attitude data output by the IMU (Inertial Measurement Unit) is recorded simultaneously, thereby forming an image-attitude data pair. That is, the image data of the calibration device is acquired by the camera, and the attitude data measured by the inertial measurement unit in the device is recorded simultaneously. Then, based on the image data, feature extraction and matching are performed to obtain the image point set and target surface number data, that is, the feature points (such as corner points or center points) of the coded target surface in the image are extracted, and the image point set with a two-dimensional-three-dimensional correspondence is established.

[0107] Further, a projection model is constructed, specifically based on the image point set to obtain projection model parameters. Then, based on the projection model parameters, combined with the image point set and the target surface numbering data, an objective function is defined. The objective function includes reprojection error (image error), IMU attitude prior error, and spatial plane geometric constraint terms. The objective function is then solved by nonlinear iterative optimization to obtain camera parameters. Specifically, the Levenberg-Marquardt Optimization Algorithm (LM algorithm) can be used to jointly optimize the objective function, solving for the camera intrinsic parameter matrix and extrinsic parameter transformation matrix (rotation matrix and translation vector). The calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index. The report includes calibration parameters such as the camera intrinsic parameter matrix and extrinsic parameter transformation matrix.

[0108] Furthermore, the camera parameters are obtained by nonlinear iterative optimization based on the objective function, wherein the optimized camera parameters include an intrinsic parameter matrix K and an extrinsic parameter matrix. Where R is the rotation matrix and t is the translation vector, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

[0109] According to an embodiment of the present invention, the step of obtaining image point sets and target surface number data by feature extraction and matching based on the image data specifically includes:

[0110] The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code;

[0111] Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code;

[0112] An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

[0113] It should be noted that, in this embodiment, as Figure 3 As shown, this diagram illustrates the acquisition of target surface numbers. A preset encoding and recognition algorithm (e.g., ArUco detection) is used to determine the target surface numbers in the image data, resulting in target surface number data. ArUco detection corresponds to Augmented Reality University of Cordoba, specifically used to detect and recognize specific binary square markers (i.e., ArUco codes). Accordingly, the identified target surfaces include checkerboard target surfaces, dot matrix target surfaces, and ArUco codes. The target surface number data is used for subsequent spatial constraints, which will be explained in detail in the following description.

[0114] Furthermore, in this embodiment, feature point extraction is performed, specifically extracting the corner points of the checkerboard target surface, the center of the dot matrix target surface, the marked corner points and ID of the ArUco code to obtain feature points. Then, the extracted feature points (2D image points) are matched with preset 3D target surface coordinates (e.g., the CAD model of the preset calibration device) to obtain an image point set, which corresponds to a two-dimensional-three-dimensional corresponding point set.

[0115] According to an embodiment of the present invention, the step of constructing a projection model based on the image point set to obtain projection model parameters, and defining an objective function based on the projection model parameters in combination with the image point set and the target surface numbering data, specifically includes:

[0116] Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is:

[0117] ;

[0118] in, For 2D image points, For 3D image points, For the camera intrinsic parameter matrix, For camera external parameters, For rotation matrix, It is a translation vector;

[0119] Visual constraints are applied based on the projection model to calculate the reprojection error;

[0120] The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints.

[0121] Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors;

[0122] The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is:

[0123] ;

[0124] in, Let be the objective function. For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface.

[0125] It should be noted that, in this embodiment, the construction of the projection model and the definition of the objective function are described. The projection model parameters are obtained by establishing a projection equation based on the image point set. The projection equation is as follows:

[0126] ;

[0127] in, For 2D image points, For 3D image points, For the camera intrinsic parameter matrix, For camera external parameters, For rotation matrix, Let be the translation vector. If IMU attitude data is used, the IMU attitude data can be converted into a rotation matrix as an extrinsic parameter. The initial estimate.

[0128] Furthermore, in this embodiment, visual constraints are applied based on the projection model to calculate the reprojection error, prior constraints are applied based on the attitude data and the attitude matrix to be optimized to obtain the attitude error, and the coplanarity of 3D points on the same target surface is forced to complete the spatial geometric constraints to obtain the spatial structure error. Finally, the objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error.

[0129] A fourth aspect of the present invention provides a computer-readable storage medium comprising a three-dimensional spatial visual calibration method program, wherein when the three-dimensional spatial visual calibration method program is executed by a processor, it implements the steps of a three-dimensional spatial visual calibration method as described in any of the preceding claims.

[0130] This invention discloses a three-dimensional spatial vision calibration device, method, system, and readable storage medium. Through innovative stereo calibration device design, multi-sensor fusion, and optimization algorithms, it effectively solves the limitations of traditional two-dimensional calibration boards, achieving automated, high-precision, and highly robust three-dimensional vision calibration, and is suitable for complex scenarios such as multi-camera systems, mobile robots, and industrial automation.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0132] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0133] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0134] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A three-dimensional spatial visual calibration method, characterized in that, The method includes the following steps: Acquire image data from the camera acquisition and calibration device, and simultaneously record attitude data measured by the inertial measurement unit within the device; Based on the image data, feature extraction and matching are performed to obtain image point sets and target surface number data; A projection model is constructed based on the image point set to obtain projection model parameters. An objective function is defined based on the projection model parameters, combined with the image point set and the target surface numbering data. Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is: ； in, For 2D image points, Here, K represents the 3D image points, and K is the camera intrinsic parameter matrix. Let R be the camera extrinsic parameter, R be the rotation matrix, and t be the translation vector; Visual constraints are applied based on the projection model to calculate the reprojection error; The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints. Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors; The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is: ； Where E is the objective function, For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface; The camera parameters are obtained by nonlinear iterative optimization based on the objective function, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

2. The three-dimensional spatial visual calibration method according to claim 1, characterized in that, The step of extracting and matching features based on the image data to obtain the image point set and target surface number data specifically includes: The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code; Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code; An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

3. A three-dimensional spatial visual calibration system, characterized in that, The system includes a memory and a processor. The memory contains a three-dimensional spatial visual calibration method program, which, when executed by the processor, performs the following steps: Acquire image data from the camera acquisition and calibration device, and simultaneously record attitude data measured by the inertial measurement unit within the device; Based on the image data, feature extraction and matching are performed to obtain image point sets and target surface number data; A projection model is constructed based on the image point set to obtain projection model parameters. An objective function is defined based on the projection model parameters, combined with the image point set and the target surface numbering data. Based on the image point set, a projection equation is established to obtain the projection model parameters, where the projection equation is: ； in, For 2D image points, Here, K represents the 3D image points, and K is the camera intrinsic parameter matrix. Let R be the camera extrinsic parameter, R be the rotation matrix, and t be the translation vector; Visual constraints are applied based on the projection model to calculate the reprojection error; The attitude error is obtained by combining the attitude data with the attitude matrix to be optimized and applying prior constraints. Force 3D points on the same target surface to be coplanar to perform target surface planar constraints in order to complete spatial geometric constraints and obtain spatial structural errors; The objective function is obtained based on the reprojection error, the attitude error, and the spatial structure error, wherein the objective function is: ； Where E is the objective function, For reprojection error, For attitude error, For spatial structure error, , These are the weighting coefficients. The vector before projection is a 2D vector. This is the projected 3D vector. The attitude data is measured by the inertial measurement unit. The pose matrix to be optimized, For measurement points, Points on the target surface; The camera parameters are obtained by nonlinear iterative optimization based on the objective function, the calibration accuracy is verified based on the camera parameters, and a calibration report is output based on the accuracy index.

4. A three-dimensional spatial visual calibration system according to claim 3, characterized in that, The step of extracting and matching features based on the image data to obtain the image point set and target surface number data specifically includes: The target surface number data is obtained by determining the target surface number in the image data based on a preset encoding and recognition algorithm, wherein the target surface includes a checkerboard target surface, a dot matrix target surface, and an ArUco code; Feature points are obtained by extracting the corner points of the chessboard target surface, the center of the circle of the dot matrix target surface, the marked corner points and ID of the ArUco code; An image point set is obtained by matching the feature points with the preset 3D target surface coordinates, wherein the feature points are 2D image points and the image point set corresponds to a two-dimensional-three-dimensional point set.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a three-dimensional spatial visual calibration method program, which, when executed by a processor, implements the steps of a three-dimensional spatial visual calibration method as described in any one of claims 1 to 2.

6. A three-dimensional spatial visual calibration device, applied in the three-dimensional spatial visual calibration method according to any one of claims 1-2, characterized in that, include: A cube structure module, the surface of which is provided with different types of visual coding target surfaces for calibration at multiple spatial angles, wherein the coding target surfaces include checkerboard, dot matrix, ArUco code or custom pattern coding; At least one inertial measurement unit is embedded inside the cube structure module for real-time detection of the device's current attitude information; A wireless communication module, communicatively connected to the inertial measurement unit, is used to send attitude data to a preset image processing system; At least one identification surface module is integrated on the surface of the cube structure module for identifying the currently visible target surface number or type.

7. A three-dimensional spatial visual calibration device according to claim 6, characterized in that, It also includes an adjustable base, which is installed at the bottom of the cubic structure module and is used to adjust the height and pitch angle of the device.

Images