Camera reference plane calibration, three-dimensional reconstruction method, device and medium

Abstract

The present application provides a reference plane calibration method, a three-dimensional reconstruction method, a device and a medium for a camera, and relates to the field of computer vision technology. The reference plane calibration method for the camera includes: determining the normal of the reference plane based on the three-dimensional point data of the reference plane; converting the normal to the camera coordinate system using the external parameters of the camera to obtain the normal vector; and adjusting the position and posture of the camera based on the angle between the normal vector and the optical axis vector of the camera. The present application can obtain a more accurate adjustment result, making the optical axis direction of the camera closer to perpendicularity with the reference plane, reducing the error caused by human judgment, and ensuring the stability of the calibration result of the reference plane of the camera.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device and medium for reference plane calibration and three-dimensional reconstruction of a camera. Background Art

[0002] A structured light 3D camera is a device used to obtain the three-dimensional shape and depth information of an object. It reconstructs the three-dimensional shape of the object by projecting a light pattern of known structure (such as stripes, dots or grids) onto the object, and then uses a camera to capture the deformation of these patterns on the surface of the object. The calibration of a structured light 3D camera is a key step to ensure that it can accurately measure three-dimensional shape and depth information. Calibration includes the calibration of the camera's internal and external parameters, as well as the determination of the reference plane. Among them, the calibration of the reference plane usually uses a level or a laser rangefinder to observe with the human eye to determine whether the camera's optical axis is level with the reference plane. The disadvantage of this solution is that the error is too large, and the human eye itself has an error that cannot be ignored. Summary of the invention

[0003] The embodiments of the present application provide a reference plane calibration method, a three-dimensional reconstruction method, a device and a medium for a camera to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a reference plane calibration method for a camera, which includes: determining the normal of the reference plane based on three-dimensional point data of the reference plane; converting the normal to a camera coordinate system using external parameters of the camera to obtain a normal vector; and adjusting the posture of the camera based on an angle between the normal vector and an optical axis vector of the camera.

[0005] In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction method, which includes: acquiring an image using a camera; wherein the camera is calibrated using the above-mentioned method; and performing three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result.

[0006] In a third aspect, an embodiment of the present application provides a reference plane calibration device for a camera, which includes: a fitting module for determining the normal of the reference plane based on the three-dimensional point data of the reference plane; a conversion module for converting the normal to a camera coordinate system using the external parameters of the camera to obtain a normal vector; and an adjustment module for adjusting the posture of the camera based on the angle between the normal vector and the optical axis vector of the camera.

[0007] In a fourth aspect, an embodiment of the present application provides a three-dimensional reconstruction device, which includes: an image module, used to acquire an image using a camera; wherein the camera is calibrated using the above-mentioned method; and a reconstruction module, used to perform three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result.

[0008] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods described above when executing the computer program.

[0009] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described above is implemented.

[0010] Compared with the related art, this application has the following advantages:

[0011] The present application provides a reference plane calibration method, a three-dimensional reconstruction method, a device and a medium for a camera. The reference plane calibration method for the camera includes: determining the normal of the reference plane based on the three-dimensional point data of the reference plane; converting the normal to the camera coordinate system using the external parameters of the camera to obtain a normal vector; and adjusting the camera's posture based on the angle between the normal vector and the optical axis vector of the camera. According to an embodiment of the present application, the reference plane can be fitted by the three-dimensional point data of the reference plane in the world coordinate system to determine the normal of the reference plane, and the normal can be converted to the camera coordinate system using the external parameters of the camera to obtain a normal vector, so that both the normal vector and the optical axis vector are located in the camera coordinate system, so as to calculate the angle between the two, and then adjust the camera's posture according to the angle, so as to obtain a more accurate adjustment result, so that the optical axis direction of the camera is closer to perpendicular to the reference plane, reduce the error caused by human judgment, and ensure the stability of the calibration result of the camera's reference plane.

[0012] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0014] Figure 1 A flow chart of a reference plane calibration method for a camera provided in an embodiment of the present application is shown;

[0015] Figure 2 A flowchart of a reference plane calibration method for a camera provided in an embodiment of the present application is shown;

[0016] Figure 3 A schematic diagram of a possible calibration plate provided in an embodiment of the present application is shown;

[0017] Figure 4 A schematic diagram of an ideal state provided in an embodiment of the present application is shown;

[0018] Figure 5 A flowchart of a three-dimensional reconstruction method provided in an embodiment of the present application is shown;

[0019] Figure 6 A structural block diagram of a reference plane calibration device for a camera provided in an embodiment of the present application is shown;

[0020] Figure 7 A structural block diagram of a three-dimensional reconstruction device provided in an embodiment of the present application is shown;

[0021] Figure 8 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0023] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0024] First, the basic knowledge of mathematical projection transformation involved in this scheme is introduced.

[0025]

[0026] In the above formula, f x 、f y is the focal length of the camera, in pixels; u0 and v0 are the principal points in the x and y directions; f x 、f y , u0, v0 constitute the internal parameter matrix. c , Y c , Z c is a 3D point in the camera coordinate system, and u and v are pixel coordinates in the pixel coordinate system.

[0027] According to formula (1), the 3D point in the camera coordinate system can be mapped to the pixel coordinate system based on the camera intrinsic parameters.

[0028] From equations (2) and (3), the depth value and pixel coordinates are known. According to the camera intrinsic parameters, the depth map can be converted to the corresponding camera coordinate system.

[0029] Camera calibration plays an important role in two-dimensional (2D) and three-dimensional (3D) computer vision. In the calibration process of structured light 3D cameras, a reference plane needs to be calibrated, and the reference plane needs to be perpendicular to the camera optical axis. How to obtain the reference plane quickly, accurately, stably and at low cost is the key to obtaining good calibration results. How to minimize human intervention, achieve automation as much as possible, and reduce calibration costs as much as possible is the key to evaluating the pros and cons of calibration methods.

[0030] The existing camera reference plane calibration scheme has poor operability, low automation and requires human participation. In addition, the calibration results have poor stability, strong randomness and poor noise resistance.

[0031] Based on this, the present application provides a camera reference plane calibration method, a three-dimensional reconstruction method, a device and a medium. The method is highly operational and can be implemented in a low-cost and high-precision engineering manner, eliminating human participation, ensuring full process automation, and improving the accuracy of the calibration results. In addition, the method can resist the influence of noise and has high stability.

[0032] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0033] The present application embodiment provides a reference plane calibration method for a camera. Figure 1 The flowchart of the reference plane calibration method of a camera according to an embodiment of the present application is shown. The method may include:

[0034] Step 101: determine a normal line of the reference plane based on the three-dimensional point data of the reference plane.

[0035] In an embodiment of the present application, three-dimensional point data on a reference plane can be obtained, and fitting data of the reference plane can be obtained based on the three-dimensional point data. The fitting data can be a set of three-dimensional point data (which can be experimental data or measurement data), and these points are distributed near a reference plane.

[0036] Preferably, the image of the reference plane can be processed to obtain three-dimensional point data on the reference plane, and the three-dimensional point data can be converted into a form suitable for linear regression to obtain matrix and vector data. After that, the least squares problem is solved, and the parameters of the plane are solved using linear algebra methods. Finally, the normal vector is extracted from the parameters of the plane equation, that is, the normal of the reference plane is determined based on the fitting data. In an embodiment of the present application, the normal of the reference plane can be represented in the form of a normal vector, which is located in the world coordinate system.

[0037] In this step, by fitting the reference plane, fitting data is obtained. Based on the fitting data, the normal of the reference plane is calculated, which can more accurately determine the position and posture of the calibration plate in three-dimensional space, thereby improving the overall accuracy of the calibration. In addition, the normal provides additional geometric constraints. In the calibration process, more geometric constraints usually mean more stable and more accurate calibration results. The normal of the reference plane can be used as an additional constraint to help optimize the calibration parameters and reduce errors.

[0038] Step 102: convert the normal to a camera coordinate system using the extrinsic parameters of the camera to obtain a normal vector.

[0039] In an embodiment of the present application, an external parameter is used to characterize the conversion relationship between the camera coordinate system and the world coordinate system. The normal is converted to the camera coordinate system using the external parameter to obtain the normal vector of the camera coordinate system, so that the normal of the reference plane can be represented by the normal vector in the camera coordinate system. The optical axis vector is determined in the camera coordinate system. The optical axis vector is a vector used to describe the direction of the optical axis of the camera. In the camera coordinate system, the optical axis is usually defined as the normal direction from the center point of the camera lens (i.e., the origin of the camera) to the imaging plane. Therefore, the normal vector and the optical axis vector are both in the camera coordinate system, which is convenient for calculating the angle between the normal vector and the optical axis vector of the camera.

[0040] Step 103: Adjust the position and posture of the camera based on the angle between the normal vector and the optical axis vector of the camera.

[0041] In this step, the angle between the normal vector and the optical axis vector of the camera can be calculated to determine whether the optical axis of the camera is perpendicular to the reference plane. By calculating the angle, the positional relationship between the optical axis of the camera and the reference plane can be effectively determined.

[0042] In an embodiment of the present application, after the angle is determined, the position and posture of the camera can be accurately adjusted so that the optical axis of the camera and the reference plane are closer to a vertical relationship, thereby improving the accuracy and efficiency of the reference plane calibration.

[0043] The embodiment of the present application provides a reference plane calibration method for a camera, and the reference plane calibration method for the camera includes: determining the normal of the reference plane based on the three-dimensional point data of the reference plane; converting the normal to the camera coordinate system using the external parameters of the camera to obtain a normal vector; and adjusting the position and posture of the camera based on the angle between the normal vector and the optical axis vector of the camera. According to the embodiment of the present application, the reference plane can be fitted by the three-dimensional point data of the reference plane in the world coordinate system to determine the normal of the reference plane, and the normal can be converted to the camera coordinate system using the external parameters of the camera to obtain the normal vector, so that both the normal vector and the optical axis vector are located in the camera coordinate system, so as to calculate the angle between the two, and then adjust the position and posture of the camera according to the angle, so as to obtain a more accurate adjustment result, so that the optical axis direction of the camera is closer to perpendicular to the reference plane, reduce the error caused by human judgment, and ensure the stability of the calibration result of the reference plane of the camera.

[0044] In a possible implementation, the following steps may be performed before the normal is converted to the camera coordinate system using the external parameters of the camera to obtain the normal vector: obtaining an image taken by the camera and the internal parameters of the camera, wherein the image includes the reference plane; calculating the external parameters of the camera based on the image and the internal parameters; wherein the external parameters are used to characterize the conversion relationship between the camera coordinate system and the world coordinate system.

[0045] In this possible implementation, the camera and the reference plane can be set at a certain angle to ensure that the reference plane falls within the range that the camera can capture. The reference plane includes information that can be used to calibrate the camera to ensure the accuracy and reliability of the calibration process. For example, the reference plane may include but is not limited to the following characteristics: clear geometric patterns, high contrast, multi-view coverage, sufficient resolution and clarity, no distortion or minimized distortion, and uniform lighting. Camera intrinsic parameters are parameters that describe the internal characteristics of the camera imaging system, and may include but are not limited to focal length, principal point position, and distortion coefficient. These parameters are determined in advance through the intrinsic calibration process, and are used to realize the conversion of points in the three-dimensional world coordinate system and points in the two-dimensional image coordinate system, thereby achieving accurate image projection and geometric correction.

[0046] It should be noted that in the embodiment of the present application, the camera used may be a structured light 3D camera, which projects known patterns (such as stripes, grids or dots) onto the surface of an object, and then uses the camera to capture the deformation of these patterns on the surface of the object, thereby calculating the three-dimensional shape of the object.

[0047] The camera's internal parameters can be calibrated in advance. In the embodiment of the present application, the pre-calibrated internal parameters are directly obtained to improve the execution efficiency of the embodiment of the present application. When pre-calibrating the internal parameters, the appropriate internal parameter calibration method selected depends on the specific application scenario and equipment conditions. The internal parameter calibration methods that can be used include but are not limited to: checkerboard calibration method, calibration method based on dot array, method based on three-dimensional calibration object, and direct linear transformation method.

[0048] In this step, by acquiring the image including the reference plane taken by the camera and the internal parameters of the camera, various data required for the subsequent reference plane calibration of the camera are obtained to ensure the efficiency of the reference plane calibration.

[0049] In the embodiment of the present application, the extrinsic parameters of the camera are parameters that describe the position and orientation of the camera in three-dimensional space. The extrinsic parameters define the transformation relationship between the world coordinate system and the camera coordinate system, and may generally include a rotation matrix and a translation vector. By obtaining the image, the pixel points in the image can be obtained, and the extrinsic parameters of the camera can be calculated based on the information of the pixel points and the obtained camera intrinsic parameters. In the embodiment of the present application, the camera coordinate system may be a three-dimensional coordinate system with the optical center (optical center) of the camera as the origin. The origin of the world coordinate system may be located on a reference plane.

[0050] In this step, the camera's extrinsic parameters are calculated based on the image and the camera's intrinsic parameters, so that the conversion relationship between the camera coordinate system and the world coordinate system can be obtained, so that the information belonging to the two coordinate systems can be unified into the same coordinate system, which is convenient for data analysis and calculation.

[0051] In a possible implementation, the reference plane is a calibration plate, and the calculation of the external parameters of the camera based on the image and the intrinsic parameters can be performed according to the following steps: acquiring the size data of the calibration plate; identifying the two-dimensional feature points in the image, and converting the two-dimensional feature points into three-dimensional feature points using the intrinsic parameters; and calculating the external parameters of the camera based on the two-dimensional feature points, the three-dimensional feature points and the size data.

[0052] In an embodiment of the present application, the calibration plate is a tool for camera calibration, which is mainly used to determine the intrinsic and extrinsic parameters of the camera. Camera calibration is an important step in computer vision. Through calibration, the focal length, principal point, distortion coefficient and other parameters of the camera can be determined, so as to correct the geometric distortion in the image and accurately measure the three-dimensional position of the object. The reference plane may include, but is not limited to, any of the following types of calibration plates: a checkerboard calibration plate, which is composed of black and white square grids. The corner points of the checkerboard calibration plate (i.e., the intersection of the black and white grids) are used as feature points for calibration. Figure 3A schematic diagram of a possible calibration plate is shown; a dot calibration plate, which is composed of regularly arranged dots, the centers of which are used as feature points. In some cases, the dot calibration plate is easier to detect and match than the checkerboard calibration plate; a ring calibration plate, which is composed of regularly arranged ring patterns, the centers of which are used as feature points. This calibration plate has advantages in certain specific applications. The steps of using a calibration plate for camera calibration include preparing the calibration plate, taking calibration images, detecting feature points, calculating camera parameters, and verification and optimization.

[0053] In an embodiment of the present application, the dimension data of the calibration plate is used to describe the dimensions of the calibration plate itself. For example, in the case where the calibration plate is a checkerboard calibration plate, the dimension data of the calibration plate includes but is not limited to the actual distance between the corner points, which can be obtained by measurement. In the case where the calibration plate is a checkerboard calibration plate, identifying the two-dimensional feature points in the image may include identifying the corner points of the checkerboard in the image, calculating the two-dimensional feature points corresponding to the corner points in combination with the intrinsic parameters of the camera to obtain the three-dimensional feature points corresponding to the corner points, and then calculating the external parameters of the camera based on the two-dimensional feature points, the three-dimensional feature points and the dimension data. The camera external parameters calculated in this way can be aligned with the actual size.

[0054] In this possible implementation, using real-size data to align camera extrinsics can significantly improve the accuracy of positioning and measurement, enhance the effect of virtual-real fusion, ensure the accuracy of 3D reconstruction, and improve the collaborative working ability of multi-camera systems. This is of great significance for application scenarios that require high precision and high reliability, such as robot navigation, augmented reality, 3D reconstruction, and industrial automation.

[0055] By using real-size data, the position and orientation of the camera relative to the object or scene can be determined more accurately. This is especially important for applications that require precise positioning, such as robot navigation, autonomous driving, and industrial automation. Real-size data can reduce positioning errors caused by measurement errors or inaccurate data, thereby improving the accuracy of the overall calibration. In augmented reality applications, virtual objects need to be accurately superimposed on real-world scenes. Using real-size data to align camera extrinsics can ensure that virtual objects are displayed in the correct position and scale, providing a more realistic experience. Real-size data ensures that the size and scale of virtual objects are consistent with the real world, avoiding the problem of scale distortion. In 3D reconstruction applications, using real-size data to align camera extrinsics can generate more accurate 3D models. This is very important for applications in the fields of architecture, cultural relics protection, and medical imaging. Real-size data ensures that the reconstructed 3D model has the correct scale information, which is crucial for subsequent analysis and processing. In a multi-camera system, using real-size data can ensure that the extrinsics between each camera are accurately aligned, thereby improving the ability of the entire system to work together. Real-size data can reduce errors in the camera calibration process and improve the overall accuracy and stability of the multi-camera system.

[0056] In industrial automation, robots need to perform precise operations based on visual data from cameras. Using real-size data to align camera extrinsics ensures that robots can accurately grasp, assemble, and manipulate objects. Accurate extrinsic alignment can optimize production processes, reduce production errors and waste caused by inaccurate positioning, and improve production efficiency and quality. In measurement and monitoring applications, using real-size data can ensure the accuracy of measurement results. For example, in building measurement, topographic surveying, and medical imaging, accurate extrinsic alignment can provide more reliable measurement data. In real-time monitoring systems, aligning camera extrinsics using real-size data can provide more accurate monitoring information, helping to detect and deal with problems in a timely manner.

[0057] In a possible implementation, calculating the external parameters of the camera based on the two-dimensional feature points, the three-dimensional feature points and the size data can be performed according to the following steps: using the two-dimensional feature points, the three-dimensional feature points and the intrinsic parameters as input parameters of a PNP algorithm, and using the PNP algorithm to calculate the initial external parameters of the camera; and using the size data to adjust the initial external parameters to obtain the external parameters of the camera.

[0058] In this possible implementation, it should be noted that the PNP (Perspective-N-Point) algorithm is a technology for calculating camera extrinsics (i.e., the camera's posture, including rotation and translation). It uses known 3D points and their 2D projection points in the image to more accurately infer the position and direction of the camera. In an embodiment of the present application, after obtaining two-dimensional feature points and three-dimensional feature points, the PNP method can be used to calculate the initial extrinsic parameters of the camera based on the camera intrinsic parameters, two-dimensional feature points, and three-dimensional feature points. The initial extrinsic parameters are not aligned with the actual size, so the size data can be reused to align the calculated extrinsic parameters with the actual size.

[0059] It should be noted that in some scenarios, the initial external parameters of the camera do not need to be adjusted, and the initial external parameters of the camera can be directly used as the external parameters of the camera. These scenarios include but are not limited to the following:

[0060] Image registration and stitching: In image registration and stitching, the main goal is to align multiple images in the same coordinate system, not necessarily to the true scale. In this case, the relative relationship of the extrinsic parameters is more important than the absolute scale. For example, the stitching of panoramic images.

[0061] Relative positioning: In some applications, we only need to know the relative position and orientation between cameras, without the absolute real scale. Relative alignment of external parameters can meet the requirements. For example, relative pose estimation in a multi-camera system.

[0062] Computer vision algorithm testing and verification: When developing and testing computer vision algorithms, you may only need to verify the correctness and stability of the algorithm without aligning it to the true scale. In this case, relative alignment of external parameters is sufficient. For example, testing of image feature matching algorithms.

[0063] Applications in virtual environments: In some virtual reality (VR) or computer-generated imagery (CGI) applications, external parameters do not need to be aligned with real scale, because the entire environment is virtual and the proportions and scales can be set arbitrarily. For example, the creation of virtual game scenes.

[0064] In a possible implementation, the identifying of the two-dimensional feature points in the image may be performed according to the following steps: obtaining the SIFT (Scale-Invariant Feature Transform) key points and / or ORB (Oriented FAST and Rotated BRIEF) key points of the image; and using the SIFT key points and / or the ORB key points as the two-dimensional feature points in the image.

[0065] In the embodiment of the present application, SIFT is an algorithm for detecting and describing local features of an image. The SIFT algorithm provides a powerful and robust tool for tasks such as image matching, object recognition, and three-dimensional reconstruction by detecting and describing local feature points in an image. Its scale invariance and rotation invariance make it have a wide range of application prospects in various complex environments. The key points calculated using the SIFT algorithm are used as SIFT key points. ORB is an algorithm for feature point detection and description, combining the advantages of FAST (an efficient corner point detection algorithm) and BRIEF (an algorithm for image feature description), and has been improved to have rotation invariance and certain scale invariance. ORB has a wide range of application prospects in applications with high real-time requirements, such as image matching, object recognition, SLAM, and augmented reality. The key points calculated using the ORB algorithm are used as ORB key points. Preferably, SIFT key points and / or the ORB key points can be used as two-dimensional feature points in the image.

[0066] In this possible implementation, by using a preferred algorithm to calculate key points and obtain two-dimensional feature points in the image, rotation invariance and scale invariance can be guaranteed, the robustness of the calibration result can be improved, and the calibrated camera can be more widely used in different application scenarios.

[0067] In a possible implementation, obtaining the intrinsic parameters of the camera can be performed according to the following steps: obtaining multiple images including a target calibration plate; the multiple images have different shooting angles or shooting positions; identifying the corner points of the multiple images to obtain the image coordinates of the corner points; and calculating the intrinsic parameters of the camera using the world coordinates of the corner points and the image coordinates.

[0068] In this possible implementation, the type of the target calibration plate can be selected according to actual needs, for example, it can be a checkerboard calibration plate, and the size of the checkerboard and the size of each square need to be known. Take multiple images containing the calibration plate from different angles and positions. Ensure that the calibration plate has sufficient viewing angle changes in each image to improve the accuracy of the calibration. Use image processing methods to detect the checkerboard corners in each image, and then optimize the detected corners for sub-pixel accuracy.

[0069] The actual physical position of the checkerboard corners is defined as a point in the world coordinate system. The corner data detected in all images and the corresponding points in the world coordinate system are stored. After that, the camera is calibrated and the intrinsic parameters of the camera are calculated.

[0070] In this possible implementation, through the above steps, the internal parameters of the camera (focal length, principal point and distortion coefficient) can be calculated using Zhang Zhengyou's method. These parameters are very important for correcting image distortion, 3D reconstruction and other computer vision tasks.

[0071] In a possible implementation, the adjustment of the camera's posture may be performed according to the following steps: determining an adjustment direction and an adjustment angle of the camera according to the angle; and adjusting the camera's posture according to the adjustment direction and the adjustment angle until the angle is less than a preset angle threshold.

[0072] In an embodiment of the present application, determining the adjustment direction and adjustment angle of the camera according to the angle may include determining the current angle, determining the target angle according to the current angle, and if the current angle is not equal to the target angle, calculating the required rotation matrix to obtain the adjustment direction and adjustment angle of the camera, adjusting the position and posture of the camera according to the adjustment direction and the adjustment angle, until the angle is less than a preset angle threshold, and verifying the adjustment effect. It should be noted that the preset angle threshold can be set according to actual needs, for example, it can be an angle value in the range of 0.4 degrees to 0.6 degrees. Preferably, the preset angle threshold can be set to 0.5 degrees.

[0073] In this possible implementation, the adjustment direction and adjustment angle of the camera are determined according to the angle; the position of the camera is adjusted according to the adjustment direction and the adjustment angle, so that the relative position relationship between the camera and the reference plane can be optimized. This adjustment can improve the viewing angle coverage, image quality, depth perception, feature matching and system robustness, and is of great significance in applications such as stereo vision, industrial inspection and robot navigation. Through reasonable geometric transformation and posture adjustment, the performance and reliability of the system can be significantly improved. By setting an angle threshold, the operation accuracy can be further improved.

[0074] In a possible implementation, after calculating the angle between the normal vector and the optical axis vector of the camera, the following steps may also be performed: calculating a first distance between the origin of the camera coordinate system and the origin of the world coordinate system; calculating a second distance between the camera and the reference plane using the angle and the first distance; and adjusting the camera's posture according to the second distance.

[0075] In the embodiment of the present application, the cosine value of the angle is calculated, and the product of the first distance and the cosine value is calculated to obtain the distance value, that is, the second distance between the camera and the reference plane is calculated using the angle and the first distance. The position of the camera is adjusted according to the second distance, so as to obtain a higher-precision distance between the camera and the reference plane.

[0076] In this possible implementation, the accuracy of the distance between the camera and the reference plane can be improved, which can improve measurement accuracy, enhance depth perception, optimize image correction, improve the reliability of feature matching, improve robot navigation and increase the degree of automation. In the fields of industrial inspection, stereo vision, robot navigation, medical imaging and autonomous driving, this improvement in accuracy can significantly improve the performance and reliability of the system, bringing higher efficiency and better user experience.

[0077] See also Figure 2 The implementation flow chart of the reference plane calibration method for a camera is shown, and the reference plane calibration method for the camera is described below in combination with a possible specific implementation method.

[0078] refer to Figure 2 Part A in the paper describes the process of obtaining the camera intrinsic parameters. The input data is the RGB data (sequence) containing the calibration plate. First, based on the image feature points (SIFT, ORB, etc.) and the corresponding real coordinate values ​​of the corner points on the calibration plate (generally, the calibration plate is a plane, and a real-scale coordinate system with Z=0 is established on the plane of the calibration plate, and all the corner points of the calibration plate are encoded); the calibration plate is moved and photographed multiple times, and the camera intrinsic parameters are obtained using Zhang Zhengyou's method.

[0079] refer to Figure 2 Part A and Part B in the figure describe the process of obtaining the calibration plate pose. The input data is RGB data (single image is sufficient) and internal parameters. Figure 3 , fix the calibration plate to the wall, shoot the RGB data containing the calibration plate, and based on the RGB data, obtain the corner points corresponding to the calibration plate; based on the calibrated internal parameters, the PNP method can be used to obtain the pose (translation matrix + rotation matrix) of the calibration plate to the camera coordinate system.

[0080] refer to Figure 2 Part C of the paper describes the process of obtaining angles and distances. Ideally, Figure 4 , Figure 4 It is a cross-sectional view of the horizontal line including the camera optical center and the camera principal point. The downward arrow indicates the direction of the camera optical center, recorded as the spatial straight line LINE Camera_in_camera , the upward arrow indicates the vertical direction of the calibration plate, recorded as the space line LINE board_in_world Ideally, LINE Camera_in_camera and LINE board_in_world The angle between LINE and LINE should be 0 degrees. However, in actual operation, this situation cannot be achieved. Through the above external parameters, LINE board_in_world Convert to the camera coordinate system, i.e. LINE board_in_cameraIn the same coordinate system, if two spatial lines are known, the angle between the two spatial lines can be obtained; similarly, the projection of the distance from the origin of the calibration plate coordinate system to the origin of the camera coordinate system in the direction of the camera optical axis can be obtained, that is, the distance from the calibration plate to the camera coordinate system.

[0081] In engineering operations, fix the calibration plate on the wall, change the direction of the camera according to the above angle feedback (the camera can be fixed on a tripod), and record the current distance when the angle is less than 0.5 degrees. This method can ensure the perpendicularity between the reference plane and the camera optical axis, and obtain a high-precision distance (less than 0.5 ten thousandths). From the trigonometric function, we can know that the error of 0.5 degrees is 3.807693582869032e-05, that is, the error is less than 0.5 ten thousandths. The calculation process is as follows:

[0082]

[0083] In the embodiments of the present application, the existing camera mathematical model (including the camera's intrinsic and extrinsic parameters) can be fully utilized, and the image corner points can be obtained based on the visual algorithm to achieve accurate measurement of angles and distances; the PNP algorithm based on the image corner points and spatial 3D points can be used to set thresholds to further improve the operation accuracy; based on the extrinsic parameters obtained by PNP and the actual size of the calibration plate, the distance from the reference plane to the camera optical center and the angle between the reference plane and the camera optical axis can be obtained; based on the angle and distance obtained by the visual algorithm, the camera position can be adjusted to guide the adjustment, and finally the optimal reference plane can be obtained to complete the camera calibration at low cost and high precision.

[0084] The present application also provides a three-dimensional reconstruction method. Figure 5 The flowchart of a three-dimensional reconstruction method according to an embodiment of the present application is shown. The method may include:

[0085] Step 501, using a camera to acquire an image; wherein the camera is calibrated using the above-mentioned camera reference plane calibration method.

[0086] Step 502: Perform three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result.

[0087] In the embodiments of the present application, 3D reconstruction refers to reconstructing the 3D shape and structure of an object or scene from a 2D image or other data. Performing 3D reconstruction based on aligned pose data can improve the reconstruction accuracy of the reconstruction result, simplify the calculation process, and improve robustness and reconstruction efficiency.

[0088] The present application provides a three-dimensional reconstruction method, which includes: acquiring an image using a camera; wherein the camera is calibrated using the reference plane calibration method of the camera described above; and performing three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result. The reference plane calibration method of the camera can fit the reference plane through the three-dimensional point data of the reference plane in the world coordinate system, determine the normal of the reference plane, and convert the normal to the camera coordinate system using the external parameters of the camera to obtain the normal vector, so that the normal vector and the optical axis vector are both located in the camera coordinate system, so as to calculate the angle between the two, and then adjust the camera's posture according to the angle, so as to obtain a more accurate adjustment result, so that the optical axis direction of the camera is closer to perpendicular to the reference plane, reduce the error caused by human judgment, and ensure the stability of the calibration result of the reference plane of the camera.

[0089] Corresponding to the application scenario and method of the reference plane calibration method for a camera provided in the embodiment of the present application, the embodiment of the present application also provides a reference plane calibration device for a camera. Figure 6 The figure is a structural block diagram of a reference plane calibration device for a camera according to an embodiment of the present application, and the device may include:

[0090] The fitting module 601 is used to determine the normal of the reference plane based on the three-dimensional point data of the reference plane; the conversion module 602 is used to convert the normal to the camera coordinate system using the external parameters of the camera to obtain the normal vector; the adjustment module 603 is used to adjust the posture of the camera based on the angle between the normal vector and the optical axis vector of the camera.

[0091] In an embodiment of the present application, three-dimensional point data on a reference plane can be obtained, and fitting data of the reference plane can be obtained based on the three-dimensional point data. The fitting data can be a set of three-dimensional point data (which can be experimental data or measurement data), and these points are distributed near a reference plane.

[0092] Preferably, the image of the reference plane can be processed to obtain three-dimensional point data on the reference plane, and the three-dimensional point data can be converted into a form suitable for linear regression to obtain matrix and vector data. After that, the least squares problem is solved, and the parameters of the plane are solved using linear algebra methods. Finally, the normal vector is extracted from the parameters of the plane equation, that is, the normal of the reference plane is determined based on the fitting data. In an embodiment of the present application, the normal of the reference plane can be represented in the form of a normal vector, which is located in the world coordinate system.

[0093] In this step, by fitting the reference plane, fitting data is obtained. Based on the fitting data, the normal of the reference plane is calculated, which can more accurately determine the position and posture of the calibration plate in three-dimensional space, thereby improving the overall accuracy of the calibration. In addition, the normal provides additional geometric constraints. In the calibration process, more geometric constraints usually mean more stable and more accurate calibration results. The normal of the reference plane can be used as an additional constraint to help optimize the calibration parameters and reduce errors.

[0094] In an embodiment of the present application, an external parameter is used to characterize the conversion relationship between the camera coordinate system and the world coordinate system. The normal is converted to the camera coordinate system using the external parameter to obtain the normal vector of the camera coordinate system, so that the normal of the reference plane can be represented by the normal vector in the camera coordinate system. The optical axis vector is determined in the camera coordinate system. The optical axis vector is a vector used to describe the direction of the optical axis of the camera. In the camera coordinate system, the optical axis is usually defined as the normal direction from the center point of the camera lens (i.e., the origin of the camera) to the imaging plane. Therefore, the normal vector and the optical axis vector are both in the camera coordinate system, which is convenient for calculating the angle between the normal vector and the optical axis vector of the camera.

[0095] In this step, the angle between the normal vector and the optical axis vector of the camera can be calculated to determine whether the optical axis of the camera is perpendicular to the reference plane. By calculating the angle, the positional relationship between the optical axis of the camera and the reference plane can be effectively determined.

[0096] In an embodiment of the present application, after the angle is determined, the position and posture of the camera can be accurately adjusted so that the optical axis of the camera and the reference plane are closer to a vertical relationship, thereby improving the accuracy and efficiency of the reference plane calibration.

[0097] The present application provides a reference plane calibration device for a camera, which includes: a fitting module for determining the normal of the reference plane based on the three-dimensional point data of the reference plane; a conversion module for converting the normal to the camera coordinate system using the external parameters of the camera to obtain a normal vector; and an adjustment module for adjusting the position and posture of the camera based on the angle between the normal vector and the optical axis vector of the camera. According to an embodiment of the present application, the reference plane can be fitted by the three-dimensional point data of the reference plane in the world coordinate system to determine the normal of the reference plane, and the normal can be converted to the camera coordinate system using the external parameters of the camera to obtain the normal vector, so that both the normal vector and the optical axis vector are located in the camera coordinate system, so as to calculate the angle between the two, and then adjust the position and posture of the camera according to the angle, so as to obtain a more accurate adjustment result, so that the optical axis direction of the camera is closer to perpendicular to the reference plane, reduce the error caused by human judgment, and ensure the stability of the calibration result of the reference plane of the camera.

[0098] In a possible implementation, the device further includes a computing module for acquiring an image captured by the camera and an intrinsic parameter of the camera, wherein the image includes the reference plane; and calculating an extrinsic parameter of the camera based on the image and the intrinsic parameter; wherein the extrinsic parameter is used to characterize a conversion relationship between the camera coordinate system and the world coordinate system.

[0099] In an embodiment of the present application, the camera and the reference plane can be set at a certain angle to ensure that the reference plane falls within the range that the camera can capture. The reference plane includes information that can be used to calibrate the camera to ensure the accuracy and reliability of the calibration process. For example, the reference plane may include but is not limited to the following characteristics: clear geometric patterns, high contrast, multi-view coverage, sufficient resolution and clarity, no distortion or minimized distortion, and uniform lighting. Camera intrinsic parameters are parameters that describe the internal characteristics of the camera imaging system, and may include but are not limited to focal length, principal point position, and distortion coefficient. These parameters are determined in advance through the intrinsic calibration process, and are used to realize the conversion of points in the three-dimensional world coordinate system and points in the two-dimensional image coordinate system, thereby achieving accurate image projection and geometric correction.

[0100] It should be noted that in the embodiment of the present application, the camera used may be a structured light 3D camera, which projects known patterns (such as stripes, grids or dots) onto the surface of an object, and then uses the camera to capture the deformation of these patterns on the surface of the object, thereby calculating the three-dimensional shape of the object.

[0101] The camera's internal parameters can be calibrated in advance. In the embodiment of the present application, the pre-calibrated internal parameters are directly obtained to improve the execution efficiency of the embodiment of the present application. When pre-calibrating the internal parameters, the appropriate internal parameter calibration method selected depends on the specific application scenario and equipment conditions. The internal parameter calibration methods that can be used include but are not limited to: checkerboard calibration method, calibration method based on dot array, method based on three-dimensional calibration object, and direct linear transformation method.

[0102] In this step, by acquiring the image including the reference plane taken by the camera and the internal parameters of the camera, various data required for the subsequent reference plane calibration of the camera are obtained to ensure the efficiency of the reference plane calibration.

[0103] In the embodiment of the present application, the extrinsic parameters of the camera are parameters that describe the position and orientation of the camera in three-dimensional space. The extrinsic parameters define the transformation relationship between the world coordinate system and the camera coordinate system, and may generally include a rotation matrix and a translation vector. By obtaining the image, the pixel points in the image can be obtained, and the extrinsic parameters of the camera can be calculated based on the information of the pixel points and the obtained camera intrinsic parameters. In the embodiment of the present application, the camera coordinate system may be a three-dimensional coordinate system with the optical center (optical center) of the camera as the origin. The origin of the world coordinate system may be located on a reference plane.

[0104] In this step, the camera's extrinsic parameters are calculated based on the image and the camera's intrinsic parameters, so that the conversion relationship between the camera coordinate system and the world coordinate system can be obtained, so that the information belonging to the two coordinate systems can be unified into the same coordinate system, which is convenient for data analysis and calculation.

[0105] In a possible implementation, the reference plane is a calibration plate, and the calculation module is specifically used to: obtain dimensional data of the calibration plate; identify two-dimensional feature points in the image, and convert the two-dimensional feature points into three-dimensional feature points using the intrinsic parameters; and calculate the external parameters of the camera based on the two-dimensional feature points, the three-dimensional feature points and the dimensional data.

[0106] In an embodiment of the present application, the calibration plate is a tool for camera calibration, which is mainly used to determine the intrinsic and extrinsic parameters of the camera. Camera calibration is an important step in computer vision. Through calibration, the focal length, principal point, distortion coefficient and other parameters of the camera can be determined, so as to correct the geometric distortion in the image and accurately measure the three-dimensional position of the object. The reference plane may include, but is not limited to, any of the following types of calibration plates: a checkerboard calibration plate, which is composed of black and white square grids. The corner points of the checkerboard calibration plate (i.e., the intersection of the black and white grids) are used as feature points for calibration. Figure 3 A schematic diagram of a possible calibration plate is shown; a dot calibration plate, which is composed of regularly arranged dots, the centers of which are used as feature points. In some cases, the dot calibration plate is easier to detect and match than the checkerboard calibration plate; a ring calibration plate, which is composed of regularly arranged ring patterns, the centers of which are used as feature points. This calibration plate has advantages in certain specific applications. The steps of using a calibration plate for camera calibration include preparing the calibration plate, taking calibration images, detecting feature points, calculating camera parameters, and verification and optimization.

[0107] In an embodiment of the present application, the dimension data of the calibration plate is used to describe the dimensions of the calibration plate itself. For example, in the case where the calibration plate is a checkerboard calibration plate, the dimension data of the calibration plate includes but is not limited to the actual distance between the corner points, which can be obtained by measurement. In the case where the calibration plate is a checkerboard calibration plate, identifying the two-dimensional feature points in the image may include identifying the corner points of the checkerboard in the image, calculating the two-dimensional feature points corresponding to the corner points in combination with the intrinsic parameters of the camera to obtain the three-dimensional feature points corresponding to the corner points, and then calculating the external parameters of the camera based on the two-dimensional feature points, the three-dimensional feature points and the dimension data. The camera external parameters calculated in this way can be aligned with the actual size.

[0108] In this possible implementation, using real-size data to align camera extrinsics can significantly improve the accuracy of positioning and measurement, enhance the effect of virtual-real fusion, ensure the accuracy of 3D reconstruction, and improve the collaborative working ability of multi-camera systems. This is of great significance for application scenarios that require high precision and high reliability, such as robot navigation, augmented reality, 3D reconstruction, and industrial automation.

[0109] By using real-size data, the position and orientation of the camera relative to the object or scene can be determined more accurately. This is especially important for applications that require precise positioning, such as robot navigation, autonomous driving, and industrial automation. Real-size data can reduce positioning errors caused by measurement errors or inaccurate data, thereby improving the accuracy of the overall calibration. In augmented reality applications, virtual objects need to be accurately superimposed on real-world scenes. Using real-size data to align camera extrinsics can ensure that virtual objects are displayed in the correct position and scale, providing a more realistic experience. Real-size data ensures that the size and scale of virtual objects are consistent with the real world, avoiding the problem of scale distortion. In 3D reconstruction applications, using real-size data to align camera extrinsics can generate more accurate 3D models. This is very important for applications in the fields of architecture, cultural relics protection, and medical imaging. Real-size data ensures that the reconstructed 3D model has the correct scale information, which is crucial for subsequent analysis and processing. In a multi-camera system, using real-size data can ensure that the extrinsics between each camera are accurately aligned, thereby improving the ability of the entire system to work together. Real-size data can reduce errors in the camera calibration process and improve the overall accuracy and stability of the multi-camera system.

[0110] In industrial automation, robots need to perform precise operations based on visual data from cameras. Using real-size data to align camera extrinsics ensures that robots can accurately grasp, assemble, and manipulate objects. Accurate extrinsic alignment can optimize production processes, reduce production errors and waste caused by inaccurate positioning, and improve production efficiency and quality. In measurement and monitoring applications, using real-size data can ensure the accuracy of measurement results. For example, in building measurement, topographic surveying, and medical imaging, accurate extrinsic alignment can provide more reliable measurement data. In real-time monitoring systems, aligning camera extrinsics using real-size data can provide more accurate monitoring information, helping to detect and deal with problems in a timely manner.

[0111] In a possible implementation, the calculation module is specifically used to: use the two-dimensional feature points, the three-dimensional feature points and the intrinsic parameters as input parameters of a PNP algorithm, and use the PNP algorithm to calculate the initial extrinsic parameters of the camera; and use the size data to adjust the initial extrinsic parameters to obtain the extrinsic parameters of the camera.

[0112] In this possible implementation, it should be noted that the PNP (Perspective-N-Point) algorithm is a technology for calculating camera extrinsics (i.e., the camera's posture, including rotation and translation). It uses known 3D points and their 2D projection points in the image to more accurately infer the position and direction of the camera. In an embodiment of the present application, after obtaining two-dimensional feature points and three-dimensional feature points, the PNP method can be used to calculate the initial extrinsic parameters of the camera based on the camera intrinsic parameters, two-dimensional feature points, and three-dimensional feature points. The initial extrinsic parameters are not aligned with the actual size, so the size data can be reused to align the calculated extrinsic parameters with the actual size.

[0113] It should be noted that in some scenarios, the initial external parameters of the camera do not need to be adjusted, and the initial external parameters of the camera can be directly used as the external parameters of the camera. These scenarios include but are not limited to the following situations:

[0114] Image registration and stitching: In image registration and stitching, the main goal is to align multiple images in the same coordinate system, not necessarily to the true scale. In this case, the relative relationship of the extrinsic parameters is more important than the absolute scale. For example, the stitching of panoramic images.

[0115] Relative positioning: In some applications, we only need to know the relative position and orientation between cameras, without the absolute real scale. Relative alignment of external parameters can meet the requirements. For example, relative pose estimation in a multi-camera system.

[0116] Computer vision algorithm testing and verification: When developing and testing computer vision algorithms, you may only need to verify the correctness and stability of the algorithm without aligning it to the true scale. In this case, relative alignment of external parameters is sufficient. For example, testing of image feature matching algorithms.

[0117] Applications in virtual environments: In some virtual reality (VR) or computer-generated imagery (CGI) applications, external parameters do not need to be aligned with real scale, because the entire environment is virtual and the proportions and scales can be set arbitrarily. For example, the creation of virtual game scenes.

[0118] In a possible implementation, the calculation module is specifically used to: obtain SIFT key points and / or ORB key points of the image; and use the SIFT key points and / or the ORB key points as two-dimensional feature points in the image.

[0119] In the embodiment of the present application, SIFT is an algorithm for detecting and describing local features of an image. The SIFT algorithm provides a powerful and robust tool for tasks such as image matching, object recognition, and three-dimensional reconstruction by detecting and describing local feature points in an image. Its scale invariance and rotation invariance make it have a wide range of application prospects in various complex environments. The key points calculated using the SIFT algorithm are used as SIFT key points. ORB is an algorithm for feature point detection and description, combining the advantages of FAST (an efficient corner point detection algorithm) and BRIEF (an algorithm for image feature description), and has been improved to have rotation invariance and certain scale invariance. ORB has a wide range of application prospects in applications with high real-time requirements, such as image matching, object recognition, SLAM, and augmented reality. The key points calculated using the ORB algorithm are used as ORB key points. Preferably, SIFT key points and / or the ORB key points can be used as two-dimensional feature points in the image.

[0120] In this possible implementation, by using a preferred algorithm to calculate key points and obtain two-dimensional feature points in the image, rotation invariance and scale invariance can be guaranteed, the robustness of the calibration result can be improved, and the calibrated camera can be more widely used in different application scenarios.

[0121] In one possible implementation, the computing module is specifically used to: acquire multiple images including a target calibration plate; the multiple images have different shooting angles or shooting positions; identify corner points of the multiple images to obtain image coordinates of the corner points; and calculate the intrinsic parameters of the camera using the world coordinates of the corner points and the image coordinates.

[0122] In this possible implementation, the type of the target calibration plate can be selected according to actual needs, for example, it can be a checkerboard calibration plate, and the size of the checkerboard and the size of each square need to be known. Take multiple images containing the calibration plate from different angles and positions. Ensure that the calibration plate has sufficient viewing angle changes in each image to improve the accuracy of the calibration. Use image processing methods to detect the checkerboard corners in each image, and then optimize the detected corners for sub-pixel accuracy.

[0123] The actual physical position of the checkerboard corners is defined as a point in the world coordinate system. The corner data detected in all images and the corresponding points in the world coordinate system are stored. After that, the camera is calibrated and the intrinsic parameters of the camera are calculated.

[0124] In this possible implementation, through the above steps, the internal parameters of the camera (focal length, principal point and distortion coefficient) can be calculated using Zhang Zhengyou's method. These parameters are very important for correcting image distortion, 3D reconstruction and other computer vision tasks.

[0125] In a possible implementation, the adjustment module is specifically used to: determine an adjustment direction and an adjustment angle of the camera according to the angle; and adjust the position and posture of the camera according to the adjustment direction and the adjustment angle until the angle is less than a preset angle threshold.

[0126] In an embodiment of the present application, determining the adjustment direction and adjustment angle of the camera according to the angle may include determining the current angle, determining the target angle according to the current angle, and if the current angle is not equal to the target angle, calculating the required rotation matrix to obtain the adjustment direction and adjustment angle of the camera, adjusting the position and posture of the camera according to the adjustment direction and the adjustment angle, until the angle is less than a preset angle threshold, and verifying the adjustment effect. It should be noted that the preset angle threshold can be set according to actual needs, for example, it can be an angle value in the range of 0.4 degrees to 0.6 degrees. Preferably, the preset angle threshold can be set to 0.5 degrees.

[0127] In this possible implementation, the adjustment direction and adjustment angle of the camera are determined according to the angle; the position of the camera is adjusted according to the adjustment direction and the adjustment angle, so that the relative position relationship between the camera and the reference plane can be optimized. This adjustment can improve the viewing angle coverage, image quality, depth perception, feature matching and system robustness, and is of great significance in applications such as stereo vision, industrial inspection and robot navigation. Through reasonable geometric transformation and posture adjustment, the performance and reliability of the system can be significantly improved. By setting an angle threshold, the operation accuracy can be further improved.

[0128] In a possible implementation, the device also includes a distance module, which is used to: calculate a first distance between the origin of the camera coordinate system and the origin of the world coordinate system; calculate a second distance between the camera and the reference plane using the angle and the first distance; and adjust the position and posture of the camera according to the second distance.

[0129] In the embodiment of the present application, the cosine value of the angle is calculated, and the product of the first distance and the cosine value is calculated to obtain the distance value, that is, the second distance between the camera and the reference plane is calculated using the angle and the first distance. The position of the camera is adjusted according to the second distance, so as to obtain a higher-precision distance between the camera and the reference plane.

[0130] In this possible implementation, the accuracy of the distance between the camera and the reference plane can be improved, which can improve measurement accuracy, enhance depth perception, optimize image correction, improve the reliability of feature matching, improve robot navigation and increase the degree of automation. In the fields of industrial inspection, stereo vision, robot navigation, medical imaging and autonomous driving, this improvement in accuracy can significantly improve the performance and reliability of the system, bringing higher efficiency and better user experience.

[0131] Corresponding to the application scenario and method of the three-dimensional reconstruction method provided in the embodiment of the present application, the embodiment of the present application also provides a three-dimensional reconstruction device. Figure 7 The figure is a structural block diagram of a 3D reconstruction device according to an embodiment of the present application, and the device may include:

[0132] The image module 701 is used to acquire an image using a camera; wherein the camera is calibrated using the above-mentioned camera reference plane calibration method; and the reconstruction module 702 is used to perform three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result.

[0133] In the embodiments of the present application, 3D reconstruction refers to reconstructing the 3D shape and structure of an object or scene from a 2D image or other data. Performing 3D reconstruction based on aligned pose data can improve the reconstruction accuracy of the reconstruction result, simplify the calculation process, and improve robustness and reconstruction efficiency.

[0134] The present application provides a three-dimensional reconstruction device, which includes: an image module for acquiring an image using a camera; wherein the camera is calibrated using the reference plane calibration method of the camera described above; and a reconstruction module for performing three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result. The reference plane calibration method of the camera can fit the reference plane through the three-dimensional point data of the reference plane in the world coordinate system, determine the normal of the reference plane, and convert the normal to the camera coordinate system using the external parameters of the camera to obtain the normal vector, so that the normal vector and the optical axis vector are both located in the camera coordinate system, so as to calculate the angle between the two, and then adjust the camera's posture according to the angle, so as to obtain a more accurate adjustment result, so that the optical axis direction of the camera is closer to perpendicular to the reference plane, reduce the error caused by human judgment, and ensure the stability of the calibration result of the reference plane of the camera.

[0135] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0136] Figure 8 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 8As shown, the electronic device includes: a memory 801 and a processor 802. The memory 801 stores a computer program that can be run on the processor 802. When the processor 802 executes the computer program, the method in the above embodiment is implemented. The number of the memory 801 and the processor 802 can be one or more.

[0137] The electronic device also includes:

[0138] The communication interface 803 is used to communicate with external devices and perform data exchange transmission.

[0139] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the memory 801, the processor 802 and the communication interface 803 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0140] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.

[0141] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0142] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0143] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0144] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0145] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0146] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0147] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0148] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0149] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0150] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0151] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0152] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0153] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0154] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A reference plane calibration method for a camera, wherein: include: Determining a normal line of the reference plane based on the three-dimensional point data of the reference plane; The normal is converted into a camera coordinate system using the camera's extrinsic parameters to obtain a normal vector; The position and posture of the camera is adjusted based on an angle between the normal vector and an optical axis vector of the camera.

2. The method according to claim 1, wherein: Before converting the normal line to the camera coordinate system using the external parameters of the camera to obtain the normal vector, the method further includes: Acquire an image captured by the camera and an internal parameter of the camera, wherein the image includes the reference plane; The external parameters of the camera are calculated according to the image and the internal parameters; wherein the external parameters are used to characterize the conversion relationship between the camera coordinate system and the world coordinate system.

3. The method according to claim 2, wherein: The reference plane is a calibration plate, and the step of calculating the external parameters of the camera according to the image and the internal parameters includes: Acquiring dimension data of the calibration plate; Identifying two-dimensional feature points in the image, and converting the two-dimensional feature points into three-dimensional feature points using the intrinsic parameters; The external parameters of the camera are calculated according to the two-dimensional feature points, the three-dimensional feature points and the size data.

4. The method according to claim 3, wherein: Calculating the external parameters of the camera according to the two-dimensional feature points, the three-dimensional feature points and the size data includes: The two-dimensional feature points, the three-dimensional feature points and the intrinsic parameters are used as input parameters of a PNP algorithm, and the initial extrinsic parameters of the camera are calculated using the PNP algorithm; The initial extrinsic parameters are adjusted using the size data to obtain the extrinsic parameters of the camera.

5. The method according to claim 3, wherein: The identifying of the two-dimensional feature points in the image includes: Obtaining SIFT key points and / or ORB key points of the image; The SIFT key points and / or the ORB key points are used as two-dimensional feature points in the image.

6. The method according to claim 2, wherein: Get the internal parameters of the camera, including: Acquire multiple images including a target calibration plate; the multiple images are taken at different angles or positions; Identify corner points of the multiple images to obtain image coordinates of the corner points; The intrinsic parameters of the camera are calculated using the world coordinates of the corner points and the image coordinates.

7. The method according to claim 1, wherein: The adjusting the position and posture of the camera includes: Determining the adjustment direction and adjustment angle of the camera according to the angle; The position and posture of the camera are adjusted according to the adjustment direction and the adjustment angle until the angle is less than a preset angle threshold.

8. The method according to any one of claims 2 to 5, wherein: Also includes: Calculating a first distance between the origin of the camera coordinate system and the origin of the world coordinate system; Calculating a second distance between the camera and the reference plane using the angle and the first distance; The position and posture of the camera is adjusted according to the second distance.

9. A three-dimensional reconstruction method, wherein: include: Acquire an image using a camera; wherein the camera is calibrated using the method described in any one of claims 1 to 8; Three-dimensional reconstruction is performed based on the image to obtain a three-dimensional reconstruction result.

10. A reference plane calibration device for a camera, wherein: include: A fitting module, used for determining a normal line of the reference plane based on the three-dimensional point data of the reference plane; A conversion module, used to convert the normal line into a camera coordinate system using an external parameter of the camera to obtain a normal line vector; An adjustment module is used to adjust the position and posture of the camera based on the angle between the normal vector and the optical axis vector of the camera.

11. A three-dimensional reconstruction device, wherein: include: An image module, used to acquire an image using a camera; wherein the camera is calibrated using the method described in any one of claims 1 to 8; The reconstruction module is used to perform three-dimensional reconstruction based on the image to obtain a three-dimensional reconstruction result.

12. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

13. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images