Three-dimensional reconstruction method and device, electronic equipment and storage medium

Abstract

The application discloses a three-dimensional reconstruction method and device, electronic equipment and storage medium, and belongs to the technical field of image acquisition. The three-dimensional reconstruction method comprises the following steps: image acquisition of a code element is performed by using multiple cameras with different visual angles, and a code element image is obtained; a reference object for camera calibration is the code element; the code element comprises multiple markers, and each marker has a corresponding identification number; identification numbers of the markers in the code element image are identified, and world coordinates corresponding to the markers are searched by using the identification numbers; an extrinsic parameter matrix of the camera is calculated according to marker coordinates of a camera coordinate system and the marker coordinates of the world coordinates, and point clouds are unified to the world coordinate system to obtain multiple point clouds with different visual angles; and the multiple point clouds are spliced to obtain a three-dimensional reconstruction image. The method converges faster and has higher precision, and the process is simple and easy to operate and implement.

Description

Technical Field

[0001] This application belongs to the field of image acquisition technology, specifically relating to a three-dimensional reconstruction method, device, electronic device, and storage medium. Background Technology

[0002] In recent years, 3D reconstruction has become a hot topic in the fields of computer vision and computer graphics. This technology has been widely used in reverse engineering, CT imaging, artifact display, mixed reality, and other fields. In 2019, Microsoft officially released the next-generation Azure-Kinect depth sensor. This device calculates distance based on the Time-of-Flight (TOF) method. Compared with the previous two generations of Kinect depth sensors, its accuracy, field of view, and resolution have been further improved. The portable and low-cost hardware has attracted more and more scholars to devote themselves to 3D reconstruction technology research.

[0003] Currently, most camera extrinsic calibration is based on calibration objects. This type of method is theoretically mature, simple to operate, and highly accurate. However, factors such as cumbersome image acquisition process, complex calibration object manufacturing, and overlapping field of view limitations greatly restrict the application scenarios of this type of method. Summary of the Invention

[0004] The purpose of this application is to provide a three-dimensional reconstruction method, apparatus, electronic device, and storage medium to solve the problem of the cumbersome and complex existing three-dimensional reconstruction process.

[0005] According to a first aspect of the embodiments of this application, a three-dimensional reconstruction method is provided, the method comprising:

[0006] Multiple cameras from different perspectives are used to acquire images of the code elements to obtain code element images. The reference objects calibrated by the cameras are the code elements. Each code element includes multiple markers, and each marker has a corresponding identification number.

[0007] Identify the identifiers marked in the symbol image and use the identifiers to search for the world coordinates corresponding to the markers;

[0008] Based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, the extrinsic parameter matrix of the camera is calculated, and the point cloud is unified under the world coordinate system to obtain multiple point clouds from different perspectives.

[0009] Multiple point clouds are stitched together to obtain a 3D reconstructed image.

[0010] In some optional embodiments of this application, the identifier number marked in the identification symbol image includes:

[0011] The symbol image is sequentially processed by grayscale conversion and binarization to obtain a symbol binary image;

[0012] Regions of interest (ROIs) are extracted and corner points are detected in the binary image of the symbol to obtain the marker;

[0013] The identifier of a marker is determined by the black and white pixel values ​​within the coded area of ​​the marker.

[0014] In some optional embodiments of this application, ROI region extraction and corner detection are performed on the binary symbol image to obtain a marker, including:

[0015] Contour lookup is performed on the binary image of the code elements to obtain multiple contours, each of which consists of a set of continuous points;

[0016] Traverse each outline and calculate the enclosed area;

[0017] When the enclosed area is outside the preset range, it is judged as unmarked, and the polygon fitting function is called to detect the number of corner points of each contour. If the number of corner points detected in the contour is 5, and the polygon fitted by the contour is non-convex, then the polygon fitted by the contour is judged as marked.

[0018] In some optional embodiments of this application, identifying the identifier number of the marker based on the black and white pixel values ​​within the coded area of ​​the marker includes:

[0019] Calculate the total pixel value within the coded region of the marker;

[0020] The identifier is obtained by decoding the total pixel value.

[0021] In some optional embodiments of this application, the extrinsic parameter matrix of the camera is calculated based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and the point cloud is unified to the world coordinate system to obtain multiple point clouds from different viewpoints, including:

[0022] Define the translation vector at each viewpoint k. Define rotation matrix The transformation between camera coordinate system and world coordinate system is shown in the following formula:

[0023]

[0024] In the formula, These are the marked point cloud models in the camera coordinate system and the marked point cloud models in the world coordinate system, respectively; the subscript C represents the camera coordinate system and the subscript M represents the world coordinate system.

[0025] Translation vector To determine the three-dimensional corner points The calculated center coordinates of the world coordinate system;

[0026] Finding the rotation matrix using orthogonal Pluke analysis

[0027] The detection markers are used to obtain the corresponding pre-inputted position priors through the identifier numbers, and the point cloud is unified to the world coordinate system, as shown in the following formula:

[0028]

[0029] in, This is a point cloud model in the world coordinate system, where the subscript W indicates the camera coordinate system; These are the rotation parameters and translation parameters, respectively.

[0030] In some optional embodiments of this application, multiple point clouds are stitched together to obtain a three-dimensional reconstructed image, including:

[0031] Based on the Colored-ICP algorithm, point clouds from different viewpoints are stitched together; the expression for Colored-ICP is:

[0032] ε(T)=(1-δ)ε C (T)+δε G (T);

[0033] In the formula, T∈SE(3), SE(3) is the estimated transformation matrix, ε C For color items, ε G For the geometric term, δ∈[0,1] is the weight variable. The selection of the corresponding point pair (p,q)∈M is the same as that in the ICP algorithm, which minimizes the Euclidean distance between points. For the corresponding point set M, the geometric term and color term of the objective function are constructed respectively, as shown in the following formula:

[0034] ε G (T)=Σ (p,q)∈M ((Tq-p) T n p ) 2 ;

[0035] ε C (T)=Σ (p,q)∈M (C p (f(Tq))-C(q)) 2 ;

[0036] In the formula, n p Let ε be the normal vector of the corresponding point p; G Equivalent to the distance from the source point to the tangent plane of the target point; f() is a function that back-projects 3D points onto the tangent plane of the target point cloud; C p () is a function that parameterizes a virtual image, and C() is the color value at the corresponding coordinates.

[0037] In some optional embodiments of this application, after stitching together multiple point clouds to obtain a three-dimensional reconstructed image, the three-dimensional reconstruction method further includes:

[0038] The Gauss-Newton method is used to iteratively optimize the 3D reconstructed image.

[0039] According to a second aspect of the embodiments of this application, a three-dimensional reconstruction apparatus is provided, the apparatus comprising:

[0040] The symbol image acquisition module is used to acquire images of symbols using multiple cameras from different perspectives to obtain symbol images. The reference object calibrated by the camera is the symbol. The symbol includes multiple markers, and each marker has a corresponding identification number.

[0041] The recognition module is used to identify the identifiers marked in the symbol image and use the identifiers to search for the world coordinates corresponding to the markers;

[0042] The point cloud computing module is used to calculate the camera's extrinsic parameter matrix based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and to unify the point cloud into the world coordinate system to obtain multiple point clouds from different perspectives.

[0043] The stitching module is used to stitch together multiple point clouds to obtain a 3D reconstructed image.

[0044] According to a third aspect of the embodiments of this application, an electronic device is provided, which may include:

[0045] processor;

[0046] Memory used to store processor-executable instructions;

[0047] The processor is configured to execute instructions to implement the three-dimensional reconstruction method as shown in any embodiment of the first aspect.

[0048] According to a fourth aspect of the embodiments of this application, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an information processing device or a server, causes the information processing device or server to implement the three-dimensional reconstruction method as shown in any embodiment of the first aspect.

[0049] The above-mentioned technical solution of this application has the following beneficial technical effects:

[0050] The method described in this application acquires symbol images using multiple cameras from different perspectives, obtaining symbol images. The reference objects calibrated by the cameras are the symbols, each consisting of multiple markers, each with a corresponding identifier. The method identifies the identifiers in the symbol images and uses these identifiers to search for the corresponding world coordinates. Based on the marker coordinates in the camera coordinate system and the marker coordinates in the world coordinate system, the camera's extrinsic parameter matrix is ​​calculated, and the point clouds are unified to the world coordinate system, resulting in multiple point clouds from different perspectives. These multiple point clouds are then stitched together to obtain a 3D reconstructed image. This method offers faster convergence, higher accuracy, and a simple, easy-to-implement process. Attached Figure Description

[0051] Figure 1 This is a flowchart of a three-dimensional reconstruction method in an exemplary embodiment of this application;

[0052] Figure 2 This is a flowchart of a three-dimensional reconstruction method in a specific exemplary embodiment of this application;

[0053] Figure 3 This is the experimental environment for three-dimensional reconstruction in an exemplary embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the chessboard grid marking in an exemplary embodiment of this application;

[0055] Figure 5 This is a schematic diagram of the coarse stitching effect of point clouds in an exemplary embodiment of this application;

[0056] Figure 6 This is a schematic diagram comparing the local point cloud optimization results between the algorithm of this application and two traditional algorithms;

[0057] Figure 7 This is a schematic diagram comparing the convergence speed of the Colored-ICP algorithm with two traditional ICP algorithms in an exemplary embodiment of this application;

[0058] Figure 8 This is an exemplary embodiment of the HoloLens2 display effect under different actions;

[0059] Figure 9 This is a schematic diagram of the structure of a three-dimensional reconstruction device in an exemplary embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the electronic device structure in an exemplary embodiment of this application;

[0061] Figure 11 This is a schematic diagram of the hardware structure of an electronic device in an exemplary embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0063] The accompanying drawings illustrate layer structure diagrams according to embodiments of this application. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0064] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0065] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0066] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0067] Point clouds are collections of discrete 3D coordinate points on a target surface. Due to limitations in the camera's field of view and occlusion, a single camera scan can only obtain local surface features of the object. To obtain a complete surface model, multiple cameras need to be positioned at 360 degrees, and point cloud data from different viewpoints need to be unified into the same world coordinate system by combining camera extrinsic parameters. Obtaining camera extrinsic parameters involves camera extrinsic parameter calibration, which is a crucial process in 3D reconstruction. The inventors have found that the classic Iterative Closest Point (ICP) algorithm and its variants suffer from problems such as inability to handle low-overlap point cloud data, excessive reliance on features like normal vectors and curvature, and slow speed. Therefore, this application provides a 3D reconstruction method to improve reconstruction speed.

[0068] The three-dimensional reconstruction method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0069] like Figure 1 As shown, in a first aspect of the embodiments of this application, a three-dimensional reconstruction method is provided, which may include:

[0070] Step S110: Use multiple cameras with different perspectives to acquire images of the code elements to obtain code element images. The reference object calibrated by the camera is the code element. The code element includes multiple markers, and each marker has a corresponding identification number.

[0071] Step S120: Identify the identifiers marked in the symbol image and use the identifiers to search for the world coordinates corresponding to the markers;

[0072] Step S130: Calculate the camera's extrinsic parameter matrix based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and unify the point cloud into the world coordinate system to obtain multiple point clouds from different perspectives;

[0073] Step S140: Stitch together multiple point clouds to obtain a 3D reconstructed image.

[0074] The above-described method uses the input tag ID (identifier number) and the corresponding rotation angle and translation distance in the world coordinate system as priors; it extracts and performs tag detection and ID recognition; it retrieves the corresponding transformation parameters based on the recognized ID, and combines analysis to inversely calculate the corresponding extrinsic parameter matrix, and achieves 3D reconstruction by stitching. This method converges faster, has higher accuracy, and is simple and easy to operate.

[0075] To provide a clearer explanation, the steps described above will be explained below:

[0076] The first step is S110: Multiple cameras from different perspectives are used to acquire images of the code elements to obtain code element images. The reference objects calibrated by the cameras are the code elements. The code elements include multiple markers, and each marker has a corresponding identification number.

[0077] This step involves pre-setting the code element as a reference for external parameter calibration. The code element contains multiple markers, each of which is a black pentagonal marker, meaning each marker has five corner points. The interior of each marker is the encoding area, which includes nine square regions of the same size. Each of the nine square regions corresponds to a nine-bit binary number, with each square region corresponding to one bit of binary data. The binary data corresponds to the color of the square region, with black representing 1 and white representing 0. The first four bits correspond to the marker ID, the next four bits represent the marker ID with the opposite color, and the last bit is the parity check bit.

[0078] For example, if the first four bits of a tag ID are 0101, then the last four bits of the tag ID should be 1010. If the last four bits of the tag ID are not 1010, it indicates that the tag ID is a false positive. The last bit is a parity bit, which is uniformly 1 or 0 to indicate that the tag ID is a false positive. This encoding method can reduce the false positive rate of tag detection.

[0079] Then, the camera is used to acquire images of the symbols, obtaining symbol images. At this point, the symbol images are color images, which can be preprocessed. Preprocessing can include: first, converting the color symbol images to grayscale to obtain grayscale symbol images; then, using a threshold segmentation function to binarize the grayscale symbol images to obtain binary symbol images. The binary symbol images obtained after preprocessing significantly reduce background noise, facilitating subsequent marker detection.

[0080] The next step is step S120: identify the identifiers marked in the symbol image and use the identifiers to search for the world coordinates corresponding to the markers.

[0081] This step performs contour lookup on the binary symbol image. Each contour is found to consist of a set of continuous points. For example, the contours can be stored in a Contours[] vector list. Next, each contour is traversed and its bounding area is calculated. Areas that are too small or too large are considered unmarked. Then, a polygon fitting function is called to detect the corner points of each contour. If a contour has 5 detected corner points and the polygon fitted by that contour is non-convex, then the polygon fitted by that contour is considered marked, and the coordinates of the 5 detected corner points are saved as a corner point vector. Where k represents the viewpoint corresponding to the camera, j represents the coordinates of the j-th corner point, j = 1, 2, 3, 4, 5.

[0082] After the marker detection, it was found that there were still a few "pseudo-markers" among the markers. At this time, further verification was carried out based on the detected marker ID. The marker ID recognition was performed on the detected markers. The marker ID recognition depends on the black and white pixel values ​​in the coding area of ​​the marker. Since it is time-consuming to traverse the pixel values ​​of every pixel in the entire coding area, the total pixel value of the nine square areas in the coding area of ​​the marker is calculated. The nine-bit binary number corresponding to the marker can be decoded using the total pixel value of the nine square areas in the coding area of ​​the marker, so as to obtain the marker ID.

[0083] The total pixel value of the square region is calculated using an integral image as follows: The coordinates of the four vertices of the square region are (x1, y1), (x2, y1), (x1, y2), and (x2, y2), and the pixel values ​​of the four vertices are I(x1, y1), I(x2, y1), I(x1, y2), and I(x2, y2), respectively. Therefore, the total pixel value of the square region is I(x1, y1) + I(x2, y2) - I(x2, y1) - I(x1, y2).

[0084] In practical use, the markers cannot always be directly facing the device lens, and the encoding area of ​​the markers is distorted in the pixel plane, leading to incorrect marker ID recognition. Therefore, the predefined standard corner coordinates of the markers are stored as standard corner vectors. Where k represents the viewpoint corresponding to the camera, j represents the coordinates of the j-th corner point, j = 1, 2, 3, 4, 5.

[0085] The detected corner vectors are calculated using homography transformation. With the corresponding standard corner vector The transformation matrix H, H∈R3x3, is used to perform perspective transformation, thereby correcting the original image, i.e. the symbol image captured by the camera, to the plane with standard corner coordinates.

[0086] Based on the color and depth images captured by the camera, combined with the corner vectors detected by the camera... Back-projecting to three-dimensional space yields three-dimensional corner vectors.

[0087] The next step is S130: Based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, calculate the camera's extrinsic parameter matrix, and unify the point cloud into the world coordinate system to obtain multiple point clouds from different viewpoints.

[0088] To remove outliers and reduce noise, this step uses n consecutive frames of images to calculate the mean coordinates. In this embodiment, n = 20. At the viewpoint k corresponding to the camera, a translation vector is defined. and defining the rotation matrix The transformation from the camera coordinate system to the marker coordinate system is shown in the following equation:

[0089]

[0090] In the formula, These are the marked point cloud model in the camera coordinate system and the marked point cloud model in the marker coordinate system, respectively; the subscript C represents the camera coordinate system, the subscript M represents the marker coordinate system, and k represents the viewpoint corresponding to the camera;

[0091] Translation vector It can be simplified to based on three-dimensional corner points The calculated coordinates of the mark center;

[0092] Rotation matrix The results were obtained using orthogonal Pluke analysis. To eliminate the influence of translation components, the detected three-dimensional corner points were analyzed. With predefined standard corner vectors Normalization is performed by subtracting the mean from each coordinate point, and then the corner sorting algorithm is used to establish the correspondence between the five points. The rotation matrix can thus be obtained as follows:

[0093]

[0094] According to Pruco's analysis, the above equation is equivalent to:

[0095]

[0096] In the formula, After performing SVD decomposition on M, we get M = USV T The above problem is equivalent to maximizing:

[0097]

[0098] If and only if The above equation satisfies the maximization condition, and the rotation matrix is ​​finally obtained. for:

[0099]

[0100] The spatial location of the marker is a parameter obtained through manual input. For example, in a size of 60×40×3c3m... 3 Four unique IDs are affixed to the four sides of the cardboard box. The front mark is set as the origin, and the back mark can be seen as rotating 180° around the Y-axis and translating 40cm along the Z-axis. In this way, once the camera at the viewpoint k detects the mark, the corresponding pre-input pose prior can be obtained by recognizing the mark ID, and finally the point cloud is unified to the world coordinate system.

[0101]

[0102] in, A point cloud model in the world coordinate system. These are the manually entered rotation and translation parameters, respectively.

[0103] Extrinsic parameter calibration was performed for cameras with different viewpoints, and the point clouds from each viewpoint were unified to the world coordinate system. Point clouds from the same viewpoint (i.e., the same camera) are located in the corresponding camera coordinate system. The purpose of extrinsic parameter calibration is to transform point clouds from different camera coordinate systems to the same world coordinate system for point cloud stitching.

[0104] Finally, let's introduce step S140: stitching together multiple point clouds to obtain a 3D reconstructed image.

[0105] This step can be given a set of target points P = {p} with overlapping regions. i p i ∈R 3 The source point set Q = {q}, i = 1, 2, ..., n} i |q i ∈R 3For each i = 1, 2, ..., n, the ICP algorithm minimizes the energy function. Finding the optimal match between two point clouds involves replacing the Euclidean distance between associated point pairs in ICP with the distance between the point and the tangent plane. This method requires fewer iterations and is less prone to getting trapped in local optima. However, the aforementioned ICP and its variants only utilize the geometric features of the object's surface. When encountering scenarios with fewer geometric features, such as smooth surfaces, a large number of mismatched points will be generated. Therefore, it is necessary to extend the joint optimization objective function based on the classic ICP algorithm:

[0106] ε(T)=(1-δ)ε C (T)+δε G (T);

[0107] In the formula, T∈SE(3) is the estimated transformation matrix, ε C For color items, ε G For the geometric term, δ∈[0,1] is the weight variable. The selection of the corresponding point pair (p,q)∈M is the same as that of the ICP algorithm, that is, minimizing the Euclidean distance between the points. Once the corresponding point set M is obtained, the geometric term and color term of the objective function are constructed respectively, as shown in the following formula:

[0108] ε G (T)=Σ (p,q)∈M ((Tq-p) T n p ) 2 ;

[0109] ε C (T)=Σ (p,q)∈M (C p (f(Tq))-C(q)) 2 ;

[0110] In the formula, n p Let ε be the normal vector of the corresponding point p, therefore, ε G It can be equivalent to the distance from the source point to the target point tangent plane; f() is a function that backprojects the 3D point to the target point cloud tangent plane; CP() is a function that parameterizes a virtual image, which can be represented as a continuous color function C(), which is the color value of the corresponding coordinates. Finally, the Gauss-Newton method is used for iterative optimization.

[0111] like Figure 2As shown in a specific embodiment of this application, a 3D reconstruction method is provided, which consists of two stages. The first stage is marker detection and extrinsic parameter calibration. In this stage, RGB images corresponding to multiple viewpoints are first extracted for corner detection. After perspective transformation of the extracted ROI regions, integral images are calculated to determine the marker IDs under the corresponding viewpoints. Then, the corresponding marker world coordinates are retrieved using the IDs, and the camera extrinsic parameters are calculated using orthogonal Prokopius analysis. The second stage is local registration optimization based on Colored-ICP, which jointly optimizes the color and geometric terms, resulting in faster convergence and higher accuracy in the fine registration stage. Finally, the 3D reconstruction model is transmitted to the HoloLens 2 mixed reality device in real time, achieving a good virtual interaction effect.

[0112] In this embodiment, the hardware consists of three Azure Kinect devices forming a depth acquisition system, and a PC server for processing point cloud data, such as... Figure 3 As shown, the Azure-Kinect TOF depth camera has a resolution of 1 million pixels, the RGB camera has a resolution of 12 million pixels, the working range is 0.25-5.46m, the computer system configuration is Intel(R) Core(TM) i5-9400F CPU@2.90GHz, 8.0GB RAM, the algorithm is implemented in C++ language, and the camera-side configuration API calls Azure-Kienct-SDK.

[0113] In camera extrinsic calibration experiments, the selection of calibration objects should meet the following conditions: regular shape, facilitating symbol detection; and minimal non-rigid deformation, thus affecting calibration accuracy. Based on these conditions, a rectangular cardboard box was selected as the calibration object, with A4-sized symbol markers of different IDs pasted on each side. Three devices were spaced approximately 120° apart, forming a scanning area with a radius of 1.5m. Figure 3 As shown.

[0114] Table 1 below lists the pre-input world coordinates of the markers. For comparison with the classic Zhang calibration algorithm, a checkerboard pattern of 6×9 corner points and 105×105mm size was used, and 13 sets of images were acquired. The rotation and translation parameters of the slave device 1 relative to the master device were calculated, as follows: Figure 4 As shown, Figure 4 a is the image captured by the main device. Figure 4 b is the image captured from device 1.

[0115] Table 1

[0116]

[0117] To avoid the influence of random errors, six experiments were conducted under the same conditions. Table 2 shows the results obtained by the classic Zhang's calibration algorithm and the calibration algorithm proposed in this paper under the same conditions, and the errors compared with the true values ​​measured manually. The true values ​​for translation of device 1 relative to the master device are 1.64m, 0.60m, and 2.58m, respectively; the true values ​​for rotation are 32°, 125°, and 12°. In the six experiments, the average rotation error of the classic Zhang's calibration algorithm was 1.7222°, and the average translation error was 0.2133m; while the average rotation error and average translation error of the algorithm in this embodiment are 1.6111° and 0.0583m, respectively, which are more accurate than the classic Zhang's calibration algorithm.

[0118] Table 2

[0119]

[0120] Experimental analysis shows that the advantages of the algorithm in this application are: no need to collect multiple sets of images, simplifying the calibration process; higher accuracy and shorter time; simple marker production; and applicability to low-overlap and non-overlapping scenes of camera field of view.

[0121] Figure 5 The coarsely calibrated point cloud model is presented. Due to the error introduced by manual priors, gaps exist in the point cloud model from certain perspectives, requiring further optimization.

[0122] In the fine registration experiment of point cloud stitching, due to the influence of prior human error, the 3D model obtained by coarse calibration has some misalignment. Therefore, local optimization is performed by comparing with the ICP variant algorithm. Figure 6 As can be seen, from the same perspective, the surface of the cardboard box, after rough splicing, has relatively large gaps. After iterative optimization using ICP, the model surface is smoother. Furthermore, from... Figure 6 As can be seen from the markings on the right half, the addition of color items plays a regularization role in the registration process because the Colored-ICP algorithm makes better use of RGB information, thus reducing mismatch points and achieving better registration results.

[0123] To better compare the advantages of the Colored-ICP algorithm in terms of speed and accuracy, under the same iteration stopping conditions: a maximum number of iterations of 30 and a root mean square error (RMSE) of less than 1e-6, a similarity curve was plotted. Figure 7 It can be seen that when the corresponding point set (p,q)∈M is about 30000, the Colored-ICP algorithm not only has the fastest convergence speed, requiring only 4 rounds of iteration, but also has the highest similarity, Fieness≈0.4802, which is better than the traditional ICP algorithm.

[0124] In this embodiment, a mixed reality experiment was also conducted based on a 3D reconstruction method. Remote immersion, as an emerging technology, allows users to collaborate remotely by placing a real-time generated 3D representation into a shared virtual space. As a verification experiment, this embodiment uses the TCP / IP communication protocol to upload point cloud data to the HoloLens2 in real time, and then uses the Unity3D particle system for 3D model mesh rendering. Furthermore, the RGB image resolution is downsampled to the depth image resolution, and a GZip decompression program is embedded in the HoloLens2, which further reduces transmission bandwidth pressure and improves the display frame rate.

[0125] Figure 8 This demonstrates the real-time reconstruction effect of the glasses when a human body performs different actions in a scene, from... Figure 8 It can be seen that the virtual human body model on the HoloLens 2 side maintains a high degree of synchronization with the real human body under different actions, and details such as facial expressions, finger joints, and clothing textures are clearly reconstructed; however, due to factors such as sensor sampling noise and lighting, the holes and color differences on the model surface are directions for further optimization; Table 3 lists the comparison of model data size and frame rate under different actions. Without GPU acceleration, by adding data downsampling and compression algorithms, an average display refresh rate of 25fps was achieved on the HoloLens 2 side, which basically ensures the real-time performance of the system.

[0126] Table 3

[0127]

[0128]

[0129] In some optional embodiments of this application, the identifier number marked in the identification symbol image includes:

[0130] The symbol image is sequentially processed by grayscale conversion and binarization to obtain a symbol binary image;

[0131] Regions of interest (ROIs) are extracted and corner points are detected in the binary image of the symbol to obtain the marker;

[0132] The identifier of a marker is determined by the black and white pixel values ​​within the coded area of ​​the marker.

[0133] In some optional embodiments of this application, ROI region extraction and corner detection are performed on the binary symbol image to obtain a marker, including:

[0134] Contour lookup is performed on the binary image of the code elements to obtain multiple contours, each of which consists of a set of continuous points;

[0135] Traverse each outline and calculate the enclosed area;

[0136] When the enclosed area is outside the preset range, it is judged as unmarked, and the polygon fitting function is called to detect the number of corner points of each contour. If the number of corner points detected in the contour is 5, and the polygon fitted by the contour is non-convex, then the polygon fitted by the contour is judged as marked.

[0137] In some optional embodiments of this application, identifying the identifier number of the marker based on the black and white pixel values ​​within the coded area of ​​the marker includes:

[0138] Calculate the total pixel value within the coded region of the marker;

[0139] The identifier is obtained by decoding the total pixel value.

[0140] In some optional embodiments of this application, the extrinsic parameter matrix of the camera is calculated based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and the point cloud is unified to the world coordinate system to obtain multiple point clouds from different viewpoints, including:

[0141] Define the translation vector at each viewpoint k. Define rotation matrix The transformation between camera coordinate system and world coordinate system is shown in the following formula:

[0142]

[0143] In the formula, These are the marked point cloud models in the camera coordinate system and the marked point cloud models in the world coordinate system, respectively; the subscript C represents the camera coordinate system and the subscript M represents the world coordinate system.

[0144] Translation vector To determine the three-dimensional corner points The calculated center coordinates of the world coordinate system;

[0145] Finding the rotation matrix using orthogonal Pluke analysis

[0146] The detection markers are used to obtain the corresponding pre-inputted position priors through the identifier numbers, and the point cloud is unified to the world coordinate system, as shown in the following formula:

[0147]

[0148] in, This is a point cloud model in the world coordinate system, where the subscript W indicates the camera coordinate system; These are the rotation parameters and translation parameters, respectively.

[0149] In some optional embodiments of this application, multiple point clouds are stitched together to obtain a three-dimensional reconstructed image, including:

[0150] Based on the Colored-ICP algorithm, point clouds from different viewpoints are stitched together; the expression for Colored-ICP is:

[0151] ε(T)=(1-δ)ε C (T)+δε G (T);

[0152] In the formula, T∈SE(3), SE(3) is the estimated transformation matrix, ε C For color items, ε G For the geometric term, δ∈[0,1] is the weight variable. The selection of the corresponding point pair (p,q)∈M is the same as that in the ICP algorithm, which minimizes the Euclidean distance between points. For the corresponding point set M, the geometric term and color term of the objective function are constructed respectively, as shown in the following formula:

[0153] ε G (T)=∑ (p,q)∈M ((Tq-p) T n p ) 2 ;

[0154] ε C (T)=∑ (p,q)∈M (C p (f(Tq))-C(q)) 2 ;

[0155] In the formula, n p Let ε be the normal vector of the corresponding point p; G Equivalent to the distance from the source point to the tangent plane of the target point; f() is a function that back-projects 3D points onto the tangent plane of the target point cloud; C p () is a function that parameterizes a virtual image, and C() is the color value at the corresponding coordinates.

[0156] In some optional embodiments of this application, after stitching together multiple point clouds to obtain a three-dimensional reconstructed image, the three-dimensional reconstruction method further includes:

[0157] The Gauss-Newton method is used to iteratively optimize the 3D reconstructed image.

[0158] It should be noted that the 3D reconstruction method provided in this application embodiment can be executed by a 3D reconstruction device or a control module within that 3D reconstruction device for executing the 3D reconstruction method. This application embodiment uses a 3D reconstruction device executing a 3D reconstruction method as an example to illustrate the 3D reconstruction device provided in this application embodiment.

[0159] like Figure 9 As shown, in a second aspect of the embodiments of this application, a three-dimensional reconstruction apparatus is provided, which may include:

[0160] The symbol image acquisition module 210 is used to acquire images of symbols using multiple cameras from different perspectives to obtain symbol images. The reference object calibrated by the camera is the symbol. The symbol includes multiple markers, and each marker has a corresponding identification number.

[0161] The identification module 220 is used to identify the identifier number marked in the symbol image and use the identifier number to search for the world coordinates corresponding to the mark;

[0162] The point cloud computing module 230 is used to calculate the external parameter matrix of the camera based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and to unify the point cloud into the world coordinate system to obtain multiple point clouds from different perspectives.

[0163] The stitching module 240 is used to stitch together multiple point clouds to obtain a three-dimensional reconstructed image.

[0164] The device described above uses the input tag ID (identifier number) and the corresponding rotation angle and translation distance in the world coordinate system as priors; it extracts and performs tag detection and ID recognition; it retrieves the corresponding transformation parameters based on the recognized ID, combines analysis to inversely calculate the corresponding extrinsic parameter matrix, and achieves three-dimensional reconstruction by stitching. This method converges faster, has higher accuracy, and is simple and easy to operate.

[0165] The 3D reconstruction device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0166] The three-dimensional reconstruction device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0167] The three-dimensional reconstruction device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0168] Optionally, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001, a memory 1002, and a program or instructions stored in the memory 1002 and executable on the processor 1001. When the program or instructions are executed by the processor 1001, they implement the various processes of the above-described three-dimensional reconstruction method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0169] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0170] Figure 11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0171] The electronic device 1100 includes, but is not limited to, components such as: radio frequency unit 1101, network module 1102, audio output unit 1103, input unit 1104, sensor 1105, display unit 1106, user input unit 1107, interface unit 1108, memory 1109, and processor 1110.

[0172] Those skilled in the art will understand that the electronic device 1100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 1110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 11 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0173] It should be understood that, in this embodiment, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042. The GPU 11041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 1106 may include a display panel 11061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here. The memory 1109 can be used to store software programs and various data, including but not limited to applications and operating systems. Processor 1110 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1110.

[0174] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described three-dimensional reconstruction method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0175] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0176] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described three-dimensional reconstruction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0177] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0180] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A three-dimensional reconstruction method, characterized in that, include: Multiple cameras from different perspectives are used to acquire images of the code element to obtain code element images. The reference object calibrated by the camera is the code element. The code element includes multiple markers, and each marker has a corresponding identification number. Identifying the identifier of the mark in the symbol image includes: sequentially performing grayscale processing and binarization processing on the symbol image to obtain a symbol binary image; performing ROI region extraction and corner detection on the symbol binary image to obtain the mark; and identifying the identifier of the mark based on the black and white pixel values ​​in the coding area of ​​the mark. The process of extracting the Region of Interest (ROI) and detecting corner points from the binary symbol image to obtain the marker includes: A contour query is performed on the binary image of the code elements to obtain multiple contours, each of which consists of a set of continuous points; Traverse each of the described contours and calculate the enclosed area; When the enclosed area is outside the preset range, it is determined to be unmarked, and the polygon fitting function is called to detect the number of corner points of each contour. If the number of corner points detected for the contour is 5, and the polygon fitted by the contour is non-convex, then the polygon fitted by the contour is determined to be marked. The inside of the tag is the encoding area, which includes nine square areas of the same size. Each of the nine square areas corresponds to a binary number of nine bits. Each square area corresponds to a binary number of one bit, and the binary number corresponds to the color of the square area, with black representing 1 and white representing 0. The first four bits correspond to the tag ID, the next four bits represent the tag ID with the opposite color, and the last bit is the parity bit. Use the identifier to search for the world coordinates corresponding to the marker; Based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, the extrinsic parameter matrix of the camera is calculated, and the point cloud is unified to the world coordinate system to obtain multiple point clouds from different perspectives. Multiple point clouds are stitched together to obtain a three-dimensional reconstructed image.

2. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of identifying the identifier number of the mark based on the black and white pixel values ​​within the coded area of ​​the mark includes: Calculate the total pixel value within the coded region of the mark; The identifier is obtained by decoding the total pixel value.

3. The three-dimensional reconstruction method according to claim 1, characterized in that, The process involves calculating the camera's extrinsic parameter matrix based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and unifying the point cloud to the world coordinate system to obtain multiple point clouds from different viewpoints, including: Define the translation vector at each viewpoint k. , Define the rotation matrix , The transformation between the camera coordinate system and the world coordinate system is shown in the following equation: ； In the formula, These are the marked point cloud models in the camera coordinate system and the marked point cloud models in the world coordinate system, respectively; the subscript C represents the camera coordinate system and the subscript M represents the world coordinate system. Translation vector To determine the three-dimensional corner points The calculated center coordinates of the world coordinates; Finding the rotation matrix using orthogonal Pluke analysis ; Detect the markers, obtain the corresponding pre-input position priors through the identifiers, and unify the point cloud to the world coordinate system, as shown in the following formula: ； in, This is a point cloud model in the world coordinate system, where the subscript W indicates the camera coordinate system; These are the rotation parameters and translation parameters, respectively.

4. The three-dimensional reconstruction method according to claim 1, characterized in that, The step of stitching together multiple point clouds to obtain a three-dimensional reconstructed image includes: The point clouds from different viewpoints are stitched together based on the Colored-ICP algorithm; the expression for Colored-ICP is: ； In the formula, , SE (3) is the transformation matrix to be estimated. For color items, For geometric terms, For weight variables, corresponding point pairs The selection of points is the same as in the ICP algorithm, which minimizes the Euclidean distance between points; for the corresponding point set M, the geometric and color terms of the objective function are constructed respectively, as shown in the following formula: ； ； In the formula, Let be the normal vector of the corresponding point p; Equivalent to the tangent plane distance from the source point to the target point; f ( ) is a function that backprojects 3D points onto the tangent plane of the target point cloud; C p ( ) is a function that parameterizes a virtual image. C ( ) represents the color value of the corresponding coordinate.

5. The three-dimensional reconstruction method according to claim 1, characterized in that, After stitching together the multiple point clouds to obtain a three-dimensional reconstructed image, the three-dimensional reconstruction method further includes: The three-dimensional reconstructed image is iteratively optimized using the Gauss-Newton method.

6. A three-dimensional reconstruction device, characterized in that, include: The symbol image acquisition module is used to acquire images of symbols using multiple cameras from different perspectives to obtain symbol images. The reference object calibrated by the camera is the symbol. The symbol includes multiple markers, and each marker has a corresponding identification number. The recognition modules are used to identify the identifiers of the markers in the symbol image, including: performing grayscale and binarization processing on the symbol image sequentially to obtain a binary symbol image; performing ROI region extraction and corner detection on the binary symbol image to obtain the marker; and identifying the identifier of the marker based on the black and white pixel values ​​in the coding area of ​​the marker. The process of extracting the Region of Interest (ROI) and detecting corner points from the binary symbol image to obtain the marker includes: A contour query is performed on the binary image of the code elements to obtain multiple contours, each of which consists of a set of continuous points; Traverse each of the described contours and calculate the enclosed area; When the enclosed area is outside the preset range, it is determined to be unmarked, and the polygon fitting function is called to detect the number of corner points of each contour. If the number of corner points detected for the contour is 5, and the polygon fitted by the contour is non-convex, then the polygon fitted by the contour is determined to be marked. The inside of the marker is the encoding area, which includes nine square regions of the same size. Each of the nine square regions corresponds to a nine-bit binary number. Each square region corresponds to one bit of binary number, and the binary number corresponds to the color of the square region, with black representing 1 and white representing 0. The first four bits correspond to the marker ID, the next four bits represent the marker ID in opposite colors, and the last bit is the parity bit. The world coordinates corresponding to the marker are searched using the identifier. The point cloud computing module is used to calculate the external parameter matrix of the camera based on the marked coordinates of the camera coordinate system and the marked coordinates of the world coordinate system, and to unify the point cloud into the world coordinate system to obtain multiple point clouds from different perspectives. The stitching module is used to stitch together multiple point clouds to obtain a three-dimensional reconstructed image.

7. An electronic device, characterized in that, include: A processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the three-dimensional reconstruction method as described in any one of claims 1-5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the three-dimensional reconstruction method as described in any one of claims 1-5.

Images