A three-dimensional scanning system and method based on head mounted display device

Abstract

A three-dimensional scanning system and method based on a head-mounted display device, the scanning system comprises at least a head-mounted display component, a data processing unit and a three-dimensional scanner; the three-dimensional scanner and the head-mounted display component are connected to the data processing unit through wired transmission or wireless transmission; the result of the three-dimensional scanning is superimposed on the scanned object in real time through the SLAM technology, the three-dimensional scanner coordinate system, the head-mounted system coordinate system and the world coordinate system are unified, the virtual space and the real scene are superimposed, and the head-mounted display device is used for real-time display, so that the user can timely browse the scanning effect and status, and adjust the scanning posture, distance, position and angle until the complete scanning is completed, so as to realize the in-situ replication of the object; the method can realize the user to perform remote control scanning, and is suitable for scanning applications requiring remote assistance or in dangerous scenes.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional measurement and computer vision technology, and in particular to a three-dimensional scanning system and method based on a head-mounted display device. Background Art

[0002] A three-dimensional (3D) scanner is an instrument used to obtain the three-dimensional shape and color of an object or environmental surface. Usually, the output data of a 3D scanner is a three-dimensional point cloud or mesh data of the object and environment. It is mainly used in industrial design, defect detection, reverse engineering, robot guidance, medical information, bioinformatics, criminal identification, digital cultural relics collection, film production, game material creation, etc.

[0003] Three-dimensional scanning technology is divided into contact and non-contact. Contact three-dimensional scanning technology usually uses a multi-axis coordinate measuring machine to scan point by point to obtain complete 3D data. Although it has high accuracy, it is very inefficient. Non-contact three-dimensional scanning technology is mainly based on optical methods. For scanning large objects at medium and long distances, laser radar scanning or photogrammetry is usually used. For three-dimensional scanning of microscopic objects, laser interferometry, white light interferometry and other technologies are usually used. For objects of common scales in life, time-of-flight method and structured light technology are mainly used for scanning. Due to the advantage of accuracy, most large-scale commercial three-dimensional scanners use structured light technology. Common forms of structured light scanning are single-line or multi-line laser three-dimensional scanning, or scanning using array structured light.

[0004] In addition to the above-mentioned 3D point cloud acquisition technology, 3D scanning technology also includes point cloud stitching, fusion, sampling, and meshing technologies to obtain a complete 3D model of the scanned object. In addition, some 3D scanning devices can obtain color, texture, material and other information while acquiring point clouds, so that the material can be reconstructed while reconstructing the geometric features.

[0005] There are many ways to use a 3D scanner. It can be fixed for scanning or it can be scanned by a robot, but the most convenient and flexible way to scan is still handheld scanning. People connect a 3D scanner through a personal computer or mobile terminal, and then use the 3D scanner to scan objects from different angles. During the scanning process, the screen of the device is used to interactively browse the scanning results, so as to adjust the scanning posture, distance, position, and angle to obtain better or more complete scanning results. However, during the scanning process, the user needs to constantly alternate between observing the screen and the object, and constantly try and calibrate to coordinate hand-eye movements. In order to guide hand movements by observing the results on the screen, the object can be scanned. However, this process is not intuitive and difficult to learn, which limits the scanning efficiency and effect of ordinary people to a certain extent. Summary of the invention

[0006] In order to solve the problems of low scanning efficiency and high operating difficulty caused by difficulty in hand-eye coordination during hand-held three-dimensional scanning, the purpose of the present invention is to provide a three-dimensional scanning system and method based on a head-mounted display device, which superimposes the results of the three-dimensional scanning on the scanned object in real time through SLAM technology, and uses the head-mounted display device to display it in real time, so that the user can browse the scanning effect and status in time, and adjust the scanning posture, distance, position, and angle until the complete scan is completed, and the in-situ replication of the object is achieved.

[0007] In order to achieve the above object, the technical solution of the present invention is:

[0008] A three-dimensional scanning system based on a head-mounted display device comprises at least a head-mounted display component 100, a data processing unit 300 and a three-dimensional scanner 200; the three-dimensional scanner 200 and the head-mounted display component 100 are connected to the data processing unit 300 through wired transmission or wireless transmission;

[0009] The head mounted display assembly 100 at least includes a lens module 103 with a display function, a camera 104 for environmental perception, a microphone 106 for sound pickup, and a mechanical mechanism 107 for supporting and connecting different components;

[0010] The data processing unit 300 has SLAM positioning, 3D point cloud processing, and 3D display processing functions;

[0011] The three-dimensional scanner 200 includes at least one two-dimensional camera 203, which acquires and outputs three-dimensional point clouds of objects and environmental surfaces within a field of view in real time.

[0012] The scanning method based on the above-mentioned three-dimensional scanning system based on the head mounted display device specifically comprises the following steps:

[0013] S1: Using the camera 104 of the head mounted display assembly 100 and the data processing unit 300 to implement SLAM positioning and establish a world coordinate system 410, and through tracking, establish a real-time transformation between the head mounted display assembly 100 and the world coordinate system 410;

[0014] S2: The 3D scanner 200 scans to obtain a 3D point cloud 402, and transforms the 3D point cloud 402 into the world coordinate system 410 in step S1;

[0015] S3: According to the posture of the head mounted display assembly 100 obtained in step S1, the point cloud is projected and rendered in real time, and superimposed and displayed on the screen of the lens module 103 of the head mounted display assembly 100;

[0016] S4: by observing the point cloud and the real object image on the screen of the lens module 103 of the display assembly 100, adjusting the handheld 3D scanner 200, repeating steps S2 and S3, and splicing the obtained point cloud until the scanning is completed;

[0017] S5- Finely stitch all 3D point clouds and perform post-processing to obtain the final 3D model.

[0018] The specific steps of step S1 are as follows:

[0019] S1-1, using the camera 104 of the head mounted display assembly 100 to continuously photograph the user's surrounding environment, and transmitting the data stream to the data processing unit 300 for feature extraction, feature matching, and feature splicing, to establish a world coordinate system 410 and an environment map under the world coordinate system 410; wherein the world coordinate system 410 is a fixed coordinate system with an origin at a certain point in the surrounding environment;

[0020] S1-2: Use the camera 104 of the head mounted display assembly 100 to collect real-time data of the environment, perform feature extraction and feature matching, and calculate the position and posture of the head mounted display assembly 100 in the world coordinate system 410 in real time, that is, obtain the transformation matrix from the head mounted display assembly coordinate system 101 to the world coordinate system 410 g T w .

[0021] An alternative to step S1-2 is: the head-mounted display assembly 100 is provided with an inertial measurement unit IMU A for posture measurement; the head-mounted display assembly 100 is used to calculate the real-time position and posture of the head-mounted display assembly 100 using the inertial measurement unit IMU A of the head-mounted display assembly 100 and the environmental map established in step S1-1.

[0022] Step S1-1 is specifically as follows:

[0023] S1-1-1: Extract feature points, that is, extract ORB features from the first two frames of images. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points; then calculate the BRIEF descriptor;

[0024] S1-1-2: Feature matching, using the FLANN algorithm to match ORB features. If the number of matches is less than 200, the match fails.

[0025] S1-1-3: By obtaining the correspondence between feature points, the epipolar constraints between frames are calculated. First, the eight-point method is used to calculate the essential matrix E. Then, the singular value decomposition method is used to calculate the rotation and translation matrix, the initial posture is selected, and the world coordinate system 410 is established.

[0026] S1-1-4: Based on the poses of the first two frames, the three-dimensional map points are restored using triangulation to create an initial map. The two frames are set as key frames, and the association between key frames and map points and between key frames is completed. A global bundle adjustment optimization is performed on the key frames and map points, and the median of the map point depth is selected as the unit size 1 to initialize the map size.

[0027] S1-1-5: Use the extended Kalman filter EKF for back-end optimization to reduce the cumulative error between frames;

[0028] S1-1-6: Use the bag-of-words model for loop detection to eliminate global accumulated errors and build a globally consistent map.

[0029] The extraction method of FAST corner points in step S1-1-1 is:

[0030] S1-1-1-1.1: Select pixel p in the image, assuming its brightness is I p ;

[0031] S1-1-1-1.2: Set a threshold TH, TH is set to I p 20%-30%;

[0032] S1-1-1-1.3: Select a pixel point on the circle with pixel p as the center;

[0033] S1-1-1-1.4: Assume that there are N consecutive points on the selected circle with a brightness greater than I p +TH, less than I p -TH, then pixel p is considered to be a feature corner point;

[0034] S1-1-1-1.5: Loop the above steps S1-1-1-1 to S1-1-1-4 and perform the same operation on each pixel.

[0035] The rotation and scale feature description in step S1-1-1 is specifically as follows:

[0036] S1-1-1-2.1: In an image block, the (p, q)-order geometric moment of the image block is defined as:

[0037]

[0038] Wherein, p, q are the orders of geometric moments, and p, q = {0, 1}, x, y represent pixel coordinate values, and I (x, y) represents the grayscale value of the pixel coordinates;

[0039] S1-1-1-2.2: Calculate the image block centroid C using the geometric moment in step S1-1-1-2.1:

[0040]

[0041] Among them, m 10 , m 01 , m 00 Respectively represent the (1,0), (0,1), (0,0) order geometric moments of the image;

[0042] S1-1-1-2.3: Connect the center O and the centroid C of the image block set to obtain a direction vector So the direction descriptor of the feature point is defined as:

[0043]

[0044] The BRIEF descriptor is calculated in step S1-1-1, specifically:

[0045] S1-1-1-3.1: First, denoise the image p by using Gaussian filtering;

[0046] S1-1-1-3.2: With the feature point as the center, take an S×S neighborhood window, randomly select two points in the window, compare the sizes of the two pixels, and perform the following binary assignment to obtain the feature descriptor:

[0047]

[0048] Among them, I1 and I2 are N pairs of random points randomly selected in the window;

[0049] S1-1-1-3.3: Repeat the binary assignment in step S1-1-1-3.2 to form a binary code, which is the description of the feature point, namely the BRIEF feature descriptor.

[0050] The specific method of step S1-2 is as follows:

[0051] S1-2-1: Extract feature points. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points, and then calculate the BRIEF descriptor.

[0052] S1-2-2: Feature matching, using the fast approximate nearest neighbor FLANN algorithm to match ORB features;

[0053] S1-2-3: By obtaining the correspondence between feature points, calculate the epipolar constraints between frames: first use the classic eight-point method to calculate the essential matrix E; then use the singular value decomposition method to calculate the rotation and translation matrices to obtain the position and posture of the head-mounted display device.

[0054] The specific steps of step S2 are as follows:

[0055] S2-1: using the camera 104 of the head mounted display assembly 100 to perform gesture recognition and start the three-dimensional scanner 200; or using the microphone 106 to perform voice control recognition and start the three-dimensional scanner;

[0056] S2-2: The 3D scanner 200 scans the scanned object 401 to obtain a 3D point cloud 402 at a current angle, and transmits the 3D point cloud 402 to the data processing unit 300;

[0057] S2-3: Use the two-dimensional camera 203 of the three-dimensional scanner 200 to shoot, extract features, and match features of the scanned object 401 to determine the position and posture of the three-dimensional scanner 200 in the map, that is, to obtain the transformation matrix from the scanner coordinate system 201 to the world coordinate system 410 s T w ; and transform the three-dimensional point cloud 402 to the world coordinate system 410. At this time, the three-dimensional point cloud 402 coincides with the scanned object 401 being photographed in the world coordinate system.

[0058] The alternative to step S2-3 is: the three-dimensional scanner 200 is provided with a B inertial measurement unit IMU for posture measurement, and the B inertial measurement unit IMU of the three-dimensional scanner 200 is used to measure the position and posture of the scanned object 401 relative to the earth, that is, the transformation matrix s T w ; and transform the three-dimensional point cloud 402 to the world coordinate system 410. At this time, the three-dimensional point cloud 402 coincides with the scanned object 401 being photographed in the world coordinate system.

[0059] The step S3 is specifically as follows:

[0060] S3-1: Convert the 3D point cloud 402 in the 3D scanner coordinate system 201 of the 3D scanner 200 to the head mounted display component coordinate system 101 of the head mounted display component 100 in real time. In the coordinate system of the head mounted display component 100, the conversion matrix s T g = s T w ·( g T w ) -1 ;

[0061] S3-2: In the coordinate system of the head-mounted display assembly 100, the three-dimensional point cloud 402 obtained by the three-dimensional scanning is projected and rendered, and the projection matrix is ​​projected using the FOV parameters of the head-mounted display assembly 100, and is further superimposed and displayed on the screen of the lens module 103. At this time, the three-dimensional point cloud 402 overlaps with the scanned object 401 in the head-mounted display assembly.

[0062] An alternative to step S3-1 is to use the inertial measurement unit IMU of the head mounted display assembly 100 to measure its position and posture relative to the earth, that is, the transformation matrix g T w , thereby obtaining the transformation relationship from the 3D scanner coordinate system 201 to the head mounted display component coordinate system 101, that is, the transformation matrix s T g = s T w ·( g T w ) -1 , and convert the original three-dimensional point cloud 402 scanned by the three-dimensional scanner 200 into the coordinate system of the head-mounted display assembly 100 to obtain the point cloud in the coordinate system of the head-mounted display assembly 100 .

[0063] The step S4 is specifically as follows:

[0064] S4-1, the scanned object 401 is stationary, and the user observes the content displayed in the head mounted display assembly 100, which at this time displays the superposition of the virtual three-dimensional point cloud 402 and the real scanned object, and then adjusts the position and posture of the scanner 200 to ensure that the scanner points to the target scanning area;

[0065] S4-2, repeatedly executing steps S2-2, S2-3, S3-1, and S-3-2 in real time, transforming the continuously collected three-dimensional point clouds 402 into the world coordinate system 410 respectively, and projecting them into the display component 100;

[0066] S4-3, the scanned object 401 is kept still, and the user of the three-dimensional scanner 200 moves until the scanned object 401 is scanned in all directions.

[0067] The step S5 is specifically as follows:

[0068] S5-1, the three-dimensional point cloud 402 determines the initial values ​​of the transformation matrix based on the rough positioning;

[0069] S5-2, using the closest point iterative ICP algorithm, combined with texture and color information, to optimize the global point cloud stitching, determine the fine pose matrix of the point cloud, and obtain a complete point cloud;

[0070] S5-3, the user surrounds the scanned object 401 and checks the image of the three-dimensional point cloud 402 scanned at the current angle projected onto the display lens in real time to confirm whether there is any missing. If there is any missing, perform a supplementary scan;

[0071] S5-4, finally fusing, resampling, meshing, and texture mapping the confirmed complete three-dimensional point cloud 402 to obtain a complete model;

[0072] S5-5, optimize the position of the complete model in the world coordinate system 410 to ensure that it matches the real object in the world coordinate system 410. At this time, the scanned object 401 is removed to achieve the replication of the real object 401; that is, after the real object is removed, the virtual three-dimensional image of the scanned object 401 can still be observed in situ through the head-mounted display component 100.

[0073] The step S5-2 is specifically as follows:

[0074] S5-2-1: The three-dimensional point cloud 402 after preliminary conversion is used as an initial value, the first frame of two adjacent frames is used as a target point cloud P, and the second frame is used as a source point cloud Q;

[0075] S5-2-2: downsampling the three-dimensional point cloud 402;

[0076] S5-2-3: Find the corresponding points in the target point cloud P in the source point cloud Q so that the line segment p between the corresponding point pairs i q i shortest;

[0077] S5-2-4: Calculate the rotation and translation matrices of the target point cloud P and the source point cloud Q to minimize the error. The error function is:

[0078]

[0079] Where R is the rotation matrix and T is the translation matrix. i ,q i is the i-th group of points in the source point cloud Q corresponding to the target point cloud;

[0080] S5-2-5: Perform rotation and translation transformation on the target point cloud and calculate the average distance d from the transformed target point cloud P′ to the source point cloud Q

[0081]

[0082] S5-2-6: Determine whether the average distance d is less than a threshold;

[0083] S5-2-7: Repeat steps S5-2-3, S5-2-4, S5-2-5, and S5-2-6 until the average distance d is less than the threshold or the number of iterations is greater than the threshold, and stop the above iterative process.

[0084] Compared with the prior art, the present invention has the following beneficial effects:

[0085] 1. This method uses a head-mounted display component to project the point cloud obtained by the scanner into the virtual space in real time during the scanning process, and overlays it with the real scene for display. What you see is what you get, so as to solve the problem of hand-eye coordination difficulty in traditional three-dimensional scanning systems, improve scanning efficiency, and reduce operation difficulty.

[0086] 2. This method reconstructs the position of the scanned object in the environment through SLAM technology while scanning the three-dimensional information of the object. Traditional methods only focus on scanning the three-dimensional information of the object itself. This method realizes the in-situ three-dimensional replication of the object; that is, after the scan is completed, the scanned object entity is removed, and the removed object (in this case, a virtual object) can still be observed in situ through the head-mounted display device. It can be widely used in exhibitions, games and other scenes that combine the virtual and the real.

[0087] 3. This method unifies the 3D scanner coordinate system, head-mounted system coordinate system, and world coordinate system through the combination of feature recognition and SLAM technology, and realizes the alignment of data between multiple systems.

[0088] In summary, the present invention superimposes the virtual space and the real scene for display, solves the problem of hand-eye coordination difficulty in traditional three-dimensional scanning systems, improves scanning efficiency, and reduces operating difficulty; this method allows users to perform remote control scanning, and is suitable for scanning applications that require remote assistance or in dangerous scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a 3D scanner system based on a head-mounted display device.

[0090] In the figure: 100 head mounted display component, 101 head mounted display component coordinate system, 103 lens module, 104 camera, 106 microphone, 107 mechanical mechanism, 200 three-dimensional scanner, 201 three-dimensional scanner coordinate system, 300 data processing unit, 401 scanned object, 410 world coordinate system. DETAILED DESCRIPTION

[0091] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0092] like Figure 1 As shown, a three-dimensional scanning system based on a head-mounted display device includes at least a head-mounted display component 100, a data processing unit 300 and a three-dimensional scanner 200; the three-dimensional scanner 200 and the head-mounted display component 100 are connected to the data processing unit 300 through wired transmission or wireless transmission; the three-dimensional scanner 200 and the head-mounted display component 100 are powered by a built-in or external power supply.

[0093] The head mounted display assembly 100 at least includes a lens module 103 with a display function, a camera 104 for environmental perception, a microphone 106 for sound pickup, and a mechanical structure device 107 for supporting and connecting different components;

[0094] The head mounted display device 100 is used for display of augmented reality (AR), and can adopt AR glasses with different principles, and its principle is one of the commonly used display technologies of AR glasses. The commonly used display technologies include but are not limited to reflective light waveguides, diffraction light waveguides, holographic light waveguides, and retinal projection; the head mounted display component 100 is powered by a built-in or external power supply. The head mounted display component 100 controls the display image through built-in and external data processing modules.

[0095] The data processing unit 300 has SLAM positioning, 3D point cloud processing, and 3D display processing functions;

[0096] The three-dimensional scanner 200 includes at least one two-dimensional camera 203, and acquires and outputs three-dimensional point clouds of objects and environmental surfaces within a field of view in real time.

[0097] The three-dimensional scanner 200 is a scanner with edge computing capabilities, so that the three-dimensional data is directly transmitted to the data processing unit 300. The three-dimensional scanner 200 is preferably a three-dimensional scanner based on MEMS technology, so that the device is more low-power, compact, lightweight, and high-precision. The three-dimensional scanner 200 is equipped with an inertial measurement unit for posture measurement, which is used to measure the posture of the three-dimensional scanner.

[0098] The camera 104 is a two-dimensional camera or a three-dimensional camera, and can be a single two-dimensional camera or a combination of multiple two-dimensional cameras. When it is a three-dimensional camera, it can be a simple three-dimensional information acquisition device or a color RGBD camera that can simultaneously obtain the geometric shape and texture color of the object.

[0099] The two-dimensional camera is a device for obtaining a two-dimensional image, which can be a charge coupled device image sensor CCD, a complementary metal oxide semiconductor image sensor CMOS, or a dynamic event camera. The three-dimensional camera is a device that uses a time-of-flight method, a structured light method, or a stereoscopic vision technology for real-time imaging. The three-dimensional camera and the two-dimensional camera can be separate or can be a combined RGBD camera. If the two-dimensional camera and the three-dimensional camera are separate, they need to be calibrated to align the coordinates.

[0100] like Figure 1 As shown, when in use, the user wears the head-mounted display device 100 and holds the three-dimensional scanner 200 to scan the target physical model 401 at different angles. During scanning, by observing the image in the head-mounted display device 100, the scanning position and posture are adjusted to ensure efficient completion of the full range of scanning and obtain a complete model.

[0101] A three-dimensional scanning method based on a head mounted display device works in the following steps:

[0102] S1: Using the camera 104 of the head mounted display assembly 100 and the data processing unit 300 to implement SLAM positioning and establish a world coordinate system 410, and through tracking, establish a real-time transformation between the head mounted display assembly 100 and the world coordinate system 410;

[0103] S2: The 3D scanner 200 scans to obtain a 3D point cloud 402, and transforms the 3D point cloud 402 into the world coordinate system 410 in step S1;

[0104] S3: According to the posture of the head mounted display assembly 100 obtained in S1, the point cloud is projected and rendered in real time, and superimposed and displayed on the screen of the lens module 103 of the head mounted display assembly 100;

[0105] S4: by observing the point cloud and the real object image on the screen of the lens module 103 of the display assembly 100, adjusting the handheld 3D scanner 200, repeating steps S2 and S3, and splicing the obtained point cloud until the scanning is completed;

[0106] S5: Finely stitch all 3D point clouds and perform post-processing to obtain the final 3D model.

[0107] The specific steps of step S1 are as follows:

[0108] S1-1. Use the camera 104 of the head-mounted display assembly 100 to continuously capture the user's surrounding environment, and transmit the data stream to the data processing unit 300 for feature extraction, feature matching, and feature stitching to establish a world coordinate system 410 and an environment map under the world coordinate system 410; wherein the world coordinate system 410 is a fixed coordinate system with its origin at a certain point in the surrounding environment.

[0109] S1-2: Use the camera 104 of the head-mounted display assembly 100 to collect real-time data of the environment, perform feature extraction and feature matching, and calculate the position and posture of the head-mounted display assembly 100 in the world coordinate system 410 in real time, that is, obtain the transformation matrix from the head-mounted display assembly coordinate system 101 to the world coordinate system 410.

[0110] An alternative to step S1-2 is: the head-mounted display assembly 100 is provided with an inertial measurement unit IMU A for posture measurement, and the head-mounted display assembly 100 is calculated in real time position and posture using the inertial measurement unit IMU A of the head-mounted display assembly 100 and the environmental map established in S1-1.

[0111] The step S1-1 is specifically as follows:

[0112] S1-1-1: Extract feature points, that is, extract ORB features from the first two frames of images. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points; then calculate the BRIEF descriptor;

[0113] S1-1-2: Feature matching, using the FLANN algorithm to match ORB features. If the number of matches is less than 200, the match fails.

[0114] S1-1-3: By obtaining the correspondence between feature points, the epipolar constraints between frames are calculated. First, the eight-point method is used to calculate the essential matrix E. Then, the singular value decomposition method is used to calculate the rotation and translation matrix, the initial posture is selected, and the world coordinate system 410 is established.

[0115] S1-1-4: Based on the poses of the first two frames, the three-dimensional map points are restored using triangulation to create an initial map. The two frames are set as key frames, and the association between key frames and map points and between key frames is completed. A global bundle adjustment optimization is performed on the key frames and map points, and the median of the map point depth is selected as the unit size 1 to initialize the map size.

[0116] S1-1-5: Use the extended Kalman filter EKF for back-end optimization to reduce the cumulative error between frames;

[0117] S1-1-6: Use the bag-of-words model for loop detection to eliminate global accumulated errors and build a globally consistent map.

[0118] The extraction method of FAST corner points in S1-1-1 is:

[0119] S1-1-1-1.1: Select pixel p in the image, assuming its brightness is I p .

[0120] S1-1-1-1.2: Set a threshold TH, TH is set to I p 20%-30% of.

[0121] S1-1-1-1.3: With pixel p as the center, select 16 pixels on a circle with a radius of 3.

[0122] S1-1-1-1.4: Assume that there are N consecutive points on the selected circle with a brightness greater than I p +TH, less than I p -TH, then pixel p is considered to be a feature corner point, and the preferred value of N is 12.

[0123] S1-1-1-1.5: Loop the above steps S1-1-1-1 to S1-1-1-4 and perform the same operation on each pixel.

[0124] The rotation and scale feature description in S1-1-1 is specifically as follows:

[0125] S1-1-1-2.1: In an image block, the (p, q)-order geometric moment of the image block is defined as:

[0126]

[0127] Wherein, p, q are the orders of geometric moments, and p, q = {0, 1}, x, y represent pixel coordinate values, and I (x, y) represents the grayscale value of the pixel coordinates;

[0128] S1-1-1-2.2: Calculate the image block centroid C using the geometric moment in S1-1-1-2.1:

[0129]

[0130] Among them, m 10 , m 01 , m 00 Respectively represent the (1,0), (0,1), (0,0) order geometric moments of the image;

[0131] S1-1-1-2.3: Connect the center 0 of the image block set and the centroid C to obtain a direction vector So the direction descriptor of the feature point is defined as:

[0132]

[0133] The BRIEF descriptor is calculated in step S1-1-1, specifically:

[0134] S1-1-1-3.1: First, reduce the noise of the image p by Gaussian filtering, using a two-dimensional Gaussian filter with a window of 7x7 and a variance of 2;

[0135] S1-1-1-3.2: With the feature point as the center, take a neighborhood window of SxS, randomly select two points in the window, compare the sizes of the two pixels, and perform the following binary assignment to obtain the feature descriptor:

[0136]

[0137] Among them, I1 and I2 are N pairs of random points randomly selected in the window;

[0138] S1-1-1-3.3: Repeat the binary assignment of S1-1-1-3.2 to form a binary code. This code is the description of the feature point, namely the BRIEF feature descriptor.

[0139] The specific method of step S1-2 is as follows:

[0140] S1-2-1: Extract feature points. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points. Then calculate the BRIEF descriptor;

[0141] S1-2-2: Feature matching, using the Fast Approximate Nearest Neighbor (FLANN) algorithm to match ORB features;

[0142] S1-2-3: By obtaining the correspondence between feature points, the epipolar constraints between frames are calculated; first, the classic eight-point method is used to calculate the essential matrix E; then the singular value decomposition method is used to calculate the rotation and translation matrices to obtain the position and posture of the head-mounted display device.

[0143] The specific steps of step S2 are as follows:

[0144] S2-1: using the camera 104 of the head mounted display assembly 100 to perform gesture recognition and start the three-dimensional scanner 200; or using the microphone 106 to perform voice control recognition and start the three-dimensional scanner;

[0145] S2-2: Scanning the scanned object 401 using the 3D scanner 200 to obtain a 3D point cloud 402 at the current angle, and transmitting the point cloud to the data processing unit 300;

[0146] S2-3: Use the two-dimensional camera 203 of the three-dimensional scanner 200 to shoot, extract features, and match features of the scanned object 401 to determine the position and posture of the three-dimensional scanner 200 in the map, that is, to obtain the transformation matrix from the scanner coordinate system 201 to the world coordinate system 410 s T w The three-dimensional point cloud 402 is transformed into the world coordinate system 410 , and the three-dimensional point cloud 402 coincides with the scanned object 401 in the world coordinate system.

[0147] The alternative to step S2-3 is: the three-dimensional scanner 200 is provided with an inertial measurement unit B IMU for posture measurement, and the inertial measurement unit B IMU of the three-dimensional scanner 200 is used to measure the position and posture of the scanned object 401 relative to the earth, that is, the transformation matrix s T w ; and transform the three-dimensional point cloud 402 to the world coordinate system 410. At this time, the three-dimensional point cloud 402 coincides with the scanned object 401 being photographed in the world coordinate system.

[0148] The specific method of using IMU for attitude and position estimation is as follows:

[0149] First, let the rotation around the IMU's Z axis be the heading angle yaw, rotate by y angle, the rotation around the IMU's Y axis be the pitch angle pitch, rotate by p angle; the rotation around the IMU's X axis be the roll angle row, rotate by r angle, and g is the gravitational acceleration.

[0150]

[0151] The attitude rotation matrix is:

[0152] R=R x ·R y ·R z

[0153] The transformation relationship from the scanner coordinate system to the head-mounted display device is:

[0154] O1=R'T'O2

[0155] The rotation matrix R′ is calculated based on the r, p, y angle difference between the scanner coordinate system and the head-mounted display device:

[0156] R′=R x (Δr)·R y (Δp)·R z (Δy)

[0157] The translation matrix is:

[0158] The translation matrix T′ is obtained by integrating the position of the IMU.

[0159] T′=P1-P2+Δp

[0160] P1, P2, and Δp are the spatial positions of the scanner and the head-mounted display device obtained through IMU integration, and their initial position difference, the position difference Δp, can be obtained through calibration.

[0161] The step S3 is specifically as follows:

[0162] S3-1: Convert the 3D point cloud 402 in the 3D scanner coordinate system 201 of the 3D scanner 200 to the head mounted display component coordinate system 101 of the head mounted display component 100 in real time. The conversion matrix is s T g = s T w ·( g T w ) -1 ;

[0163] S3-2: In the coordinate system of the head-mounted display assembly 100, the three-dimensional point cloud 402 obtained by the three-dimensional scanning is projected and rendered, and the projection matrix is ​​projected using the FOV parameters of the head-mounted display assembly 100, and is further superimposed and displayed on the screen of the lens module 103; at this time, the three-dimensional point cloud 402 overlaps with the scanned object 401 in the head-mounted display assembly.

[0164] An alternative to step S3-1 is to use the inertial measurement unit IMU of the head mounted display assembly 100 to measure its position and posture relative to the earth, that is, the transformation matrix g T w , thereby obtaining the transformation relationship from the 3D scanner coordinate system 201 to the head mounted display component coordinate system 101, that is, the transformation matrix s T g = s T w ·( g T w ) -1 The original three-dimensional point cloud 402 scanned by the three-dimensional scanner 200 is converted into the coordinate system of the head mounted display assembly 100 to obtain the point cloud in the coordinate system of the head mounted display assembly 100 .

[0165] The step S4 is specifically as follows:

[0166] S4-1, the scanned object 401 is stationary, and the user observes the content displayed in the head mounted display assembly 100, which at this time displays the superposition of the virtual three-dimensional point cloud 402 and the real scanned object, and then adjusts the position and posture of the scanner 200 to ensure that the scanner points to the target scanning area;

[0167] S4-2, repeatedly executing steps S2-2, S2-3, S3-1, and S-3-2 in real time, transforming the continuously collected three-dimensional point clouds 402 into the world coordinate system 410 respectively, and projecting them into the display component 100;

[0168] S4-3, keeping the scanned object 401 still, the user of the three-dimensional scanner 200 moves until the scanned object 401 is fully scanned.

[0169] The step S5 is specifically as follows:

[0170] S5-1, the three-dimensional point cloud 402 determines the initial values ​​of the transformation matrix based on the above rough positioning;

[0171] S5-2, using the closest point iteration (ICP) algorithm, combined with texture and color information, to optimize the global point cloud stitching, determine the fine pose matrix of the point cloud, and obtain a complete point cloud;

[0172] S5-3, the user can view the image of the three-dimensional point cloud 402 scanned at the current angle projected onto the display lens around the scanned object 401 in real time to confirm whether there is any missing. If there is any missing, perform a supplementary scan;

[0173] S5-4, finally the confirmed complete three-dimensional point cloud 402 is fused, resampled, meshed, and texture mapped to obtain a complete model;

[0174] S5-5, the position of the complete model in the world coordinate system 410 is optimized to ensure that it matches the real object in the world coordinate system 410, so that the user can browse it at any time. At this time, the scanned object 401 can be removed to achieve the copy of the real object 401. That is, after the real object is removed, the virtual three-dimensional image of the scanned object 401 can still be observed in situ through the head-mounted display assembly 100.

[0175] As an alternative to this method, a robot is used to replace a human arm for scanning. The user remotely controls the robot to complete scanning of the scanned object 401 by remotely using another remote head-mounted display device to share the display information of the on-site head-mounted display component.

[0176] The S5-2 is specifically:

[0177] S5-2-1: The three-dimensional point cloud 402 after preliminary conversion is used as the initial value, the previous frame of two adjacent frames is used as the target point cloud, and the next frame is used as the source point cloud.

[0178] S5-2-2: downsampling the dimensional point cloud 402;

[0179] S5-2-3: Find the corresponding points in the target point cloud P in the source point cloud Q so that the line segment p between the corresponding point pairs i q i Shortest.

[0180] S5-2-4: Calculate the rotation and translation matrices of the target point cloud P and the source point cloud Q to minimize the error. The error function is:

[0181] Where R is the rotation matrix and T is the translation matrix. i ,q i is the i-th group of points in the source point cloud Q corresponding to the target point cloud;

[0182] S5-2-5: Perform rotation and translation transformation on the target point cloud, and calculate the average distance d from the transformed target point cloud P′ to the source point cloud Q:

[0183]

[0184] S5-2-6: Determine whether the average distance d is less than a threshold.

[0185] S5-2-7: Repeat S5-2-3, S5-2-4, S5-2-5, and S5-2-6 until the average distance d is less than the threshold or the number of iterations is greater than the threshold, and then stop the above iterative process.

[0186] This method not only scans the three-dimensional information of the object, but also reconstructs the position of the scanned object in the environment, that is, this method realizes the in-situ three-dimensional replication of the object. After the scanning is completed, the scanned object is removed, and the removed object can still be seen through the head-mounted display device, and what is seen at this time is the superposition display of the virtual three-dimensional image and the actual environment.

Claims

1. A scanning method of a three-dimensional scanning system based on a head-mounted display device, wherein the three-dimensional scanning system comprises at least a head-mounted display component (100), a data processing unit (300) and a three-dimensional scanner (200); the three-dimensional scanner (200) and the head-mounted display component (100) are connected to the data processing unit (300) via wired transmission or wireless transmission; The head mounted display assembly (100) at least comprises a lens module (103) with a display function, a camera (104) for environmental perception, a microphone (106) for sound pickup, and a mechanical mechanism (107) for supporting and connecting different components; The data processing unit (300) has SLAM positioning, three-dimensional point cloud processing, and three-dimensional display processing functions; The three-dimensional scanner (200) comprises at least one two-dimensional camera (203) for acquiring and outputting three-dimensional point clouds of objects and environmental surfaces within a field of view in real time; It is characterized in that The scanning method specifically includes the following steps: S1: using a camera (104) of a head mounted display assembly (100) and a data processing unit (300) to implement SLAM positioning to establish a world coordinate system (410), and establishing a real-time transformation between the head mounted display assembly (100) and the world coordinate system (410) through tracking; S2: The three-dimensional scanner (200) scans to obtain a three-dimensional point cloud (402), and transforms the three-dimensional point cloud (402) into the world coordinate system (410) in step S1; S3: According to the position and posture of the head-mounted display assembly (100) obtained in step S1, the point cloud is projected and rendered in real time, and superimposed and displayed on the screen of the lens module (103) of the head-mounted display assembly (100); S4: by observing the point cloud and the real-object image on the screen of the lens module (103) of the display assembly (100), adjusting the handheld three-dimensional scanner (200), repeating steps S2 and S3, and splicing the obtained point cloud until the scanning is completed; S5: finely stitch all 3D point clouds and perform post-processing to obtain the final 3D model; The specific steps of step S1 are as follows: S1-1, using the camera (104) of the head mounted display assembly (100) to continuously photograph the user's surrounding environment, and transmitting the data stream to the data processing unit (300) for feature extraction, feature matching, and feature splicing, thereby establishing a world coordinate system (410) and an environment map under the world coordinate system (410); wherein the world coordinate system (410) is a fixed coordinate system with its origin at a certain point in the surrounding environment; S1-2: Using the camera (104) of the head mounted display assembly (100), collecting real-time data of the environment, performing feature extraction and feature matching, and calculating in real time the position and posture of the head mounted display assembly (100) in the world coordinate system (410), that is, obtaining a transformation matrix from the head mounted display assembly coordinate system (101) to the world coordinate system (410) g T w ; An alternative to step S1-2 is: an inertial measurement unit A IMU for position and posture measurement is provided on the head-mounted display component (100), and the head-mounted display component (100) is used to calculate the real-time position and posture of the head-mounted display component (100) using the inertial measurement unit A IMU and the environment map established in step S1-1.

2. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 1, characterized in that: Step S1-1 is specifically as follows: S1-1-1: Extract feature points, that is, extract ORB features from the first two frames of images. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points; then calculate the BRIEF descriptor; S1-1-2: Feature matching, using the FLANN algorithm to match ORB features. If the number of matches is less than 200, the match fails. S1-1-3: By obtaining the correspondence between feature points, the epipolar constraints between frames are calculated. First, the eight-point method is used to calculate the essential matrix E. Then, the singular value decomposition method is used to calculate the rotation and translation matrices, and the initial posture is selected to establish the world coordinate system (410). S1-1-4: Based on the poses of the first two frames, the three-dimensional map points are restored using triangulation to create an initial map. The two frames are set as key frames, and the association between key frames and map points and between key frames is completed. A global bundle adjustment optimization is performed on the key frames and map points, and the median of the map point depth is selected as the unit size 1 to initialize the map size. S1-1-5: Use the extended Kalman filter EKF for back-end optimization to reduce the cumulative error between frames; S1-1-6: Use the bag-of-words model to perform loop detection, eliminate global accumulated errors, and build a globally consistent map; The extraction method of FAST corner points in step S1-1-1 is: S1-1-1-1.1: Select pixel p in the image, assuming its brightness is I p ; S1-1-1-1.2: Set a threshold TH, TH is set to I p 20%-30%; S1-1-1-1.3: Select a pixel point on the circle with pixel p as the center; S1-1-1-1.4: Assume that there are N consecutive points on the selected circle with a brightness greater than I p +TH, less than I p -TH, then pixel p is considered to be a feature corner point; S1-1-1-1.5: loop the above steps S1-1-1-1 to S1-1-1-4, and perform the same operation on each pixel; The rotation and scale feature description in step S1-1-1 is specifically as follows: S1-1-1-2.1: In an image block B, the (p,q)-order geometric moment of the image block is defined as: Wherein, p, q are the orders of geometric moments, and p, q = {0, 1}, x, y represent pixel coordinate values, and I (x, y) represents the grayscale value of the pixel coordinates; B is the set of pixels of the image block; S1-1-1-2.2: Calculate the centroid C of the image block through the geometric moment in step S1-1-1-2.1: Among them, m 10 , m 01 , m 00 Respectively represent the (1,0), (0,1), (0,0) order geometric moments of the image; S1-1-1-2.3: Connect the center O and the centroid C of the image block set to obtain a direction vector So the direction descriptor of the feature point is defined as: The BRIEF descriptor is calculated in step S1-1-1, specifically: S1-1-1-3.1: First, denoise the image p by using Gaussian filtering; S1-1-1-3.2: With the feature point as the center, take an S×S neighborhood window, randomly select two points in the window, compare the sizes of the two pixels, and perform the following binary assignment to obtain the feature descriptor: Among them, I1 and I2 are N pairs of random points randomly selected in the window; S1-1-1-3.3: Repeat the binary assignment in step S1-1-1-3.2 to form a binary code, which is the description of the feature point, namely the BRIEF feature descriptor.

3. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 1, characterized in that: The specific method of step S1-2 is as follows: S1-2-1: Extract feature points. First, extract FAST corner points, then calculate the rotation and scale feature descriptions of the corner points, and then calculate the BRIEF descriptor. S1-2-2: Feature matching, using the fast approximate nearest neighbor FLANN algorithm to match ORB features; S1-2-3: By obtaining the correspondence between feature points, the epipolar constraints between frames are calculated; first, the classic eight-point method is used to calculate the essential matrix E; then the singular value decomposition method is used to calculate the rotation and translation matrices to obtain the position and posture of the head-mounted display device.

4. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 1, characterized in that: The specific steps of step S2 are as follows: S2-1: using the camera (104) of the head mounted display assembly (100) to perform gesture recognition and start the three-dimensional scanner (200); or using the microphone (106) to perform voice control recognition and start the three-dimensional scanner; S2-2: The three-dimensional scanner (200) scans the scanned object (401) to obtain a three-dimensional point cloud (402) at a current angle, and transmits the three-dimensional point cloud (402) to the data processing unit (300); S2-3: Using the two-dimensional camera (203) of the three-dimensional scanner (200) to photograph, extract features, and match features of the scanned object (401), determine the position and posture of the three-dimensional scanner (200) in the map, and obtain a conversion matrix from the scanner coordinate system (201) to the world coordinate system (410) s T w ; and transforming the three-dimensional point cloud (402) into a world coordinate system (410), whereby the three-dimensional point cloud (402) coincides with the scanned object (401) being photographed in the world coordinate system; An alternative to step S2-3 is: the three-dimensional scanner (200) is provided with a B inertial measurement unit IMU for position and posture measurement, and the B inertial measurement unit IMU of the three-dimensional scanner (200) is used to measure the position and posture of the scanned object (401) relative to the earth, that is, the transformation matrix s T w ; and transforming the three-dimensional point cloud (402) into a world coordinate system (410), whereby the three-dimensional point cloud (402) coincides with the scanned object (401) being photographed in the world coordinate system.

5. The scanning method of the three-dimensional scanning system based on the head mounted display device according to claim 4, characterized in that: The step S3 is specifically as follows: S3-1: Convert the three-dimensional point cloud (402) in the three-dimensional scanner coordinate system 201 of the three-dimensional scanner (200) to the head mounted display component coordinate system (101) of the head mounted display component (100) in real time. The conversion matrix of the head mounted display component (100) is s T g = s T w ·( g T w ) -1 ; S3-2: In the coordinate system of the head-mounted display component (100), the three-dimensional point cloud (402) obtained by the three-dimensional scanning is projected and rendered, and the projection matrix is ​​projected using the FOV parameters of the head-mounted display component (100), and is further superimposed and displayed on the screen of the lens module (103). At this time, the three-dimensional point cloud (402) overlaps with the scanned object (401) in the head-mounted display component.

6. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 5, characterized in that: The alternative to step S3-1 is: The inertial measurement unit (IMU) of the head mounted display assembly (100) is used to measure its position and attitude relative to the earth, that is, the transformation matrix g T w , thereby obtaining the transformation relationship from the three-dimensional scanner coordinate system (201) to the head mounted display component coordinate system (101), that is, the transformation matrix s T g = s T w ·( g T w ) -1 and converting the original three-dimensional point cloud (402) scanned by the three-dimensional scanner (200) into the coordinate system of the head-mounted display assembly (100) to obtain the point cloud in the coordinate system of the head-mounted display assembly (100).

7. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 5, characterized in that: The step S4 is specifically as follows: S4-1, the scanned object (401) is stationary, and the user observes the content displayed in the head mounted display assembly (100), which at this time displays a superposition of a virtual three-dimensional point cloud (402) and the real scanned object, and then adjusts the position and posture of the scanner (200) to ensure that the scanner points to the target scanned area; S4-2, repeatedly executing steps S2-3, S3-1, and S-3-2 in real time, transforming the continuously collected three-dimensional point clouds (402) into the world coordinate system (410) respectively, and projecting them into the display component (100); S4-3, keeping the scanned object (401) still, the user of the three-dimensional scanner (200) moves until the scanned object (401) is fully scanned.

8. The scanning method of a three-dimensional scanning system based on a head mounted display device according to claim 1, characterized in that: The step S5 is specifically as follows: S5-1, the three-dimensional point cloud (402) determines the initial values ​​of the transformation matrix based on the rough positioning; S5-2, using the closest point iterative ICP algorithm, combined with texture and color information, to optimize the global point cloud stitching, determine the fine pose matrix of the point cloud, and obtain a complete point cloud; S5-3, the user surrounds the scanned object (401), and checks in real time the image of the scanned three-dimensional point cloud (402) at the current angle projected onto the display lens to confirm whether there is any missing, and if so, performs a supplementary scan; S5-4, finally fusing, resampling, meshing, and texture mapping the confirmed complete three-dimensional point cloud (402) to obtain a complete model; S5-5, optimizing the position of the complete model in the world coordinate system (410) to ensure that it matches the real object in the world coordinate system (410), and then removing the scanned object (401) to achieve replication of the real object; that is, after the real object is removed, the virtual three-dimensional image of the scanned object (401) can still be observed in situ through the head-mounted display assembly (100); The step S5-2 is specifically as follows: S5-2-1: taking the initially converted three-dimensional point cloud (402) as the initial value, taking the previous frame of two adjacent frames as the target point cloud P, and taking the next frame as the source point cloud Q; S5-2-2: downsampling the three-dimensional point cloud (402); S5-2-3: Find the corresponding points in the target point cloud P in the source point cloud Q so that the line segment p between the corresponding point pairs i q i shortest; S5-2-4: Calculate the rotation and translation matrices of the target point cloud P and the source point cloud Q to minimize the error. The error function is: Among them, R is the rotation matrix, T is the translation matrix; p i ,q i is the i-th group of points in the source point cloud Q corresponding to the target point cloud; n is the number of points in the point cloud; S5-2-5: Perform rotation and translation transformation on the target point cloud, and calculate the average distance d from the transformed target point cloud P′ to the source point cloud Q: p′ i is the i-th group of points in the point cloud P′ after the target point cloud P is transformed; S5-2-6: Determine whether the average distance d is less than a threshold; S5-2-7: Repeat steps S5-2-3, S5-2-4, S5-2-5, and S5-2-6 until the average distance d is less than the threshold or the number of iterations is greater than the threshold, and stop the above iterative process.

Images