Three-dimensional scene reconstruction method, device and XR equipment

Abstract

This invention provides a method, apparatus, and XR device for three-dimensional scene reconstruction, relating to the field of image processing technology. The three-dimensional scene reconstruction method, applied to an XR device, includes: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmarks in a global coordinate system; and performing mesh reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model. This invention enables real-time reconstruction of indoor scene three-dimensional models without relying on a dedicated depth camera.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a three-dimensional scene reconstruction method, apparatus, and XR device. Background Technology

[0002] With the development of smart hardware and artificial intelligence technologies, XR (Extended Reality) devices have been widely used in various scenarios such as gaming, entertainment, holographic conferencing, virtual shopping, and virtual home decoration. In practical applications, using XR devices to perform 3D reconstruction of the user's real-world environment to achieve an immersive experience that blends the virtual and real worlds has become an important technological trend. The key to achieving this function lies in the efficient and accurate 3D reconstruction of indoor scenes, providing a foundation for subsequent virtual content generation and interaction.

[0003] Currently, most mainstream 3D reconstruction algorithms rely on ToF (Time-of-Flight) cameras or RGB-D (Red, Green, Blue-Depth) cameras to acquire scene depth maps and assist in the real-time reconstruction of 3D models. However, due to limitations in size, power consumption, and cost, some XR devices (especially portable XR devices) are not equipped with these dedicated depth cameras, or cannot continuously operate high-power dedicated depth cameras. Therefore, existing 3D reconstruction technologies based on ToF or RGB-D cameras are difficult to directly apply to such XR devices.

[0004] Therefore, how to achieve real-time reconstruction of 3D models of indoor scenes without relying on dedicated depth cameras has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] This invention provides a three-dimensional scene reconstruction method, apparatus, and XR device, which can realize real-time reconstruction of indoor scene three-dimensional models without relying on a dedicated depth camera.

[0006] This invention provides a three-dimensional scene reconstruction method, applied to XR devices, comprising:

[0007] Receive scene images acquired by the XR device;

[0008] Feature extraction and feature matching are performed on the scene image to obtain matching point pairs;

[0009] Convert the matched point pairs into three-dimensional landmarks in the global coordinate system;

[0010] Based on the three-dimensional landmarks in the global coordinate system, a grid reconstruction is performed to obtain a three-dimensional grid model.

[0011] According to a three-dimensional scene reconstruction method provided by the present invention, the scene image includes a left-eye scene image and a right-eye scene image, wherein feature extraction and feature matching of the scene image to obtain matching point pairs includes:

[0012] The left-eye scene image is subjected to scale-invariant feature transformation (SIFT) feature extraction to obtain a first keypoint set and a first SIFT feature vector set;

[0013] SIFT feature extraction is performed on the right-eye scene image to obtain a second keypoint set and a second SIFT feature vector set;

[0014] Based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching point pairs are searched from the first keypoint set and the second keypoint set, and the matching distance value between the multiple candidate matching point pairs is obtained.

[0015] A ratio test is performed on the matching distance value to obtain the ratio test result;

[0016] Based on the ratio test results, the multiple candidate matching point pairs are filtered, and matching point pairs are obtained based on the filtering results.

[0017] According to a three-dimensional scene reconstruction method provided by the present invention, the step of performing mesh reconstruction based on three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model includes:

[0018] The three-dimensional landmarks in the global coordinate system are divided into spatial grids to obtain a grid map;

[0019] For the three-dimensional landmarks of each grid in the grid map, local grid reconstruction is performed to obtain a local three-dimensional grid model;

[0020] The 3D landmarks of the neighboring grids of each grid in the grid map are reconstructed across grids and then fused with the local 3D grid model to obtain a 3D grid model.

[0021] According to a three-dimensional scene reconstruction method provided by the present invention, the step of performing local mesh reconstruction on the three-dimensional landmarks of each grid in the mesh map to obtain a local three-dimensional mesh model includes:

[0022] Based on the three-dimensional landmarks of each grid in the grid map, determine the fitting plane and points in the plane for each grid;

[0023] The points in the plane are projected onto the fitting plane to obtain the first two-dimensional points after projection.

[0024] The first two-dimensional point is triangulated to obtain the index of the first triangle;

[0025] Based on the first triangle index, the three-dimensional landmarks of each grid are connected to generate three-dimensional triangular patches to obtain a local three-dimensional mesh model.

[0026] According to a three-dimensional scene reconstruction method provided by the present invention, the three-dimensional landmark points of the neighboring grids of each grid in the grid map are reconstructed across grids to be fused with the local three-dimensional grid model to obtain a three-dimensional grid model, comprising:

[0027] Obtain the 3D landmarks of the neighboring grids of each grid in the grid map;

[0028] Calculate the distance between the 3D landmarks of the neighborhood grid and the fitting plane of the corresponding grid;

[0029] Based on the distance, candidate neighborhood points are selected from the three-dimensional landmark points of the neighborhood grid.

[0030] The neighborhood candidate points are merged with the boundary points of the corresponding grid to obtain the points to be merged; wherein the boundary points are selected from the points in the plane.

[0031] The points to be fused are projected onto the fitting plane of the corresponding grid to obtain the second two-dimensional points after projection;

[0032] Triangulation is performed on the second two-dimensional point to obtain the index of the second triangle;

[0033] Based on the second triangle index, the points to be fused are connected to generate three-dimensional triangular patches to obtain a cross-mesh three-dimensional model;

[0034] The local 3D mesh model and the cross-mesh 3D model are fused to obtain a 3D mesh model.

[0035] According to a three-dimensional scene reconstruction method provided by the present invention, the step of converting the matching point pairs into three-dimensional landmarks in a global coordinate system includes:

[0036] Obtain camera calibration parameters and camera pose;

[0037] Based on the camera calibration parameters, the matching point pairs are triangulated to obtain three-dimensional landmark points in the camera coordinate system;

[0038] Based on the camera pose, the 3D landmarks in the camera coordinate system are converted into 3D landmarks in the global coordinate system.

[0039] According to a three-dimensional scene reconstruction method provided by the present invention, before converting the matching point pairs into three-dimensional landmarks in a global coordinate system, the method further includes:

[0040] The coordinates of the matching point pairs are normalized to obtain normalized matching point pairs;

[0041] Based on the essential matrix, the symmetric transfer error corresponding to each normalized matching point pair is calculated; wherein, the essential matrix is ​​calculated based on the camera pose.

[0042] Based on the symmetric transfer error, the matching point pairs are filtered to obtain filtered matching point pairs.

[0043] According to a three-dimensional scene reconstruction method provided by the present invention, before performing feature extraction and feature matching on the scene image to obtain matching point pairs, the method further includes:

[0044] The scene image is subjected to data augmentation processing to obtain an enhanced scene image; wherein the data augmentation processing includes at least one of gamma correction, contrast stretching, and noise reduction filtering.

[0045] The present invention also provides a three-dimensional scene reconstruction device, comprising:

[0046] The receiving module is used to receive scene images acquired by the XR device;

[0047] The acquisition module is used to extract and match features from the scene image to obtain matching point pairs;

[0048] The conversion module is used to convert the matching point pairs into three-dimensional landmark points in the global coordinate system;

[0049] The reconstruction module is used to perform mesh reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model.

[0050] The present invention also provides an XR device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional scene reconstruction method as described above.

[0051] The 3D scene reconstruction method, apparatus, and XR device provided by this invention acquire scene images using a pure binocular camera in an XR device. Then, feature extraction and feature matching are performed on the scene images to obtain matching point pairs between the left and right scene images. These two-dimensional matching point pairs are converted into 3D landmarks in a global coordinate system to obtain 3D point cloud data. Subsequently, based on these 3D landmarks in the global coordinate system, a mesh reconstruction is performed to obtain a 3D mesh model. Through this method, this invention eliminates the need for a dedicated depth camera, utilizing only the standard pure binocular camera included with XR devices to achieve real-time reconstruction of indoor scene 3D models. This significantly simplifies the hardware architecture and greatly reduces costs, greatly promoting the lightweight and compact design of XR devices. Simultaneously, the mesh reconstruction method significantly reduces computational complexity and computational power consumption. In summary, this invention provides native, real-time scene 3D perception and reconstruction capabilities for lightweight, consumer-grade XR devices, which is conducive to promoting the implementation of indoor XR applications. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is one of the flowcharts illustrating the three-dimensional scene reconstruction method provided by the present invention;

[0054] Figure 2 This is the second flowchart illustrating the three-dimensional scene reconstruction method provided by the present invention;

[0055] Figure 3 This is the third flowchart of the three-dimensional scene reconstruction method provided by the present invention;

[0056] Figure 4 This is the fourth flowchart of the three-dimensional scene reconstruction method provided by the present invention;

[0057] Figure 5 This is a schematic diagram of the structure of the three-dimensional scene reconstruction device provided by the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of the XR device provided by the present invention. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0060] This invention proposes a three-dimensional scene reconstruction method, apparatus, and XR equipment, which are described below in conjunction with... Figures 1-6 Describe it.

[0061] Figure 1 This is one of the flowcharts illustrating the three-dimensional scene reconstruction method provided by this invention, such as... Figure 1 As shown, the three-dimensional scene reconstruction method includes steps S110, S120, S130 and S140.

[0062] Step S110: Receive the scene image acquired by the XR device.

[0063] In this embodiment, the three-dimensional scene reconstruction method is applied to an XR device.

[0064] XR devices refer to wearable or portable devices that integrate virtual and real environments through hardware and software technologies to enable human-computer interaction. XR devices include, but are not limited to: VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices. VR devices use computer technology to simulate and generate a three-dimensional virtual space, allowing users to immerse themselves in and interact with it, gaining a truly immersive experience. AR devices use technology to merge virtual information with the real world, overlaying it onto real-world scenes in real time to enhance sensory experience. MR devices mix the real and virtual worlds to create a new visual environment that simultaneously contains physical entities and virtual information, allowing users to interact with these physical entities and virtual information in real time.

[0065] This 3D scene reconstruction method is applicable to scenarios such as games, entertainment, holographic meetings, virtual shopping, and virtual home decoration.

[0066] Here, the scene image is an image of the user's current location captured by an XR device. It can be captured by the XR device's binocular camera, which consists of a left and a right camera; correspondingly, the scene image includes a left-eye scene image and a right-eye scene image. It should be understood that this embodiment is applicable to the 3D reconstruction of real-time scenes, and the corresponding scene image can be a real-time scene image.

[0067] Furthermore, after acquiring the scene image, distortion correction and / or stereo correction can be performed on the scene image.

[0068] Distortion correction uses distortion coefficients to correct the distortion of the original left and right scene images in order to eliminate the effects of lens distortion and ensure the accuracy of subsequent feature extraction and geometric calculations.

[0069] Stereo correction uses calibrated extrinsic parameters to perform stereo correction on distortion-free left and right eye scene images. The goal is to horizontally align the epipolar lines of the two images, greatly simplifying subsequent feature matching searches and outputting a row-aligned corrected image. Stereo correction significantly simplifies the subsequent feature matching process, reducing a two-dimensional search problem to a one-dimensional search, thus improving matching efficiency and accuracy.

[0070] Step S120: Perform feature extraction and feature matching on the scene image to obtain matching point pairs.

[0071] Feature extraction and feature matching are performed on the scene images to obtain matching point pairs from the left and right scene images. The matching point pairs contain the feature point coordinates of the left scene image and the corresponding feature point coordinates of the right scene image.

[0072] Specifically, the left-eye scene image undergoes Scale-Invariant Feature Transform (SIFT) feature extraction to obtain a first keypoint set and a first SIFT feature vector set; the right-eye scene image undergoes SIFT feature extraction to obtain a second keypoint set and a second SIFT feature vector set; based on the first and second SIFT feature vector sets, multiple candidate matching point pairs are searched from the first and second keypoint sets, and the matching distance values ​​between the multiple candidate matching point pairs are obtained; a ratio test is performed on the matching distance values ​​to obtain the ratio test results; based on the ratio test results, the multiple candidate matching point pairs are filtered, and the matching point pairs are obtained based on the filtering results. The specific execution process can be referred to in the following embodiment, and will not be elaborated here.

[0073] Step S130: Convert the matched point pairs into three-dimensional landmarks in the global coordinate system.

[0074] The matching point pairs in the 2D image are converted into 3D landmarks in the global coordinate system to obtain the 3D coordinates of the corresponding points in the scene, so as to facilitate the subsequent 3D reconstruction of the scene.

[0075] Specifically, the camera calibration parameters and camera pose are first obtained; then, based on the camera calibration parameters, the matching point pairs are triangulated to obtain 3D landmarks in the camera coordinate system; subsequently, based on the camera pose, the 3D landmarks in the camera coordinate system are converted into 3D landmarks in the global coordinate system. The specific execution process can be found in the following embodiment, and will not be elaborated here.

[0076] Step S140: Based on the three-dimensional landmarks in the global coordinate system, perform grid reconstruction to obtain a three-dimensional grid model.

[0077] After obtaining the three-dimensional landmarks in the global coordinate system, the three-dimensional landmarks are used to perform mesh reconstruction to convert the point cloud in three-dimensional space into a continuous surface mesh model composed of polygons (usually triangles), that is, a three-dimensional mesh model.

[0078] It should be noted that mesh reconstruction typically consumes less computational power than other 3D scene reconstruction methods, such as TSDF (Truncated Signed Distance Function) and MVS (Multiple View Stereo), making it particularly suitable for XR devices and real-time applications.

[0079] As one implementation method, three-dimensional landmarks in a global coordinate system are divided into spatial grids to obtain a grid map; the three-dimensional landmarks in each grid of the grid map are reconstructed into a grid model.

[0080] As another implementation method, spatial gridding is performed on the 3D landmarks in the global coordinate system to obtain a grid map. Local grid reconstruction is then performed on the 3D landmarks of each grid in the grid map to obtain a local 3D grid model. Cross-grid reconstruction is then performed on the 3D landmarks of the neighboring grids of each grid in the grid map to fuse them with the local 3D grid model to obtain a 3D grid model. The specific execution process can be referred to in the following embodiments, which will not be elaborated here. Compared to the previous implementation method which only performs local grid reconstruction on the 3D landmarks in each grid, this implementation method first performs local grid reconstruction and then cross-grid reconstruction to achieve global grid fusion, which improves the accuracy of the 3D grid model and results in a better reconstructed 3D scene.

[0081] The 3D scene reconstruction method provided in this invention acquires scene images using a pure binocular camera of an XR device. Then, feature extraction and feature matching are performed on the scene images to obtain matching point pairs between the left and right scene images. These 2D matching point pairs are converted into 3D landmarks in a global coordinate system to obtain 3D point cloud data. Subsequently, based on these 3D landmarks in the global coordinate system, a mesh reconstruction is performed to obtain a 3D mesh model. Through this method, this invention eliminates the need for a dedicated depth camera, utilizing only the standard pure binocular camera included with XR devices to achieve real-time reconstruction of indoor scene 3D models. This significantly simplifies the hardware architecture and greatly reduces costs, greatly promoting the lightweight and compact design of XR devices. Simultaneously, the mesh reconstruction method significantly reduces computational complexity and computational power consumption. In summary, this invention provides lightweight, consumer-grade XR devices with native, real-time scene 3D perception and reconstruction capabilities, which is conducive to promoting the implementation of indoor XR applications.

[0082] Based on any of the above embodiments, the scene image includes a left-eye scene image and a right-eye scene image. Figure 2 This is the second flowchart illustrating the three-dimensional scene reconstruction method provided by this invention, as shown below. Figure 2 As shown, step S120 includes: step S121, step S122, step S123, step S124 and step S125.

[0083] Step S121: Scale-Invariant Feature Transform (SIFT) is performed on the left-eye scene image to extract features, resulting in a first set of key points and a first set of SIFT feature vectors.

[0084] In this embodiment, the binocular camera consists of a left-eye camera and a right-eye camera. The scene images captured by the binocular camera include a left-eye scene image and a right-eye scene image. The left-eye scene image refers to the scene image captured by the left-eye camera, and the right-eye scene image refers to the scene image captured by the right-eye camera.

[0085] SIFT (Scale-Invariant Feature Transform) feature extraction is performed on the left-eye scene image to obtain a keypoint set (denoted as the first keypoint set, to be distinguished from the keypoint set of the right-eye scene image) and a SIFT feature vector set (denoted as the first SIFT feature vector set, to be distinguished from the SIFT feature vector set of the right-eye scene image).

[0086] Specifically, the SIFT algorithm is used to detect key points in the left-eye scene image, resulting in a key point set. For each key point in the detected key point set, its SIFT descriptor is calculated. This is a high-dimensional (usually 128-dimensional) vector, denoted as the SIFT feature vector, which describes the gradient direction distribution of the local region around the key point.

[0087] Step S122: Perform SIFT feature extraction on the right eye scene image to obtain a second key point set and a second SIFT feature vector set.

[0088] Similarly, SIFT features are extracted from the right-eye scene image to obtain a keypoint set (denoted as the second keypoint set, to distinguish it from the keypoint set of the left-eye scene image) and a SIFT feature vector set (denoted as the second SIFT feature vector set, to distinguish it from the SIFT feature vector set of the left-eye scene image).

[0089] It should be noted that SIFT features possess scale invariance, rotation invariance, and partial illumination invariance, making them robust in complex scenes. By extracting SIFT features, stable and highly discriminative image feature vectors can be constructed, laying a solid foundation for subsequent feature matching.

[0090] Step S123: Based on the first SIFT feature vector set and the second SIFT feature vector set, search for multiple candidate matching point pairs from the first keypoint set and the second keypoint set, and obtain the matching distance value between the multiple candidate matching point pairs.

[0091] As one implementation method, based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching points corresponding to each key point in the second key point set are searched to obtain multiple candidate matching point pairs, and the matching distance value between each key point in the second key point set and its corresponding multiple candidate matching points is obtained, which is the matching distance value between multiple candidate matching point pairs.

[0092] As another implementation, based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching points corresponding to each key point in the first key point set are searched from the second key point set to obtain multiple candidate matching point pairs, and the matching distance value between each key point in the first key point set and its corresponding multiple candidate matching points is obtained, which is the matching distance value between multiple candidate matching point pairs.

[0093] When performing feature matching, the KNN (K-Nearest Neighbors) algorithm can be used.

[0094] Specifically, taking the second implementation method as an example, for each keypoint in the first keypoint set, the distance between its SIFT feature vector and each SIFT feature vector in the second SIFT feature vector set is calculated. This distance can be Euclidean distance.

[0095] Then, the calculated distances are sorted in ascending order, and the K keypoints with the smallest distances are selected as candidate matching points. Here, K can be 2. Correspondingly, there are two matching distance values, denoted as the best matching distance d1 and the second best matching distance d2. Combining the keypoints with the candidate matching points yields candidate matching point pairs. It should be understood that each keypoint corresponds to K candidate matching points. Multiple candidate matching point pairs can be divided into multiple groups based on the keypoints, with each group including one keypoint and its corresponding K candidate matching points.

[0096] Step S124: Perform a ratio test on the matching distance value to obtain the ratio test result.

[0097] Step S125: Based on the ratio test results, filter the multiple candidate matching point pairs, and obtain matching point pairs based on the filtering results.

[0098] When performing ratio verification, the verification is performed on each pair of candidate matching points, that is, between the candidate matching point pairs corresponding to each key point.

[0099] The ratio verification process is as follows:

[0100] For each pair of candidate matching points, first calculate the ratio between the matching distance values. Specifically, calculate the ratio of the best matching distance d1 to the second best matching distance d2.

[0101] Then, it is checked whether the ratio is less than a preset threshold. The preset threshold can be set to any value in the range of 0.7 to 0.8.

[0102] If the ratio is less than a preset threshold, the candidate matching point pair corresponding to the best matching distance will be retained as the best matching point pair, and the candidate matching point pair consisting of the second-best matching point and its corresponding key point will be filtered out.

[0103] If the ratio is greater than or equal to the preset threshold, it is determined to be an ambiguous match. At this time, all candidate matching point pairs corresponding to the key point are filtered out, and the search continues for the next set of candidate matching point pairs in the first key point set to perform ratio verification and filtering on the K candidate matching points of the next key point.

[0104] By using ratio testing, unreliable matches generated in areas with repetitive textures, blurred regions, or indistinct features can be effectively filtered out, resulting in higher quality matching point pairs and providing more reliable data for subsequent processing.

[0105] The 3D scene reconstruction method provided in this invention can efficiently obtain scene key points and their corresponding stable and highly discriminative SIFT feature vectors through SIFT feature extraction, laying a solid foundation for subsequent feature matching. During feature matching, multiple candidate matching points are found for each key point using the KNN algorithm, and further combined with ratio checks, significantly improving matching reliability while ensuring real-time performance.

[0106] Based on any of the above embodiments Figure 3 This is the third flowchart of the three-dimensional scene reconstruction method provided by the present invention, as shown below. Figure 3 As shown, step S140 includes: step S141, step S142 and step S143.

[0107] Step S141: Divide the three-dimensional landmarks in the global coordinate system into a spatial grid to obtain a grid map.

[0108] In this embodiment, a spatial grid can be used to divide three-dimensional landmarks in the global coordinate system using a grid map structure.

[0109] Specifically, iterate through all the 3D landmarks in the global coordinate system and calculate the minimum and maximum values ​​of these 3D landmarks on the X-axis, Y-axis and Z-axis in the global coordinate system, denoted as min_x, max_x, min_y, max_y, min_z and max_z, respectively.

[0110] Then, based on the minimum and maximum values ​​on the X, Y, and Z axes, and the preset grid resolution, the number of grids in each dimension is calculated. Specifically, the number of grids in the X-axis dimension is calculated as num_x = ceil((max_x - min_x) / r), the number of grids in the Y-axis dimension is calculated as num_y = ceil((max_y - min_y) / r), and the number of grids in the Z-axis dimension is calculated as num_z = ceil((max_z - min_z) / r). Here, ceil stands for ceiling function, which rounds up, and r represents the preset grid resolution. Furthermore, it should be understood that the preset grid resolution affects the size of the grid. The grid should not be too large, otherwise a single grid may contain multiple planes; nor should it be too small, otherwise there may not be enough points to fit the planes. The preset grid resolution can be a static preset value or dynamically adjusted based on the density of 3D landmarks and the current scene.

[0111] Next, a 3D grid container is created to store 3D landmarks. The 3D grid container can be a 3D array or a hash table data structure.

[0112] Iterate through all 3D landmarks in the global coordinate system, calculate their respective grid indices, and add each 3D landmark to the grid point list of its corresponding index to obtain a grid map. It should be understood that this grid map comprises multiple grids, each grid maintaining multiple 3D landmarks; that is, each grid corresponds to a set of 3D landmarks.

[0113] By managing 3D landmarks in a global coordinate system through a grid map structure, massive amounts of point cloud data can be distributed across different grids based on their spatial location. This allows subsequent local and cross-grid reconstructions to be efficiently confined to one or a few relevant grids, avoiding the need to process the entire global map and improving query and computational efficiency. Furthermore, it enables the reconstruction of large-scale scenes to be processed independently or in parallel.

[0114] Step S142: Perform local grid reconstruction on the three-dimensional landmarks of each grid in the grid map to obtain a local three-dimensional grid model.

[0115] For the 3D landmarks of each grid, the discrete point cloud in each grid is transformed into a continuous triangular mesh surface through grid reconstruction, thus obtaining a local 3D mesh model.

[0116] Specifically, based on the 3D landmarks of each grid in the grid map, the fitting plane and points within the plane of each grid are determined; the points within the plane are projected onto the fitting plane to obtain the first 2D points after projection; the first 2D points are triangulated to obtain the first triangle index; based on the first triangle index, the 3D landmarks of each grid are connected to generate 3D triangular patches to obtain a local 3D mesh model. The specific execution process can be found in the following embodiment, which will not be elaborated here.

[0117] Step S143: Perform cross-grid reconstruction on the 3D landmarks of the neighboring grids of each grid in the grid map, so as to merge them with the local 3D grid model to obtain a 3D grid model.

[0118] Since the planes of adjacent grids may have slight angular differences, boundary cracks may exist between grids. Therefore, the 3D landmarks of the neighboring grids of each grid in the grid map are further reconstructed across grids to eliminate the boundary cracks between the original individual grids and generate a smoother grid surface with a more reasonable topology.

[0119] As one implementation method, the 3D landmarks of neighboring grids of each grid in the grid map are obtained, and the distance between the 3D landmarks of the neighboring grids and the fitting plane of the corresponding grid is calculated. Based on the distance, candidate neighboring points are selected from the 3D landmarks of the neighboring grids. The candidate neighboring points and the in-plane points of the corresponding grids are merged to obtain the points to be fused. The points to be fused are projected onto the fitting plane of the corresponding grid to obtain the projected second 2D points. The second 2D points are triangulated to obtain the second triangle index. Based on the second triangle index, the points to be fused are connected to generate 3D triangular patches to obtain a cross-grid 3D model. The local 3D mesh model and the cross-grid 3D model are fused to obtain a 3D mesh model. The specific execution process can be referred to in the following embodiments, which will not be elaborated here.

[0120] As another implementation method, the 3D landmarks of the neighboring grids of each grid in the grid map are obtained; the distance between the 3D landmarks of the neighboring grids and the fitting plane of the corresponding grid is calculated; based on the distance, candidate neighboring points are selected from the 3D landmarks of the neighboring grids; the candidate neighboring points are merged with the boundary points of the corresponding grids to obtain the points to be fused; wherein, the boundary points are selected from points in the plane; the points to be fused are projected onto the fitting plane of the corresponding grid to obtain the projected second 2D points; the second 2D points are triangulated to obtain the second triangle index; based on the second triangle index, the points to be fused are connected to generate 3D triangular patches to obtain a cross-grid 3D model; the local 3D mesh model and the cross-grid 3D model are fused to obtain a 3D mesh model. The specific execution process can be referred to in the following embodiments, which will not be elaborated here.

[0121] It should be noted that both implementation methods can address the issue of cracks or discontinuities that may occur at mesh boundaries. However, compared to the first implementation method, the second method, by selecting boundary points from the in-plane points of each mesh and then merging them with corresponding neighboring candidate points to obtain the points to be fused, can significantly reduce the computational complexity of subsequent triangulation, save computational resources, and avoid redundant reconstruction. Furthermore, since boundary points are the only regions where cracks or discontinuities may occur between meshes, focusing on this region for cross-mesh fusion can more accurately stitch adjacent meshes together and avoid smoothing out originally clear geometric features (such as corners and edges) at the boundaries due to the introduction of a large number of internal points.

[0122] The 3D scene reconstruction method provided in this invention first performs local mesh reconstruction of the 3D landmarks of each grid in a grid map, and then performs cross-mesh reconstruction of the 3D landmarks of the neighboring grids of each grid to eliminate boundary cracks between individual grids and achieve global mesh fusion. This method improves the accuracy of the 3D mesh model, resulting in a better reconstructed 3D scene.

[0123] Based on any of the above embodiments, step S142 includes: step S1421, step S1422, step S1423 and step S1424.

[0124] Step S1421: Determine the fitting plane and in-plane points of each grid based on the three-dimensional landmarks of each grid in the grid map.

[0125] As one implementation method, the RANSAC (Random Sample Consensus) algorithm is used to determine the fitting plane for each grid.

[0126] Specifically, for any given grid, three 3D landmarks within the grid are randomly sampled. A candidate plane is then calculated based on these sampled landmarks. Next, the distances from all 3D landmarks within the grid to this candidate plane are calculated, and the number of inliers whose distances are less than a preset distance threshold is counted. Euclidean distance can be used. This process is iterated multiple times, and the candidate plane with the highest number of inliers is retained as the fitted plane for that grid, i.e., the optimal plane. Simultaneously, 3D landmarks within the grid whose distances to this fitted plane are less than the preset distance threshold are identified and denoted as inliers within the plane.

[0127] By using the RANSAC algorithm to determine the fitting plane and distance filtering, points that do not belong to the fitting plane can be effectively filtered out, reducing the interference of noise and outliers on subsequent mesh construction and avoiding the generation of false planes in space.

[0128] As another implementation method, the fitting plane of each grid can be determined by using the PCA (Principal Component Analysis) algorithm.

[0129] Specifically, for any grid, the centroid, i.e. the geometric center of all three-dimensional landmarks, is calculated based on the three-dimensional landmarks in the grid. Then, a covariance matrix is ​​constructed based on the centroid and the three-dimensional landmarks. Eigenvalue decomposition is performed on the covariance moments. Based on the obtained eigenvalues ​​and eigenvectors, the plane normal vector is determined. Based on the centroid and the plane normal vector, the fitting plane can be determined.

[0130] Furthermore, three-dimensional landmarks within the grid whose distance to the fitted plane is less than a preset distance threshold are identified and denoted as points within the plane.

[0131] Compared to the RANSAC algorithm, the PCA algorithm is faster and more suitable for real-time reconstruction of indoor scenes.

[0132] Furthermore, parallel processing can be used for processing each grid to improve processing efficiency and reduce latency.

[0133] Step S1422: Project the points in the plane onto the fitting plane to obtain the first two-dimensional points after projection.

[0134] Projecting points in the plane onto the fitted plane yields two-dimensional points, which are denoted as the first two-dimensional point.

[0135] Step S1423: Triangulate the first two-dimensional point to obtain the first triangle index.

[0136] Step S1424: Connect the three-dimensional landmarks of each grid according to the first triangle index to generate a three-dimensional triangular patch to obtain a local three-dimensional mesh model.

[0137] The Delaunay Triangulation algorithm is used to triangulate the first two-dimensional point to obtain a triangle index, denoted as the first triangle index. Then, based on the first triangle index, the three-dimensional landmarks of each mesh are connected to generate three-dimensional triangular patches, thus obtaining a local three-dimensional mesh model.

[0138] Among them, the Delaunay Triangulation algorithm is a method for connecting a set of points on a plane into a triangular mesh.

[0139] The 3D scene reconstruction method provided in this invention determines the fitting plane and in-plane points of each grid based on the 3D landmarks of each grid in the grid map. This filters out 3D landmarks that do not belong to the fitting plane, reducing noise and outliers from interfering with subsequent mesh construction and avoiding the generation of false planes in space, resulting in a more accurate local 3D mesh model. Then, the in-plane points are projected onto the fitting plane, and the first two-dimensional points after projection are triangulated to generate 3D triangular patches, thus obtaining the local 3D mesh model. By transforming the 3D surface fitting problem into a 2D plane triangulation problem, the calculation is greatly simplified, saving computational resources. Furthermore, since triangulation is performed on a 2D plane, it generates triangular meshes that avoid elongated triangles as much as possible, resulting in a higher quality local 3D mesh model.

[0140] Based on any of the above embodiments, step S143 includes: step S1431, step S1432, step S1433, step S1434, step S1435, step S1436, step S1437 and step S1438.

[0141] Step S1431: Obtain the three-dimensional landmarks of the neighboring grids of each grid in the grid map.

[0142] Step S1432: Calculate the distance between the three-dimensional landmarks of the neighborhood grid and the fitting plane of the corresponding grid.

[0143] Since the planes of adjacent grids may have slight angular differences, boundary cracks may exist between grids. Therefore, in this embodiment, the 3D landmarks of the neighboring grids of each grid in the grid map are further reconstructed across grids to eliminate the boundary cracks between the original individual grids.

[0144] Retrieve the 3D landmarks of the neighboring grids of each grid in the grid map. The neighboring grids are the six grids adjacent to the current grid. For example, if the center coordinates of the current grid are (x, y, z), then its neighboring grids are (x+1, y, z), (x, y+1, z), (x, y, z+1), (x-1, y, z), (x, y-1, z), and (x, y, z-1).

[0145] Then, the distance between the 3D landmarks of the neighboring grid and the fitting plane of the corresponding grid is calculated to filter points in the neighboring grid that are closer to the fitting plane of the current grid. The distance can be Euclidean distance.

[0146] Step S1433: Based on the distance, filter out candidate neighborhood points from the three-dimensional landmark points of the neighborhood grid.

[0147] From the 3D landmarks of the neighborhood grid, points whose distance is less than a preset distance threshold are selected as neighborhood candidate points. This selection ensures that subsequent fusion only merges those neighborhood candidate points that are spatially close to the current grid's fitted plane, further avoiding the generation of false planes in space.

[0148] Step S1434: Merge the neighborhood candidate points with the boundary points of the corresponding grid to obtain the points to be merged; wherein the boundary points are selected from the points in the plane.

[0149] The method for determining the boundary points of each grid is as follows: First, the boundary edges are determined based on the first triangle index obtained during the local network reconstruction process through triangulation. Specifically, based on the first triangle index, the number of triangles sharing each edge is counted; edges shared by only one triangle are considered boundary edges. Then, the corresponding two-dimensional points are obtained based on the boundary edges. A two-dimensional point is a vertex connected to at least one boundary edge. Finally, this two-dimensional point is mapped back onto the fitting plane to filter out boundary points from the points within the plane.

[0150] After selecting the boundary points, the neighboring candidate points are merged with the boundary points of the corresponding grid to obtain the points to be merged.

[0151] Step S1435: Project the point to be fused onto the fitting plane of the corresponding mesh to obtain the second two-dimensional point after projection.

[0152] The points to be fused are projected onto the fitting plane of the corresponding mesh to obtain the projected two-dimensional points, which are denoted as the second two-dimensional points.

[0153] Step S1436: Triangulate the second two-dimensional point to obtain the second triangle index.

[0154] Step S1437: Connect the points to be fused according to the second triangle index to generate a three-dimensional triangular patch to obtain a cross-mesh three-dimensional model.

[0155] The Delaunay Triangulation algorithm is used to triangulate the second two-dimensional points to obtain triangle indices, denoted as the second triangle index. Then, based on the second triangle index, the points to be merged are connected to generate three-dimensional triangular patches, resulting in a cross-mesh 3D model.

[0156] Step S1438: The local three-dimensional mesh model and the cross-mesh three-dimensional model are fused to obtain a three-dimensional mesh model.

[0157] The local 3D mesh model is fused with the cross-mesh 3D model to obtain a new 3D mesh model. Specifically, the 3D triangular faces of the original local 3D mesh model are replaced with the 3D triangular faces of the newly generated cross-mesh 3D model to achieve global mesh fusion.

[0158] The 3D scene reconstruction method provided in this invention calculates the distance between the 3D landmarks of neighboring grids and the fitting plane of the corresponding grid to select candidate neighboring points from the 3D landmarks of the neighboring grids. This allows subsequent fusion of only those neighboring points that are spatially close to the fitting plane of the current grid, further avoiding the generation of false planes in space. Then, the points to be fused, obtained by merging the candidate neighboring points with the boundary points of the corresponding grid, are projected onto the fitting plane of the corresponding grid. Triangulation is performed on the projected second 2D points to generate 3D triangular patches to obtain a cross-grid 3D model. Subsequently, the local 3D grid model and the cross-grid 3D model are fused to obtain a 3D grid model. Through the above method, points of adjacent grids and the current grid that are close to the same plane can be fused, and triangulation can be uniformly performed on these points to generate continuous and seamless triangular patches across grid boundaries, thereby solving the problem of cracks or discontinuities that may occur at grid boundaries. Furthermore, compared to directly merging neighboring candidate points with in-plane points of the corresponding mesh to obtain the point to be fused, this embodiment first selects boundary points from the in-plane points of each mesh, and then merges them with the corresponding neighboring candidate points to obtain the point to be fused. This can greatly reduce the computational complexity of subsequent triangulation, save computational resources, and avoid redundant reconstruction. At the same time, since boundary points are the only regions where cracks or discontinuities may occur between meshes, focusing on this region for cross-mesh fusion can more accurately stitch adjacent meshes together, and can also avoid smoothing out originally clear geometric features (such as corners and edges) at the boundary due to the introduction of a large number of internal points.

[0159] Based on any of the above embodiments Figure 4 This is the fourth flowchart illustrating the three-dimensional scene reconstruction method provided by this invention, as shown below. Figure 4 As shown, step S130 includes: step S131, step S132 and step S133.

[0160] Step S131: Obtain camera calibration parameters and camera pose.

[0161] Camera calibration parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include the intrinsic parameter matrix of the left camera and the intrinsic parameter matrix of the right camera, while the camera extrinsic parameters include the rotation matrix and translation vector of the right camera relative to the left camera.

[0162] The front-end VIO (Visual-Inertial Odometry) of an XR device estimates the camera's pose in the global coordinate system in real time. The camera pose is represented by a rotation matrix and a translation vector, where the rotation matrix R... global It is a 3×3 matrix representing the rotation from the camera coordinate system to the global coordinate system; the translation vector T global It is a 3×1 vector representing the position of the camera's optical center in the global coordinate system.

[0163] It should be understood that the cameras of XR devices are pre-calibrated. Through calibration, the real-world space and the camera's imaging space can be aligned to ensure the geometric consistency of subsequent data acquisition.

[0164] Step S132: Based on the camera calibration parameters, the matching point pair is triangulated to obtain three-dimensional landmark points in the camera coordinate system.

[0165] The triangulation process is as follows:

[0166] First, a projection matrix is ​​constructed based on the camera calibration parameters.

[0167] Based on the intrinsic parameter matrix of the left-eye camera, construct the projection matrix of the left-eye camera. Details are as follows:

[0168] P l =K l ·[I|O];

[0169] Where: P l K represents the projection matrix of the left eye camera. l The left eye camera intrinsic parameter matrix is ​​represented by I, which represents a 3x3 identity matrix, and O represents a 3×1 zero vector. The symbol [I|O] indicates that I and O are horizontally concatenated to form a 3x4 matrix.

[0170] Simultaneously, based on the intrinsic and extrinsic parameters of the right-eye camera, a projection matrix for the right-eye camera is constructed. The details are as follows:

[0171] P r =K r ·[R|T];

[0172] Where: P r K represents the projection matrix of the right eye camera. r Let R represent the intrinsic parameter matrix of the right camera, R represent the rotation matrix of the right camera relative to the left camera, and T represent the translation vector of the right camera relative to the left camera. The symbol [R|T] indicates that R and T are horizontally concatenated into a 3x4 matrix.

[0173] Then, based on the matching point pairs and the projection matrix, the coordinates of the 3D points corresponding to each matching point pair in space are solved in the coordinate system of the left eye camera.

[0174] Specifically, for each pair of matching points, a homogeneous equation is constructed; then, SVD (Singular Value Decomposition) is performed on the constructed homogeneous equation to obtain the optimal 3D point coordinates. It should be understood that the solution yields 3D point coordinates in the camera coordinate system (with the left eye camera as the origin).

[0175] Step S133: Based on the camera pose, convert the 3D landmarks in the camera coordinate system into 3D landmarks in the global coordinate system.

[0176] The conversion process is as follows:

[0177] X global =R global ·X cam +T global .

[0178] Among them, X global Represents a 3D landmark in the global coordinate system, X cam R represents a 3D landmark in the camera coordinate system. global Let T represent the rotation matrix. global This represents the translation vector.

[0179] Furthermore, after obtaining the 3D landmarks in the global coordinate system, depth filtering can be applied. For example, 3D landmarks with depth values ​​≤ 0 and / or depth values ​​> a preset depth value can be removed. Depth filtering further improves the robustness of subsequent 3D reconstruction tasks and enhances the accuracy of the reconstructed scene.

[0180] The three-dimensional scene reconstruction method provided in this embodiment of the invention performs triangulation processing on matching point pairs according to camera calibration parameters to obtain three-dimensional landmarks in the camera coordinate system; then, according to the camera pose, the three-dimensional landmarks in the camera coordinate system are converted into three-dimensional landmarks in the global coordinate system. Through the above conversion, a data foundation can be provided for the subsequent construction of a physically and spatiotemporally consistent scene map.

[0181] Based on any of the above embodiments, before step S130, the three-dimensional scene reconstruction method further includes: steps S151, S152, S153, S154, S155 and S156.

[0182] Step S151: Perform coordinate normalization processing on the matching point pairs to obtain normalized matching point pairs.

[0183] In this embodiment, the coordinates of the matching point pairs are normalized according to the camera calibration parameters to obtain normalized matching point pairs.

[0184] Among them, the camera calibration parameters include the left eye camera intrinsic parameter matrix K. l and the intrinsic parameter matrix K of the right eye camera r For matching point pairs (p) l _i, p r _i), where p l _i and p r _i represents the coordinates of a point in the left and right view scene images, respectively.

[0185] Calculate the normalized coordinates x in the left-eye scene image based on the left-eye camera intrinsic parameter matrix. l _i, x l _i=K l -1 ·p l _i; Simultaneously, based on the intrinsic parameter matrix of the right eye camera, the normalized coordinates x in the right eye scene image are calculated. r _i, x r _i=K r -1 ·p r _i. Furthermore, we obtain the normalized matching point pairs (x... l _i, x r _i).

[0186] Step S152: Calculate the symmetric transfer error corresponding to each normalized matching point pair based on the essential matrix. The essential matrix is ​​calculated based on the camera pose.

[0187] Camera pose, including rotation matrix R global Translation vector T global .

[0188] The essential matrix describes the geometric relationship of the same point in space from two different camera perspectives. Its formula is: E = [T] global ] × R global Where E represents the essential matrix, [T global ] × Represents the translation vector T global An antisymmetric matrix.

[0189] For all normalized matching point pairs (x l _i, x r _i), calculate the symmetric transition error, which serves as an indicator of whether the normalized matching point pairs satisfy the essential matrix.

[0190] Specifically, based on the essential matrix and normalized matching point pairs, the epipolar line (denoted as l) corresponding to the points in the left-eye scene image is calculated. r The epipolar lines (denoted as l) corresponding to points in the right eye scene image and the point in the right eye scene image. l _i). Specifically:

[0191] l r _i=E·x l _i;

[0192] l l _i=E·x r _i.

[0193] Then, calculate the point x in the right eye scene image. r The epipolar line l corresponding to the point in the left-eye scene image. r The distance to _i, denoted as dist l _i. Simultaneously, calculate point x in the left-eye scene image. l _i to point x in the right view scene image r The epipolar line l corresponding to _i l The distance to _i, denoted as dist r _i.

[0194] For the two calculated distances (dist l _i and dist r _i) are summed to obtain the symmetric transfer error.

[0195] Step S153: Based on the symmetric transfer error, filter the matching point pairs to obtain filtered matching point pairs.

[0196] The system checks whether the symmetric transition error is less than a preset threshold. If it is, the normalized matching point pair and its corresponding original matching point pair are considered interior points of the essential matrix, thus determining the filtered matching point pair. It should be noted that since the original matching point pairs are filtered using the above method, they are denoted as the filtered matching point pair.

[0197] It should be noted that when determining the interior point, the index of the interior point should be recorded for later use.

[0198] At this point, step S130 includes: converting the filtered matching point pairs into three-dimensional landmarks in the global coordinate system.

[0199] After filtering the matching point pairs using the aforementioned epipolar geometric constraints, the filtered matching point pairs are converted into 3D landmarks in the global coordinate system. The specific execution process can be found in the above embodiment and will not be elaborated upon here.

[0200] The three-dimensional scene reconstruction method provided in this invention uses epipolar geometry constraints to robustly eliminate erroneous matches caused by duplicate textures, occlusion, noise, etc., which greatly improves the reliability of matching point pairs. This is the key to ensuring the accuracy of subsequent triangulation.

[0201] Based on any of the above embodiments, before step S120, the three-dimensional scene reconstruction method may further include: performing data enhancement processing on the scene image to obtain an enhanced scene image; wherein, the data enhancement processing includes at least one of gamma correction, contrast stretching, and noise reduction filtering.

[0202] In this embodiment, gamma correction refers to performing a non-linear operation on the grayscale values ​​of the scene image, so that the grayscale values ​​of the processed scene image have an exponential relationship with those of the unprocessed scene image. Gamma correction enhances the overall brightness and contrast adaptability of the scene image, improving its performance in excessively dark or bright environments.

[0203] Contrast stretching maps the original grayscale range of a scene image to a target range using a linear mapping function, thus fully utilizing the dynamic range of the display device. Its core principle is to redistribute existing grayscale values, making details in both dark and bright areas clearer, thereby improving visual discernibility.

[0204] Denoising filtering refers to using edge-preserving filtering algorithms (such as nonlocal mean denoising or bilateral filtering) to suppress noise in scene images while maximizing the preservation of edge structures and texture details. Its core mechanism achieves a balance between noise smoothing and feature protection through dual weight constraints of spatial proximity and gray-level similarity.

[0205] At this point, step S120 includes: performing feature extraction and feature matching on the enhanced scene image to obtain matching point pairs.

[0206] After data augmentation of the scene image, feature extraction and feature matching are further performed on the augmented scene image to obtain matching point pairs. The specific execution process can be found in the above embodiment and will not be elaborated here.

[0207] The three-dimensional scene reconstruction method provided in this invention can enhance the robustness of scene image features and improve image quality through data augmentation processing, making the feature vectors obtained from subsequent feature extraction more discriminative, thereby reducing ambiguity in the subsequent feature matching stage and helping to improve the effect of three-dimensional scene reconstruction.

[0208] The three-dimensional scene reconstruction device provided by the present invention is described below. The three-dimensional scene reconstruction device described below and the three-dimensional scene reconstruction method described above can be referred to in correspondence.

[0209] Figure 5 This is a schematic diagram of the structure of the three-dimensional scene reconstruction device provided by the present invention, as shown below. Figure 5 As shown, the device includes a receiving module 510, an acquiring module 520, a conversion module 530, and a reconstruction module 540; wherein:

[0210] The receiving module 510 is used to receive scene images acquired by the XR device;

[0211] The acquisition module 520 is used to extract and match features from the scene image to obtain matching point pairs;

[0212] The conversion module 530 is used to convert the matching point pairs into three-dimensional landmark points in the global coordinate system;

[0213] The reconstruction module 540 is used to perform grid-based reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model.

[0214] The 3D scene reconstruction device provided in this invention acquires scene images using a pure binocular camera of an XR device. Then, it performs feature extraction and feature matching on the scene images to obtain matching point pairs between the left and right scene images. These two-dimensional matching point pairs are then converted into 3D landmarks in a global coordinate system to obtain 3D point cloud data. Based on these 3D landmarks, a mesh reconstruction is performed to obtain a 3D mesh model. Through this method, this invention eliminates the need for a dedicated depth camera, utilizing only the standard pure binocular camera included with XR devices to achieve real-time reconstruction of indoor scene 3D models. This significantly simplifies the hardware architecture and greatly reduces costs, greatly promoting the lightweight and compact design of XR devices. Simultaneously, the mesh reconstruction method significantly reduces computational complexity and computational power consumption. In summary, this invention provides native, real-time scene 3D perception and reconstruction capabilities for lightweight, consumer-grade XR devices, which is beneficial for promoting the implementation of indoor XR applications.

[0215] Furthermore, the scene image includes a left-eye scene image and a right-eye scene image, and the acquisition module 520 is specifically used for:

[0216] The left-eye scene image is subjected to scale-invariant feature transformation (SIFT) feature extraction to obtain a first keypoint set and a first SIFT feature vector set;

[0217] SIFT feature extraction is performed on the right-eye scene image to obtain a second keypoint set and a second SIFT feature vector set;

[0218] Based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching points corresponding to each key point in the first key point set are searched from the second key point set, and the matching distance value between each key point in the first key point set and its corresponding multiple candidate matching points is obtained.

[0219] A ratio test is performed on the matching distance value to obtain the ratio test result;

[0220] Based on the ratio test results, the multiple candidate matching points are filtered, and matching point pairs are obtained based on the filtering results.

[0221] Furthermore, the reconstruction module 540 includes:

[0222] The partitioning unit is used to divide the three-dimensional landmark points in the global coordinate system into a spatial grid to obtain a grid map;

[0223] The local reconstruction unit is used to perform local grid reconstruction of the three-dimensional landmarks of each grid in the grid map to obtain a local three-dimensional grid model.

[0224] The cross-grid reconstruction unit is used to perform cross-grid reconstruction of the three-dimensional landmarks of the neighboring grids of each grid in the grid map, so as to merge them with the local three-dimensional grid model to obtain a three-dimensional grid model.

[0225] Furthermore, the local reconstruction unit is specifically used for:

[0226] Based on the three-dimensional landmarks of each grid in the grid map, determine the fitting plane and points in the plane for each grid;

[0227] The points in the plane are projected onto the fitting plane to obtain the first two-dimensional points after projection.

[0228] The first two-dimensional point is triangulated to obtain the index of the first triangle;

[0229] Based on the first triangle index, the three-dimensional landmarks of each grid are connected to generate three-dimensional triangular patches to obtain a local three-dimensional mesh model.

[0230] Furthermore, the cross-grid reconstruction unit is specifically used for:

[0231] Obtain the 3D landmarks of the neighboring grids of each grid in the grid map;

[0232] Calculate the distance between the 3D landmarks of the neighborhood grid and the fitting plane of the corresponding grid;

[0233] Based on the distance, candidate neighborhood points are selected from the three-dimensional landmark points of the neighborhood grid.

[0234] The neighborhood candidate points are merged with the boundary points of the corresponding grid to obtain the points to be merged; wherein the boundary points are selected from the points in the plane.

[0235] The points to be fused are projected onto the fitting plane of the corresponding grid to obtain the second two-dimensional points after projection;

[0236] Triangulation is performed on the second two-dimensional point to obtain the index of the second triangle;

[0237] Based on the second triangle index, the points to be fused are connected to generate three-dimensional triangular patches to obtain a cross-mesh three-dimensional model;

[0238] The local 3D mesh model and the cross-mesh 3D model are fused to obtain a 3D mesh model.

[0239] Furthermore, the conversion module 530 is specifically used for:

[0240] Obtain camera calibration parameters and camera pose;

[0241] Based on the camera calibration parameters, the matching point pairs are triangulated to obtain three-dimensional landmark points in the camera coordinate system;

[0242] Based on the camera pose, the 3D landmarks in the camera coordinate system are converted into 3D landmarks in the global coordinate system.

[0243] Furthermore, the 3D scene reconstruction device also includes a filtering module; the filtering module is used for:

[0244] The coordinates of the matching point pairs are normalized to obtain normalized matching point pairs;

[0245] Based on the essential matrix, the symmetric transfer error corresponding to each normalized matching point pair is calculated; wherein, the essential matrix is ​​calculated based on the camera pose.

[0246] Based on the symmetric transfer error, the matching point pairs are filtered to obtain filtered matching point pairs;

[0247] The conversion module 530 is specifically used for:

[0248] The filtered matching point pairs are converted into three-dimensional landmarks in the global coordinate system.

[0249] Furthermore, the three-dimensional scene reconstruction device also includes:

[0250] The processing module is used to perform data enhancement processing on the scene image to obtain an enhanced scene image; wherein the data enhancement processing includes at least one of gamma correction, contrast stretching, and noise reduction filtering;

[0251] The acquisition module 520 is specifically used for:

[0252] Feature extraction and feature matching are performed on the enhanced scene image to obtain matching point pairs.

[0253] It should be noted that the three-dimensional scene reconstruction device provided in this embodiment of the invention can implement all the method steps implemented in the above three-dimensional scene reconstruction method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0254] Figure 6An example is a schematic diagram of the physical structure of an XR device, such as... Figure 6 As shown, the XR device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a 3D scene reconstruction method. This method includes: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into 3D landmarks in a global coordinate system; and performing mesh reconstruction based on the 3D landmarks in the global coordinate system to obtain a 3D mesh model.

[0255] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0256] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional scene reconstruction method provided by the above methods. The method includes: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmarks in a global coordinate system; and performing mesh reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model.

[0257] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the three-dimensional scene reconstruction method provided by the above methods. The method includes: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmarks in a global coordinate system; and performing mesh reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model.

[0258] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0259] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0260] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reconstructing a three-dimensional scene, characterized in that, Applications in extended reality XR devices include: Receive scene images acquired by the XR device; Feature extraction and feature matching are performed on the scene image to obtain matching point pairs; Convert the matched point pairs into three-dimensional landmarks in the global coordinate system; Based on the three-dimensional landmarks in the global coordinate system, a grid reconstruction is performed to obtain a three-dimensional grid model; The step of reconstructing a three-dimensional mesh model based on three-dimensional landmarks in the global coordinate system includes: The three-dimensional landmarks in the global coordinate system are divided into spatial grids to obtain a grid map; For the three-dimensional landmarks of each grid in the grid map, local grid reconstruction is performed to obtain a local three-dimensional grid model; The 3D landmarks of the neighboring grids of each grid in the grid map are reconstructed across grids and then fused with the local 3D grid model to obtain a 3D grid model. The process of locally reconstructing the 3D landmarks of each grid in the grid map to obtain a local 3D grid model includes: Based on the three-dimensional landmarks of each grid in the grid map, determine the fitting plane and points in the plane for each grid; The points in the plane are projected onto the fitting plane to obtain the first two-dimensional points after projection. The first two-dimensional point is triangulated to obtain the index of the first triangle; Based on the first triangle index, the three-dimensional landmarks of each grid are connected to generate three-dimensional triangular patches to obtain a local three-dimensional mesh model.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The scene images include a left-eye scene image and a right-eye scene image. The process of extracting and matching features from the scene images to obtain matching point pairs includes: The left-eye scene image is subjected to scale-invariant feature transformation (SIFT) feature extraction to obtain a first keypoint set and a first SIFT feature vector set; SIFT feature extraction is performed on the right-eye scene image to obtain a second keypoint set and a second SIFT feature vector set; Based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching point pairs are searched from the first keypoint set and the second keypoint set, and the matching distance value between the multiple candidate matching point pairs is obtained. A ratio test is performed on the matching distance value to obtain the ratio test result; Based on the ratio test results, the multiple candidate matching point pairs are filtered, and matching point pairs are obtained based on the filtering results.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of reconstructing the 3D landmarks of neighboring grids of each grid in the grid map across grids, and then fusing them with the local 3D grid model to obtain a 3D grid model, includes: Obtain the 3D landmarks of the neighboring grids of each grid in the grid map; Calculate the distance between the 3D landmarks of the neighborhood grid and the fitting plane of the corresponding grid; Based on the distance, candidate neighborhood points are selected from the three-dimensional landmark points of the neighborhood grid. The neighborhood candidate points are merged with the boundary points of the corresponding grid to obtain the points to be merged; wherein the boundary points are selected from the points in the plane. The points to be fused are projected onto the fitting plane of the corresponding grid to obtain the second two-dimensional points after projection; Triangulation is performed on the second two-dimensional point to obtain the index of the second triangle; Based on the second triangle index, the points to be fused are connected to generate three-dimensional triangular patches to obtain a cross-mesh three-dimensional model; The local 3D mesh model and the cross-mesh 3D model are fused to obtain a 3D mesh model.

4. The three-dimensional scene reconstruction method according to claim 1, characterized in that, The step of converting the matched point pairs into three-dimensional landmarks in the global coordinate system includes: Obtain camera calibration parameters and camera pose; Based on the camera calibration parameters, the matching point pairs are triangulated to obtain three-dimensional landmark points in the camera coordinate system; Based on the camera pose, the 3D landmarks in the camera coordinate system are converted into 3D landmarks in the global coordinate system.

5. The three-dimensional scene reconstruction method according to any one of claims 1 to 4, characterized in that, Before converting the matched point pairs into 3D landmarks in the global coordinate system, the method further includes: The coordinates of the matching point pairs are normalized to obtain normalized matching point pairs; Based on the essential matrix, the symmetric transfer error corresponding to each normalized matching point pair is calculated; wherein, the essential matrix is ​​calculated based on the camera pose. Based on the symmetric transfer error, the matching point pairs are filtered to obtain filtered matching point pairs.

6. The three-dimensional scene reconstruction method according to any one of claims 1 to 4, characterized in that, Before performing feature extraction and feature matching on the scene image to obtain matching point pairs, the method further includes: The scene image is subjected to data augmentation processing to obtain an enhanced scene image; wherein the data augmentation processing includes at least one of gamma correction, contrast stretching, and noise reduction filtering.

7. A three-dimensional scene reconstruction device, characterized in that, include: The receiving module is used to receive scene images acquired by the XR device; The acquisition module is used to extract and match features from the scene image to obtain matching point pairs; The conversion module is used to convert the matching point pairs into three-dimensional landmark points in the global coordinate system; The reconstruction module is used to perform mesh reconstruction based on the three-dimensional landmarks in the global coordinate system to obtain a three-dimensional mesh model; The reconstruction module is specifically used for: The three-dimensional landmarks in the global coordinate system are divided into spatial grids to obtain a grid map; For the three-dimensional landmarks of each grid in the grid map, local grid reconstruction is performed to obtain a local three-dimensional grid model; The 3D landmarks of the neighboring grids of each grid in the grid map are reconstructed across grids and then fused with the local 3D grid model to obtain a 3D grid model. The reconstruction module is also specifically used for: Based on the three-dimensional landmarks of each grid in the grid map, determine the fitting plane and points in the plane for each grid; The points in the plane are projected onto the fitting plane to obtain the first two-dimensional points after projection. The first two-dimensional point is triangulated to obtain the index of the first triangle; Based on the first triangle index, the three-dimensional landmarks of each grid are connected to generate three-dimensional triangular patches to obtain a local three-dimensional mesh model.

8. An XR device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional scene reconstruction method as described in any one of claims 1 to 6.

Images