Three-dimensional scene reconstruction method and apparatus, and XR device

Abstract

The invention provides a three-dimensional scene reconstruction method and apparatus, and an XR device, and relates to the technical field of image processing, the three-dimensional scene reconstruction method is applied to the XR device, and comprises the steps of receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain a matching point pair; converting the matching point pairs into three-dimensional landmark points under a global coordinate system; and performing gridding reconstruction according to the three-dimensional landmark points under the global coordinate system to obtain a three-dimensional grid model. According to the method, real-time reconstruction of the indoor scene three-dimensional model can be realized on the premise of not depending on a special depth camera.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional scene reconstruction method, device and XR equipment. Background Art

[0002] With the development of intelligent hardware and artificial intelligence technologies, XR (Extended Reality) devices have become widely used in various scenarios, including gaming, entertainment, holographic conferencing, virtual shopping, and virtual home renovation. In practical applications, using XR devices to reconstruct the user's real-world environment in 3D, achieving an immersive experience that blends virtual and real life, has become a key technological trend. The key to achieving this functionality lies in efficient and accurate 3D reconstruction of indoor scenes, providing the foundation for subsequent virtual content generation and interaction.

[0003] Currently, most mainstream 3D reconstruction algorithms rely on ToF (Time-of-Flight) cameras or RGB-D (Red, Green, Blue-Depth) cameras to acquire scene depth maps to assist in real-time reconstruction of 3D models. However, due to size, power consumption, and cost constraints, some XR devices (especially portable ones) lack these dedicated depth cameras or are unable to operate these high-power dedicated depth cameras continuously. Therefore, existing 3D reconstruction technologies based on ToF or RGB-D cameras are difficult to directly apply to these XR devices.

[0004] Therefore, how to achieve real-time reconstruction of three-dimensional models of indoor scenes without relying on dedicated depth cameras has become a technical challenge that urgently needs to be solved in this field. Summary of the Invention

[0005] The present invention provides a three-dimensional scene reconstruction method, device and XR equipment, which can realize real-time reconstruction of the three-dimensional model of the indoor scene without relying on a dedicated depth camera.

[0006] The present invention provides a three-dimensional scene reconstruction method, which is applied to XR equipment, comprising: Receiving a scene image acquired by the XR device; Performing feature extraction and feature matching on the scene image to obtain matching point pairs; Converting the matching point pairs into three-dimensional landmark points in a global coordinate system; Grid reconstruction is performed based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0007] According to a three-dimensional scene reconstruction method provided by the present invention, the scene image includes a left-view scene image and a right-view scene image, and the feature extraction and feature matching of the scene image are performed to obtain matching point pairs, including: Performing scale-invariant feature transform (SIFT) feature extraction on the left-view scene image to obtain a first key point set and a first SIFT feature vector set; Performing SIFT feature extraction on the right scene image to obtain a second key point set and a second SIFT feature vector set; Searching for a plurality of candidate matching point pairs from the first key point set and the second key point set according to the first SIFT feature vector set and the second SIFT feature vector set, and obtaining matching distance values ​​between the plurality of candidate matching point pairs; Performing a ratio test on the matching distance value to obtain a ratio test result; The plurality of candidate matching point pairs are filtered according to the ratio check result, and matching point pairs are obtained according to the filtering result.

[0008] According to a three-dimensional scene reconstruction method provided by the present invention, gridding reconstruction is performed based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model, including: Performing spatial grid division on the three-dimensional landmark points in the global coordinate system to obtain a grid map; Performing local grid reconstruction on the three-dimensional landmark points of each grid in the grid map to obtain a local three-dimensional grid model; The three-dimensional landmark points of the neighborhood grids of each grid in the grid map are reconstructed across the grid to be fused with the local three-dimensional grid model to obtain a three-dimensional grid model.

[0009] According to a three-dimensional scene reconstruction method provided by the present invention, the three-dimensional landmark points of each grid in the grid map are locally gridded and reconstructed to obtain a local three-dimensional grid model, including: Determining the fitting plane and the points within the plane of each grid according to the three-dimensional landmark points of each grid in the grid map; Projecting the points in the plane onto the fitting plane to obtain a first two-dimensional point after projection; Performing triangulation on the first two-dimensional point to obtain a first triangle index; According to the first triangle index, the three-dimensional landmark points of each mesh are connected to generate a three-dimensional triangular facet to obtain a local three-dimensional mesh model.

[0010] According to a 3D scene reconstruction method provided by the present invention, cross-grid reconstruction of 3D landmark points of neighboring grids of each grid in the grid map is performed to fuse with the local 3D grid model to obtain a 3D grid model, including: Obtaining three-dimensional landmark points of neighboring grids of each grid in the grid map; Calculating the distance between the three-dimensional landmark point of the neighborhood grid and the fitting plane of the corresponding grid; According to the distance, screening the neighborhood candidate points from the three-dimensional landmark points of the neighborhood grid; Merging the candidate points in the neighborhood with the boundary points of the corresponding grid to obtain points to be fused; wherein the boundary points are obtained by screening the points in the plane; Projecting the point to be fused onto the fitting plane of the corresponding grid to obtain a second two-dimensional point after projection; Performing triangulation on the second two-dimensional point to obtain a second triangle index; Connecting the points to be fused according to the second triangle index to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; The local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model.

[0011] According to a three-dimensional scene reconstruction method provided by the present invention, converting the matching point pairs into three-dimensional landmark points in a global coordinate system includes: Get camera calibration parameters and camera pose; triangulate the matching point pairs according to the camera calibration parameters to obtain three-dimensional landmark points in the camera coordinate system; According to the camera pose, the three-dimensional landmark points in the camera coordinate system are converted into three-dimensional landmark points in the global coordinate system.

[0012] According to a three-dimensional scene reconstruction method provided by the present invention, before converting the matching point pairs into three-dimensional landmark points in a global coordinate system, the method further includes: performing coordinate normalization processing on the matching point pairs to obtain normalized matching point pairs; Calculate the symmetric transfer error corresponding to each normalized matching point pair according to the essential matrix; wherein the essential matrix is ​​calculated according to the camera pose; The matching point pairs are filtered according to the symmetric transfer errors to obtain filtered matching point pairs.

[0013] According to a three-dimensional scene reconstruction method provided by the present invention, before extracting and matching features of the scene image to obtain matching point pairs, the method further includes: Performing data enhancement processing on the scene image to obtain an enhanced scene image; wherein the data enhancement processing includes at least one of gamma correction, contrast stretching, and denoising filtering.

[0014] The present invention also provides a three-dimensional scene reconstruction device, comprising: A receiving module, used to receive scene images acquired by the XR device; An acquisition module is used to perform feature extraction and feature matching on the scene image to obtain matching point pairs; A conversion module, used to convert the matching point pairs into three-dimensional landmark points in a global coordinate system; The reconstruction module is used to perform grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0015] The present invention also provides an XR device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the three-dimensional scene reconstruction method described above is implemented.

[0016] The three-dimensional scene reconstruction method, device and XR device provided by the present invention acquire scene images through the pure binocular camera of the XR device, then perform feature extraction and feature matching on the scene images to obtain matching point pairs between the left-eye scene image and the right-eye scene image, and convert the two-dimensional matching point pairs into three-dimensional landmark points in the global coordinate system to obtain three-dimensional point cloud data, and then perform grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model. Through the above-mentioned method, the present invention does not need to rely on a dedicated depth camera, but only uses the ordinary pure binocular camera that is standard with the XR device to achieve real-time reconstruction of the three-dimensional model of the indoor scene, achieving significant simplification of the hardware architecture and a substantial reduction in cost, which greatly promotes the lightweight and small-volume design of the XR device. At the same time, the use of a grid reconstruction method can significantly reduce computational complexity and reduce computing power loss. In summary, the present invention provides lightweight, consumer-grade XR devices with native, real-time scene three-dimensional perception and reconstruction capabilities, which is conducive to promoting the implementation of indoor XR applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the three-dimensional scene reconstruction method provided by the present invention; Figure 2 This is the second flow chart of the three-dimensional scene reconstruction method provided by the present invention; Figure 3 This is the third flow chart of the three-dimensional scene reconstruction method provided by the present invention; Figure 4 This is the fourth flow chart of the three-dimensional scene reconstruction method provided by the present invention; Figure 5 It is a structural schematic diagram of the three-dimensional scene reconstruction device provided by the present invention; Figure 6 It is a structural schematic diagram of the XR device provided by the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The present invention proposes a three-dimensional scene reconstruction method, device and XR equipment. Figures 1-6 Provide a description.

[0021] Figure 1 This is one of the flow charts of the three-dimensional scene reconstruction method provided by the present invention, such as Figure 1 As shown, the three-dimensional scene reconstruction method includes: step S110, step S120, step S130 and step S140.

[0022] Step S110: Receive a scene image acquired by the XR device.

[0023] In this embodiment, the three-dimensional scene reconstruction method is applied to an XR device.

[0024] XR devices are wearable or portable devices that achieve human-computer interaction by integrating virtual and real environments through hardware and software technologies. XR devices include, but are not limited to, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and MR (Mixed Reality) devices. VR devices use computer technology to simulate and generate three-dimensional virtual spaces, allowing users to immerse themselves in and interact with them for an immersive experience. AR devices use technology to fuse virtual information with the real world, overlaying it onto real scenes in real time to enhance the sensory experience. MR devices blend the real and virtual worlds to create a new visual environment that combines physical entities and virtual information, allowing users to interact with these entities and virtual information in real time.

[0025] This three-dimensional scene reconstruction method is suitable for scenarios such as gaming, entertainment, holographic conferencing, virtual shopping, and virtual home decoration.

[0026] Here, the scene image is an image of the user's current scene captured by the XR device. This image can be captured using the XR device's binocular camera, which consists of a left camera and a right camera. Accordingly, the scene image includes both the left and right scene images. It should be understood that this embodiment is applicable to 3D reconstruction of real-time scenes, and accordingly, the scene image can be a real-time scene image.

[0027] Furthermore, after the scene image is acquired, distortion correction and / or stereo correction may be performed on the scene image.

[0028] Distortion correction is the process of using distortion coefficients to correct the distortion of the original left-eye scene images and right-eye scene images to eliminate the influence of lens distortion and ensure the accuracy of subsequent feature extraction and geometric calculations.

[0029] Stereo correction uses calibrated extrinsic parameters to perform stereo correction on the undistorted left and right scene images. The goal is to horizontally align the epipolar lines of the two images, greatly simplifying the subsequent feature matching search and outputting a corrected image with aligned rows. Stereo correction significantly simplifies the subsequent feature matching process, reducing the two-dimensional search problem to a one-dimensional search, improving matching efficiency and accuracy.

[0030] Step S120 , performing feature extraction and feature matching on the scene image to obtain matching point pairs.

[0031] Feature extraction and feature matching are performed on the scene image to obtain matching point pairs from the left scene image and the right scene image. The matching point pairs include the feature point coordinates of the left scene image and the corresponding feature point coordinates of the right scene image.

[0032] Specifically, scale-invariant feature transform (SIFT) feature extraction is performed on the left-eye scene image to obtain a first key point set and a first SIFT feature vector set; SIFT feature extraction is performed on the right-eye scene image to obtain a second key point set and a second SIFT feature vector set; based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching point pairs are searched from the first key point set and the second key point set, and matching distance values ​​between the multiple candidate matching point pairs are obtained; a ratio test is performed on the matching distance values ​​to obtain a ratio test result; based on the ratio test result, the multiple candidate matching point pairs are filtered, and matching point pairs are obtained based on the filtering result. The specific execution process can be referred to the following embodiment and will not be described in detail here.

[0033] Step S130: convert the matching point pairs into three-dimensional landmark points in a global coordinate system.

[0034] The matching point pairs of the two-dimensional image are converted into three-dimensional landmark points in the global coordinate system to obtain the three-dimensional coordinate points of the corresponding points in the scene, so as to facilitate the subsequent three-dimensional reconstruction of the scene.

[0035] Specifically, the camera calibration parameters and camera pose are first obtained. Then, based on the camera calibration parameters, the matching point pairs are triangulated to obtain 3D landmark points in the camera coordinate system. Furthermore, based on the camera pose, the 3D landmark points in the camera coordinate system are converted to 3D landmark points in the global coordinate system. The specific implementation process can be found in the following examples and is not detailed here.

[0036] Step S140 , performing grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0037] After obtaining the three-dimensional landmark points in the global coordinate system, mesh reconstruction is performed based on these three-dimensional landmark points to convert the point cloud in the three-dimensional space into a continuous surface mesh model composed of polygons (usually triangles), that is, a three-dimensional mesh model.

[0038] It should be noted that grid reconstruction usually has lower computing power consumption than other 3D scene reconstruction methods, such as TSDF (Truncated Signed Distance Function) and MVS (Multiple View Stereo), and is particularly suitable for XR devices and real-time applications.

[0039] As an implementation method, the three-dimensional landmark points in the global coordinate system are spatially gridded to obtain a grid map; the three-dimensional landmark points of each grid in the grid map are grid-reconstructed to obtain a three-dimensional grid model.

[0040] As another implementation method, the three-dimensional landmark points in the global coordinate system are spatially gridded to obtain a grid map; the three-dimensional landmark points of each grid in the grid map are locally gridded and reconstructed to obtain a local three-dimensional grid model; the three-dimensional landmark points of the neighborhood grids of each grid in the grid map are cross-grid reconstructed to fuse with the local three-dimensional grid model to obtain a three-dimensional grid model. The specific implementation process can be referred to the following embodiments and will not be described here. Compared with the previous implementation method of only locally gridding the three-dimensional landmark points in each grid, in this implementation method, by first performing local gridding reconstruction and then cross-gridding reconstruction to achieve global grid fusion, the accuracy of the three-dimensional grid model can be improved, and the reconstructed three-dimensional scene effect is better.

[0041] The three-dimensional scene reconstruction method provided by the embodiment of the present invention obtains the scene image through the pure binocular camera of the XR device, and then performs feature extraction and feature matching on the scene image to obtain the matching point pair between the left-eye scene image and the right-eye scene image, and converts the two-dimensional matching point pair into a three-dimensional landmark point in the global coordinate system to obtain three-dimensional point cloud data, and then performs grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model. Through the above method, the present invention does not need to rely on a dedicated depth camera, but only uses the ordinary pure binocular camera that is standard on the XR device to achieve real-time reconstruction of the three-dimensional model of the indoor scene, which achieves a significant simplification of the hardware architecture and a substantial reduction in cost, and greatly promotes the lightweight and small-volume design of the XR device. At the same time, the use of a grid reconstruction method can significantly reduce the computational complexity and reduce the computing power loss. In summary, the present invention provides lightweight, consumer-grade XR devices with native, real-time scene three-dimensional perception and reconstruction capabilities, which is conducive to promoting the implementation of indoor XR applications.

[0042] Based on any of the above embodiments, the scene image includes a left-eye scene image and a right-eye scene image, Figure 2 This is the second flow chart of the three-dimensional scene reconstruction method provided by the present invention, such as Figure 2 As shown, step S120 includes: step S121, step S122, step S123, step S124 and step S125.

[0043] Step S121 , performing scale-invariant feature transform (SIFT) feature extraction on the left-view scene image to obtain a first key point set and a first SIFT feature vector set.

[0044] In this embodiment, the binocular camera consists of a left-eye camera and a right-eye camera. The scene image captured by the binocular camera includes a left-eye scene image and a right-eye scene image, wherein the left-eye scene image refers to the scene image captured by the left camera, and the right-eye scene image refers to the scene image captured by the right camera.

[0045] Perform SIFT (Scale-Invariant Feature Transform) feature extraction on the left scene image to obtain a key point set (referred to as the first key point set to distinguish it from the key point set of the right scene image) and a SIFT feature vector set (referred to as the first SIFT feature vector set to distinguish it from the SIFT feature vector set of the right scene image).

[0046] Specifically, the SIFT algorithm is used to detect key points in the left-view scene image, generating a key point set. For each key point in the detected key point set, its SIFT descriptor is calculated. This is a high-dimensional (usually 128-dimensional) vector, denoted as a SIFT feature vector. This SIFT feature vector describes the distribution of gradient directions in the local area surrounding the key point.

[0047] Step S122 , performing SIFT feature extraction on the right-eye scene image to obtain a second key point set and a second SIFT feature vector set.

[0048] Similarly, SIFT features are extracted from the right scene image to obtain a key point set (recorded as the second key point set to distinguish it from the key point set of the left scene image) and a SIFT feature vector set (recorded as the second SIFT feature vector set to distinguish it from the SIFT feature vector set of the left scene image).

[0049] It should be noted that SIFT features are scale-invariant, rotation-invariant, and partially illumination-invariant, making them highly robust in complex scenes. By extracting SIFT features, we can construct a stable and highly discriminative image feature vector, laying a solid foundation for subsequent feature matching.

[0050] Step S123 : searching for multiple candidate matching point pairs from the first key point set and the second key point set according to the first SIFT feature vector set and the second SIFT feature vector set, and obtaining matching distance values ​​between the multiple candidate matching point pairs.

[0051] As an implementation method, based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching points corresponding to each key point in the second key point set are searched from the second key point set to obtain multiple candidate matching point pairs, and the matching distance values ​​between each key point in the second key point set and its corresponding multiple candidate matching points are obtained, that is, the matching distance values ​​between the multiple candidate matching point pairs.

[0052] As another implementation method, based on the first SIFT feature vector set and the second SIFT feature vector set, multiple candidate matching points corresponding to each key point in the first key point set are searched from the second key point set to obtain multiple candidate matching point pairs, and the matching distance values ​​between each key point in the first key point set and its corresponding multiple candidate matching points are obtained, that is, the matching distance values ​​between the multiple candidate matching point pairs.

[0053] When matching features, the KNN (K-Nearest Neighbors) algorithm can be used.

[0054] Specifically, taking the second embodiment as an example, first, for each key point in the first key point set, the distance between its SIFT feature vector and each SIFT feature vector in the second SIFT feature vector set is calculated, wherein the distance may be a Euclidean distance.

[0055] The calculated distances are then sorted from smallest to largest, and the top K keypoints with the smallest distances are selected as candidate matching points. K can be 2. Correspondingly, there are two matching distance values, denoted as the best matching distance d1 and the second-best matching distance d2. Keypoints are combined with candidate matching points to obtain candidate matching point pairs. It should be understood that each keypoint corresponds to K candidate matching points. Multiple candidate matching point pairs can be divided into multiple groups based on keypoints, with each group consisting of a keypoint and its corresponding K candidate matching points.

[0056] Step S124: performing a ratio test on the matching distance value to obtain a ratio test result.

[0057] Step S125 : filtering the plurality of candidate matching point pairs according to the ratio check result, and obtaining matching point pairs according to the filtering result.

[0058] When performing ratio verification, verification is performed on each set of candidate matching point pairs, that is, verification is performed between the candidate matching point pairs corresponding to each key point.

[0059] The ratio verification process is as follows: For each set of candidate matching point pairs, the ratio between the matching distance values ​​is first calculated. Specifically, the ratio of the best matching distance d1 to the second best matching distance d2 is calculated.

[0060] Then, it is detected whether the ratio is less than a preset threshold value, wherein the preset threshold value can be set to any value in the range of 0.7 to 0.8.

[0061] If the ratio is less than a preset threshold, the candidate matching point pair corresponding to the best matching distance is retained as the best matching point pair, and the candidate matching point pair consisting of the second best matching point and its corresponding key point is filtered out.

[0062] If the ratio is greater than or equal to the preset threshold, it is determined to be a fuzzy match. At this time, all candidate matching point pairs corresponding to the key point are filtered out, and the search continues for the next set of candidate matching point pairs in the first key point set to perform ratio testing and filtering on the K candidate matching points of the next key point.

[0063] The ratio test can effectively filter out unreliable matches generated in repeated textures, fuzzy areas or areas with unclear features, making the matching point pairs higher in quality and providing more reliable data for subsequent processing.

[0064] The 3D scene reconstruction method provided by the present invention uses SIFT feature extraction to efficiently obtain scene key points and their corresponding stable and highly discriminative SIFT feature vectors, laying a solid foundation for subsequent feature matching. During feature matching, multiple candidate matching points are found for each key point using the KNN algorithm. This is further combined with a ratio test to significantly improve matching reliability while ensuring real-time performance.

[0065] Based on any of the above embodiments, Figure 3 This is the third flow chart of the three-dimensional scene reconstruction method provided by the present invention, such as Figure 3 As shown, the step S140 includes: step S141, step S142 and step S143.

[0066] Step S141 , performing spatial grid division on the three-dimensional landmark points in the global coordinate system to obtain a grid map.

[0067] In this embodiment, the three-dimensional landmark points in the global coordinate system may be spatially gridded using a grid map structure.

[0068] Specifically, traverse all three-dimensional landmark points in the global coordinate system, and calculate the minimum and maximum values ​​of these three-dimensional landmark points on the X-axis, Y-axis, and Z-axis in the global coordinate system, which are recorded as min_x, max_x, min_y, max_y, min_z, and max_z respectively.

[0069] Then, according to the minimum and maximum values ​​on the X-axis, Y-axis and Z-axis, and the preset grid resolution, the number of grids in each dimension is calculated. Among them, the number of grids in the X-axis dimension num_x=ceil((max_x-min_x) / r), the number of grids in the Y-axis dimension num_y=ceil((max_y-min_y) / r), and the number of grids in the Z-axis dimension num_z=ceil((max_z-min_z) / r). Among them, the full name of ceil is ceiling function, which is a rounding function, and r represents the preset grid resolution. In addition, it should be understood that the preset grid resolution will affect the size of the grid. The grid should not be too large, otherwise a grid may contain multiple planes; it should not be too small, otherwise there may not be enough points to fit the plane. The preset grid resolution can be a static preset value, or it can be dynamically adjusted according to the density of the three-dimensional landmark points and the current scene.

[0070] Next, create a 3D grid container to store 3D landmark points. The 3D grid container can be a 3D array or hash table data structure.

[0071] Traverse all 3D landmark points in the global coordinate system, calculate the grid index to which they belong, and add each 3D landmark point to the grid point list of the corresponding index to obtain a grid map. It should be understood that the grid map includes multiple grids, each of which maintains multiple 3D landmark points, that is, each grid corresponds to a 3D landmark point set.

[0072] By managing 3D landmarks in a global coordinate system through a grid map structure, massive amounts of point cloud data can be distributed across different grids based on spatial location. This allows subsequent localized and cross-grid reconstruction to be efficiently confined to one or several relevant grids, avoiding the need to process the entire global map and improving query and computational efficiency. Furthermore, it allows the reconstruction of large-scale scenes to be processed independently or in parallel.

[0073] Step S142 , performing local grid reconstruction on the three-dimensional landmark points of each grid in the grid map to obtain a local three-dimensional grid model.

[0074] For the three-dimensional landmark points of each grid, the discrete point clouds in each grid are converted into continuous triangular mesh surfaces through mesh reconstruction to obtain a local three-dimensional mesh model.

[0075] Specifically, based on the 3D landmark points of each grid in the grid map, a fitting plane and points within the plane are determined for each grid; the points within the plane are projected onto the fitting plane to obtain the projected first 2D points; the first 2D points are triangulated to obtain first triangle indices; and based on the first triangle indices, the 3D landmark points of each grid are connected to generate 3D triangular facets to obtain a local 3D mesh model. The specific implementation process can be referred to in the following embodiment and is not described in detail here.

[0076] Step S143 , performing cross-grid reconstruction on the three-dimensional landmark points of the neighboring grids of each grid in the grid map, so as to fuse them with the local three-dimensional grid model to obtain a three-dimensional grid model.

[0077] Since the planes of adjacent meshes may have slight angle differences, boundary cracks may exist between meshes. Therefore, we further perform cross-grid reconstruction on the 3D landmark points of the neighboring meshes of each mesh in the mesh map to eliminate the boundary cracks between the original individual meshes and generate a smoother and more topologically reasonable mesh surface.

[0078] As an implementation method, the three-dimensional landmark points of the neighborhood grids of each grid in the grid map are obtained, and the distance between the three-dimensional landmark points of the neighborhood grid and the fitting plane of the corresponding grid is calculated; based on the distance, the neighborhood candidate points are screened from the three-dimensional landmark points of the neighborhood grid; the neighborhood candidate points and the points in the plane of the corresponding grid are merged to obtain the points to be fused; the points to be fused are projected onto the fitting plane of the corresponding grid to obtain the second two-dimensional points after projection; the second two-dimensional points are triangulated to obtain the second triangle index; based on the second triangle index, the points to be fused are connected to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; the local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model. The specific execution process can be referred to the following embodiment, which will not be described in detail here.

[0079] As another implementation method, the three-dimensional landmark points of the neighborhood grids of each grid in the grid map are obtained; the distance between the three-dimensional landmark points of the neighborhood grid and the fitting plane of the corresponding grid is calculated; based on the distance, the neighborhood candidate points are screened from the three-dimensional landmark points of the neighborhood grid; the neighborhood candidate points are merged with the boundary points of the corresponding grid to obtain the points to be fused; wherein the boundary points are screened from the points in the plane; the points to be fused are projected onto the fitting plane of the corresponding grid to obtain the second two-dimensional points after projection; the second two-dimensional points are triangulated to obtain the second triangle index; based on the second triangle index, the points to be fused are connected to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; the local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model. The specific execution process can be referred to the following embodiment, which will not be described in detail here.

[0080] It should be noted that both implementations can solve the problem of cracks or discontinuities that may occur at the boundaries of the grids. However, compared to the first implementation, the second implementation obtains boundary points by screening the in-plane points of each grid, and then merges them with the corresponding neighborhood candidate points to obtain the points to be fused. This can greatly reduce the computational complexity of subsequent triangulation, save computing resources, and avoid redundant reconstruction. At the same time, since the boundary points are the only areas where cracks or discontinuities may occur between grids, focusing on this area for cross-grid fusion can more accurately stitch adjacent grids, and can also avoid the original clear geometric features (such as corners and edges) being smoothed out at the boundaries due to the introduction of a large number of internal points.

[0081] The 3D scene reconstruction method provided by the present invention first performs local grid reconstruction of the 3D landmark points of each grid in a grid map, and then performs cross-grid reconstruction of the 3D landmark points of each grid's neighboring grids. This eliminates boundary cracks between individual grids and achieves global grid fusion. This approach improves the accuracy of the 3D grid model and produces a better reconstructed 3D scene.

[0082] Based on any of the above embodiments, step S142 includes: step S1421, step S1422, step S1423 and step S1424.

[0083] Step S1421 : determining the fitting plane and the points within the plane of each grid according to the three-dimensional landmark points of each grid in the grid map.

[0084] As an implementation method, a RANSAC (Random Sample Consensus) algorithm is used to determine the fitting plane of each grid.

[0085] Specifically, for any grid, three 3D landmark points are randomly sampled within the grid. A candidate plane is calculated based on the sampled 3D landmark points. The distances from all 3D landmark points within the grid to the candidate plane are then calculated, and the number of inliers whose distances are less than a preset distance threshold is counted. Euclidean distance can be used as the distance. This process is repeated multiple times, and the candidate plane with the largest number of inliers is retained as the fitting plane for the grid, also known as the optimal plane. Simultaneously, 3D landmark points within the grid whose distances to the fitting plane are less than a preset distance threshold are identified and recorded as inliers.

[0086] The RANSAC algorithm is used to determine the fitting plane and distance screening, which can effectively filter out points that do not belong to the fitting plane, reduce the interference of noise and outliers on subsequent grid construction, and avoid the generation of false planes in space.

[0087] As another implementation, the fitting plane of each grid may be determined by using a PCA (Principal Component Analysis) algorithm.

[0088] Specifically, for any grid, the centroid, that is, the geometric center of all three-dimensional landmark points, is calculated based on the three-dimensional landmark points in the grid; then, a covariance matrix is ​​constructed based on the centroid and the three-dimensional landmark points, and the covariance moment is decomposed by eigenvalues. The plane normal vector is determined based on the obtained eigenvalues ​​and eigenvectors; based on the centroid and the plane normal vector, the fitting plane can be determined.

[0089] Furthermore, three-dimensional landmark points within the grid whose distance to the fitting plane is less than a preset distance threshold are determined and recorded as in-plane points.

[0090] Compared with the RANSAC algorithm, the PCA algorithm is faster and more suitable for real-time reconstruction of indoor scenes.

[0091] In addition, parallel processing can be used for the processing of each grid to improve processing efficiency and reduce latency.

[0092] Step S1422: Project the points in the plane onto the fitting plane to obtain a first two-dimensional point after projection.

[0093] Project the points in the plane onto the fitting plane to obtain the projected two-dimensional point, which is recorded as the first two-dimensional point.

[0094] Step S1423: triangulate the first two-dimensional point to obtain a first triangle index.

[0095] Step S1424: Connect the three-dimensional landmark points of each mesh according to the first triangle index to generate a three-dimensional triangular facet to obtain a local three-dimensional mesh model.

[0096] The first two-dimensional point is triangulated using the Delaunay triangulation algorithm to obtain a triangle index, recorded as the first triangle index. Furthermore, based on the first triangle index, the three-dimensional landmark points of each mesh are connected to generate three-dimensional triangular facets to obtain a local three-dimensional mesh model.

[0097] Among them, the Delaunay Triangulation algorithm is a method of connecting a set of points on a plane into a triangular mesh.

[0098] The three-dimensional scene reconstruction method provided by the embodiment of the present invention determines the fitting plane and in-plane points of each grid based on the three-dimensional landmark points of each grid in the grid map, so as to filter out the three-dimensional landmark points that do not belong to the fitting plane, reduce the interference of noise and outliers on the subsequent grid construction, avoid the generation of false planes in space, and make the constructed local three-dimensional grid model more accurate. Then, the in-plane points are projected onto the fitting plane, and the first two-dimensional points after projection are triangulated to generate three-dimensional triangular facets to obtain a local three-dimensional grid model. In the above manner, the three-dimensional surface fitting problem is converted into a two-dimensional plane triangulation problem, which can greatly simplify the calculation and save computing power. At the same time, the triangulation is performed on a two-dimensional plane, and a triangular mesh that avoids narrow and long triangles as much as possible can be generated, so that the constructed local three-dimensional grid model is of higher quality.

[0099] Based on any of the above embodiments, step S143 includes: step S1431, step S1432, step S1433, step S1434, step S1435, step S1436, step S1437 and step S1438.

[0100] Step S1431, obtaining three-dimensional landmark points of the neighborhood grids of each grid in the grid map.

[0101] Step S1432: Calculate the distance between the three-dimensional landmark point of the neighborhood grid and the fitting plane of the corresponding grid.

[0102] Since the planes of adjacent grids may have slight angle differences, boundary cracks may exist between grids. Therefore, this embodiment further performs cross-grid reconstruction on the 3D landmark points of the neighboring grids of each grid in the grid map to eliminate the boundary cracks between the original individual grids.

[0103] Get the 3D landmark points of the neighboring grids of each grid in the grid map. The neighboring grids are the six grids adjacent to the current grid. Assuming the center coordinates of the current grid are (x, y, z), then its neighboring grids are (x+1, y, z), (x, y+1, z), (x, y, z+1), (x-1, y, z), (x, y-1, z), and (x, y, z-1).

[0104] Then, the distance between the 3D landmark points of the neighboring grid and the fitting plane of the corresponding grid is calculated to filter the points of the neighboring grid that are closest to the fitting plane of the current grid. The distance can be calculated using Euclidean distance.

[0105] Step S1433 , screening and obtaining neighborhood candidate points from the three-dimensional landmark points of the neighborhood grid according to the distance.

[0106] From the 3D landmark points of the neighborhood grid, points with distances less than a preset distance threshold are selected as candidate neighborhood points. This selection ensures that only those candidate neighborhood points that are spatially close to the current grid's fitted plane are subsequently fused, further avoiding the generation of false planes in space.

[0107] Step S1434 , merging the neighborhood candidate points with the boundary points of the corresponding grid to obtain points to be fused; wherein the boundary points are obtained by screening the points in the plane.

[0108] The boundary points of each mesh are determined by first determining the boundary edges based on the first triangle index obtained through triangulation during the local network reconstruction process. Specifically, based on the first triangle index, the number of triangles sharing each edge is counted. Edges shared by only one triangle are considered boundary edges. Then, based on the boundary edges, the corresponding 2D points are obtained. A 2D point is a vertex connected to at least one boundary edge. Finally, these 2D points are mapped back onto the fitting plane to filter the boundary points from the points within the plane.

[0109] After filtering out the boundary points, the neighborhood candidate points are merged with the boundary points of the corresponding grid to obtain the points to be fused.

[0110] Step S1435 : Projecting the point to be fused onto the fitting plane of the corresponding grid to obtain a projected second two-dimensional point.

[0111] Project the point to be fused onto the fitting plane of the corresponding grid to obtain the projected two-dimensional point, which is recorded as the second two-dimensional point.

[0112] Step S1436: triangulate the second two-dimensional point to obtain a second triangle index.

[0113] Step S1437: Connect the points to be fused according to the second triangle index to generate three-dimensional triangle facets to obtain a cross-grid three-dimensional model.

[0114] The second two-dimensional point is triangulated using the Delaunay Triangulation algorithm to obtain a triangle index, which is recorded as the second triangle index. Then, the points to be fused are connected based on the second triangle index to generate a three-dimensional triangular facet to obtain a cross-grid three-dimensional model.

[0115] Step S1438: Fusing the local three-dimensional grid model with the cross-grid three-dimensional model to obtain a three-dimensional grid model.

[0116] The local 3D mesh model is fused with the cross-mesh 3D model to obtain a 3D mesh model. That is, the 3D triangular facets of the original local 3D mesh model are replaced with the 3D triangular facets of the newly generated cross-mesh 3D model to achieve global mesh fusion.

[0117] The three-dimensional scene reconstruction method provided by the embodiment of the present invention calculates the distance between the three-dimensional landmark points of the neighborhood grids of each grid and the fitting plane of the corresponding grid, so as to screen out neighborhood candidate points from the three-dimensional landmark points of the neighborhood grids, so that only those neighborhood points that are spatially close to the fitting plane of the current grid are subsequently fused, further avoiding the generation of false planes in space. Then, the points to be fused obtained by merging the neighborhood candidate points with the boundary points of the corresponding grid are projected onto the fitting plane of the corresponding grid, and the second two-dimensional points after projection are triangulated to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; then, the local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model. In this way, the points of the adjacent grids and the current grid close to the same plane can be fused, and triangulation can be uniformly performed on these points to generate cross-grid boundary, continuous, seamless triangular facets, thereby solving the problem of cracks or discontinuities that may appear at the grid boundaries. Furthermore, compared to directly merging neighborhood candidate points with the in-plane points of the corresponding mesh to obtain the points to be fused, this embodiment first filters boundary points from the in-plane points of each mesh, then merges them with the corresponding neighborhood candidate points to obtain the points to be fused. This significantly reduces the computational complexity of subsequent triangulation, conserves computing resources, and avoids redundant reconstruction. Furthermore, because boundary points are the only areas where cracks or discontinuities may occur between meshes, focusing on these areas for cross-mesh fusion allows for more precise stitching of adjacent meshes and avoids the smoothing of previously clear geometric features (such as corners and edges) at the boundaries caused by the inclusion of a large number of internal points.

[0118] Based on any of the above embodiments, Figure 4 This is the fourth flow chart of the three-dimensional scene reconstruction method provided by the present invention, such as Figure 4 As shown, step S130 includes: step S131, step S132 and step S133.

[0119] Step S131, obtaining camera calibration parameters and camera pose.

[0120] The camera calibration parameters include camera intrinsic parameters and camera extrinsic parameters. The camera intrinsic parameters include the left camera intrinsic parameter matrix and the right camera intrinsic parameter matrix, and the camera extrinsic parameters include the rotation matrix and translation vector of the right camera relative to the left camera.

[0121] The front-end VIO (Visual-Inertial Odometry) of the XR device estimates the camera's pose in the global coordinate system in real time. The camera pose is represented by a rotation matrix and a translation vector, where the rotation matrix R global Is a 3×3 matrix representing the rotation from the camera coordinate system to the global coordinate system; the translation vector T globalis a 3×1 vector representing the position of the camera's optical center in the global coordinate system.

[0122] It should be understood that the camera of the XR device is calibrated in advance. Through the calibration process, the real-world space and the camera imaging space can be aligned to ensure the geometric consistency of the subsequent collected data.

[0123] Step S132 : triangulating the matching point pairs according to the camera calibration parameters to obtain three-dimensional landmark points in the camera coordinate system.

[0124] The triangulation process is as follows: First, the projection matrix is ​​constructed based on the camera calibration parameters.

[0125] According to the left camera intrinsic parameter matrix, the left camera projection matrix is ​​constructed. The details are as follows: P l =K l ·[I|O]; Where: P l Represents the left camera projection matrix, K l Represents the intrinsic parameter matrix of the left camera, I represents the 3x3 identity matrix, O represents the 3x1 zero vector, and the symbol [I|O] means horizontally concatenating I and O into a 3x4 matrix.

[0126] At the same time, the right camera projection matrix is ​​constructed based on the right camera intrinsic parameter matrix and camera extrinsic parameters. The details are as follows: P r =K r [R|T]; Where: P r Represents the right camera projection matrix, K r Represents the intrinsic parameter matrix of the right camera, R represents the rotation matrix of the right camera relative to the left camera, T represents the translation vector of the right camera relative to the left camera, and the symbol [R|T] means horizontally splicing R and T into a 3x4 matrix.

[0127] Then, based on the matching point pairs and the projection matrix, the coordinates of the three-dimensional points corresponding to each matching point pair in space in the left camera coordinate system are solved.

[0128] Specifically, for each matching point pair, a homogeneous equation is constructed. Then, the constructed homogeneous equation is solved using SVD (Singular Value Decomposition) to obtain the optimal 3D point coordinates. It should be understood that the solution obtains the 3D point coordinates in the camera coordinate system (with the left camera as the origin).

[0129] Step S133 : converting the three-dimensional landmark points in the camera coordinate system into three-dimensional landmark points in the global coordinate system according to the camera pose.

[0130] The conversion process is as follows: X global =R global ·X cam +T global .

[0131] Among them, X global Represents a three-dimensional landmark point in the global coordinate system, X cam Represents the three-dimensional landmark point in the camera coordinate system, R global represents the rotation matrix, T global Represents the translation vector.

[0132] Furthermore, after converting the 3D landmark points to the global coordinate system, depth filtering can be performed on them. For example, 3D landmark points with depth values ​​≤ 0 and / or depth values ​​greater than a preset depth value can be eliminated. This depth filtering can further improve the robustness of subsequent 3D reconstruction tasks and enhance the accuracy of the reconstructed scene.

[0133] The three-dimensional scene reconstruction method provided by an embodiment of the present invention triangulates the matching point pairs according to the camera calibration parameters to obtain three-dimensional landmark points in the camera coordinate system; then, according to the camera posture, the three-dimensional landmark points in the camera coordinate system are converted into three-dimensional landmark points in the global coordinate system. Through the above conversion, a data foundation can be provided for the subsequent three-dimensional construction of a physically consistent and spatiotemporally consistent scene map.

[0134] Based on any of the above embodiments, before the above step S130, the three-dimensional scene reconstruction method further includes: step S151, step S152, step S153, step S154, step S155 and step S156.

[0135] Step S151 : performing coordinate normalization processing on the matching point pairs to obtain normalized matching point pairs.

[0136] In this embodiment, coordinate normalization processing is performed on the matching point pairs according to the camera calibration parameters to obtain normalized matching point pairs.

[0137] Among them, the camera calibration parameters include the left camera intrinsic parameter matrix K l And the right camera intrinsic parameter matrix K r For matching point pairs (p l _i, p r _i), where p l _i and p r _i are the point coordinates in the left scene image and the right scene image respectively.

[0138] According to the left camera intrinsic parameter matrix, calculate the normalized coordinate x in the left scene image l_i,x l _i=K l -1 ·p l _i; At the same time, according to the right camera internal parameter matrix, calculate the normalized coordinate x in the right scene image r _i,x r _i=K r -1 ·p r _i. Then, we get the normalized matching point pair (x l _i,x r _i).

[0139] Step S152: Calculate the symmetric transfer error corresponding to each normalized matching point pair based on the essential matrix, wherein the essential matrix is ​​calculated based on the camera pose.

[0140] Camera pose, including rotation matrix R global and the translation vector T global .

[0141] The Essential Matrix describes the geometric relationship between the same spatial point under two different camera perspectives. Its calculation formula is: E=[T global ] × R global . Where E represents the essential matrix, [T global ] × Represents the translation vector T global The antisymmetric matrix of .

[0142] For all normalized matching point pairs (x l _i,x r _i), calculate the symmetric transfer error as an indicator to measure whether the normalized matching point pair satisfies the essential matrix.

[0143] Specifically, according to the essential matrix and the normalized matching point pair, the epipolar line corresponding to the point in the left scene image is calculated (denoted as l r _i) and the epipolar line corresponding to the point in the right scene image (denoted as l l _i). The details are as follows: l r _i=E·x l _i; l l _i=E·x r _i.

[0144] Then, calculate the point x in the right scene image r _i to the epipolar line l corresponding to the point in the left scene image r The distance of _i, denoted as dist l_i. At the same time, calculate the point x in the left scene image l _i to point x in the right scene image r _i corresponding epipolar line l l The distance of _i, denoted as dist r _i.

[0145] The two distances calculated (dist l _i and dist r _i) are added to obtain the symmetrical transfer error.

[0146] Step S153: filtering the matching point pairs according to the symmetric transfer error to obtain filtered matching point pairs.

[0147] Check whether the symmetric transfer error is less than a preset threshold. If so, the normalized matching point pair and its corresponding original matching point pair are considered to be interior points of the essential matrix, and the filtered matching point pair is determined. It should be noted that since the original matching point pair is filtered in the above manner, it is recorded as the filtered matching point pair.

[0148] It should be noted that, while determining the inliers, the index of the inliers is recorded for subsequent use.

[0149] At this time, step S130 includes: converting the filtered matching point pairs into three-dimensional landmark points in a global coordinate system.

[0150] After filtering the matching point pairs using the epipolar geometry constraints, the filtered matching point pairs are converted into three-dimensional landmark points in the global coordinate system. The specific execution process can be referred to the above embodiment and will not be described in detail here.

[0151] The three-dimensional scene reconstruction method provided by the embodiment of the present invention utilizes epipolar geometry constraints to robustly eliminate erroneous matches caused by repeated textures, occlusions, noise, etc., greatly improving the reliability of matching point pairs, which is the key to ensuring the accuracy of subsequent triangulation.

[0152] Based on any of the above embodiments, before step S120, the three-dimensional scene reconstruction method may further include: performing data enhancement processing on the scene image to obtain an enhanced scene image; wherein the data enhancement processing includes at least one of gamma correction, contrast stretching and denoising filtering.

[0153] In this embodiment, gamma correction applies a nonlinear operation to the grayscale values ​​of a scene image, resulting in an exponential relationship between the grayscale values ​​of the processed scene image and the grayscale values ​​of the pre-processed scene image. Gamma correction enhances the adaptability of the scene image's overall brightness and contrast, improving its performance in both dark and bright environments.

[0154] Contrast stretching uses a linear mapping function to map the original grayscale range of a scene image to a target range, fully utilizing the dynamic range of the display device. Its core goal is to redistribute existing grayscale values, making dark and bright details clearer and improving visual legibility.

[0155] Denoising filtering uses edge-preserving filtering algorithms (such as non-local means denoising or bilateral filtering) to suppress scene image noise while maximally preserving edge structure and texture details. Its core mechanism uses a dual weight constraint of spatial proximity and grayscale similarity to achieve a balance between noise smoothing and feature preservation.

[0156] At this time, step S120 includes: performing feature extraction and feature matching on the enhanced scene image to obtain matching point pairs.

[0157] After data enhancement processing is performed on the scene image, feature extraction and feature matching are further performed on the enhanced scene image to obtain matching point pairs. The specific execution process can be referred to the above embodiment and will not be described in detail here.

[0158] The three-dimensional scene reconstruction method provided by the embodiment of the present invention can enhance the robustness of scene image features and improve image quality through data enhancement processing, so that the feature vectors obtained by subsequent feature extraction are more discriminative, thereby reducing the ambiguity in the subsequent feature matching stage, and helping to improve the effect of three-dimensional reconstruction of the scene.

[0159] The three-dimensional scene reconstruction device provided by the present invention is described below. The three-dimensional scene reconstruction device described below and the three-dimensional scene reconstruction method described above can be referenced to each other.

[0160] Figure 5 Schematic diagram of the structure of the three-dimensional scene reconstruction device provided by the present invention. Figure 5 As shown, the apparatus includes a receiving module 510, an acquiring module 520, a converting module 530 and a reconstructing module 540; wherein: A receiving module 510 is configured to receive a scene image acquired by an XR device; An acquisition module 520 is configured to perform feature extraction and feature matching on the scene image to obtain matching point pairs; A conversion module 530 is used to convert the matching point pairs into three-dimensional landmark points in a global coordinate system; The reconstruction module 540 is used to perform grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0161] The three-dimensional scene reconstruction device provided by the embodiment of the present invention obtains the scene image through the pure binocular camera of the XR device, and then performs feature extraction and feature matching on the scene image to obtain the matching point pair between the left-eye scene image and the right-eye scene image, and converts the two-dimensional matching point pair into a three-dimensional landmark point in the global coordinate system to obtain three-dimensional point cloud data, and then performs grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model. Through the above method, the present invention does not need to rely on a dedicated depth camera, but only uses the ordinary pure binocular camera that is standard on the XR device to achieve real-time reconstruction of the three-dimensional model of the indoor scene, which achieves a significant simplification of the hardware architecture and a substantial reduction in cost, and greatly promotes the lightweight and small-volume design of the XR device. At the same time, the use of a grid reconstruction method can significantly reduce the computational complexity and reduce the computing power loss. In summary, the present invention provides lightweight, consumer-grade XR devices with native, real-time scene three-dimensional perception and reconstruction capabilities, which is conducive to promoting the implementation of indoor XR applications.

[0162] Furthermore, the scene image includes a left-eye scene image and a right-eye scene image, and the acquisition module 520 is specifically configured to: Performing scale-invariant feature transform (SIFT) feature extraction on the left-view scene image to obtain a first key point set and a first SIFT feature vector set; Performing SIFT feature extraction on the right scene image to obtain a second key point set and a second SIFT feature vector set; Searching, from the second key point set, based on the first SIFT feature vector set and the second SIFT feature vector set, for a plurality of candidate matching points corresponding to each key point in the first key point set, and obtaining matching distance values ​​between each key point in the first key point set and its corresponding plurality of candidate matching points; Performing a ratio test on the matching distance value to obtain a ratio test result; The plurality of candidate matching points are filtered according to the ratio check result, and matching point pairs are obtained according to the filtering result.

[0163] Furthermore, the reconstruction module 540 includes: A division unit, configured to perform spatial grid division on the three-dimensional landmark points in the global coordinate system to obtain a grid map; A local reconstruction unit is used to perform local grid reconstruction on the three-dimensional landmark points of each grid in the grid map to obtain a local three-dimensional grid model; The cross-grid reconstruction unit is used to perform cross-grid reconstruction on the three-dimensional landmark points of the neighboring grids of each grid in the grid map to fuse them with the local three-dimensional grid model to obtain a three-dimensional grid model.

[0164] Furthermore, the local reconstruction unit is specifically configured to: Determining the fitting plane and the points within the plane of each grid according to the three-dimensional landmark points of each grid in the grid map; Projecting the points in the plane onto the fitting plane to obtain a first two-dimensional point after projection; Performing triangulation on the first two-dimensional point to obtain a first triangle index; According to the first triangle index, the three-dimensional landmark points of each mesh are connected to generate a three-dimensional triangular facet to obtain a local three-dimensional mesh model.

[0165] Furthermore, the cross-grid reconstruction unit is specifically configured to: Obtaining three-dimensional landmark points of neighboring grids of each grid in the grid map; Calculating the distance between the three-dimensional landmark point of the neighborhood grid and the fitting plane of the corresponding grid; According to the distance, screening the neighborhood candidate points from the three-dimensional landmark points of the neighborhood grid; Merging the candidate points in the neighborhood with the boundary points of the corresponding grid to obtain points to be fused; wherein the boundary points are obtained by screening the points in the plane; Projecting the point to be fused onto the fitting plane of the corresponding grid to obtain a second two-dimensional point after projection; Performing triangulation on the second two-dimensional point to obtain a second triangle index; Connecting the points to be fused according to the second triangle index to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; The local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model.

[0166] Furthermore, the conversion module 530 is specifically configured to: Get camera calibration parameters and camera pose; triangulate the matching point pairs according to the camera calibration parameters to obtain three-dimensional landmark points in the camera coordinate system; According to the camera pose, the three-dimensional landmark points in the camera coordinate system are converted into three-dimensional landmark points in the global coordinate system.

[0167] Furthermore, the three-dimensional scene reconstruction device further includes a filtering module; the filtering module is used to: performing coordinate normalization processing on the matching point pairs to obtain normalized matching point pairs; Calculate the symmetric transfer error corresponding to each normalized matching point pair according to the essential matrix; wherein the essential matrix is ​​calculated according to the camera pose; filtering the matching point pairs according to the symmetric transfer error to obtain filtered matching point pairs; The conversion module 530 is specifically configured to: The filtered matching point pairs are converted into three-dimensional landmark points in a global coordinate system.

[0168] Furthermore, the three-dimensional scene reconstruction device further includes: a processing module, configured to perform data enhancement processing on the scene image to obtain an enhanced scene image; wherein the data enhancement processing includes at least one of gamma correction, contrast stretching, and denoising filtering; The acquisition module 520 is specifically configured to: Feature extraction and feature matching are performed on the enhanced scene image to obtain matching point pairs.

[0169] It should be noted here that the above-mentioned three-dimensional scene reconstruction device provided by the embodiment of the present invention can implement all the method steps implemented by the above-mentioned three-dimensional scene reconstruction method embodiment, and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as the method embodiment will not be described in detail here.

[0170] Figure 6 An example of a physical structure diagram of an XR device is shown below. Figure 6 As shown, the XR device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call logic instructions in the memory 630 to execute a three-dimensional scene reconstruction method, which includes: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmark points in a global coordinate system; and performing mesh reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional mesh model.

[0171] Furthermore, the logic instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0172] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional scene reconstruction method provided by the above methods, which includes: receiving the scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmark points in a global coordinate system; and performing grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0173] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the three-dimensional scene reconstruction method provided by the above-mentioned methods, the method including: receiving a scene image acquired by the XR device; performing feature extraction and feature matching on the scene image to obtain matching point pairs; converting the matching point pairs into three-dimensional landmark points in a global coordinate system; and performing grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

[0174] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0175] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A three-dimensional scene reconstruction method, characterized in that: Applied to extended reality (XR) devices, including: Receiving a scene image acquired by the XR device; Performing feature extraction and feature matching on the scene image to obtain matching point pairs; Converting the matching point pairs into three-dimensional landmark points in a global coordinate system; Grid reconstruction is performed based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

2. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The scene image includes a left-eye scene image and a right-eye scene image, and the feature extraction and feature matching of the scene image are performed to obtain matching point pairs, including: Performing scale-invariant feature transform (SIFT) feature extraction on the left-view scene image to obtain a first key point set and a first SIFT feature vector set; Performing SIFT feature extraction on the right scene image to obtain a second key point set and a second SIFT feature vector set; Searching for a plurality of candidate matching point pairs from the first key point set and the second key point set according to the first SIFT feature vector set and the second SIFT feature vector set, and obtaining matching distance values ​​between the plurality of candidate matching point pairs; Performing a ratio test on the matching distance value to obtain a ratio test result; The plurality of candidate matching point pairs are filtered according to the ratio check result, and matching point pairs are obtained according to the filtering result.

3. The three-dimensional scene reconstruction method according to claim 1, characterized in that: The step of performing grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model includes: Performing spatial grid division on the three-dimensional landmark points in the global coordinate system to obtain a grid map; Performing local grid reconstruction on the three-dimensional landmark points of each grid in the grid map to obtain a local three-dimensional grid model; The three-dimensional landmark points of the neighborhood grids of each grid in the grid map are reconstructed across the grid to be fused with the local three-dimensional grid model to obtain a three-dimensional grid model.

4. The three-dimensional scene reconstruction method according to claim 3, characterized in that: The method of performing local grid reconstruction on the three-dimensional landmark points of each grid in the grid map to obtain a local three-dimensional grid model includes: Determining the fitting plane and the points within the plane of each grid according to the three-dimensional landmark points of each grid in the grid map; Projecting the points in the plane onto the fitting plane to obtain a first two-dimensional point after projection; Performing triangulation on the first two-dimensional point to obtain a first triangle index; According to the first triangle index, the three-dimensional landmark points of each mesh are connected to generate a three-dimensional triangular facet to obtain a local three-dimensional mesh model.

5. The three-dimensional scene reconstruction method according to claim 4, characterized in that: The cross-grid reconstruction of the three-dimensional landmark points of the neighboring grids of each grid in the grid map to fuse with the local three-dimensional grid model to obtain a three-dimensional grid model includes: Obtaining three-dimensional landmark points of neighboring grids of each grid in the grid map; Calculating the distance between the three-dimensional landmark point of the neighborhood grid and the fitting plane of the corresponding grid; According to the distance, screening the neighborhood candidate points from the three-dimensional landmark points of the neighborhood grid; Merging the candidate points in the neighborhood with the boundary points of the corresponding grid to obtain points to be fused; wherein the boundary points are obtained by screening the points in the plane; Projecting the point to be fused onto the fitting plane of the corresponding grid to obtain a second two-dimensional point after projection; Performing triangulation on the second two-dimensional point to obtain a second triangle index; Connecting the points to be fused according to the second triangle index to generate three-dimensional triangular facets to obtain a cross-grid three-dimensional model; The local three-dimensional grid model and the cross-grid three-dimensional model are fused to obtain a three-dimensional grid model.

6. The three-dimensional scene reconstruction method according to claim 1, characterized in that: Converting the matching point pairs into three-dimensional landmark points in a global coordinate system includes: Get camera calibration parameters and camera pose; triangulate the matching point pairs according to the camera calibration parameters to obtain three-dimensional landmark points in the camera coordinate system; According to the camera pose, the three-dimensional landmark points in the camera coordinate system are converted into three-dimensional landmark points in the global coordinate system.

7. The three-dimensional scene reconstruction method according to any one of claims 1 to 5, characterized in that: Before converting the matching point pairs into three-dimensional landmark points in a global coordinate system, the method further includes: performing coordinate normalization processing on the matching point pairs to obtain normalized matching point pairs; Calculate the symmetric transfer error corresponding to each normalized matching point pair according to the essential matrix; wherein the essential matrix is ​​calculated according to the camera pose; The matching point pairs are filtered according to the symmetric transfer errors to obtain filtered matching point pairs.

8. The three-dimensional scene reconstruction method according to any one of claims 1 to 5, characterized in that: Before extracting and matching features on the scene image to obtain matching point pairs, the method further includes: Performing data enhancement processing on the scene image to obtain an enhanced scene image; wherein the data enhancement processing includes at least one of gamma correction, contrast stretching, and denoising filtering.

9. A three-dimensional scene reconstruction device, characterized in that: include: A receiving module, used to receive scene images acquired by the XR device; An acquisition module is used to perform feature extraction and feature matching on the scene image to obtain matching point pairs; A conversion module, used to convert the matching point pairs into three-dimensional landmark points in a global coordinate system; The reconstruction module is used to perform grid reconstruction based on the three-dimensional landmark points in the global coordinate system to obtain a three-dimensional grid model.

10. An XR device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the three-dimensional scene reconstruction method according to any one of claims 1 to 8 is implemented.

Images