Multi-modal data three-dimensional scene reconstruction method and device

Abstract

The invention relates to the technical field of three-dimensional scene reconstruction methods, in particular to a multi-modal data three-dimensional scene reconstruction method and device, and the method specifically comprises the following steps: 1, carrying out the data collection of a target scene through a plurality of sensors, comprising a color image sequence acquired by an optical camera, point cloud data acquired by a laser radar and a depth image sequence acquired by a depth camera; 2, denoising processing is carried out on the collected color image sequence, and a median filtering algorithm is adopted; 3, rough alignment is carried out on preprocessed multi-modal data by adopting a method based on feature points; according to the method, the alignment precision is remarkably improved, accurate matching of different modal data in space is ensured, the geometric structure of a reconstruction model is accurate, and the relative position of an object fits an actual scene.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional scene reconstruction methods, and in particular to a method and apparatus for multimodal data three-dimensional scene reconstruction. Background Technology

[0002] With the rapid development of computer vision and sensor technology, 3D scene reconstruction has been widely applied in many fields such as virtual reality, augmented reality, robot navigation, and digital twins. Traditional 3D scene reconstruction methods mainly rely on single-modal data, such as optical camera-based visual reconstruction methods, which obtain geometric information of the scene through multi-view image matching; and lidar-based reconstruction methods, which use lidar point cloud data to construct scene models. However, these single-modal methods have obvious limitations.

[0003] Visual reconstruction methods based on optical cameras are easily affected by factors such as lighting conditions and regions with similar textures. In scenes with low texture and drastic lighting changes, image feature matching is difficult and prone to mismatches, leading to a decrease in reconstruction accuracy. Summary of the Invention

[0004] To address the technical problem that existing optical camera-based visual reconstruction methods are easily affected by factors such as lighting conditions and similar texture regions, and that image feature matching is difficult and prone to mismatch in low-texture and drastically changing lighting scenes, leading to a decrease in reconstruction accuracy, this invention provides a multimodal data 3D scene reconstruction method and apparatus.

[0005] The technical solution adopted in this invention is: a method for reconstructing a three-dimensional scene from multimodal data, specifically including the following steps:

[0006] Step 1: Collect data on the target scene using multiple sensors, including color image sequences acquired by an optical camera, point cloud data acquired by a lidar, and depth image sequences acquired by a depth camera.

[0007] Step 2: Denoise the acquired color image sequence using a median filtering algorithm.

[0008] Step 3: Perform coarse alignment on the preprocessed multimodal data using a feature point-based method;

[0009] Step 4: Construct a multimodal deep learning network based on deep learning feature extraction to extract features from the coarsely aligned color image, point cloud data, and depth image respectively;

[0010] Step 5: Use an attention mechanism to fuse the extracted multimodal features;

[0011] Step 6: Based on the fused features, perform fine alignment of multimodal data using an improved version of the iterative nearest point algorithm;

[0012] Step 7: Construct a 3D scene model using the finely aligned multimodal data;

[0013] Step 8: Optimize the initially constructed 3D scene model;

[0014] Step 9: Render the optimized 3D scene model to generate a realistic 3D scene image;

[0015] Step 10: Output the rendered 3D scene model in a common 3D model format (such as OBJ, STL, PLY, etc.) and save it locally or upload it to the server.

[0016] In one embodiment, step one involves: acquiring data about the target scene using multiple sensors, including a sequence of color images {I1, I2, ..., I...} acquired by an optical camera. n Point cloud data P = {p1, p2, ..., p} acquired by lidar m} and the depth image sequence {D1,D2,…,D} acquired by the depth camera n}; where n represents the number of images, m represents the number of points in the point cloud data, and I i Let p represent the i-th color image. j D represents the j-th point in the point cloud data. i Let i represent the depth image of the i-th image.

[0017] In one embodiment, step two involves: processing the acquired color image sequence {I1,I2,…,I...} n For noise reduction, a median filtering algorithm is used. The specific method and formula are as follows:

[0018] I' i (x,y)=median{I i (xk,yl),(x,y)∈N k×l};

[0019] Among them, I′ i This represents the i-th color image after denoising, where (x, y) are the pixel coordinates of the image, and N... k×l It is a neighborhood window of size k×l centered at (x,y), and median represents the median calculation operation;

[0020] Outlier removal is performed on the point cloud data P acquired by lidar, and the value of each point p is calculated. j The average distance d between it and its k nearest neighbors j :

[0021]

[0022] Where, N k (p j ) represents point p j The set of k nearest neighbors, ||·|| represents the Euclidean distance; set a threshold T, if d j >T, then point p is considered j Outliers are removed from the point cloud data to obtain preprocessed point cloud data P′;

[0023] For the depth image sequence {D1,D2,…,D…} n Hole filling is performed using a hole repair algorithm based on the fast traversal method to fill the hole regions in the depth image, resulting in a complete depth image sequence {D′1,D′2,…,D′}. n};

[0024] By calculating the Euclidean distance between the descriptors of feature points, four-way matching is performed on feature points of different modalities. This is particularly relevant for feature points in color images. and feature points f in point cloud data Pk Calculate its descriptor and d P Distance of k:

[0025]

[0026] Set a distance threshold t, if Then it is believed and f Pk To match point pairs, we obtain the set M of matching point pairs between color image and point cloud data, color image and depth image, and point cloud data and depth image. IP M ID M PD ;

[0027] Based on the set of matching point pairs, the coarse alignment transformation matrix between multimodal data is calculated using the least squares method. Let the set of matching point pairs be... By minimizing the following objective function:

[0028]

[0029] Where R is the rotation matrix and t is the translation vector, solving the above objective function yields the coarse alignment transformation matrix T between the color image and the point cloud data. IP =[R IP ,t IP Similarly, the coarse alignment transformation matrix T between the color image and the depth image is obtained. ID The coarse alignment transformation matrix T between point cloud data and depth image PDCoarse alignment is performed on multimodal data.

[0030] In one embodiment, the specific method for coarsely aligning the preprocessed multimodal data using a feature point-based approach in step three is as follows:

[0031] In color image I′ i In this study, the SIFT algorithm was used to extract feature points.

[0032] Where q is the number of feature points; in the point cloud data P′, the PFH algorithm is used to extract feature points F. P ={f P1 ,f P2 ,…,f Pr};

[0033] Where r is the number of feature points in the point cloud; in the depth image D′ i In this study, the SURF algorithm was used to extract feature points. s represents the number of feature points in the depth image.

[0034] In one embodiment, a multimodal deep learning network is constructed based on deep learning feature extraction. The specific method for extracting features from the coarsely aligned color image, point cloud data, and depth image is as follows:

[0035] The color image is input into the convolutional neural network, and after multiple layers of convolution and pooling operations, the semantic and geometric features of the image are extracted.

[0036] Let the input color image be I. i "After L layers of convolution and pooling operations, the image features are obtained."

[0037]

[0038] Where, θ CNN These are the parameters of the CNN network;

[0039] Point cloud data is converted into a voxel mesh representation and then fed into a 3D convolutional neural network. After multiple layers of 3D convolution and pooling operations, the spatial geometric features of the point cloud data are extracted.

[0040]

[0041] Where V(P″) is the voxel mesh after point cloud data conversion, and θ 3D-CNN These are the parameters of the 3D-CNN network;

[0042] The depth image is input into an improved U-Net network, and its encoder-decoder structure is used to extract multi-scale features from the depth image.

[0043]

[0044] Where, θ U-Net These are the parameters of the U-Net network.

[0045] In one embodiment, the specific method for fusing the extracted multimodal features using an attention mechanism in step five is as follows:

[0046] For color image features Point cloud data features and depth image features Calculate the attention weights between them separately, and calculate the attention weight w. IP :

[0047]

[0048] Where s(·,·) is the similarity calculation function, using cosine similarity:

[0049]

[0050] Similarly, the attention weight w is calculated. ID and w PD ;

[0051] Based on the calculated attention weights, the multimodal features are weighted and fused to obtain the fused features.

[0052] In one embodiment, step six involves using an improved version of the iterative nearest point algorithm to perform fine alignment of multimodal data based on the fused features, as follows:

[0053] Based on the fused features, a more accurate correspondence is established between different modal data. For point p in point cloud data P′... j Find the regions with the most similar features in the color and depth images and establish corresponding point pairs;

[0054] The fine alignment transformation matrix is ​​calculated by minimizing the distance between corresponding point pairs;

[0055] Let the set of corresponding point pairs be C = {(p j ,q j By minimizing the following objective function:

[0056]

[0057] Where R′ is the rotation matrix and t′ is the translation vector, the fine alignment transformation matrix T is obtained by iteratively solving the above objective function. final =[R final ,t final [This is used to perform fine alignment on multimodal data.]

[0058] In one embodiment, step seven involves constructing a 3D scene model using finely aligned multimodal data, as follows:

[0059] A voxel-based method is used to fuse point cloud data, color image texture information, and depth image information into a voxel mesh;

[0060] Voxelize the point cloud data P′ to obtain the voxel mesh V. P Based on the spatial distribution of the point cloud, the size of the voxels is determined so that the voxels can better represent the geometric details of the scene.

[0061] For each voxel in the voxel mesh, the texture information of the color image is mapped onto the voxel according to its corresponding position in the color image; let voxel v be in the color image I. i The corresponding coordinates in "" are (x v ,y v ), then I i (x v ,y v The color information of the voxel is assigned to the voxel.

[0062] Depth image information is fused into a voxel mesh. Based on the depth values ​​in the depth image, the depth information of the voxels is determined, further refining the geometric structure of the 3D scene model. Through these operations, a preliminary 3D scene model M is obtained. init .

[0063] In one embodiment, step eight involves: processing the initially constructed 3D scene model M init The specific methods for optimization are as follows:

[0064] The Laplacian smoothing algorithm is used to smooth the surface of the model; for each vertex v on the model surface, its Laplacian operator Δv is calculated:

[0065]

[0066] Where N(v) represents the set of neighboring vertices of vertex v, and k is the number of neighboring vertices; the vertex position is updated iteratively:

[0067] v new =v+αΔv

[0068] Where α is a smoothing factor, which smooths the model surface through multiple iterations;

[0069] The hole regions in the detection model were repaired using a Poisson surface reconstruction method, resulting in an optimized 3D scene model M. opt ;

[0070] In step nine: the optimized 3D scene model M opt The specific method for rendering and generating realistic 3D scene images is as follows:

[0071] Physically based rendering techniques are used to calculate the color and brightness of the model's surface, taking into account physical factors such as lighting, materials, and reflection.

[0072] Based on the light source information in the scene, calculate the illumination intensity at each point on the model surface; for point p, its illumination intensity L(p) is:

[0073]

[0074] Where, N l L represents the number of light sources. l (p) represents the light intensity produced by the l-th light source at point p; based on the material properties of different objects in the model, the reflection and refraction of light on the object surface are calculated to obtain the rendered 3D scene image.

[0075] The beneficial effects of this invention are as follows: Compared with the prior art, in the data acquisition and preprocessing stage, multiple sensors collaboratively acquire multimodal data, covering color images, point cloud data, and depth images, comprehensively obtaining the texture, geometry, and depth information of the scene, making up for the deficiencies of single-modality data, and laying a solid foundation for subsequent processing. In the data alignment stage, a strategy of coarse alignment followed by fine alignment is adopted. Coarse alignment is based on feature point matching to initially establish data correspondence, significantly reducing the initial positional differences between data, providing good initial values ​​for fine alignment, and reducing the risk of getting trapped in local optima. Fine alignment uses fusion features to improve the ICP algorithm, establishes accurate correspondence based on multimodal comprehensive information, significantly improves alignment accuracy, ensures accurate spatial matching of different modal data, makes the geometric structure of the reconstructed model accurate and error-free, and ensures that the relative positions of objects fit the actual scene. In the process of constructing the 3D scene model, a voxel-based method is used to deeply fuse multimodal data, voxelize and regularize point cloud data to construct a geometric framework, texture mapping gives the model realistic color texture, and deep fusion refines the object shape, comprehensively presenting the real appearance of the scene, meeting the stringent requirements of virtual reality, augmented reality, and other fields for high-precision and high-realism models. The model optimization process eliminates surface defects and ensures model integrity by smoothing surfaces and repairing holes, thereby improving visual effects, engineering application reliability, and expanding the model's applicability. Attached Figure Description

[0076] Figure 1 This is a three-dimensional structural schematic diagram of the present invention; Detailed Implementation

[0077] In the description of this invention, it should be noted that the terms "front", "up", "down", "left", "right", "vertical", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0078] refer to Figure 1 In order to solve the problems existing in the background art, this application proposes the following technical solution:

[0079] Step 1: Multimodal Data Acquisition: Data is acquired from the target scene using multiple sensors, including color image sequences {I1, I2, ..., I...} captured by an optical camera. n Point cloud data P = {p1, p2, ..., p} acquired by lidar m} and the depth image sequence {D1,D2,…,D} acquired by the depth camera n}. Where n represents the number of images, m represents the number of points in the point cloud data, and I i Let p represent the i-th color image. j D represents the j-th point in the point cloud data. i Let i represent the depth image of the i-th image.

[0080] The above technical solution can be explained as follows: By utilizing data collected from multiple sensors such as optical cameras, LiDAR, and depth cameras, a rich source of information is provided for subsequent 3D scene reconstruction. Color images acquired by optical cameras contain texture, color, and semantic information of the scene, giving the reconstructed model a realistic visual appearance; point cloud data acquired by LiDAR provides high-precision 3D geometric structure information, accurately describing the shape and position of objects in the scene; depth images acquired by depth cameras further supplement the spatial depth information of the scene, helping to refine the contours and distance relationships of objects. The complementary nature of multiple modal data overcomes the limitations of single-modal data and avoids inaccurate reconstruction due to missing information. For example, in low-texture scenes, relying solely on optical cameras may be insufficient to acquire enough features for accurate reconstruction, but combining data from LiDAR and depth cameras provides more constraints from geometric and depth dimensions, thus laying a solid data foundation for subsequent steps and significantly improving the reliability and comprehensiveness of 3D scene reconstruction.

[0081] Step 2: Multimodal data preprocessing:

[0082] Color image preprocessing: Preprocessing the acquired color image sequence {I1, I2, ..., I...} n For noise reduction, a median filtering algorithm is used, as shown in the following formula:

[0083] I' i (x,y)=median{I i (xk,yl),(x,y)∈N k×l};

[0084] Among them, I′ i This represents the i-th color image after denoising, where (x, y) are the pixel coordinates of the image, and N... k×l This is a k×l neighborhood window centered at (x,y), where `median` represents the median calculation. The denoised image is then subjected to histogram equalization to enhance image contrast and improve the accuracy of subsequent feature extraction. 2. Point cloud data preprocessing: Outlier removal is performed on the point cloud data P acquired by the LiDAR. Calculate the median for each point p. j The average distance d between it and its k nearest neighbors j :

[0085]

[0086] Where, N k (p j ) represents point p j Let ||·|| represent the set of k-nearest neighbors, and ||·|| denote the Euclidean distance. Set a threshold T; if d j >T, then point p is considered j Outliers are removed from the point cloud data to obtain preprocessed point cloud data P′.

[0087] Depth image preprocessing: processing the depth image sequence {D1,D2,…,D…} n Hole filling is performed using a hole repair algorithm based on the fast traversal method to fill the hole regions in the depth image, resulting in a complete depth image sequence {D′1,D′2,…,D′}. n}

[0088] Feature point matching: By calculating the Euclidean distance between feature point descriptors, feature points from different modalities are matched. Taking color images and point cloud data as an example, for feature points in a color image... and feature points f in point cloud data Pk Calculate its descriptor and d P Distance of k:

[0089]

[0090] Set a distance threshold t, if Then it is believed and f Pk To match point pairs, the above method is used to obtain a set M of matching point pairs between color images and point cloud data, color images and depth images, and point cloud data and depth images. IP M ID M PD .

[0091] Coarse alignment transformation calculation: Based on the set of matching point pairs, the coarse alignment transformation matrix between multimodal data is calculated using the least squares method. Taking color image and point cloud data as an example, let the set of matching point pairs be... By minimizing the following objective function:

[0092]

[0093] Where R is the rotation matrix and t is the translation vector. By solving the above objective function, the coarse alignment transformation matrix T between the color image and the point cloud data is obtained. IP =[R IP ,t IP Similarly, the coarse alignment transformation matrix T between the color image and the depth image is obtained. ID The coarse alignment transformation matrix T between point cloud data and depth image PD Coarse alignment is performed on multimodal data.

[0094] The above technical solutions are explained as follows: In the color image preprocessing stage, the median filtering algorithm effectively removes noise from the image, preventing noise from interfering with subsequent feature extraction and matching processes, making image features clearer and more accurate, and improving the accuracy and stability of feature point detection. Histogram equalization enhances image contrast, making image details more prominent, especially in scenes with uneven lighting or low light, better presenting the edges and textures of objects, facilitating the extraction of more representative image features. In the point cloud data preprocessing stage, outlier removal eliminates abnormal points caused by sensor errors or environmental interference, purifying the point cloud data and improving its quality and reliability. After removing outliers, the point cloud data more realistically reflects the geometric structure of the scene, avoiding the negative impact of outliers on subsequent point cloud registration and model construction, making point cloud-based 3D reconstruction more accurate. Hole filling in depth image preprocessing solves the problem of holes in depth images caused by occlusion, reflection, etc., ensuring the integrity of the depth image and enabling the depth information to accurately reflect the spatial structure of the scene. This provides accurate depth data support for subsequent 3D modeling and improves the geometric accuracy of 3D scene reconstruction.

[0095] Step 3: Coarse alignment of multimodal data:

[0096] A feature-point-based method is used to perform coarse alignment on the preprocessed multimodal data. 1. Feature point extraction: In the color image I′ i In this study, the SIFT (Scale Invariant Feature Transform) algorithm was used to extract feature points. Where q represents the number of feature points; in the point cloud data P′, the PFH (Point Feature Histogram) algorithm is used to extract feature points F. P ={f P1 ,f P2 ,…,f Pr}, r is the number of feature points in the point cloud; in the depth image D′ i In this paper, the SURF (Accelerated Robust Features) algorithm is used to extract feature points. s represents the number of feature points in the depth image.

[0097] The above technical solution is explained as follows: The feature-point-based coarse alignment method establishes a preliminary correspondence between multimodal data by extracting and matching feature points from different modal data. Feature extraction algorithms (SIFT, PFH, SURF) suitable for the characteristics of color images, point cloud data, and depth images are employed respectively to fully exploit the feature information of each modality, improving the accuracy and robustness of feature point matching. This coarse alignment method effectively reduces the initial positional differences between multimodal data, lowering the difficulty and computational cost of subsequent fine alignment. For example, in the processing of multimodal data in large-scale scenes, without coarse alignment, direct fine alignment may lead to local optima. However, the initial transformation matrix provided by coarse alignment provides better initial values ​​for fine alignment, guiding the fine alignment process to converge to the global optimum more quickly, thereby improving the overall efficiency and accuracy of multimodal data alignment and laying a solid foundation for constructing accurate 3D scene models.

[0098] Step 4: Construct a multimodal deep learning network based on deep learning feature extraction to extract features from the coarsely aligned color image, point cloud data, and depth image respectively.

[0099] Color image feature extraction: The color image is input into the convolutional neural network (CNN) part, and after multiple convolution and pooling operations, semantic and geometric features of the image are extracted. Let the input color image be I. i "After L layers of convolution and pooling operations, the image features are obtained."

[0100]

[0101] Where, θ CNN These are the parameters of the CNN network.

[0102] Point cloud data feature extraction: The point cloud data is converted into a voxel mesh representation and then input into a 3D convolutional neural network (3D-CNN). After multiple layers of 3D convolution and pooling operations, the spatial geometric features of the point cloud data are extracted.

[0103] Where V(P″) is the voxel mesh after point cloud data conversion, and θ 3D-CNN These are the parameters of the 3D-CNN network. 3. Depth Image Feature Extraction: The depth image is input into the improved U-Net network, and its encoder-decoder structure is used to extract multi-scale features from the depth image.

[0104]

[0105] Where, θ U-Net These are the parameters of the U-Net network.

[0106] The above technical solution is explained as follows: A multimodal deep learning network is constructed, with specialized network structures designed for the characteristics of different modalities of data. For color images, the convolutional neural network can automatically learn the semantic and geometric features of the image, extracting representative features from a large amount of image data. Compared to traditional manually designed feature extraction methods, it can capture more complex and abstract image information, exhibiting stronger adaptability to changes in illumination and viewing angle, thus improving the expressive power of image features. Point cloud data, converted into a voxel grid and input into a 3D convolutional neural network, can fully utilize 3D spatial information to extract the spatial geometric features of the point cloud data, effectively describing the shape and structure of the point cloud, overcoming the limitations of traditional point cloud feature extraction methods in handling complex spatial relationships. The improved U-Net network is used for depth image feature extraction. Its encoder-decoder structure can extract multi-scale depth features, understanding depth images at different levels and resolutions. It preserves image detail information while acquiring overall structural information, providing rich and accurate depth features for subsequent feature fusion, enhancing the understanding and processing capabilities of depth images, and contributing to improved quality of 3D scene reconstruction.

[0107] Step 5: Multimodal feature fusion: The extracted multimodal features are fused using an attention mechanism.

[0108] Calculate attention weights: for color image features Point cloud data features and depth image features Calculate the attention weights between them separately. Taking color image features and point cloud data features as an example, calculate the attention weight w. IP :

[0109]

[0110] Where s(·,·) is the similarity calculation function, using cosine similarity:

[0111]

[0112] Similarly, the attention weight w is calculated. ID and w PD .

[0113] Feature fusion: Based on the calculated attention weights, multimodal features are weighted and fused to obtain the fused features.

[0114]

[0115] The above technical solution is explained as follows: Employing an attention mechanism for multimodal feature fusion dynamically allocates weights based on the similarity between different modal features. By calculating attention weights, modal features more relevant to the current scene are highlighted, while irrelevant or redundant feature information is suppressed, achieving an organic combination and complementary advantages of different modal features. For example, when reconstructing a scene containing objects with complex textures, the texture features of the color image may be more important, and the attention mechanism will assign it a higher weight, allowing it to play a dominant role in feature fusion. Conversely, when reconstructing a scene with complex geometric structures, the weights of the geometric features of the point cloud data and the depth features of the depth image will be correspondingly increased. This adaptive feature fusion method enables the fused features to more comprehensively and accurately describe the scene. Compared to simple feature stitching or weighted averaging methods, it better preserves the advantages of each modal data, improves the detail richness and realism of the 3D scene reconstruction model, and enhances the model's adaptability to different scenes.

[0116] Step Six: Fine Alignment of Multimodal Data

[0117] Based on the fused features, an improved version of the Iterative Closest Point (ICP) algorithm is used for fine alignment of multimodal data.

[0118] Establish correspondences: Based on the fused features, establish more accurate correspondences between different modal data. For point p in point cloud data P′... j Find the regions with the most similar features in the color and depth images and establish corresponding point pairs.

[0119] Fine alignment transformation calculation: The fine alignment transformation matrix is ​​calculated by minimizing the distance between corresponding point pairs. Let the set of corresponding point pairs be C = {(p j ,q j By minimizing the following objective function:

[0120]

[0121] Where R′ is the rotation matrix and t′ is the translation vector. By iteratively solving the above objective function, the fine alignment transformation matrix T is obtained. final =[R final ,t final [This is used to perform fine alignment of multimodal data.]

[0122] The above technical solution is explained as follows: The improved ICP algorithm, based on fused features, achieves fine alignment by establishing a more accurate correspondence using fused features, effectively overcoming the problems of traditional ICP algorithms being sensitive to initial values ​​and prone to getting trapped in local optima. In establishing the correspondence, the fused features integrate information from multimodal data, more accurately reflecting the true matching relationship between different modalities, allowing the fine alignment process to start from a more reasonable starting point. By minimizing the distance between corresponding point pairs to calculate the fine alignment transformation matrix, the alignment accuracy of multimodal data is further improved, making the spatial positions and orientations of different modalities more accurately matched. This fine alignment method significantly reduces data alignment errors, ensuring accurate fusion of different modal information in the 3D scene reconstruction model, resulting in a more accurate geometric structure of the reconstructed 3D model, and the relative positional relationships between objects more consistent with the actual scene, thus improving the overall accuracy and reliability of 3D scene reconstruction.

[0123] Step 7: 3D Scene Model Construction: A 3D scene model is constructed using finely aligned multimodal data. A voxel-based method is employed to fuse point cloud data, color image texture information, and depth image information into a voxel mesh.

[0124] Voxelization: The point cloud data P′ is voxelized to obtain a voxel mesh V. P Based on the spatial distribution of the point cloud, the size of the voxels is determined so that the voxels can better represent the geometric details of the scene.

[0125] Texture mapping: For each voxel in the voxel mesh, the texture information of the color image is mapped onto the voxel based on its corresponding position in the color image. Let voxel v be located in color image I. i The corresponding coordinates in "" are (x v ,y v ), then I i (x v ,y v The color information of the voxel is assigned to the voxel.

[0126] Deep fusion: This process fuses depth image information into a voxel mesh. Based on the depth values ​​in the depth image, the depth information of the voxels is determined, further refining the geometry of the 3D scene model. Through these operations, a preliminary 3D scene model M is obtained.init .

[0127] The above technical solution is explained as follows: The voxel-based 3D scene model construction method organically integrates point cloud data, color image texture information, and depth image information into a voxel mesh. Voxelization converts point cloud data into a regular mesh structure, facilitating subsequent processing and computation, while also preserving the spatial distribution information of the point cloud and accurately constructing the geometric framework of the scene. The texture mapping process assigns texture information from the color image to the voxels, giving the model rich color and texture details, enhancing its realism and visual effect. The depth fusion operation utilizes the depth information from the depth image to further refine the model's geometric structure, detailing the shape and contour of objects. Especially when dealing with objects with complex shapes and depth variations, it can more accurately describe the object's 3D morphology. This multimodal information fusion model construction method, compared to models constructed from single-modal data, can more comprehensively present the true appearance of the scene, providing users with a more realistic and detailed 3D scene experience, meeting the needs of virtual reality, augmented reality, and other fields for high-precision 3D scene models.

[0128] Step 8: Model Optimization

[0129] For the initially constructed 3D scene model M init Optimize the model to improve its quality.

[0130] Surface smoothing: The Laplacian smoothing algorithm is used to smooth the surface of the model. For each vertex v on the model surface, its Laplacian operator Δv is calculated:

[0131]

[0132] Where N(v) represents the set of neighboring vertices of vertex v, and k is the number of neighboring vertices. Vertex positions are updated iteratively:

[0133] v new =v + αΔv;

[0134] Where α is a smoothing factor, which makes the model surface smoother through multiple iterations. 2. Hole Repair: Hole areas in the model are detected, and a method based on Poisson surface reconstruction is used to repair the holes, resulting in an optimized 3D scene model M. opt .

[0135] The above technical solutions are explained as follows: The surface smoothing operation employs the Laplacian smoothing algorithm, effectively removing unevenness and sharp edges on the model surface caused by data noise or reconstruction errors, making the model surface smoother and more natural, conforming to the appearance characteristics of real objects. After smoothing, the visual effect of the model is significantly improved, making it more aesthetically pleasing during rendering and visualization, and enhancing the user's visual experience. The hole repair operation is based on the Poisson surface reconstruction method, which can accurately detect and repair hole areas in the model, ensuring the integrity and closure of the model. A complete model is not only more visually perfect, but also avoids calculation errors or inaccurate results caused by model incompleteness in subsequent applications, such as finite element analysis and fluid simulation, improving the applicability and reliability of the model in engineering and scientific computing fields, and further enhancing the quality and practicality of 3D scene reconstruction models.

[0136] Step Nine: Model Rendering

[0137] For the optimized 3D scene model M opt Rendering is performed to generate realistic 3D scene images. Physically based rendering (PBR) technology is used, considering physical factors such as lighting, materials, and reflections to calculate the color and brightness of the model surface. 1. Lighting calculation: Based on the light source information in the scene, the illumination intensity received by each point on the model surface is calculated. For point p, its illumination intensity L(p) is:

[0138]

[0139] Where, N l L represents the number of light sources. l (p) represents the light intensity produced by the l-th light source at point p. 2. Application of material properties: Based on the material properties of different objects in the model, such as diffuse reflection coefficient and specular reflection coefficient, calculate the reflection and refraction of light on the object surface to obtain the rendered 3D scene image.

[0140] The above technical solution is explained as follows: Physically Based Rendering (PBR) technology fully considers physical factors such as lighting, materials, and reflection during the model rendering process, making the rendered 3D scene images more consistent with the optical laws of the real world. Lighting calculations accurately simulate the lighting effects produced by different light sources in the scene, including direct and indirect lighting, realistically representing the changes in brightness and shadow effects of objects under different lighting conditions, enhancing the scene's three-dimensionality and realism. The application of material properties accurately simulates the reflection, refraction, and scattering processes of light on the object's surface based on the actual material characteristics, allowing objects of different materials to exhibit their unique appearance and texture in the rendered image, such as the luster of metal and the diffuse reflection of plastic. Compared to traditional rendering methods, this physically based rendering method can generate higher quality and more realistic 3D scene images, providing more lifelike visual effects for fields such as virtual reality, film and television production, and game development, and enhancing the application value of 3D scene reconstruction results.

[0141] Step 10: Model Output and Storage

[0142] The rendered 3D scene model is output in common 3D model formats (such as OBJ, STL, PLY, etc.) and stored locally or uploaded to a server for subsequent application and sharing.

[0143] The above technical solution is explained as follows: Outputting and storing the rendered 3D scene model in a common 3D model format facilitates sharing and interactive use of the model across different platforms and software. Common 3D model formats (such as OBJ, STL, PLY, etc.) have broad compatibility and are supported by most 3D modeling, rendering, and analysis software, allowing users to further edit, process, and apply the model in different software environments according to their needs.

[0144] In summary, this embodiment systematically solves many problems existing in traditional methods from data acquisition to model output, and shows significant beneficial effects in terms of data processing accuracy, feature fusion effect, reconstruction efficiency and application expansion.

[0145] During the data acquisition and preprocessing stage, multiple sensors collaborate to collect multimodal data, encompassing color images, point cloud data, and depth images. This comprehensive acquisition of texture, geometry, and depth information of the scene compensates for the limitations of single-modal data, laying a solid foundation for subsequent processing. Targeted data preprocessing operations, such as color image denoising and enhancement, outlier removal from point cloud data, and hole filling in depth images, effectively improve data quality, reduce data errors and interference, provide reliable data assurance for accurate reconstruction, and avoid reconstruction deviations caused by data flaws.

[0146] In the data alignment stage, an innovative strategy of first coarse alignment and then fine alignment is adopted. Coarse alignment is based on feature point matching to initially establish the correspondence between data, which significantly reduces the initial positional differences between data and provides a good initial value for fine alignment, reducing the risk of getting trapped in local optima. Fine alignment improves the ICP algorithm by using fused features and establishes accurate correspondence based on multimodal comprehensive information, which significantly improves the alignment accuracy and ensures that data from different modalities are accurately matched in space. This makes the geometric structure of the reconstructed model accurate and the relative positions of objects fit the actual scene.

[0147] In terms of feature extraction and fusion, deep learning networks customize their architecture based on the characteristics of data from each modality, efficiently extracting rich features such as semantics, geometry, and multi-scale depth. Compared with traditional methods, they have greater abstract representation and environmental adaptability. Attention-driven feature fusion dynamically balances the weights of features from each modality, achieving complementary advantages, highlighting key features, and suppressing redundant information. This enables the fused features to comprehensively and accurately describe the scene, endowing the reconstructed model with rich details and realistic texture, and enhancing its adaptability to complex and diverse scenes.

[0148] In the process of constructing a 3D scene model, a voxel-based method is used to deeply fuse multimodal data. Voxelized and regularized point cloud data is used to construct a geometric framework, texture mapping gives the model realistic colors and textures, and deep fusion refines object shapes, presenting a comprehensive and realistic view of the scene. This meets the stringent requirements of virtual reality, augmented reality, and other fields for high-precision and highly realistic models. The model optimization steps, through surface smoothing and hole repair, eliminate surface defects, ensure model integrity, improve visual effects and engineering application reliability, and expand the model's applicability.

[0149] In the model rendering stage, physically based rendering technology follows real optical laws, accurately simulating the physical properties of lighting and materials. This results in rendered images that highly reproduce real-world scenes, providing an immersive and realistic visual experience for virtual reality, film and games, and other fields, thus enhancing the application value of the reconstruction results. Finally, the universal model output format and convenient storage method break down platform software barriers, enabling efficient model sharing and interaction. This facilitates secondary development and utilization by users, promoting the widespread dissemination and in-depth application of 3D scene reconstruction technology across multiple industries, and fostering continuous technological innovation and development.

[0150] Although embodiments of the invention have been shown and described, the scope of the invention will be defined by the appended claims and their equivalents by those skilled in the art.

Claims

1. A method for reconstructing a three-dimensional scene from multimodal data, characterized in that, Specifically, the following steps are included: Step 1: Collect data on the target scene using multiple sensors, including color image sequences acquired by an optical camera, point cloud data acquired by a lidar, and depth image sequences acquired by a depth camera. Step 2: Denoise the acquired color image sequence using a median filtering algorithm. Step 3: Perform coarse alignment on the preprocessed multimodal data using a feature point-based method; Step 4: Construct a multimodal deep learning network based on deep learning feature extraction to extract features from the coarsely aligned color image, point cloud data, and depth image respectively; Step 5: Use an attention mechanism to fuse the extracted multimodal features; Step 6: Based on the fused features, perform fine alignment of multimodal data using an improved version of the iterative nearest point algorithm; Step 7: Construct a 3D scene model using the finely aligned multimodal data; Step 8: Optimize the initially constructed 3D scene model; Step 9: Render the optimized 3D scene model to generate a realistic 3D scene image; Step 10: Output the rendered 3D scene model in a common 3D model format and save it locally or upload it to the server.

2. The method for reconstructing a three-dimensional scene from multimodal data according to claim 1, characterized in that, In step one: data is collected from the target scene using multiple sensors, including a sequence of color images {I1, I2, ..., I...} acquired by an optical camera. n Point cloud data P = {p1, p2, ..., p} acquired by lidar m } and the depth image sequence {D1,D2,…,D} acquired by the depth camera n }; where n represents the number of images, m represents the number of points in the point cloud data, and I i Let p represent the i-th color image. j D represents the j-th point in the point cloud data. i Let i represent the depth image of the i-th image.

3. The method for reconstructing a three-dimensional scene from multimodal data according to claim 2, characterized in that, In step two: the acquired color image sequence {I1,I2,…,I...} n For noise reduction, a median filtering algorithm is used. The specific method and formula are as follows: I′ i (x,y)=median{I i (x-k,y-l),(x,y)∈N k×l }; Among them, I i This represents the i-th color image after denoising, where (x, y) are the pixel coordinates of the image, and N... k×l It is a neighborhood window of size k×l centered at (x,y), and median represents the median calculation operation; Outlier removal is performed on the point cloud data P acquired by lidar, and the value of each point p is calculated. j The average distance d between it and its k nearest neighbors j ; Where, N k (p j ) represents point p j The set of k nearest neighbors, ||·|| represents the Euclidean distance; set a threshold T, if d j >T, then point p is considered j Outliers are removed from the point cloud data to obtain preprocessed point cloud data P′; For the depth image sequence {D1,D2,…,D…} n Hole filling is performed using a hole repair algorithm based on the fast traversal method to fill the hole regions in the depth image, resulting in a complete depth image sequence {D′1,D′2,…,D′}. n }; By calculating the Euclidean distance between the descriptors of feature points, four-way matching is performed on feature points of different modalities. This is particularly relevant for feature points in color images. and feature points f in point cloud data Pk Calculate its descriptor and d P Distance of k: Set a distance threshold t, if Then it is believed and f Pk To match point pairs, we obtain the set M of matching point pairs between color image and point cloud data, color image and depth image, and point cloud data and depth image. IP M ID M PD ; Based on the set of matching point pairs, the coarse alignment transformation matrix between multimodal data is calculated using the least squares method. Let the set of matching point pairs be... By minimizing the following objective function: Where R is the rotation matrix and t is the translation vector, solving the above objective function yields the coarse alignment transformation matrix T between the color image and the point cloud data. IP =[R IP ,t IP Similarly, the coarse alignment transformation matrix T between the color image and the depth image is obtained. ID The coarse alignment transformation matrix T between point cloud data and depth image PD Coarse alignment is performed on multimodal data.

4. The method for reconstructing a three-dimensional scene from multimodal data according to claim 3, characterized in that, In step three, the specific method for coarse alignment of the preprocessed multimodal data using a feature point-based approach is as follows: In color image I′ i In this study, the SIFT algorithm was used to extract feature points. Where q is the number of feature points; In the point cloud data P′, the PFH algorithm is used to extract feature points F. P ={f P1 ,f P2 ,…,f Pr }; Where r is the number of feature points in the point cloud; In depth image D′ i In this study, the SURF algorithm was used to extract feature points. This represents the number of feature points in the depth image.

5. The method for reconstructing a three-dimensional scene from multimodal data according to claim 4, characterized in that, A multimodal deep learning network is constructed based on deep learning feature extraction. The specific methods for feature extraction from coarsely aligned color images, point cloud data, and depth images are as follows: The color image is input into the convolutional neural network, and after multiple layers of convolution and pooling operations, the semantic and geometric features of the image are extracted. Let the input color image be I. i "After L layers of convolution and pooling operations, the image features are obtained." Where, θ CNN These are the parameters of the CNN network; Point cloud data is converted into a voxel mesh representation and then fed into a 3D convolutional neural network. After multiple layers of 3D convolution and pooling operations, the spatial geometric features of the point cloud data are extracted. Where V(P″) is the voxel mesh after point cloud data conversion, and θ 3D-CNN These are the parameters of the 3D-CNN network; The depth image is input into an improved U-Net network, and its encoder-decoder structure is used to extract multi-scale features from the depth image. Where, θ U-Net These are the parameters of the U-Net network.

6. The method for reconstructing a three-dimensional scene from multimodal data according to claim 5, characterized in that, In step five, the specific method for fusing the extracted multimodal features using an attention mechanism is as follows: For color image features Point cloud data features and depth image features Calculate the attention weights between them separately, and calculate the attention weight w. IP : Where s(·,·) is the similarity calculation function, using cosine similarity: Similarly, the attention weight w is calculated. ID and w PD ; Based on the calculated attention weights, the multimodal features are weighted and fused to obtain the fused features.

7. The method for reconstructing a three-dimensional scene from multimodal data according to claim 6, characterized in that, In step six: Based on the fused features, the specific method for fine alignment of multimodal data using an improved version of the iterative nearest point algorithm is as follows: Based on the fused features, a more accurate correspondence is established between different modal data. For point p in point cloud data P′... j Find the regions with the most similar features in the color and depth images and establish corresponding point pairs; The fine alignment transformation matrix is ​​calculated by minimizing the distance between corresponding point pairs; Let the set of corresponding point pairs be C = {(p j ,q j The objective function is minimized by minimizing the following: Where R′ is the rotation matrix and t′ is the translation vector, the fine alignment transformation matrix T is obtained by iteratively solving the above objective function. final =[R final ,t final [This is used to perform fine alignment of multimodal data.] 8. The method for reconstructing a three-dimensional scene from multimodal data according to claim 7, characterized in that, In step seven: the specific method for constructing a 3D scene model using the finely aligned multimodal data is as follows: A voxel-based method is used to fuse point cloud data, color image texture information, and depth image information into a voxel mesh; Voxelize the point cloud data P′ to obtain the voxel mesh V. P Based on the spatial distribution of the point cloud, the size of the voxels is determined so that the voxels can better represent the geometric details of the scene. For each voxel in the voxel mesh, the texture information of the color image is mapped onto the voxel according to its corresponding position in the color image; let voxel v be in the color image I. i The corresponding coordinates in "" are (x v ,y v ), then I i "(x v ,y v The color information of the voxel is assigned to the voxel. Depth image information is fused into a voxel mesh. Based on the depth values ​​in the depth image, the depth information of the voxels is determined, resulting in a preliminary 3D scene model M. init .

9. The method for reconstructing a three-dimensional scene from multimodal data according to claim 8, characterized in that, In step eight: the initially constructed 3D scene model M init The specific methods for optimization are as follows: The Laplacian smoothing algorithm is used to smooth the surface of the model; for each vertex v on the model surface, its Laplacian operator Δv is calculated: Where N(v) represents the set of neighboring vertices of vertex v, and k is the number of neighboring vertices; the vertex position is updated iteratively: v new =v+αΔv Where α is a smoothing factor, which smooths the model surface through multiple iterations; The hole regions in the detection model were repaired using a Poisson surface reconstruction method, resulting in an optimized 3D scene model M. opt ; In step nine: the optimized 3D scene model M opt The specific method for rendering and generating realistic 3D scene images is as follows: Physically based rendering techniques are used to calculate the color and brightness of the model's surface; Based on the light source information in the scene, calculate the illumination intensity at each point on the model surface; for point p, its illumination intensity L(p) is: Where, N l L represents the number of light sources. l (p) represents the light intensity produced by the l-th light source at point p; based on the material properties of different objects in the model, the reflection and refraction of light on the object surface are calculated to obtain the rendered 3D scene image.

Images