Complex environment three-dimensional scene reconstruction method based on multi-view neural network

Abstract

The patent application provides a complex environment three-dimensional scene reconstruction method based on a multi-view neural network, and the method mainly comprises two steps: scene information capturing: obtaining scene data from a plurality of views through a high-resolution camera, and ensuring that all details of a complex scene are covered; the captured image is standardized, including color correction and noise removal, to ensure image quality and consistency. And three-dimensional model reconstruction: performing three-dimensional reconstruction on the acquired scene data by using a neural network. Firstly, light sampling is performed on a scene through camera parameters and pose information, and an occupancy grid is created to optimize the sampling efficiency. Thirdly, decomposing the sampling data into feature vectors and inputting the feature vectors into a neural network which is divided into a density network and a color network and is used for generating volume density and color information of the scene; and finally, obtaining a three-dimensional scene model through a volume rendering technology. The method has the advantage that a high-precision three-dimensional model can be quickly generated in a complex environment through an efficient sampling and data processing technology.

Description

Technical Field

[0001] This invention relates to the field of image processing and analysis technology, and more specifically, to a method for reconstructing three-dimensional scenes in complex environments. Background Technology

[0002] 3D reconstruction is an advanced computational process designed to automatically recover the 3D structure of objects or scenes from 2D data. With technological advancements and increasing demands for depth information, accurately reconstructing a complete 3D model from a limited viewpoint has become a key challenge in this research field. 3D reconstruction focuses on utilizing algorithms and computer vision techniques to automatically construct an accurate representation of 3D space by analyzing multiple images or other forms of measurement data. This method not only provides richer spatial and shape information than 2D images but also reproduces the true appearance and dimensions of objects in environments where direct measurement or access is impossible.

[0003] As a crucial component of computer vision technology, 3D reconstruction holds immense promise for applications in fields such as cultural relic preservation, medical imaging, and robot navigation. Neural network-based 3D reconstruction represents a significant breakthrough in this field, aiming to overcome the limitations of traditional methods. By learning a continuous volumetric representation of a scene from sparse image data using deep neural networks, it can generate high-quality images from previously unseen perspectives, offering new possibilities for 3D scene rendering. Unlike traditional 3D reconstruction methods based on geometric or optical principles, neural network-based methods do not directly rely on feature point matching, direct measurement of depth information, or specific hardware. Instead, it trains a deep neural network to simulate the color and density of every point in the scene, using a set of sparse 2D images as input. This network learns how to predict the color and density of light rays passing through the scene, enabling the rendering of high-quality images from any viewpoint.

[0004] At present, there are still many problems in the 3D reconstruction technology based on neural networks: (1) Due to the complex structure of neural networks, the optimization calculation of neural networks requires a lot of time; (2) Due to the insufficient precision of the multilayer perceptron in neural networks in processing high-frequency details, the accuracy of the 3D reconstructed model in complex environments is low; (3) There is a lack of ability to quickly and accurately process scene details under different scales of observation. Summary of the Invention

[0005] like Figures 1 to 3As shown, the present invention provides a method for reconstructing three-dimensional scenes in complex environments, used for three-dimensional reconstruction of scenes. The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for reconstructing three-dimensional scenes in complex environments. This three-dimensional scene reconstruction method can not only effectively generate three-dimensional scene models in complex environments, but also has high efficiency, enabling relatively rapid reconstruction of three-dimensional scene models in complex environments, thereby improving the accuracy and effectiveness of three-dimensional models.

[0006] To achieve the above objectives, the present invention provides a method for reconstructing a three-dimensional scene in a complex environment, used to generate a three-dimensional model of a complex scene; characterized by comprising the following two steps: The first step is to capture scene information by using a high-resolution camera to capture the scene from multiple perspectives to ensure that enough perspectives are obtained to cover all the details and depth information in the complex scene. Then, the captured images are standardized, including color correction and noise removal, to obtain high-quality scene data information.

[0007] The second step involves using a neural network to construct a 3D model of the complex scene. Based on the collected scene data, corresponding pose information and camera parameters are obtained. Then, using the pose information and camera parameters, light is simulated to sample the scene data. The intersection of the light rays passing through the camera origin and the sampling point is assumed to be a sphere, and the area within the sphere is sampled. An occupancy grid is created to optimize the light sampling process, allowing blank areas to skip sampling and improving efficiency. Next, spatial decomposition is used to decompose the sampled spatial information into feature vectors of three orthogonal multi-level texture planes to represent the spatial position information. The obtained feature vectors are used as input to the neural network for training. The neural network consists of two parts: the first part is a density neural network, where the feature vectors representing spatial information are used as input, and the output is the volume density of the scene information; the second part is a color neural network, where the input is the output of the first neural network plus the direction information processed by spherical harmonic encoding, and the output is the color of the scene information. Finally, volume rendering technology is used to render the 3D model of the scene using the volume density and color.

[0008] In the first step, capturing scene information involves using a high-resolution camera to capture the scene from multiple perspectives to ensure sufficient coverage of all details and depth information in a complex scene. Then, the captured images undergo standardization processing, including color correction and noise removal, to obtain high-quality scene data information. This includes the following steps: (1.1) Capture scene information and set up multiple cameras to capture scene information from different angles to ensure coverage of all areas in the complex scene; (1.2) Ensure high-quality image data and perform color correction on the captured images to ensure that the colors are consistent across different cameras. Then perform noise removal to remove interference information in the images and improve the clarity and accuracy of the images. (1.3) Extract depth information and calculate the depth information of each pixel based on the geometric layout of the camera; (1.4) Generate standardized scene data. Standardize all captured images and depth information to ensure that the resolution, size and coordinate system of all data are consistent. Then, use common image processing techniques, such as smoothing, sharpening and contrast adjustment, to improve data quality. (1.5) Organize the data to ensure the integrity and consistency of the scene information, and delete redundant or unnecessary data to obtain high-quality scene data information; In the second step, a neural network is used to construct a 3D model of a complex scene. Based on the collected scene data, corresponding pose information and camera parameters are obtained. Then, using the pose information and camera parameters, light is simulated to sample the scene data. The intersection of the light rays passing through the camera origin and the sampling point is assumed to be a sphere, and the area within the sphere is sampled. An occupancy grid is created to optimize the light sampling process, allowing blank areas to skip sampling and improving efficiency. Next, spatial decomposition is used to decompose the sampled spatial information into feature vectors of three orthogonal multi-level texture planes to represent the spatial position information. The obtained feature vectors are used as input to the neural network for training. The neural network consists of two parts: the first part is a density neural network, where the feature vectors representing spatial information are used as input, and the output is the volume density of the scene information; the second part is a color neural network, where the input is the output of the first neural network plus the direction information processed by spherical harmonic encoding, and the output is the color of the scene information. Finally, volume rendering technology is used to render the volume density and color to create a 3D model of the scene, which includes the following steps: (2.1) Using the camera's pose information and parameters, simulate the light rays emitted from the camera's origin. This is achieved through the formula... ,in o It is the camera's origin. d It is a direction vector. t It is the distance that determines the path of light in the scene.

[0009] (2.2) Multi-level texture coding uses a cone projection method to determine the part where the light rays projected from the camera origin intersect with the scene. During the ray sampling process, the intersection point between the light ray and the scene is assumed to be a sphere. The radius of the sphere is determined according to the focal length and the distance between the camera origin and the sampling point to ensure that the sampling is within a specific area of ​​space.

[0010] (2.3) Optimize the ray sampling process by creating an occupancy grid. The occupancy grid uses binary markers to ensure that blank areas are skipped, avoiding unnecessary sampling in areas where there are no objects. Using an occupancy grid can significantly improve the efficiency of ray sampling.

[0011] (2.4) Spatial decomposition is used to decompose the spatial information of the sampling points into feature vectors of three orthogonal multi-level texture planes, thereby realizing multi-resolution feature description of three-dimensional space. Each plane has a feature vector to represent the spatial position and features.

[0012] (2.5) The 3D model of the scene is represented by a neural network, which consists of two parts. The first part is a density neural network, which takes feature vectors obtained from multi-level texture planes as input and outputs the volume density of scene information. The output of this part of the network can represent the density of objects in the scene.

[0013] (2.6) The second part is the color neural network. The input is the output of the density neural network and the direction information processed by spherical harmonic encoding. The output is the color information of the scene. The color neural network combines the direction of light and the features in the scene to output the color.

[0014] (2.7) Use volume rendering technology to weight and integrate the density and color of each sampling point to synthesize the density and color information into a three-dimensional scene model, thereby rendering the three-dimensional model of the scene.

[0015] In step (2.2), determining the radius of the sphere based on the focal length and the distance between the camera origin and the sampling point means calculating the radius of the sphere using formula (1): (1) in, f It is the focal length, and the radius of the circle where the camera ray intersects the sampling point is determined by... Seeking, o It is the camera's origin. d It is a direction vector; In step (2.4), the spatial decomposition is used to decompose the spatial information of the sampling points into feature vectors of three orthogonal multi-level texture planes, and the corresponding multi-level texture query level is calculated by formula (2): (2) in It is a multi-level texture The radius of the feature element in the basic level, B max and B min These are the maximum and minimum angles of the axis-aligned bounding box (AABB) in the 3D space of interest, respectively.

[0016] Compared with the prior art, the present invention has the following advantages and beneficial effects: 1. Unlike existing methods, this method captures scene information from multiple perspectives, ensuring that every part of the scene is fully captured. This multi-view setup helps achieve comprehensive visual coverage and reduces blind spots.

[0017] The captured images undergo standardization processing, including color correction and noise removal, to ensure data consistency and quality. This helps reduce errors and noise in the images, providing a more stable data source for subsequent 3D reconstruction.

[0018] When representing complex scenes in 3D, neural networks can automate the construction of 3D data. This reduces human intervention and ensures automation and efficiency in the reconstruction process. Light sampling is simulated using pose information and camera parameters, and the sampling process is optimized using the sphere assumption and occupancy mesh. This multi-layered information processing method ensures sampling accuracy and avoids invalid sampling of blank areas. Spatial decomposition is used to break down spatial information into three orthogonal multi-level texture planes. This effectively captures spatial feature vectors, providing rich input data for the neural network. Finally, volume rendering technology integrates density and color information to present a 3D model. This technique ensures high quality and realism in the final rendering result.

[0019] 2. This 3D scene reconstruction method can not only accurately predict the volume density and color information of the scene, but also is applicable to various 3D reconstruction applications, including virtual reality, augmented reality, and movie special effects.

[0020] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention. Attached Figure Description Figure 1 This is the flowchart framework of the three-dimensional scene reconstruction method of the present invention; Figure 2 This is the first step in the process of acquiring scene images from complex environments; Figure 3 This is the second step, the method and process of reconstructing a 3D model from the collected scene images; Figure 4 This is an appearance model diagram of a spherical volume sample; Figure 5 This is a schematic diagram of the 3D model of the scene reconstruction.

Claims

1. A method for reconstructing complex 3D scenes based on multi-view neural networks, used for reconstructing 3D scenes, characterized in that... It includes the following two steps: The first step is to capture scene information by using a high-resolution camera to capture the scene from multiple perspectives to ensure that enough perspectives are obtained to cover all the details and depth information in the complex scene; then the captured images are standardized, including color correction and noise removal, to obtain high-quality scene data information. The second step involves using neural networks to construct a 3D model of the complex scene. Based on the collected scene data, corresponding pose information and camera parameters are obtained. Then, using the pose information and camera parameters, light is simulated to sample the scene data. The intersection of the light rays passing through the camera origin and the sampling point is assumed to be a sphere, and the region within this sphere is sampled. The process of ray sampling is optimized by creating an occupancy grid, allowing blank areas to skip sampling and improving efficiency. Next, spatial decomposition is used to decompose the sampled spatial information into feature vectors of three orthogonal multi-level texture planes to represent spatial location information. These feature vectors are then used as input to a neural network for training. The neural network consists of two parts: a density neural network, where the feature vectors representing spatial information are input, and the output is the volume density of the scene information; and a color neural network, where the input is the output of the first neural network plus the directional information processed by spherical harmonic encoding, and the output is the color of the scene information. Finally, volume rendering technology is used to render the 3D model of the scene using the volume density and color. In the first step, capturing scene information involves using a high-resolution camera to capture the scene from multiple perspectives to ensure sufficient coverage of all details and depth information in a complex scene. Then, the captured images undergo standardization processing, including color correction and noise removal, to obtain high-quality scene data information. This includes the following steps: (1.1) Capture scene information and set up multiple cameras to capture scene information from different angles to ensure coverage of all areas in the complex scene; (1.2) Ensure high-quality image data and perform color correction on the captured images to ensure that the colors are consistent across different cameras. Then perform noise removal to remove interference information in the images and improve the clarity and accuracy of the images. (1.3) Extract depth information and calculate the depth information of each pixel based on the geometric layout of the camera; (1.4) Generate standardized scene data. Standardize all captured images and depth information to ensure that the resolution, size and coordinate system of all data are consistent. Then, use common image processing techniques, such as smoothing, sharpening and contrast adjustment, to improve data quality. (1.5) Organize the data to ensure the integrity and consistency of the scene information, and delete redundant or unnecessary data to obtain high-quality scene data information; In the second step, a neural network is used to construct a 3D model of the complex scene. Based on the collected scene data, corresponding pose information and camera parameters are obtained. Then, using the pose information and camera parameters, light is simulated to sample the scene data. The intersection of the light rays passing through the camera origin and the sampling point is assumed to be a sphere, and the region within the sphere is sampled. The process of light sampling is optimized by creating an occupancy grid, allowing blank areas to skip sampling and improving efficiency. Next, spatial decomposition is used to decompose the sampled spatial information into feature vectors of three orthogonal multi-level texture planes to represent spatial location information. These feature vectors are then used as input to a neural network for training. The neural network consists of two parts: the first is a density neural network, where the feature vectors representing spatial information are input, and the output is the volume density of the scene information; the second is a color neural network, where the input is the output of the first neural network plus the directional information processed by spherical harmonic encoding, and the output is the color of the scene information. Finally, rendering the 3D model of the scene using volume density and color through volume rendering technology involves the following steps: (2.1) Using the camera's pose information and parameters, simulate the light rays emitted from the camera's origin. This is achieved through the formula... ,in o It is the camera's origin. d It is a direction vector. t It is the distance that determines the path of light in the scene; (2.2) Multi-level texture coding uses a cone projection method to determine the part where the light rays projected from the camera origin intersect with the scene. During the ray sampling process, the intersection point between the light ray and the scene is assumed to be a sphere. The radius of the sphere is determined based on the focal length and the distance between the camera origin and the sampling point to ensure that the sampling is within a specific area of ​​space; (2.3) Optimize the ray sampling process by creating an occupancy grid. The occupancy grid uses binary markers to ensure that blank areas are skipped, avoiding unnecessary sampling in areas without objects. Using an occupancy grid can significantly improve the efficiency of ray sampling; (2.4) Spatial decomposition is used to decompose the spatial information of the sampling points into feature vectors of three orthogonal multi-level texture planes, thereby realizing multi-resolution feature description of three-dimensional space. Each plane has a feature vector to represent the spatial position and features; (2.5) The 3D model of the scene is represented by a neural network, which consists of two parts. The first part is a density neural network, which takes feature vectors obtained from a multi-level texture plane as input and outputs the volume density of scene information. The output of this part of the network can represent the density of objects in the scene; (2.6) The second part is the color neural network. The input is the output of the density neural network and the direction information processed by spherical harmonic encoding. The output is the color information of the scene. The color neural network combines the direction of light and the features in the scene to output the color. (2.7) Use volume rendering technology to weight and integrate the density and color of each sampling point to synthesize the density and color information into a three-dimensional scene model, thereby rendering the three-dimensional model of the scene; In step (2.2), determining the radius of the sphere based on the focal length and the distance between the camera origin and the sampling point means calculating the radius of the sphere using formula (1): (1) in, f It is the focal length, and the radius of the circle where the camera ray intersects the sampling point is determined by... Seeking, o It is the camera's origin. d It is a direction vector; In step (2.4), the spatial decomposition is used to decompose the spatial information of the sampling points into feature vectors of three orthogonal multi-level texture planes, and the corresponding multi-level texture query level is calculated by formula (2): (2) in It is a multi-level texture The radius of the feature element in the basic level, B max and B min These are the maximum and minimum angles of the axis-aligned bounding box (AABB) in the 3D space of interest, respectively.

2. The method for reconstructing a three-dimensional scene in a complex environment according to claim 1, characterized in that: In steps (2.5) to (2.7), the representation of the three-dimensional model of the scene by a neural network means that the feature vector based on the spatial information of multi-level texture encoding is learned by the neural network, so as to better represent the voxel information and color of the scene. Finally, the density and color of each sampling point are weighted and integrated by volume rendering technology to synthesize the density and color information into a three-dimensional scene model.

Images