A scene building method and system based on three-dimensional reconstruction

Abstract

The application discloses a scene building method and system based on three-dimensional reconstruction, comprising collecting multi-view two-dimensional images of an environment; pre-processing the obtained two-dimensional images, and automatically identifying and labeling different objects and regions in the images by using a semantic segmentation technology to understand semantic information in the images; according to the labeled two-dimensional image data, a multi-view geometric three-dimensional reconstruction algorithm based on a bundle adjustment is used to reconstruct a three-dimensional model from the two-dimensional images, and the reconstructed three-dimensional model is stored as a sub-scene model; the application accurately reconstructs a three-dimensional model by collecting multi-view two-dimensional images and using a semantic segmentation and bundle adjustment technology, solves the technical bottleneck of real-time rendering of a complex scene, and improves the scene building efficiency and accuracy; a generative adversarial network and a variational autoencoder optimization model are combined to enhance the authenticity, detail expression and diversity, and provide a rich and realistic scene experience for a virtual reality field.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically a scene construction method and system based on 3D reconstruction. Background Technology

[0002] With the development of computer graphics, virtual reality (VR) and augmented reality (AR) technologies have been widely applied in entertainment, education, and healthcare. However, existing technologies still face significant technical bottlenecks when rendering large-scale, complex scenes in real time. Traditional 3D scene construction methods typically rely on specialized 3D modeling software, requiring designers to manually create 3D models and perform texture mapping. This is not only time-consuming and labor-intensive but also fails to meet the efficiency and accuracy requirements of real-time rendering of large-scale scenes, resulting in an inadequate user experience in complex scenarios. Summary of the Invention

[0003] The purpose of this invention is to provide a scene building method and system based on 3D reconstruction. By using multi-view image acquisition, preprocessing, semantic segmentation and a 3D reconstruction algorithm based on bundle adjustment, it effectively solves the technical bottleneck of real-time rendering of large-scale complex scenes and improves the efficiency and accuracy of scene building.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] This application provides a scene construction method based on 3D reconstruction, including the following steps:

[0006] Capture multi-view two-dimensional images of the environment using drones, mobile phone cameras, and image acquisition devices;

[0007] The acquired two-dimensional images are preprocessed, and semantic segmentation technology is used to automatically identify and label different objects and regions in the images to understand the semantic information in the images;

[0008] Based on the labeled 2D image data, a 3D model is reconstructed from the 2D image using a multi-viewpoint geometric 3D reconstruction algorithm based on bundle adjustment, and the reconstructed 3D model is stored as a sub-scene model.

[0009] The multi-viewpoint geometric 3D reconstruction algorithm based on bundle adjustment reconstructs a 3D model from a 2D image and stores the reconstructed 3D model as a sub-scene model. Specifically, this includes:

[0010] Match corresponding feature points between images from different viewpoints, determine the position of feature points in different images, and determine the camera's intrinsic and extrinsic parameters using known calibration board images;

[0011] Using multi-view feature point matching and camera parameters, the three-dimensional coordinates of each feature point in space are determined by triangulation.

[0012] The 3D coordinates and camera parameters are optimized by minimizing the reprojection error. The least squares method is used to update the parameter estimates. The reprojection error refers to the difference between the actual position of the 3D point after it is projected back into the 2D image and the actual position of the feature point.

[0013] The minimized reprojection error is expressed as: ;

[0014] in, It refers to the location of feature points in the image. It corresponds to the position of a three-dimensional point. These are camera parameters. Represents the projection function. It is a robust kernel function used to handle outliers;

[0015] The optimized 3D point coordinates and camera parameters of the image are stored as a sub-scene model.

[0016] Furthermore, the acquired two-dimensional image undergoes preprocessing, specifically including:

[0017] Noise in two-dimensional images is removed using bilateral filtering;

[0018] Specifically, it is expressed as follows:

[0019] ;

[0020] in, It represents the pixel value of the filtered image at point (x, y). The original image at points pixel values, It is a spatial Gaussian function. It is the spatial standard deviation, which controls the size of the filter. It is an intensity Gaussian function. It is the intensity standard deviation, which controls the similarity between pixel values. It is a normalization factor;

[0021] Color correction is performed by adjusting the gain of each channel using the grayscale world hypothesis algorithm, and then the color is adjusted using the white balance algorithm based on the grayscale card in the scene or based on the information of the image itself.

[0022] The specific algorithm for the gray-scale world hypothesis is expressed as follows: ;in It is the average brightness of the entire image. It is the average brightness of channel c;

[0023] The specific white balance algorithm is expressed as follows: Where I(c) is a channel of the original image. This is the value of that channel after white balance;

[0024] The acquired two-dimensional image is registered using feature matching algorithm and image registration algorithm;

[0025] Specifically, a feature detection algorithm is used to detect key points and descriptors, and then matching point pairs are found by matching descriptors; the transformation matrix is ​​used to align one image to another; the transformation matrix solution includes homography matrix and affine transformation.

[0026] The homography matrix is ​​represented as follows: ;where x and It is a matching point pair. It is a robust kernel function;

[0027] The affine transformation is expressed as: ;in and These are the coordinates of the matching point pairs.

[0028] Furthermore, the application of semantic segmentation technology to automatically identify and label different objects and regions in the image specifically includes:

[0029] Features are extracted from preprocessed images using convolutional and pooling layers, and pixel-level classification is performed using a fully convolutional network.

[0030] The softmax algorithm is used to classify each pixel, and the feature map is enlarged to its original size through upsampling and deconvolution techniques to generate a segmentation map.

[0031] By using thresholding and connectivity analysis, pixel labels are aggregated into coherent object regions, enabling the identification and localization of boundaries for different objects.

[0032] Furthermore, the method of using least squares to update the parameter estimates specifically includes:

[0033] Where X is the design matrix and Y is the observation vector. It is a parameter vector.

[0034] Furthermore, a scene building method based on 3D reconstruction also includes: automatically selecting appropriate storage sub-scene models by specifying the required scene type and specific requirements, and performing preliminary combination according to the automatically selected instructions to form an initial scene model;

[0035] The process involves indexing the stored sub-scene models by defining keywords and tags, using intelligent retrieval to perform keyword matching and tag filtering based on project requirements analysis results, and identifying sub-scene models that match the requirements. Then, computer graphics technology is used to perform spatial analysis and calculations on the selected sub-scene models to determine their relative positions and orientations in the virtual scene. Finally, a constraint-based layout algorithm is used to automatically arrange and combine the models.

[0036] Furthermore, a scene construction method based on 3D reconstruction also includes: automatically making minor adjustments to the initial scene model to generate a target scene description file for the target scene model, wherein the target scene description file includes information for rendering and displaying the scene.

[0037] Furthermore, during the automatic fine-tuning of the initial scene model, training was conducted by adding or removing objects, modifying materials, and adjusting lighting.

[0038] Specifically, computer graphics techniques are used to automatically add or remove objects in a scene;

[0039] In the Blinn-Phong lighting model, adjust parameters such as reflectivity to modify material parameters; adjust the material properties of the object's surface, such as color, texture, and roughness, to match the style of the target scene.

[0040] Using a lighting model, physically based rendering technology simulates real-world lighting effects and optimizes lighting conditions in a scene, including adding or adjusting the position, intensity, and type of light sources.

[0041] In the physical rendering technique described above, the lighting calculations using BRDF (Brown-Based Lighting Rendering) to represent the lighting model are specifically represented as follows: ;

[0042] in, It is the Fresnel coefficient, which represents the reflectivity of the material. It is the normal distribution function, which describes the effect of surface roughness on illumination. It is a geometric function that describes the degree to which light is blocked by a surface. Here, N is the surface roughness, L is the surface normal, and V is the direction vector from the surface point to the light source and from the surface point to the camera. It refers to the intensity of the light source.

[0043] Furthermore, after automatically making minor adjustments to the initial scene model and generating the target scene description file of the target scene model, the target scene model is optimized by combining generative adversarial network and variational autoencoder. Specifically, the loss function of variational autoencoder and loss function of generative adversarial network are fused to obtain a joint optimization objective, which is used to obtain a three-dimensional model.

[0044] The joint loss function is expressed as:

[0045] ;

[0046] in, and These are the weighting coefficients that balance the two loss terms. Represented as reconstruction loss, it is used to measure the similarity between the 3D point cloud reconstructed by the latent variable z and the original point cloud;

[0047] Represented as KL divergence, it is used to measure the distribution of latent variables. With prior distribution The differences between them; This is to encourage the discriminator to correctly identify real 3D models. This is to encourage the generator to produce realistic fake 3D models.

[0048] Furthermore, a scene building method based on 3D reconstruction also includes: checking whether the generated target scene description file conforms to preset rules based on the target scene description file of the generated target scene model, and deploying the model that conforms to the rules to the virtual platform.

[0049] A scene building system based on 3D reconstruction includes an image acquisition module, an image preprocessing module, a feature extraction and image registration module, a semantic segmentation module, a 3D reconstruction and optimization module, and a scene combination and adjustment module.

[0050] The image acquisition module acquires multi-view two-dimensional images of the environment through drones, mobile phone cameras, and image acquisition devices;

[0051] The image preprocessing module performs preprocessing operations on the acquired two-dimensional images, specifically including bilateral filtering for noise removal, and gray-world hypothesis and white balance algorithm for color correction.

[0052] The feature extraction and image registration module uses a feature detection algorithm to detect key points and descriptors, and uses a feature matching algorithm to find matching point pairs.

[0053] Among them, an image registration algorithm using homography matrix and affine transformation is used to accurately align multi-view images;

[0054] The semantic segmentation module uses deep learning technology with fully convolutional networks to perform pixel-level semantic segmentation, automatically identify and label different objects and regions in the image, and understand the semantic information in the image.

[0055] The 3D reconstruction and optimization module, based on the multi-viewpoint geometric 3D reconstruction algorithm of bundle adjustment, reconstructs a 3D model from a 2D image and stores the reconstructed 3D model as a sub-scene model.

[0056] The scene combination and adjustment module automatically selects suitable sub-scene models through intelligent retrieval technology, performs preliminary combination to form an initial scene model, and then makes fine adjustments to the initial scene model, including adding or removing objects, modifying materials and adjusting lighting, generating a target scene description file, and then deploying the model that meets the rules to the virtual platform.

[0057] This also includes optimizing the initial scene model using generative adversarial networks and variational autoencoders.

[0058] The beneficial effects of this invention are as follows:

[0059] (1) This invention acquires multi-view two-dimensional images of the environment, preprocesses the acquired two-dimensional images, and then uses advanced semantic segmentation technology to automatically identify and label different objects and regions in the images. Based on the labeled two-dimensional image data, a multi-view geometric three-dimensional reconstruction algorithm based on bundle adjustment is used to accurately reconstruct a three-dimensional model from the two-dimensional images. This effectively solves the technical bottlenecks of traditional three-dimensional scene building methods in real-time rendering of large-scale complex scenes, such as large computational load and low accuracy. By improving the efficiency and accuracy of scene building, the user experience is fully guaranteed.

[0060] (2) Users only need to specify the required scene type and specific requirements, and the system will automatically select the appropriate storage sub-scene model and perform a preliminary combination according to the automatically selected instructions to form an initial scene model. The initial scene model can be finely adjusted to realize the automation and intelligence of scene building, which greatly improves the speed and efficiency of scene building. At the same time, due to the reduction in the need for manual intervention, the cost and error rate are also reduced.

[0061] (3) Combining generative adversarial networks and variational autoencoders to optimize the initial scene model, and obtaining a more realistic and detailed 3D model through the joint optimization objective obtained by fusion, not only improves the realism and detail of the 3D model, but also enhances the diversity and flexibility of the model. By combining GAN and VAE, it is possible to generate 3D models with higher quality and more realism, providing richer and more realistic scene experiences for virtual reality and augmented reality. This automated scene combination strategy provides users with a fast, efficient and accurate way to build complex 3D scenes. Attached Figure Description

[0062] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings.

[0063] Figure 1 A flowchart illustrating a scene construction method based on 3D reconstruction provided in Embodiment 1 of this application;

[0064] Figure 2 This is a schematic diagram illustrating the process of preprocessing two-dimensional images in a scene construction method based on three-dimensional reconstruction, as provided in Embodiment 1 of this application.

[0065] Figure 3 This is a flowchart illustrating the automatic identification and annotation of different objects and regions in an image using a scene construction method based on 3D reconstruction, as provided in Embodiment 1 of this application.

[0066] Figure 4 This is a schematic diagram illustrating the process of storing the reconstructed 3D model as a sub-scene model in a scene building method based on 3D reconstruction, as provided in Embodiment 1 of this application. Detailed Implementation

[0067] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.

[0068] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0069] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0070] Example 1

[0071] Please see Figures 1-4 This embodiment provides a scene building method and system based on 3D reconstruction. By using multi-view image acquisition, preprocessing, semantic segmentation and a 3D reconstruction algorithm based on bundle adjustment, it effectively solves the technical bottleneck of real-time rendering of large-scale complex scenes and improves the efficiency and accuracy of scene building.

[0072] This invention provides a scene construction method based on 3D reconstruction, comprising the following steps:

[0073] S1. Collect multi-view two-dimensional images of the real-world environment using drones, mobile phone cameras, and image acquisition devices;

[0074] Specifically, to acquire multi-view 2D images of the real-world environment, drones can be used to capture panoramic images of vast areas from an overhead perspective; mobile phone cameras can be used for close-up, detailed shooting. For the acquisition of multi-view images, algorithms can be used for image fusion and stitching. For example, combining image registration and stitching algorithms from computer vision can merge images from multiple perspectives into a unified scene view for subsequent processing and analysis. The resulting multi-view 2D images provide rich scene information, offering necessary input data for subsequent 3D reconstruction and scene construction.

[0075] S2. Preprocess the acquired two-dimensional image, such as denoising and color correction, to improve the quality of subsequent three-dimensional reconstruction. Use semantic segmentation technology to automatically identify and label different objects and regions in the image, understand the semantic information in the image, and thus preserve the category and functional characteristics of the object during the reconstruction process.

[0076] Furthermore, the acquired two-dimensional image undergoes preprocessing, specifically including:

[0077] S11. Remove noise from the two-dimensional image using bilateral filtering;

[0078] Specifically, it is expressed as follows:

[0079] ;

[0080] in, It represents the pixel value of the filtered image at point (x, y). The original image at points pixel values, It is a spatial Gaussian function. It is the spatial standard deviation, which controls the size of the filter. It is an intensity Gaussian function. It is the intensity standard deviation, which controls the similarity between pixel values. It is a normalization factor;

[0081] Specifically, bilateral filtering is a commonly used nonlinear filtering technique in image processing. It combines spatial Gaussian weights and intensity Gaussian weights to smooth the image while preserving edge information. By cleverly combining spatial and intensity Gaussian functions, it achieves noise removal while preserving image edges, resulting in a natural and clear image. The adjustable parameters of bilateral filtering allow the algorithm to adapt to different image characteristics and noise environments, exhibiting excellent robustness. The effect of bilateral filtering is to remove noise while maintaining the edge sharpness and details of the image, making the image more visually pleasing, especially suitable for scenarios where edge information needs to be preserved.

[0082] S12. Color correction is achieved by adjusting the gain of each channel through the grayscale world hypothesis algorithm, and then the color is adjusted according to the grayscale card in the scene or according to the information of the image itself through the white balance algorithm.

[0083] The specific algorithm for the gray-scale world hypothesis is expressed as follows: ;in It is the average brightness of the entire image. It is the average brightness of channel c;

[0084] The specific white balance algorithm is expressed as follows: Where I(c) is a channel of the original image, This is the value of that channel after white balance;

[0085] Specifically, color correction using the gray-world hypothesis algorithm and white balance algorithm can significantly improve the color consistency of an image under different lighting conditions, thereby enhancing the realism and visual quality of the image. The white balance algorithm further adjusts colors based on information from a grayscale card in the scene or the image itself to ensure that white objects in the image appear truly white. The advantage of using the gray-world hypothesis and white balance algorithm for color correction lies in their ability to automatically adjust image colors to adapt to different lighting conditions, thus ensuring color consistency and realism. By calculating and adjusting the RGB channel gain of the image to achieve color balance, white objects appear truly white in the image, improving the visual quality of the acquired image, enhancing the naturalness and accuracy of colors, and exhibiting broad applicability and strong robustness.

[0086] S13. Register the acquired two-dimensional image using feature matching algorithm and image registration algorithm.

[0087] Specifically, a feature detection algorithm is used to detect key points and descriptors, and then matching point pairs are found by matching descriptors; the transformation matrix is ​​used to align one image to another; the transformation matrix solution includes homography matrix and affine transformation.

[0088] The homography matrix is ​​represented as follows: ;where x and It is a matching point pair. It is a robust kernel function;

[0089] The affine transformation is expressed as: ;in and These are the coordinates of the matching point pairs.

[0090] Specifically, through feature point matching, accurate correspondences can be found between different images, making the registration results very precise. Different feature detection algorithms and transformation matrix solving methods, such as homography matrix and affine transformation, can be selected according to different application requirements to adapt to different scenarios and requirements.

[0091] Furthermore, the application of semantic segmentation technology to automatically identify and label different objects and regions in the image specifically includes:

[0092] S21. Extract features from the preprocessed image through convolutional and pooling layers, and perform pixel-level classification using a fully convolutional network;

[0093] S22. Use the softmax algorithm to classify each pixel, and use upsampling and deconvolution techniques to enlarge the feature map to its original size to generate a segmentation map;

[0094] S23. By using threshold processing and connection component analysis, pixel labels are aggregated into coherent object regions to identify and locate the boundaries of different objects.

[0095] Specifically, by extracting features through convolutional and pooling layers and then using a fully convolutional network for pixel-level classification, semantic segmentation technology can accurately identify objects and regions in an image. The softmax algorithm is used to classify each pixel, ensuring classification accuracy. The application of upsampling and deconvolution techniques allows the resolution of the segmentation result to match that of the original image, preserving the image's detailed information. Finally, through thresholding and connection component analysis, scattered pixel labels are aggregated into continuous regions, achieving clear identification and localization of object boundaries, thereby generating an accurate object segmentation map visually.

[0096] S3. Based on the labeled two-dimensional image data, a three-dimensional model is reconstructed from the two-dimensional image using a multi-viewpoint geometric three-dimensional reconstruction algorithm based on bundle adjustment, and the reconstructed three-dimensional model is stored as a sub-scene model.

[0097] Each sub-scene model contains its position, size, and material information;

[0098] Furthermore, the method of using a multi-viewpoint geometric 3D reconstruction algorithm based on bundle adjustment to reconstruct a 3D model from a 2D image and storing the reconstructed 3D model as a sub-scene model specifically includes:

[0099] S31. Match corresponding feature points between images from different viewpoints, determine the position of feature points in different images, and determine the camera's intrinsic and extrinsic parameters using known calibration board images;

[0100] S32. Using multi-view feature point matching and camera parameters, the three-dimensional coordinates of each feature point in space are determined by triangulation. The triangulation utilizes the different positions of the same feature point in different images when viewed from different perspectives, thereby calculating its actual position in three-dimensional space.

[0101] The formula for trigonometric measurements is expressed as a solution to a linear system. , where A is a matrix constructed based on the coordinates of the corresponding point and the camera projection matrix, and P is the homogeneous coordinates of the 3D point.

[0102] S33. Optimize the 3D coordinates and camera parameters by minimizing the reprojection error, and use the least squares method to update the parameter estimates. The reprojection error refers to the difference between the actual position of the 3D point after it is projected back into the 2D image and the actual feature point position.

[0103] The minimized reprojection error is expressed as: ;

[0104] in, It refers to the location of feature points in the image. It corresponds to the position of a three-dimensional point. These are camera parameters. Represents the projection function. It is a robust kernel function used to handle outliers;

[0105] S34. Store the optimized 3D point coordinates and image camera parameters as a sub-scene model.

[0106] Specifically, a multi-viewpoint geometric 3D reconstruction algorithm based on bundle adjustment can accurately reconstruct 3D models from 2D images and store these models as sub-scene models. First, feature point matching technology is used to determine corresponding points in images from different viewpoints. Then, a calibration board image is used to calibrate camera parameters, ensuring the accuracy of the reconstruction process. Next, triangulation methods, combined with multi-viewpoint feature point matching and accurate camera parameters, are used to calculate the position of each feature point in 3D space. Furthermore, an optimization process that minimizes reprojection errors further improves the accuracy of 3D point positions and camera parameters. The application of a robust kernel function effectively handles outliers, enhancing the algorithm's stability. Finally, the optimized 3D point coordinates and camera parameters are used to construct sub-scene models, providing high-quality 3D data for subsequent applications such as graphics rendering, virtual reality, and augmented reality. The advantages of this entire process lie in its high precision, high degree of automation, and ability to effectively handle complex scenes, providing users with a reliable 3D reconstruction solution.

[0107] Furthermore, the method of using least squares to update the parameter estimates specifically includes:

[0108] Where X is the design matrix and Y is the observation vector. It is a parameter vector.

[0109] S4. By specifying the required scene type and specific requirements, the system automatically selects the appropriate storage sub-scene model and performs a preliminary combination according to the automatically selected instructions to form the initial scene model.

[0110] Furthermore, the automatic selection of suitable storage sub-scene models and the preliminary combination according to the automatically selected instructions to form an initial scene model specifically includes:

[0111] By defining keywords and tags, the stored sub-scenario models are indexed. Based on the project requirements analysis results, intelligent retrieval is used to perform keyword matching and tag filtering to find sub-scenario models that match the requirements.

[0112] Using computer graphics technology, spatial analysis and calculations are performed on selected sub-scene models to determine their relative positions and orientations in the virtual scene. The models are then automatically arranged and combined using a constraint-based layout algorithm.

[0113] Specifically, the advantages of the automated sub-scene model selection and combination process lie in its efficiency, accuracy, and scalability. Through intelligent retrieval and computer graphics technology, manual operations can be significantly reduced, improving the speed and quality of scene construction. Meanwhile, the constraint-based layout algorithm ensures the rationality and diversity of model combinations, while the real-time preview and fine-tuning functions provide an intuitive interactive method, allowing users to easily adjust the scene to achieve the desired effect.

[0114] S5. Automatically make minor adjustments to the initial scene model to generate a target scene description file for the target scene model. The target scene description file includes information for rendering and displaying the scene.

[0115] Furthermore, during the automatic fine-tuning of the initial scene model, training was conducted by adding or removing objects, modifying materials, and adjusting lighting.

[0116] Specifically, computer graphics techniques are used to automatically add or remove objects in a scene;

[0117] In the Blinn-Phong lighting model, adjust parameters such as reflectivity to modify material parameters; adjust the material properties of the object's surface, such as color, texture, and roughness, to match the style of the target scene.

[0118] Using a lighting model, physically based rendering technology simulates real-world lighting effects and optimizes lighting conditions in a scene, including adding or adjusting the position, intensity, and type of light sources.

[0119] In the physical rendering technique described above, the lighting calculations using BRDF (Brown-Based Lighting Rendering) to represent the lighting model are specifically represented as follows: ;

[0120] in, It is the Fresnel coefficient, which represents the reflectivity of the material. It is the normal distribution function, which describes the effect of surface roughness on illumination. It is a geometric function that describes the degree to which light is blocked by a surface. Here, N is the surface roughness, L is the surface normal, and V is the direction vector from the surface point to the light source and from the surface point to the camera. It refers to the intensity of the light source.

[0121] Furthermore, after automatically making minor adjustments to the initial scene model to generate the target scene description file for the target scene model, the generated target scene model is optimized using generative adversarial networks and variational autoencoders. This makes the 3D model more realistic, diverse, and possesses richer details and expressiveness. Specific details include:

[0122] The loss function of variational autoencoder and the loss function of generative adversarial network are fused to obtain a joint optimization objective, which is used to obtain a more realistic and atmospheric 3D model. Specifically, the reconstruction capability of variational autoencoder and the realism enhancement capability of generative adversarial network are utilized.

[0123] The joint loss function is expressed as:

[0124] ;

[0125] in, and These are the weighting coefficients that balance the two loss terms. Represented as reconstruction loss, it is used to measure the similarity between the 3D point cloud reconstructed by the latent variable z and the original point cloud; Represented as KL divergence, it is used to measure the distribution of latent variables. With prior distribution The differences between them;

[0126] This is to encourage the discriminator to correctly identify real 3D models. This is to encourage the generator to produce realistic fake 3D models. Through the loss function of a generative adversarial network, the generator learns to produce 3D models with rich detail and high-quality surfaces.

[0127] Specifically, Generative Adversarial Networks (GANs) can learn the style and structure of target objects through adversarial mechanisms, thereby improving the visual expressiveness and realism of the generated 3D models. Combined with Variational Autoencoders (VAEs), the model can be modeled and optimized in the latent space, enabling model generation and reconstruction. This technique generates smoother and more continuous latent representations, improving the diversity and flexibility of the generated template scene model.

[0128] Furthermore, a scene building method based on 3D reconstruction also includes: S6, checking whether the generated target scene description file conforms to preset rules based on the generated target scene model target scene description file, and deploying the model that conforms to the rules to the virtual platform;

[0129] Furthermore, a scene building method based on 3D reconstruction also includes: significantly improving the speed of scene building through automated 3D reconstruction technology and intelligent scene combination strategies; and using real 2D materials for 3D reconstruction to make the generated scene more realistic.

[0130] This embodiment provides a scene construction method based on 3D reconstruction, aiming to address the challenges encountered by traditional technologies in real-time rendering of large-scale complex scenes. The method first acquires multi-view 2D images of the real-world environment using devices such as drones and mobile phone cameras. These images undergo preprocessing, including denoising and color correction, to improve the quality of subsequent 3D reconstruction. In the preprocessing step, bilateral filtering is used to smooth the images and preserve edge information, while the gray-world hypothesis and white balance algorithm are used to adjust image colors, ensuring color consistency under different lighting conditions. Next, feature matching and image registration algorithms are used to accurately register the images, ensuring correct alignment of multi-view images. With the help of semantic segmentation, objects and regions in the images are automatically identified and labeled, and the semantic information in the images is understood.

[0131] Subsequently, a multi-view geometric 3D reconstruction algorithm based on bundle adjustment is used to reconstruct 3D models from 2D images, and these models are stored as sub-scene models. During this process, 3D coordinates and camera parameters are optimized by minimizing reprojection errors to ensure the accuracy of the reconstructed 3D models.

[0132] In the sub-scene model selection and combination stage, intelligent retrieval technology automatically selects suitable sub-scene models based on project requirements. Computer graphics technology is then used for spatial analysis and calculation, automatically arranging and combining the models. Finally, fine-tuning of the initial scene model is performed, including adding or removing objects, modifying materials, and adjusting lighting, generating a target scene description file for the target scene model. To further enhance the model's realism and diversity, generative adversarial networks and variational autoencoders are combined to optimize the generated target scene model, resulting in a more realistic 3D model that is more detailed and expressive. After verification and checking to ensure the generated target scene description file conforms to preset rules, the model is deployed to a virtual platform, providing high-quality 3D scenes for virtual reality and augmented reality applications. This automated 3D reconstruction and scene building process not only improves efficiency but also ensures the realism and visual quality of the scene.

[0133] Example 2

[0134] This embodiment provides a scene building system based on 3D reconstruction, which is applied to a scene building process based on 3D reconstruction. It is used to form an automated 3D scene building system that can efficiently reconstruct realistic 3D scenes from real-world 2D images and provide high-quality 3D data for applications such as virtual reality and augmented reality.

[0135] A scene building system based on 3D reconstruction, specifically including:

[0136] The image acquisition module uses drones, mobile phone cameras, and image acquisition devices to acquire multi-view two-dimensional images of the real world environment, enabling both overhead shots of large areas and close-up detail shots, providing rich scene information for 3D reconstruction;

[0137] The image preprocessing module performs preprocessing operations such as denoising and color correction on the acquired two-dimensional images. Specifically, it includes bilateral filtering technology for noise removal and gray-world hypothesis and white balance algorithm for color correction.

[0138] The feature extraction and image registration module uses a feature detection algorithm to detect key points and descriptors, and uses a feature matching algorithm to find matching point pairs.

[0139] Among them, an image registration algorithm using homography matrix and affine transformation is used to accurately align multi-view images;

[0140] The semantic segmentation module uses deep learning technology with fully convolutional networks to perform pixel-level semantic segmentation, automatically identify and label different objects and regions in the image, and understand the semantic information in the image;

[0141] The 3D reconstruction and optimization module uses a multi-viewpoint geometric 3D reconstruction algorithm based on bundle adjustment to reconstruct a 3D model from a 2D image and store the reconstructed 3D model as a sub-scene model.

[0142] The scene combination and adjustment module automatically selects suitable sub-scene models through intelligent retrieval technology and performs preliminary combination to form an initial scene model; it is used to make fine adjustments to the initial scene model, including adding or removing objects, modifying materials and adjusting lighting, as well as generating target scene description files, and finally deploying the model that meets the rules to the virtual platform;

[0143] This also includes using generative adversarial networks and variational autoencoders to optimize the initial scene model in order to improve the model's realism and diversity.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for scene building based on three-dimensional reconstruction, characterized in that: Comprise the following steps: Collecting multi-view two-dimensional images of the environment through a drone or a mobile phone camera; Pretreating the obtained two-dimensional images and automatically identifying and labeling different objects and regions in the images using semantic segmentation technology to understand the semantic information in the images; According to the labeled two-dimensional image data, a multi-view geometric three-dimensional reconstruction algorithm based on bundle adjustment is used to reconstruct a three-dimensional model from the two-dimensional images and store the reconstructed three-dimensional model as a sub-scene model; The multi-view geometric three-dimensional reconstruction algorithm based on bundle adjustment is used to reconstruct a three-dimensional model from the two-dimensional images and store the reconstructed three-dimensional model as a sub-scene model, specifically comprising: Matching corresponding feature points between images at different angles, determining the positions of the feature points in different images, and determining the internal and external parameters of the camera through the known calibration board image; Using multi-view feature point matching and camera parameters, the three-dimensional coordinates of each feature point in space are determined by triangulation method; Optimizing the three-dimensional coordinates and camera parameters by minimizing the re-projection error, which is the difference between the three-dimensional point projected back to the two-dimensional image and the actual feature point position, and updating the parameter estimates using the least squares method; The minimization of the re-projection error is expressed as: ； wherein, is a feature point position in the image, is a corresponding three-dimensional point position, is a camera parameter, denotes a projection function, is a robust kernel function for handling outliers; According to the optimized three-dimensional point coordinates and image camera parameters, the reconstructed three-dimensional model is stored as a sub-scene model; Pretreating the obtained two-dimensional images, specifically including: Removing noise from the two-dimensional images through bilateral filtering; Adjusting the gain of each channel for color correction through the gray world assumption algorithm, and then adjusting the color according to the gray card in the scene or according to the information of the image itself through the white balance algorithm; Registering the obtained two-dimensional images through feature matching algorithm and image registration algorithm, specifically including: using feature detection algorithm to detect key points and descriptors, and then finding matching point pairs by matching descriptors; solving the transformation matrix to align one image to another image; the transformation matrix solving includes homography matrix and affine transformation.

2. The scene building method based on three-dimensional reconstruction according to claim 1, characterized in that: The semantic segmentation technology is used to automatically identify and label different objects and regions in the images, specifically including: Extracting features from the pretreated images through convolutional layers and pooling layers, and performing pixel-level classification using a fully convolutional network; Using a softmax algorithm to classify each pixel, and using upsampling and deconvolution techniques to enlarge the feature map to the original size to generate a segmentation map; Through threshold processing and connected component analysis, pixel labels are aggregated into coherent object regions, and the boundaries of different objects are identified and located.

3. The scene building method based on three-dimensional reconstruction according to claim 1, characterized in that: The least squares method is used to update the parameter estimates, specifically including: ; where X is a design matrix, Y is a vector of observations, is a parameter vector.

4. The scene building method based on three-dimensional reconstruction according to claim 1, characterized in that: Further comprising: By specifying the required scene type and specific requirements, automatically selecting appropriate storage sub-scene models, and preliminarily combining according to the automatically selected instructions to form an initial scene model; The sub-scene model is indexed by defining keywords and tags, keyword matching and tag screening are performed by intelligent retrieval according to the project demand analysis result, and the sub-scene model conforming to the demand is found out; and the spatial analysis and calculation of the selected sub-scene model are performed by using computer graphics technology, the relative position and orientation of the sub-scene model in the virtual scene are determined, and the arrangement and combination of the model are automatically performed by using the constraint-based layout algorithm.

5. The scene building method based on three-dimensional reconstruction according to claim 4, characterized in that: Further comprising: The initial scene model is automatically fine-tuned to generate a target scene description file of the target scene model, and the target scene description file includes information for rendering and displaying the scene.

6. The scene building method based on three-dimensional reconstruction according to claim 1, characterized in that: Further comprising: According to the target scene description file of the generated target scene model, it is checked whether the generated target scene description file conforms to the preset rules, and the model conforming to the rules is deployed to the virtual platform.

7. A scene building system based on three-dimensional reconstruction, applied to the scene building method based on three-dimensional reconstruction in any one of claims 1-6, characterized in that: The image acquisition module, the image preprocessing module, the feature extraction and image registration module, the semantic segmentation module, the three-dimensional reconstruction and optimization module, and the scene combination and adjustment module are included. The image acquisition module acquires multi-view two-dimensional images of the environment by using a drone and a mobile phone camera. The image preprocessing module performs preprocessing operations on the acquired two-dimensional images, specifically including noise removal by using a bilateral filtering technique and color correction by using a gray world assumption and a white balance algorithm. The feature extraction and image registration module detects key points and descriptors by using a feature detection algorithm, and finds matching point pairs by using a feature matching algorithm. The multi-view images are accurately aligned by using a homography matrix and an affine transformation image registration algorithm. The semantic segmentation module performs pixel-level semantic segmentation by using a deep learning technology of a fully convolutional network, automatically identifies and labels different objects and regions in the image, and understands semantic information in the image. The three-dimensional reconstruction and optimization module reconstructs a three-dimensional model from the two-dimensional images based on a multi-view geometric three-dimensional reconstruction algorithm of a bundle adjustment method, and stores the reconstructed three-dimensional model as a sub-scene model. The scene combination and adjustment module automatically selects appropriate sub-scene models by using an intelligent retrieval technology, and performs preliminary combination to form an initial scene model, fine-tunes the initial scene model, including adding or removing objects, modifying materials, and adjusting lighting, generates a target scene description file, and deploys the model conforming to the rules to a virtual platform. The initial scene model is further optimized by using a generative adversarial network and a variational autoencoder.

Images