Building indoor and outdoor integrated three-dimensional model construction method, system and equipment

Abstract

The invention relates to the technical field of three-dimensional modeling, and discloses a building indoor and outdoor integrated three-dimensional model construction method, system and equipment. The method comprises the steps of collecting video data in real time based on cameras inside and outside a building; constructing a federated learning framework; converting two-dimensional image information of the video data into radiation field representation in a three-dimensional space through a training neural network by adopting a NeRF technology based on a federated learning framework, and obtaining a trained 3D field model; and performing 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an indoor and outdoor integrated three-dimensional model of the building. According to the invention, two-dimensional image information is converted into radiation field representation in a three-dimensional space through training a neural network by utilizing the advantages of the NeRF technology, and a highly-vivid 3D field model is constructed. And model parameter sharing and data privacy protection among the sub-computing nodes are realized in combination with a federated learning framework. And obtaining an integrated three-dimensional model through 3D Gaussian function transformation and data fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a method, system and equipment for constructing an integrated indoor and outdoor three-dimensional model of a building. Background Art

[0002] With the rapid development of 3D modeling technology, the integrated indoor and outdoor 3D model of buildings has shown unprecedented and extensive application prospects in the fields of architectural design, navigation, virtual reality, etc. However, traditional 3D modeling methods face many challenges and limitations in achieving this goal.

[0003] Traditional 3D modeling methods often rely on a large amount of on-site measurement data and manual modeling processes. This not only requires a lot of manpower, material resources and time, but is also inefficient and costly. During the on-site measurement process, professionals need to use various measurement tools to accurately measure the building point by point and face by face, which is not only time-consuming and labor-intensive, but also easily affected by environmental, weather and other factors, resulting in inaccurate or missing measurement data. The manual modeling process requires technicians to gradually build a 3D model of the building based on the measurement data through professional modeling software, which not only requires superb modeling skills, but also makes it difficult to ensure the accuracy and completeness of the model.

[0004] In addition, traditional 3D modeling methods are even more inadequate for indoor scenes involving privacy protection. Since a large amount of on-site measurement data needs to be collected and processed, these data may contain sensitive information or personal privacy. Once the data is leaked or abused, it may cause serious losses and risks to the relevant parties. Therefore, how to quickly build an integrated indoor and outdoor 3D model of a building while ensuring data security and privacy has become an urgent problem to be solved. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, system and equipment for constructing an integrated indoor and outdoor three-dimensional model of a building, aiming to solve at least one of the above-mentioned technical problems.

[0006] To achieve the above object, the present invention provides a method for constructing an integrated indoor and outdoor three-dimensional model of a building, comprising:

[0007] Real-time video data collection based on multiple cameras inside and outside the building;

[0008] Build a federated learning framework;

[0009] Based on the federated learning framework, NeRF technology is used to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network to obtain a trained 3D field model;

[0010] The trained 3D field model is subjected to 3D Gaussian function transformation and data fusion processing to obtain a three-dimensional model integrating indoor and outdoor aspects of the building.

[0011] In some embodiments, constructing a federated learning framework includes:

[0012] Setting up a main service center; wherein the main service center is used for model parameter aggregation and distribution;

[0013] Sub-computing nodes are set; wherein each of the sub-computing nodes independently performs training of the NeRf model, and the sub-computing nodes transmit model parameters through a secure communication protocol.

[0014] In some embodiments, the sub-computing nodes include a first sub-computing node and a second sub-computing node; wherein a plurality of outdoor cameras are set as the first sub-computing nodes, and a plurality of indoor cameras are set as the second sub-computing nodes.

[0015] In some embodiments, the method of converting the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network based on the federated learning framework using NeRF technology to obtain a trained 3D field model includes:

[0016] On each sub-computing node of the federated learning framework, a NeRf model is constructed using a deep learning framework;

[0017] The two-dimensional image information of the video data is converted into a radiation field representation in a three-dimensional space according to the NeRf model to obtain a trained 3D field model.

[0018] In some embodiments, the method further comprises:

[0019] During the training process of the NeRf model, the model training is performed using the optimization strategies of distributed training, gradient clipping, and learning rate scheduling;

[0020] Use pre-trained models for initialization and combine transfer learning technology to transfer the learned knowledge to each computing node;

[0021] During the training process of the NeRf model, data enhancement technology is used and regularization terms are introduced to increase the generalization ability of the NeRf model and prevent overfitting.

[0022] In some embodiments, the model architecture of the NeRf model includes: a multi-layer fully connected neural network, a position encoding module, and a volume rendering module.

[0023] In some embodiments, the 3D Gaussian function transformation and data fusion processing are performed on the trained 3D field model to obtain an integrated indoor and outdoor 3D model of the building, including:

[0024] The trained 3D field model is transformed into a 3D Gaussian point cloud model with spatial coordinate positions through a 3D Gaussian function transformation; wherein the 3D Gaussian point cloud model includes a 3D Gaussian point cloud model trained by an outdoor camera and a 3D Gaussian point cloud model trained by an indoor camera;

[0025] Based on the service main center of the federated learning framework, the 3D Gaussian point cloud model trained by the outdoor camera and the 3D Gaussian point cloud model trained by the indoor camera are fused according to the spatial position information to obtain a fused three-dimensional model;

[0026] The fused three-dimensional model is imported into a visualization platform to obtain an integrated three-dimensional model of the indoor and outdoor of the building.

[0027] In some embodiments, the step of converting the trained 3D field model into a 3D Gaussian point cloud model having a spatial coordinate position through a 3D Gaussian function transformation includes:

[0028] Calculating the spatial coordinates and radiation intensity of each point according to the radiation field representation of the trained 3D field model;

[0029] Based on the 3D Gaussian function transformation, each point is fitted with a 3D Gaussian function to obtain a 3D Gaussian point cloud model.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also proposes a building indoor and outdoor integrated three-dimensional model construction system, comprising:

[0031] A data acquisition module, used to collect video data in real time based on multiple cameras inside and outside the building;

[0032] Framework building module, used to build a federated learning framework;

[0033] A training conversion module, used to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network based on the federated learning framework using NeRF technology to obtain a trained 3D field model;

[0034] The transformation and fusion module is used to perform 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated three-dimensional model of the building's indoor and outdoor areas.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also proposes an electronic device, which includes: a memory, a processor, and a program for building an integrated indoor and outdoor three-dimensional model of a building stored in the memory and run on the processor, and the program for building an integrated indoor and outdoor three-dimensional model of a building is configured to implement the method for building an integrated indoor and outdoor three-dimensional model of a building as described above.

[0036] The present invention provides a method for constructing an indoor and outdoor integrated three-dimensional model of a building, comprising: collecting video data in real time based on multiple cameras inside and outside the building; constructing a federated learning framework; using NeRF technology based on the federated learning framework to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network to obtain a trained 3D field model; performing 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an indoor and outdoor integrated three-dimensional model of the building. In the present invention, NeRF technology is applied to the construction of an indoor and outdoor integrated three-dimensional model of a building, and the advantages of NeRF technology are fully utilized to convert the two-dimensional image information into a radiation field representation in a three-dimensional space by training a neural network to construct a highly realistic trained 3D field model. At the same time, combined with the federated learning framework, the sharing of model parameters between each sub-computing node and the protection of data privacy are realized. Through 3D Gaussian function transformation and data fusion processing, an indoor and outdoor integrated three-dimensional model of a building is obtained, providing a more convenient and efficient three-dimensional modeling solution for architectural design, navigation, virtual reality and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of the structure of an electronic device in a hardware operating environment involved in an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for constructing an indoor and outdoor integrated three-dimensional model of a building according to the present invention;

[0039] Figure 3 It is a technical flow chart involved in the embodiment of the present invention;

[0040] Figure 4 This is a structural block diagram of an embodiment of a system for constructing an indoor and outdoor integrated three-dimensional model of a building according to the present invention.

[0041] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0044] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] Reference Figure 1 , Figure 1 The figure is a schematic diagram of the structure of an electronic device of the hardware operating environment involved in the embodiment of the present invention.

[0046] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM memory) or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0048] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a program for constructing an integrated indoor and outdoor three-dimensional model of a building.

[0049] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device, and the electronic device calls the indoor and outdoor integrated three-dimensional model construction program of the building stored in the memory 1005 through the processor 1001, and executes the indoor and outdoor integrated three-dimensional model construction method of the building provided in an embodiment of the present invention.

[0050] In recent years, with the rise of Neural Radiance Fields (NeRF) technology, new breakthroughs have been made in the field of 3D modeling. NeRF is a 3D reconstruction technology based on deep learning. It can transform 2D image information into a radiation field representation in 3D space by training a neural network, thereby constructing a highly realistic 3D model. Compared with traditional 3D modeling methods, NeRF technology has higher modeling efficiency and accuracy, and can handle complex scenes and lighting conditions to generate more realistic and delicate 3D models.

[0051] However, applying NeRF technology to the construction of integrated indoor and outdoor 3D models of buildings also faces some challenges. For example, how to efficiently collect and process a large amount of indoor and outdoor video data, how to use distributed machine learning methods such as federated learning to protect data privacy, and how to combine NeRF models with technologies such as 3D Gaussian function transformation to achieve efficient fusion and visualization of 3D models.

[0052] In view of this, the present invention proposes a method, system and device for constructing an integrated indoor and outdoor three-dimensional model of a building.

[0053] The embodiment of the present invention provides a method for constructing an integrated indoor and outdoor three-dimensional model of a building, referring to Figure 2 , Figure 2 The present invention is a flow chart of an embodiment of a method for constructing an integrated indoor and outdoor three-dimensional model of a building.

[0054] like Figure 2 As shown, the method for constructing an integrated indoor and outdoor three-dimensional model of a building includes:

[0055] Step S100: collecting video data in real time based on multiple cameras inside and outside the building;

[0056] Step S200: constructing a federated learning framework;

[0057] Step S300: Based on the federated learning framework, NeRF technology is used to train a neural network to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space to obtain a trained 3D field model;

[0058] Step S400: performing 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated three-dimensional model of the building indoors and outdoors.

[0059] It should be noted that the execution subject in this embodiment may be an electronic device, which may be a computer device with data processing functions, or other devices that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment, a computer device is used as an example for explanation.

[0060] It is understandable that this embodiment proposes a method for quickly constructing an integrated three-dimensional model of indoor and outdoor buildings based on federated learning, which fully utilizes the advantages of NeRF technology, and converts two-dimensional image information into a radiation field representation in three-dimensional space by training a neural network to construct a highly realistic 3D field model. At the same time, combined with the federated learning framework, the sharing of model parameters between each sub-computing node and the protection of data privacy are realized. Finally, through 3D Gaussian function transformation and data fusion processing, a complete three-dimensional model of the building's indoor and outdoor integration is obtained, providing a more convenient and efficient three-dimensional modeling solution for architectural design, navigation, virtual reality and other fields. The following is an explanation in conjunction with specific steps.

[0061] In one embodiment, video data is collected in real time based on multiple cameras inside and outside the building.

[0062] Specifically, Figure 3 As shown in the figure, indoor and outdoor camera data collection. Camera placement strategy: Carefully arrange multiple cameras inside and outside the building to ensure that they can fully cover key areas, such as every corner of the building, entrances, exits, and important indoor facilities. In order to obtain high-quality video data, the camera should use a high-resolution, wide-angle lens and be equipped with the function of automatically adjusting the focus and aperture to adapt to different lighting and environmental conditions.

[0063] Specifically, data collection and management: This embodiment designs an efficient data collection system to ensure that video data can be transmitted to each sub-computing node in real time and stably. At the same time, a complete data management mechanism is established to classify, store and back up video data for subsequent processing and analysis.

[0064] For example, camera location planning: Conduct a detailed analysis of the layout of the building to determine the key areas that need to be monitored, including but not limited to every corner of the building, main entrances and exits, parking lots, stairwells, elevator entrances, etc. Consider the structure and environmental factors of the building and choose a suitable camera installation location to avoid blind spots and dead angles. Camera technical specification selection: Choose a high-resolution camera, at least 1080p or higher, to ensure that the acquired video data is clear and easy to analyze and process later. Use a wide-angle lens to expand the monitoring range and reduce the number of cameras required, while ensuring that key areas are fully covered. The camera should have the function of automatically adjusting the focus and aperture to maintain the clarity of the video under different lighting conditions. In addition, in outdoor areas, the camera should be waterproof and dustproof, and be able to adapt to various adverse weather conditions, such as rain, high temperature, low temperature, etc. In areas with large changes in light, such as entrances and exits, the camera should have the ability to automatically switch the aperture to adapt to changes in the environment from bright to dim. In areas with insufficient light, the camera should be equipped with infrared lights or low-light performance sensors to ensure that clear images can still be obtained at night or in low light.

[0065] Exemplarily, data acquisition: deploy a dedicated data acquisition system responsible for receiving video stream data from each camera in real time. The data acquisition system needs to have high bandwidth and network traffic management capabilities to support the transmission of large amounts of data. Configure a suitable encoder for each camera to convert the original video data into a format suitable for network transmission, such as H.264 or H.265 encoding, to reduce the data size and maintain video quality. Design redundant data transmission paths to ensure that when the main path fails, the data can be transmitted through the backup path to ensure the real-time and stability of the data. Efficient network transmission protocols, such as QUIC or TCP, can be used to reduce delays and packet loss in data transmission, and firewall rules and VPN tunnels can be configured for data transmission to ensure the security of data during transmission.

[0066] Exemplarily, data management mechanism: Classify and manage video data according to its content and importance, such as storing video data in sensitive areas separately from video data in other areas. Establish an efficient data storage system and use distributed storage solutions such as SAN or NAS to support the storage needs of large-scale video data. Implement data backup strategies, including local backup and remote backup, to ensure that data can be restored in the event of loss or damage. Build an efficient, secure and easy-to-manage video data acquisition and management system to provide a solid foundation for subsequent data processing and analysis.

[0067] In one embodiment, a federated learning framework is constructed, including: setting up a service main center; wherein the service main center is used for model parameter aggregation and distribution; setting sub-computing nodes; wherein each of the sub-computing nodes independently performs NeRf model training, and the model parameters are transmitted between the sub-computing nodes through a secure communication protocol.

[0068] In one embodiment, the sub-computing nodes include a first sub-computing node and a second sub-computing node; wherein a plurality of outdoor cameras are set as the first sub-computing nodes, and a plurality of indoor cameras are set as the second sub-computing nodes.

[0069] Specifically, Figure 3 As shown in the figure, a federated learning framework is constructed. Design of the service main center: A highly reliable service main center is established as the core for model parameter aggregation and distribution. The service main center should have strong computing and storage capabilities, be able to process model parameters from various computing nodes, and perform effective integration and updates.

[0070] Specifically, the sub-computing nodes are set as follows: multiple outdoor cameras are set as sub-computing nodes A (first sub-computing nodes), and multiple indoor cameras are set as sub-computing nodes B (second sub-computing nodes). Each sub-computing node is equipped with sufficient computing resources to independently train the NeRf model. At the same time, the model parameters are transmitted between the sub-computing nodes through a secure communication protocol to ensure the security and privacy of the data.

[0071] Specifically, the data confidentiality mechanism: each sub-computing node only shares the model parameters and does not upload the original video data. In this embodiment, in order to further enhance data confidentiality, advanced technologies such as differential privacy and dynamic encryption can be used to encrypt the model parameters to prevent data leakage.

[0072] For example, by analyzing the computing and storage requirements of the entire system, determine the hardware specifications required for the service center, configure the server network, ensure high-bandwidth and low-latency data transmission, select the appropriate server operating system and containerization technology, and design the software architecture of the service center, including database, message queue, and model training / fusion modules. According to the layout of the cameras, determine the location and number of sub-computing nodes, ensure that each node can independently access the required data resources, and configure appropriate computing resources for each sub-computing node.

[0073] For example, data confidentiality: configure secure communication protocols between nodes, use SSL / TLS to encrypt communications, and implement authentication mechanisms to ensure that only authorized nodes can communicate. Before nodes process video data, implement data desensitization strategies, such as pixelating or blurring sensitive information. Design a sharing mechanism for model parameters to ensure that only model parameters are transmitted in the network and that original video data is not involved. Encrypt shared model parameters using symmetric or asymmetric encryption algorithms, manage encryption keys, and ensure that only the service center and distributed computing nodes can decrypt. Introduce a differential privacy mechanism during model training to add a certain degree of noise to protect personal privacy.

[0074] In this embodiment, by establishing an efficient and secure federated learning framework, data privacy is protected and distributed computing resources can be fully utilized for model training and fusion. The main advantages of using a federated learning framework include: since there is no need to share original data, personal privacy and sensitive information can be better protected. Compared with transmitting large amounts of data to a central server, transmitting only model parameters can significantly reduce communication costs. The federated learning framework can be easily expanded to a large number of nodes, so that model training can be dispersed to various geographical locations, which is scalable. By aggregating information from multiple nodes, the federated learning framework can improve the robustness and generalization ability of the model.

[0075] In one embodiment, based on the federated learning framework, NeRF technology is used to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network to obtain a trained 3D field model, including: on each sub-computing node of the federated learning framework, a deep learning framework is used to construct a NeRf model; according to the NeRf model, the two-dimensional image information of the video data is converted into a radiation field representation in a three-dimensional space to obtain a trained 3D field model.

[0076] In one embodiment, the method further includes: during the training process of the NeRf model, using distributed training, gradient clipping, and learning rate scheduling optimization strategies to perform model training; using a pre-trained model for initialization, and combining transfer learning technology to migrate the learned knowledge to each sub-computing node; during the training process of the NeRf model, using data enhancement technology and introducing regularization terms to increase the generalization ability of the NeRf model and prevent overfitting.

[0077] In one embodiment, the model architecture of the NeRf model includes: a multi-layer fully connected neural network, a position encoding module, and a volume rendering module.

[0078] Specifically, Figure 3As shown in the figure, NeRf model training. Model architecture design: On each computing node, an advanced deep learning framework (such as TensorFlow, PyTorch, etc.) is used to build the NeRf model. The NeRf model architecture should include: a multi-layer fully connected neural network, a position encoding module, and a volume rendering module, etc., to ensure that the model can accurately convert two-dimensional image information into a radiation field representation in three-dimensional space.

[0079] Specifically, training strategy optimization: In order to solve the problems of large amount of computation and slow convergence speed in the training process of NeRf model, this embodiment adopts optimization strategies such as distributed training, gradient clipping, and learning rate scheduling to improve the training efficiency and accuracy of the model. In order to improve the training efficiency and accuracy of the NeRf model, a pre-trained model is used for initialization, and combined with transfer learning technology, the learned knowledge is transferred to each sub-scenario computing node. During the training process, data enhancement techniques such as random cropping, rotation, flipping, etc. are used to increase the generalization ability of the model. At the same time, regularization terms such as L2 regularization are introduced to prevent overfitting of the model.

[0080] For example, deep learning frameworks such as TensorFlow and PyTorch are installed and configured on each computing node. Build the NeRf model architecture: design a multi-layer fully connected neural network, determine the number of layers, the number of neurons in each layer, and the activation function; implement a position encoding module to encode the position information of the input image into a learnable feature vector; develop a volume rendering module, which is responsible for combining the encoded features with the light path to generate the final color and density prediction; integrate each module into the NeRf model to ensure that the model can work properly.

[0081] Exemplarily, training strategy optimization: configure a distributed training environment to ensure that each computing node can process training tasks in parallel, use synchronous or asynchronous training strategies, and balance the training load according to the computing power of the node. Implement gradient clipping technology to prevent gradient explosion, ensure the stability of the model, and set an appropriate gradient clipping threshold to avoid excessive gradient clipping. Design a learning rate scheduling strategy, such as learning rate decay or periodic adjustment, to improve the convergence speed and accuracy of the model, monitor the loss curve during training, and adjust the learning rate strategy as needed.

[0082] Exemplarily, pre-training and transfer learning: Select a pre-training model related to the NeRf model, such as a model that has been trained on a large-scale image dataset. Use the weights of the pre-training model to initialize some layers of the NeRf model. Adjust the structure or parameters of the pre-training model according to the characteristics of the current task.

[0083] For example, data enhancement and regularization: use data enhancement techniques, including but not limited to random cropping, rotation, flipping, etc., to simulate different data perspectives and conditions; automatically apply these techniques during training to increase the generalization ability of the model. Regularization: introduce L2 regularization terms in the loss function to prevent model overfitting; adjust regularization parameters to balance model complexity and generalization ability.

[0084] In this embodiment, by combining federated learning and NeRf technology, efficient three-dimensional model construction is achieved, which greatly shortens the modeling time.

[0085] In one embodiment, the trained 3D field model is subjected to 3D Gaussian function transformation and data fusion processing to obtain an integrated indoor and outdoor three-dimensional model of the building, including: converting the trained 3D field model into a 3D Gaussian point cloud model with a spatial coordinate position through a 3D Gaussian function transformation; wherein the 3D Gaussian point cloud model includes a 3D Gaussian point cloud model trained by an outdoor camera and a 3D Gaussian point cloud model trained by an indoor camera; based on the service main center of the federated learning framework, the 3D Gaussian point cloud model trained by the outdoor camera and the 3D Gaussian point cloud model trained by the indoor camera are fused according to spatial position information to obtain a fused three-dimensional model; the fused three-dimensional model is imported into a visualization platform to obtain an integrated indoor and outdoor three-dimensional model of the building.

[0086] In one embodiment, the trained 3D field model is converted into a 3D Gaussian point cloud model with a spatial coordinate position through a 3D Gaussian function transformation, including: calculating the spatial coordinates and radiation intensity of each point based on the radiation field representation of the trained 3D field model; fitting each point with a 3D Gaussian function based on the 3D Gaussian function transformation to obtain a 3D Gaussian point cloud model.

[0087] Specifically, Figure 3 As shown, 3D Gaussian function transformation. It should be noted that the transformation principle is: the trained 3D field model is transformed into a 3D Gaussian point cloud model with spatial coordinate positions through a 3D Gaussian function transformation. 3D Gaussian function transformation is an effective three-dimensional data representation method that can discretize the continuous three-dimensional space into a series of Gaussian distributed point clouds, thereby more intuitively representing the three-dimensional structure of the building.

[0088] Specifically, the transformation is implemented as follows: During the transformation process, the spatial coordinates and radiation intensity of each point are first calculated based on the radiation field representation of the 3D field model. Then, these points are fitted using a 3D Gaussian function to obtain a 3D Gaussian point cloud model. By adjusting the parameters of the Gaussian function (such as mean, variance, etc.), the distribution and density of the point cloud can be optimized and the representation accuracy of the model can be improved.

[0089] Specifically, transformation strategy optimization: During the 3D Gaussian function transformation process, the parameters of the Gaussian function, such as mean and variance, can be adjusted according to actual needs to optimize the distribution and density of the point cloud model. In order to improve the quality of the point cloud model, post-processing techniques such as filtering and denoising are used to remove noise and abnormal points in the point cloud.

[0090] Exemplarily, the 3D field model is obtained by training the NeRf model, which can represent the radiation intensity at each position in the three-dimensional space. The continuous three-dimensional space is discretized into a point cloud using a 3D Gaussian function transformation, and each point represents the center of a Gaussian distribution. For each point in the 3D field model, its coordinates in the three-dimensional space and the corresponding radiation intensity are calculated, and these coordinates and intensity values ​​can be directly obtained through the radiation field representation of the model. Define a 3D Gaussian function, fit each point using the Gaussian function, and match the radiation intensity of the point by optimizing the parameters of the Gaussian function. Use an optimization algorithm (such as gradient descent or Newton's method) to adjust the parameters of the Gaussian function to minimize the fitting error.

[0091] Exemplarily, transformation strategy optimization: adjust the Gaussian function parameters according to the distribution and density requirements of the point cloud model, adjust the mean and variance of the Gaussian function, the mean can be used to control the position of the point cloud, and the variance can be used to control the diffusion degree of the point cloud. Use filtering techniques such as Gaussian filtering or bilateral filtering to smooth the point cloud and remove noise. Use denoising algorithms such as statistical filtering or clustering algorithms to remove abnormal points in the point cloud. Post-process the point cloud model, such as removing points that are too far away, merging points that are close in distance, etc., to improve the accuracy of the model.

[0092] In this embodiment, the 3D Gaussian function transformation can convert the 3D field model into a 3D Gaussian point cloud model with a spatial position, thereby providing an intuitive representation of the three-dimensional structure of the building.

[0093] Specifically, Figure 3 As shown in the figure, data fusion processing. Fusion strategy: Through the service main center, the 3D Gaussian point cloud model trained by the outdoor camera and the 3D Gaussian point cloud model trained by the indoor camera are fused according to the spatial position information. During the fusion process, precise spatial alignment algorithm and point cloud registration technology are used to ensure seamless splicing and consistency of indoor and outdoor models.

[0094] Specifically, post-fusion model optimization: the fused model is further optimized, such as denoising, smoothing, detail enhancement, etc., to process the overlapping areas and eliminate splicing traces to improve the overall quality and visualization effect of the model.

[0095] Exemplarily, ensure that the 3D Gaussian point cloud model obtained by training the outdoor camera and the indoor camera is prepared and stored in the service main center. The 3D Gaussian point cloud model should contain spatial coordinate information and radiation intensity for subsequent spatial alignment and registration. Select a suitable spatial alignment algorithm such as an iterative closest point (ICP) algorithm, a feature-based registration algorithm, or a deep learning method. Align the outdoor and indoor 3D Gaussian point cloud models according to the selected spatial alignment algorithm. Calculate the transformation matrix to transform one point cloud model into the coordinate system of another point cloud model. According to the transformation matrix, adjust the spatial position of the outdoor point cloud model so that it is aligned with the indoor point cloud model. Fuse the adjusted outdoor point cloud with the indoor point cloud to form a complete 3D model (fused three-dimensional model).

[0096] Exemplarily, the post-fusion model optimization: adopt a denoising algorithm, such as statistical filtering, density-based filtering or deep learning denoising method, to remove noise points in the fused model. Use a smoothing algorithm, such as Gaussian smoothing, bilateral filtering or surface reconstruction method, to reduce the unevenness and discontinuity of the model surface. Identify the overlapping areas in the fused model and use appropriate algorithms to eliminate the splicing marks, such as using weighted average or least squares method to smooth the point cloud of the overlapping area. Enhance the details of the model, especially for the fine structures in the model, point cloud refinement technology can be used to increase the density and details of the point cloud.

[0097] In this embodiment, accurate fusion and optimization of indoor and outdoor 3D Gaussian point cloud models can be achieved, thereby obtaining a high-quality, seamlessly spliced ​​three-dimensional model for further analysis and visualization.

[0098] Specifically, Figure 3 As shown, the 3D model is visualized. Visualization platform selection: Export the fused 3D model and import it into advanced visualization platforms such as Three.js or UE (Unreal Engine). These platforms provide rich 3D rendering and interactive functions, which can achieve high-fidelity visualization of 3D models.

[0099] Specifically, the interactive and analytical functions: in the visualization platform, interactive controls and analytical tools are added to enable users to easily rotate, zoom, and translate the 3D model, as well as perform analytical tasks such as distance measurement and area calculation. These functions will greatly improve users' understanding and application of 3D models.

[0100] Exemplarily, the fused 3D model is exported from the main service center to a common 3D file format, and a suitable visualization platform is selected according to the requirements. Three.js is suitable for Web-side display, while Unreal Engine is suitable for more complex interactions and high-quality rendering. The exported 3D model file is imported into the selected visualization platform. According to the visualization requirements, the size, position and rotation of the model are adjusted to ensure that the model is correctly displayed in the scene. Add materials and textures to the model to enhance its realism. This can include properties such as color, glossiness, transparency, etc. Set the lighting of the scene, including ambient light, directional light and point light, and enable shadow rendering to improve the visual effect of the model. Add interactive controls to the visualization platform to allow users to rotate, zoom and pan the model by mouse or touch screen. Integrate distance measurement and area calculation tools to allow users to select points or faces on the model and calculate the distance or area between them. The visualization effect of the model can be optimized by rendering technology, such as using lighting models, texture mapping and shadow effects to enhance the realism of the model.

[0101] The method described in this embodiment has the following advantages: Improve efficiency: Through federated learning and NeRf technology, efficient three-dimensional model construction is achieved, greatly shortening the modeling time. Protect privacy: The federated learning framework ensures that data is processed locally and only shares model parameters, effectively protecting data privacy. Integrated modeling: Combining indoor and outdoor data, the construction of an integrated indoor and outdoor three-dimensional model of the building is achieved, improving the integrity and practicality of the model. Visual display: Supports importing models into mainstream three-dimensional engines, realizing the visual display and interaction of models, and broadening the application scenarios.

[0102] It should be noted that the combination of federated learning and NeRf technology: This embodiment combines the federated learning framework with NeRf technology to achieve efficient use of indoor and outdoor camera data for three-dimensional reconstruction of buildings under the premise of protecting data privacy. This combination not only improves the efficiency and accuracy of three-dimensional modeling, but also expands the application scope of federated learning in the field of three-dimensional vision. Introduction of 3D Gaussian function transformation: This embodiment proposes to convert the 3D field model into a 3D Gaussian point cloud model through a 3D Gaussian function transformation, which provides a new idea for the representation of the three-dimensional structure of the building, which not only simplifies the processing flow of three-dimensional data, but also improves the representation accuracy and visualization effect of the model.

[0103] The present embodiment provides a method for constructing an integrated indoor and outdoor three-dimensional model of a building, including: collecting video data in real time based on multiple cameras inside and outside the building; constructing a federated learning framework; using NeRF technology based on the federated learning framework to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network to obtain a trained 3D field model; performing 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated indoor and outdoor three-dimensional model of the building. In this embodiment, NeRF technology is applied to the construction of an integrated indoor and outdoor three-dimensional model of a building, and the advantages of NeRF technology are fully utilized to convert the two-dimensional image information into a radiation field representation in a three-dimensional space by training a neural network to construct a highly realistic trained 3D field model. At the same time, combined with the federated learning framework, the sharing of model parameters and the protection of data privacy between each sub-computing node are realized. Through 3D Gaussian function transformation and data fusion processing, an integrated indoor and outdoor three-dimensional model of a building is obtained, providing a more convenient and efficient three-dimensional modeling solution for architectural design, navigation, virtual reality and other fields.

[0104] In addition, an embodiment of the present invention also proposes a storage medium, on which is stored a program for constructing an integrated indoor and outdoor three-dimensional model of a building. When the program for constructing an integrated indoor and outdoor three-dimensional model of a building is executed by a processor, the steps of the method for constructing an integrated indoor and outdoor three-dimensional model of a building as described above are implemented.

[0105] Reference Figure 4 , Figure 4 This is a structural block diagram of an embodiment of a system for constructing an indoor and outdoor integrated three-dimensional model of a building according to the present invention.

[0106] like Figure 4 As shown, the indoor and outdoor integrated three-dimensional model construction system of the building includes:

[0107] A data acquisition module 10 is used to collect video data in real time based on multiple cameras inside and outside the building;

[0108] A framework building module 20, used to build a federated learning framework;

[0109] A training conversion module 30, configured to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network based on the federated learning framework using NeRF technology to obtain a trained 3D field model;

[0110] The transformation and fusion module 40 is used to perform 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated three-dimensional model of the building indoors and outdoors.

[0111] This embodiment provides a system for constructing an integrated three-dimensional model of indoor and outdoor buildings, which applies NeRF technology to the construction of an integrated three-dimensional model of indoor and outdoor buildings, and fully utilizes the advantages of NeRF technology to convert two-dimensional image information into a radiation field representation in three-dimensional space by training a neural network, thereby constructing a highly realistic trained 3D field model. At the same time, combined with the federated learning framework, the sharing of model parameters between each computing node and the protection of data privacy are achieved. Through 3D Gaussian function transformation and data fusion processing, an integrated three-dimensional model of indoor and outdoor buildings is obtained, providing a more convenient and efficient three-dimensional modeling solution for architectural design, navigation, virtual reality and other fields.

[0112] It should be noted that the technical details that are not described in detail in the embodiment of the system for constructing an indoor and outdoor integrated three-dimensional model of a building can be referred to the method for constructing an indoor and outdoor integrated three-dimensional model of a building as described above provided in any embodiment of the present invention, and will not be repeated here.

[0113] In the embodiments of the present application, data acquisition and data collection (for example, real-time collection of video data based on multiple cameras inside and outside a building) are all carried out within the scope of compliance with relevant laws and regulations.

[0114] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.

[0115] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.

[0116] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0117] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0119] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for constructing an integrated indoor and outdoor three-dimensional model of a building, characterized in that: include: Real-time video data collection based on multiple cameras inside and outside the building; Build a federated learning framework; Based on the federated learning framework, NeRF technology is used to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network to obtain a trained 3D field model; The trained 3D field model is subjected to 3D Gaussian function transformation and data fusion processing to obtain a three-dimensional model integrating indoor and outdoor aspects of the building.

2. The method according to claim 1, characterized in that The construction of the federated learning framework includes: Setting up a main service center; wherein the main service center is used for model parameter aggregation and distribution; Sub-computing nodes are set; wherein each of the sub-computing nodes independently performs training of the NeRf model, and the sub-computing nodes transmit model parameters through a secure communication protocol.

3. The method according to claim 2, characterized in that The sub-computing nodes include a first sub-computing node and a second sub-computing node; wherein a plurality of outdoor cameras are set as the first sub-computing nodes, and a plurality of indoor cameras are set as the second sub-computing nodes.

4. The method according to claim 1, characterized in that The method of converting the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network based on the federated learning framework using NeRF technology to obtain a trained 3D field model includes: On each sub-computing node of the federated learning framework, a NeRf model is constructed using a deep learning framework; The two-dimensional image information of the video data is converted into a radiation field representation in a three-dimensional space according to the NeRf model to obtain a trained 3D field model.

5. The method according to claim 4, characterized in that The method further comprises: During the training process of the NeRf model, the model training is performed using the optimization strategies of distributed training, gradient clipping, and learning rate scheduling; Use pre-trained models for initialization and combine transfer learning technology to transfer the learned knowledge to each computing node; During the training process of the NeRf model, data enhancement technology is used and regularization terms are introduced to increase the generalization ability of the NeRf model and prevent overfitting.

6. The method according to claim 4, characterized in that The model architecture of the NeRf model includes: a multi-layer fully connected neural network, a position encoding module and a volume rendering module.

7. The method according to claim 1, characterized in that The step of performing 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated indoor and outdoor three-dimensional model of the building includes: The trained 3D field model is transformed into a 3D Gaussian point cloud model with spatial coordinate positions through a 3D Gaussian function transformation; wherein the 3D Gaussian point cloud model includes a 3D Gaussian point cloud model trained by an outdoor camera and a 3D Gaussian point cloud model trained by an indoor camera; Based on the service main center of the federated learning framework, the 3D Gaussian point cloud model trained by the outdoor camera and the 3D Gaussian point cloud model trained by the indoor camera are fused according to the spatial position information to obtain a fused three-dimensional model; The fused three-dimensional model is imported into a visualization platform to obtain an integrated three-dimensional model of the indoor and outdoor of the building.

8. The method according to claim 7, characterized in that The step of converting the trained 3D field model into a 3D Gaussian point cloud model having a spatial coordinate position through a 3D Gaussian function transformation includes: Calculating the spatial coordinates and radiation intensity of each point according to the radiation field representation of the trained 3D field model; Based on the 3D Gaussian function transformation, each point is fitted with a 3D Gaussian function to obtain a 3D Gaussian point cloud model.

9. A building indoor and outdoor integrated three-dimensional model construction system, characterized in that: include: A data acquisition module for real-time video data acquisition based on multiple cameras inside and outside the building; Framework building module, used to build a federated learning framework; A training conversion module, used to convert the two-dimensional image information of the video data into a radiation field representation in a three-dimensional space by training a neural network based on the federated learning framework using NeRF technology to obtain a trained 3D field model; The transformation and fusion module is used to perform 3D Gaussian function transformation and data fusion processing on the trained 3D field model to obtain an integrated three-dimensional model of the building's indoor and outdoor areas.

10. An electronic device, characterized in that: The electronic device includes: a memory, a processor, and a program for constructing an integrated indoor and outdoor three-dimensional model of a building stored in the memory and executable on the processor. The program for constructing an integrated indoor and outdoor three-dimensional model of a building is configured to implement a method for constructing an integrated indoor and outdoor three-dimensional model of a building as described in any one of claims 1 to 8.

Images