Three-dimensional reconstruction method, graphical interface, system and related devices

Abstract

The application discloses a three-dimensional reconstruction method, a graphical interface, a system and related devices. In the method, a neural radiance field model can be trained using images or videos of a target scene to obtain an implicit three-dimensional model corresponding to the target scene. Then, the implicit three-dimensional model is converted into an explicit three-dimensional model that can be rendered in real time by a mobile terminal device. Finally, a picture presented by the explicit three-dimensional model under a certain viewing angle can be rendered in real time on the mobile terminal device. It can be seen that the method realizes real-time rendering of a three-dimensional model trained by a neural radiance field model on a mobile terminal device. A user can obtain a realistic three-dimensional model without manual modeling, and three-dimensional reconstruction is truly introduced into the public view.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to three-dimensional reconstruction methods, graphical interfaces, systems and related devices. Background Technology

[0002] 3D reconstruction refers to the creation of mathematical models of 3D objects suitable for computer recognition and processing. 3D object reconstruction is an important research area in computer graphics and computer vision, with wide applications in animation, film, games, and other fields. Using 3D reconstruction, real-world scenes can be "brought" into virtual space, allowing users to remotely experience the immersive feeling of being in a real-world scene without leaving home.

[0003] Currently, common 3D reconstruction technologies include: using professional cameras to take 2D images of real-world spatial scenes, and then using software to stitch together multiple 2D images to simulate a 360° panoramic view of 3D space, achieving the effect of quickly reconstructing 3D scenes. However, simulating a panoramic view using 2D images still results in a 2D image, leading to poor user engagement and failing to realistically simulate the effect of being in a real-world scene. Alternatively, expensive instruments can be used to acquire high-quality color images, and high-quality 3D models can be reconstructed using stereo matching methods. However, this method is very costly and only suitable for high-end applications, making it difficult to integrate into people's daily lives.

[0004] Therefore, how to reconstruct realistic 3D scenes and bring them to the public eye is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a three-dimensional reconstruction method, graphical interface, system and related devices, which realizes the conversion of implicit three-dimensional models obtained based on neural radiation field training into explicit three-dimensional models that can be rendered in real time by mobile devices, bringing three-dimensional reconstruction into the public eye.

[0006] In a first aspect, this application provides a three-dimensional reconstruction method applied to a server. The method includes: the server acquiring multiple images of a target scene; the server training a neural radiation field model based on the multiple images and the camera pose when the multiple images were captured to obtain an implicit three-dimensional model; wherein the input of the implicit three-dimensional model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point; the server converting the implicit three-dimensional model into an explicit three-dimensional model, the explicit three-dimensional model describing the density and color of some spatial points in the target scene under different viewing directions; and the server sending the explicit three-dimensional model to a first device.

[0007] Implementing the method provided in the first aspect, the server can use neural radiation fields to train an implicit 3D model of the target scene, and then lightweight and real-time convert this implicit 3D model into an explicit 3D model that can be rendered in real-time by mobile devices. Compared with the implicit 3D model, which only represents the coordinates, viewing direction, and input-output relationship of the color and density of spatial points, the explicit 3D model directly describes the density of spatial points and the color under different viewing directions. This avoids the trouble of querying the information of each spatial point when rendering the 3D model, thus demonstrating the real-time nature of the implicit 3D model. Furthermore, the explicit 3D model only describes the information of a portion of the spatial points in the target scene, and the number of spatial points it describes is less than that represented by the implicit 3D model, thus demonstrating the lightweight nature of the implicit 3D model. In this way, users do not need expensive mobile devices; they only need a mobile device to view the 3D model reconstructed using neural radiation fields. This lowers the barrier to 3D reconstruction, enabling ordinary consumers to achieve 3D reconstruction of target scenes using mobile devices such as mobile phones, tablets, and computers. This expands the application scenarios of 3D reconstruction, truly bringing 3D reconstruction into people's lives and introducing it into the public eye.

[0008] In conjunction with the first aspect, in some implementations, during the training of the neural radiation field model, the coordinates of the spatial points corresponding to the pixels in the image are determined by the image and the camera pose when the image was captured; the viewing direction is indicated by the camera position when the image was captured, pointing towards the pixels in the image; the integral of the color and density of one or more spatial points in the viewing direction in the target scene is the pixel value of the pixel.

[0009] In conjunction with the first aspect, in some implementations, the server transforms the implicit 3D model into an explicit 3D model, specifically including: the server using spherical harmonic functions to represent the relationship between the viewing direction and color represented in the implicit 3D model, where each spherical harmonic function represents the relationship between different viewing directions and colors of a spatial point; the server filtering some spatial points in the target scene to retain those spatial points with density values ​​greater than a preset value, where the density value of the spatial points in the target scene is obtained by inputting the coordinates of the spatial points into the implicit 3D model; and the server generating an explicit 3D model containing the following information: the density of the spatial points and the coefficients of the spherical harmonic functions of the spatial points.

[0010] In other words, the server can use spherical harmonic functions to represent the input-output relationship of the implicit 3D model, avoiding the need to query the 3D model information through the input-output relationship of the implicit 3D model when rendering the 3D model, thus speeding up the model rendering speed. In addition, the server can reduce the model's complexity by filtering spatial points with low density values, further speeding up the model rendering speed.

[0011] In conjunction with the first aspect, in some implementations, the explicit 3D model stores the information in an octree data structure, the octree including multiple nodes, one node corresponding to a spatial point in the subset of spatial points, and one node storing index information used to find the density of the spatial point corresponding to the node, as well as the coefficient of the spherical harmonic function of the spatial point corresponding to the node.

[0012] Since an octree is a tree-like structure for describing three-dimensional space, using an octree to describe this explicit 3D model is beneficial because the leaf nodes of an octree inherently represent their relative positional relationships with other leaf nodes. Thus, when rendering this explicit 3D model, the device does not need to determine the position of a spatial point in actual physical space based on its coordinates. It only needs to reconstruct the image of the target scene presented on the device based on the relative positional relationships of these spatial points, thereby speeding up the device's model rendering process.

[0013] In conjunction with the first aspect, in some embodiments, the explicit 3D model is implemented as a texture map, which includes: an index map and a model parameter map. The pixel values ​​of the index map are used to indicate the index information of nodes, and the pixel values ​​of the model parameter map are used to indicate the density of spatial points or the coefficients of spherical harmonic functions. The first node indicated by a pixel value of the index map is used to find the density of the first spatial point corresponding to the first node indicated by multiple pixel values ​​in the model parameter map, as well as the coefficients of the spherical harmonic function of the first spatial point.

[0014] In other words, when the server sends an explicit 3D model to the first device, it can directly send a texture map to the first device. The first device can obtain information about the explicit 3D model through the texture map and render the explicit 3D model.

[0015] In conjunction with the first aspect, in some implementations, the camera pose when capturing an image is determined by the server based on the image and the images adjacent to it, wherein any two adjacent images among the plurality of images contain overlapping content in a preset proportion.

[0016] In conjunction with the first aspect, in some implementations, after the server acquires multiple images of the target scene, the method further includes: the server determining the camera pose corresponding to each of the multiple images based on the multiple images and prior parameters, wherein the prior parameters include one or more of the following: the position and pose of the camera that captured the image, the position of the object captured in the image, and the depth information of the image.

[0017] The server uses prior parameters to determine the camera pose corresponding to the image, which enables the server to obtain a more accurate camera pose and achieve alignment of the constructed 3D model with the position, angle and scale of the target scene in real 3D space.

[0018] In conjunction with the first aspect, in some implementations, the server acquires multiple images of a target scene, specifically including: the server acquires multiple images of the target scene captured by the first device; or, before the server acquires multiple images of the target scene, the method further includes: the server acquires a video of the target scene captured by the first device, wherein the multiple images are multiple frames of the video.

[0019] In other words, the server can achieve 3D reconstruction of the target scene by using images or videos collected by the first device.

[0020] Secondly, embodiments of this application provide a three-dimensional reconstruction method applied to a first device. The method includes: the first device acquiring an explicit three-dimensional model sent by a server, the explicit three-dimensional model being a model obtained by the server after converting an implicit three-dimensional model, the implicit three-dimensional model being obtained by training a neural radiation field model based on multiple images of a target scene and the camera pose when the multiple images were captured, wherein the input of the implicit three-dimensional model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point, and the explicit three-dimensional model describes the density and color of some spatial points in the target scene under different viewing directions; the first device displaying a first user interface, the first user interface displaying a first image presented by the explicit three-dimensional model from a first viewing perspective.

[0021] The first device can refer to a user-facing mobile device that implements the method provided in the first aspect. The first device directly uses an explicit 3D model generated by a server to display the image presented by the explicit 3D model from a certain perspective, giving the user an immersive experience. The 3D model obtained through neural radiation field training can accurately reproduce the lighting and environment of the real scene, resulting in a realistic virtual scene and improving the accuracy and quality of the reconstructed 3D scene.

[0022] In conjunction with the second aspect, in some embodiments, the pixel value of a pixel in the first image is obtained by integrating the color and density of a spatial point in a first viewing direction described by the explicit three-dimensional model, where the first viewing direction is the direction in which the spatial point corresponding to the pixel is observed from the first viewing angle.

[0023] In conjunction with the second aspect, in some embodiments, the method further includes: the first device detecting that the user adjusts the first viewing angle to a second viewing angle; and the first device displaying, in the first user interface, a second view of the explicit 3D model presented from the second viewing angle.

[0024] In other words, the first device can update the display of the explicit 3D model from different perspectives based on the user's adjustment of the viewing angle. In this way, the user can view the 3D model of the target scene in the real environment presented in the electronic device from different angles by adjusting the new perspective, thereby achieving a roaming effect in the target scene.

[0025] In conjunction with the second aspect, in some embodiments, the multiple images of the target scene are multiple images of the target scene captured by the first device; or, the method further includes: the first device sending a video of the target scene captured to the server, wherein the multiple images of the target scene are multiple frames of the video.

[0026] In other words, the first device can participate in the early data acquisition stage of reconstructing the 3D model. Users can collect images or videos of the target scene themselves to achieve 3D reconstruction of the target scene, enhance the user's sense of participation in 3D reconstruction, and increase the fun of 3D reconstruction.

[0027] In conjunction with the second aspect, in some embodiments, before the first device sends multiple images or videos of the target scene to the server, the method further includes: the first device displaying a second user interface containing a first control; the first device detecting a first operation performed on the first control; in response to the first operation, the first device activating a camera and displaying a third user interface, the third user interface including: a first preview frame and a second control, the first preview frame being used to display images captured by the camera in real time; the first device detecting a second operation performed on the second control, the second operation being used to trigger the first device to capture multiple images or videos of the target scene.

[0028] In conjunction with the second aspect, in some embodiments, the first user interface further includes a third control. After the first device displays the first user interface, the method further includes: the first device detecting a third operation performed on the third control; in response to the third operation, the first device displaying one or more model icons in the third user interface, the one or more model icons including the first model icon; the first device detecting a fourth operation performed on the first model icon; in response to the fourth operation, the first device adding a first model corresponding to the first model icon to the first user interface, and, from the first viewing perspective, the first device displaying a third screen in the first user interface showing the explicit three-dimensional model after adding the first model.

[0029] During the process of the first device displaying an explicit 3D model of the target scene, the first device can adjust the explicit 3D model according to user operations, such as adding virtual objects to the explicit 3D model. In this way, users can view the effect of adding objects to the target scene by adding virtual objects to the virtual scene. For example, after a user captures an image of their living room and obtains a corresponding explicit 3D model, the user can add virtual furniture, change wallpaper styles, etc., to the explicit 3D model. This allows users to see the effects of placing furniture and changing wallpaper styles in the living room without expending any financial or material resources or changing the layout of the living room, providing a reference for decorating the target scene and offering convenience to the user's life.

[0030] In conjunction with the second aspect, in some embodiments, after the first device displays the third screen, the method further includes: the first device sending the explicit 3D model after adding the first model to the second device, so that the second device overlays the first model on the first image of the target scene acquired in real time, and the position of the first model in the target scene displayed in the first image corresponds to the position of the first model in the explicit 3D model after adding the first model.

[0031] In other words, the first device can send the adjusted explicit 3D model to other devices, such as a second device. The second device can then overlay the added virtual objects from the adjusted explicit 3D model onto images of the target scene captured in real time by a camera. This allows users of the first device to adjust the target scene without being physically present, enabling on-site users to view the newly added virtual objects. This achieves a convergence of the virtual and real worlds, introducing virtual reality and mixed reality into 3D reconstruction, increasing user interaction with the 3D model, and enhancing the fun of 3D reconstruction.

[0032] Thirdly, embodiments of this application provide a three-dimensional reconstruction method applied to a second device. The method includes: the second device acquiring an explicit three-dimensional model sent by a first device after adding a first model, the explicit three-dimensional model being a model of a target scene trained based on a neural radiation field model; the second device displaying a fourth user interface, the fourth user interface displaying: a first image of the target scene acquired in real time by the second device, and the first model superimposed on the first image, the position of the first model in the target scene displayed in the first image corresponding to the position of the first model in the explicit three-dimensional model after adding the first model.

[0033] By implementing the method provided in the third aspect, the second device can view the virtual objects added by the first device in the virtual 3D model in the real scene, realizing the intersection of the virtual world and the real world, increasing the fun of 3D reconstruction, and expanding the application scenarios of 3D reconstruction.

[0034] In conjunction with the third aspect, in some implementations, the explicit 3D model is a model obtained by the server after converting the implicit 3D model. The implicit 3D model is obtained by training a neural radiation field model based on multiple images of the target scene and the camera pose when the multiple images were taken. The input of the implicit 3D model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point. The explicit 3D model describes the density of some spatial points in the target scene and their colors under different viewing directions.

[0035] Fourthly, embodiments of this application provide a three-dimensional reconstruction system, which includes a server and a first device. The system includes: the server acquiring multiple images of a target scene; the server training a neural radiation field model based on the multiple images and the camera pose when the multiple images were captured to obtain an implicit three-dimensional model; wherein the input of the implicit three-dimensional model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point; the server converting the implicit three-dimensional model into an explicit three-dimensional model, the explicit three-dimensional model describing the density and color of some spatial points in the target scene under different viewing directions; the server sending the explicit three-dimensional model to the first device; the first device displaying a first user interface, the first user interface displaying: a first image presented by the explicit three-dimensional model from a first viewing perspective, the pixel value of the pixel in the first image being obtained by integrating the color and density of the spatial point in the first viewing direction described by the explicit three-dimensional model, the first viewing direction being the direction of observing the spatial point corresponding to the pixel from the first viewing perspective.

[0036] As can be seen, the server generates an explicit 3D model that can be rendered in real time on mobile devices, and the first device displays this explicit 3D model. Since this explicit 3D model is based on neural radiation field reconstruction, the first device can display a realistic virtual model of the target scene that accurately reproduces real-world ambient lighting. Furthermore, this explicit 3D model is obtained by the server through lightweighting and real-time rendering of the implicit 3D model, allowing the first device to render it more quickly. This approach introduces 3D models based on neural radiation scene reconstruction to the public, lowers the barrier to entry for 3D reconstruction, expands its application scenarios, and truly brings 3D reconstruction into people's lives.

[0037] In conjunction with the fourth aspect, in some implementations, during the training of the neural radiation field model, the coordinates of the spatial points corresponding to the pixels in the image are determined by the image and the camera pose when the image was captured; the viewing direction is indicated by the camera position when the image was captured, pointing towards the pixels in the image; the integral of the color and density of one or more spatial points in the viewing direction in the target scene is the pixel value of the pixel.

[0038] In conjunction with the fourth aspect, in some embodiments, the system further includes: the first device detecting that the user adjusts the first viewing angle to a second viewing angle; and the first device displaying, in the first user interface, a second view of the explicit 3D model presented from the second viewing angle.

[0039] In other words, the first device can update the display of the explicit 3D model from different perspectives based on the user's adjustment of the viewing angle. In this way, the user can view the 3D model of the target scene in the real environment presented in the electronic device from different angles by adjusting the new perspective, thereby achieving a roaming effect in the target scene.

[0040] In conjunction with the fourth aspect, in some implementations, the server transforms the implicit 3D model into an explicit 3D model, specifically including: the server using spherical harmonic functions to represent the relationship between the viewing direction and color represented in the implicit 3D model, where each spherical harmonic function represents the relationship between different viewing directions and colors of a spatial point; the server filtering some spatial points in the target scene to retain those spatial points with density values ​​greater than a preset value, where the density value of the spatial points in the target scene is obtained by inputting the coordinates of the spatial points into the implicit 3D model; and the server generating an explicit 3D model containing the following information: the density of the spatial points and the coefficients of the spherical harmonic functions of the spatial points.

[0041] In other words, the server can use spherical harmonic functions to represent the input-output relationship of the implicit 3D model, avoiding the need to query the 3D model information through the input-output relationship of the implicit 3D model when rendering the 3D model, thus speeding up the model rendering speed. In addition, the server can reduce the model's complexity by filtering spatial points with low density values, further speeding up the model rendering speed.

[0042] In conjunction with the fourth aspect, in some implementations, the explicit three-dimensional model stores the information in an octree data structure, the octree including multiple nodes, one node corresponding to a spatial point in the part of spatial points, and one node storing index information used to find the density of the spatial point corresponding to the node, and the coefficient of the spherical harmonic function of the spatial point corresponding to the node.

[0043] Since an octree is a tree-like structure for describing three-dimensional space, using an octree to describe this explicit 3D model is beneficial because the leaf nodes of an octree inherently represent their relative positional relationships with other leaf nodes. Thus, when rendering this explicit 3D model, the device does not need to determine the position of a spatial point in actual physical space based on its coordinates. It only needs to reconstruct the image of the target scene presented on the device based on the relative positional relationships of these spatial points, thereby speeding up the device's model rendering process.

[0044] In conjunction with the fourth aspect, in some embodiments, the explicit 3D model is implemented as a texture map, which includes: an index map and a model parameter map. The pixel values ​​of the index map are used to indicate the index information of nodes, and the pixel values ​​of the model parameter map are used to indicate the density of spatial points or the coefficients of spherical harmonic functions. The first node indicated by a pixel value of the index map is used to find the density of the first spatial point corresponding to the first node indicated by multiple pixel values ​​in the model parameter map, as well as the coefficients of the spherical harmonic function of the first spatial point.

[0045] In other words, when the server sends an explicit 3D model to the first device, it can directly send a texture map to the first device. The first device can obtain information about the explicit 3D model through the texture map and render the explicit 3D model.

[0046] In conjunction with the fourth aspect, in some implementations, the camera pose when capturing an image is determined by the server based on the image and the images adjacent to it, wherein any two adjacent images among the plurality of images contain overlapping content in a preset proportion.

[0047] In conjunction with the fourth aspect, in some implementations, after the server acquires multiple images of the target scene, the system further includes: the server determining the camera pose corresponding to each of the multiple images based on the multiple images and prior parameters, wherein the prior parameters include one or more of the following: the position and pose of the camera that captured the image, the position of the object captured in the image, and the depth information of the image.

[0048] The server uses prior parameters to determine the camera pose corresponding to the image, which enables the server to obtain a more accurate camera pose and achieve alignment of the constructed 3D model with the position, angle and scale of the target scene in real 3D space.

[0049] In conjunction with the fourth aspect, in some embodiments, before the server acquires multiple images of the target scene, the system further includes: the first device sending the acquired multiple images of the target scene to the server; or, the first device sending the acquired video of the target scene to the server, wherein the multiple images are multiple frames of the video.

[0050] In other words, the server can achieve 3D reconstruction of the target scene by collecting images or videos of the target scene through the first device. The first device can participate in the early data collection stage of reconstructing the 3D model. Users can collect images or videos of the target scene themselves to achieve 3D reconstruction of the target scene, enhance the user's sense of participation in 3D reconstruction, and increase the fun of 3D reconstruction.

[0051] In conjunction with the fourth aspect, in some embodiments, before the first device sends multiple images or videos of the target scene to the server, the system further includes: the first device displaying a second user interface containing a first control; the first device detecting a first operation performed on the first control; in response to the first operation, the first device activating a camera and displaying a third user interface, the third user interface including: a first preview frame and a second control, the first preview frame being used to display images captured by the camera in real time; and the first device detecting a second operation performed on the second control, the second operation being used to trigger the first device to capture multiple images or videos of the target scene.

[0052] In conjunction with the fourth aspect, in some embodiments, the first user interface further includes: a third control; after the first device displays the first user interface, the system further includes: the first device detecting a third operation performed on the third control; in response to the third operation, the first device displaying one or more model icons in the first user interface; the first device detecting a fourth operation performed on a first model icon among the one or more model icons; in response to the fourth operation, the first device adding a first model corresponding to the first model icon to the first user interface, and, from the first viewing perspective, the first device displaying a third screen in the first user interface showing the explicit 3D model after adding the first model.

[0053] During the process of the first device displaying an explicit 3D model of the target scene, the first device can adjust the explicit 3D model according to user operations, such as adding virtual objects to the explicit 3D model. In this way, users can view the effect of adding objects to the target scene by adding virtual objects to the virtual scene. For example, after a user captures an image of their living room and obtains a corresponding explicit 3D model, the user can add virtual furniture, change wallpaper styles, etc., to the explicit 3D model. This allows users to see the effects of placing furniture and changing wallpaper styles in the living room without expending any financial or material resources or changing the layout of the living room, providing a reference for decorating the target scene and offering convenience to the user's life.

[0054] In conjunction with the fourth aspect, in some embodiments, the communication system further includes: a second device, and after the first device displays the third screen, the system further includes: the first device sending the explicit 3D model with the first model added to the second device.

[0055] In other words, the first device can share the adjusted explicit 3D model with other devices so that other devices can also view the adjusted explicit 3D model.

[0056] In conjunction with the fourth aspect, in some embodiments, the system further includes: the second device displaying a fourth user interface, the fourth user interface displaying: a first image of the target scene acquired in real time by the second device, and a first model superimposed on the first image, the position of the first model in the target scene displayed in the first image corresponding to the position of the first model in the explicit 3D model after the first model is added.

[0057] As can be seen, the second device can overlay and display the added virtual objects in the adjusted explicit 3D model onto the real-time images of the target scene captured by the camera. In this way, users of the first device can adjust the target scene without being physically present, allowing on-site users to view the newly added virtual objects. This achieves a convergence of the virtual and real worlds, introducing virtual reality and mixed reality into 3D reconstruction, increasing user interaction with the 3D model, and enhancing the fun of 3D reconstruction.

[0058] Fifthly, embodiments of this application provide a three-dimensional reconstruction system, which includes a first device and a second device. The system includes: the first device displaying a first user interface, which displays a first image of an explicit three-dimensional model from a first viewing perspective, the explicit three-dimensional model being trained based on a neural radiation field model; the first device detecting an operation to adjust the explicit three-dimensional model, the operation being an operation applied to a first model icon; in response to the operation, the first device adding a first model corresponding to the first model icon to the first user interface, and, from the first viewing perspective, the first device displaying a third image of the explicit three-dimensional model after adding the first model to the first user interface; the first device sending the explicit three-dimensional model after adding the first model to the second device; and the second device displaying a fourth user interface, which displays a first image of the target scene acquired in real time by the second device, and the first model superimposed on the first image, the position of the first model in the target scene displayed in the first image corresponding to the position of the first model in the explicit three-dimensional model after adding the first model.

[0059] As can be seen, the first device can adjust the explicit 3D model of the target scene, such as adding virtual objects. The second device can view the virtual objects added by the first device in the corresponding position in the virtual scene by capturing images of the target scene in real time through a camera. This realizes the intersection of the virtual world and the real world. In this way, users can remotely adjust the model of the target scene and affect the on-site viewing of the target scene, increasing the fun of 3D reconstruction.

[0060] In conjunction with the fifth aspect, in some embodiments, after the first device displays the first user interface, the system further includes: the first device detecting that the user adjusts the first viewing angle to a second viewing angle; and the first device displaying, in the first user interface, a second view of the explicit three-dimensional model presented from the second viewing angle.

[0061] In other words, the first device can update the display of the explicit 3D model from different perspectives based on the user's adjustment of the viewing angle. In this way, the user can view the 3D model of the target scene in the real environment presented in the electronic device from different angles by adjusting the new perspective, thereby achieving a roaming effect in the target scene.

[0062] In conjunction with the fifth aspect, in some embodiments, the first user interface further includes: a third control; after the first device displays the first user interface, the system further includes: the first device detecting a third operation acting on the third control; in response to the third operation, the first device displays one or more model icons in the first user interface, the one or more model icons including the first model icon.

[0063] In other words, the first device can provide multiple virtual models for users to choose from. Users can add virtual models to the explicit 3D model of the target scene according to their own needs and preferences, thereby enhancing user participation and improving user experience.

[0064] In conjunction with the fifth aspect, in some embodiments, the communication system further includes: a server, and before the first device displays the first user interface, the system further includes: the server acquiring multiple images of a target scene; the server training a neural radiation field model based on the multiple images and the camera pose when the multiple images were captured to obtain an implicit three-dimensional model; wherein the input of the implicit three-dimensional model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point; the server converting the implicit three-dimensional model into an explicit three-dimensional model, the explicit three-dimensional model describing the density and color of some spatial points in the target scene under different viewing directions; and the server sending the explicit three-dimensional model to the first device.

[0065] In other words, the server can use neural radiation fields to train and obtain an implicit 3D model of the target scene, and then lightweight and real-time convert this implicit 3D model into an explicit 3D model that can be rendered in real time by mobile devices. This will improve the rendering speed of mobile devices, expand the application scenarios of 3D models obtained from neural radiation fields, and truly bring 3D reconstruction into the public eye.

[0066] In conjunction with the fifth aspect, in some embodiments, during the training of the neural radiation field model, the coordinates of the spatial points corresponding to the pixels in the image are determined by the image and the camera pose when the image was captured; the viewing direction is indicated by the camera position when the image was captured, pointing towards the pixels in the image; the integral of the color and density of one or more spatial points in the viewing direction in the target scene is the pixel value of the pixel.

[0067] In conjunction with the fifth aspect, in some implementations, the server transforms the implicit 3D model into an explicit 3D model, specifically including: the server using spherical harmonic functions to represent the relationship between the viewing direction and color represented in the implicit 3D model, where each spherical harmonic function represents the relationship between different viewing directions and colors of a spatial point; the server filtering some spatial points in the target scene to retain those spatial points with density values ​​greater than a preset value, where the density value of the spatial points in the target scene is obtained by inputting the coordinates of the spatial points into the implicit 3D model; and the server generating an explicit 3D model containing the following information: the density of the spatial points and the coefficients of the spherical harmonic functions of the spatial points.

[0068] In other words, the server can use spherical harmonic functions to represent the input-output relationship of the implicit 3D model, avoiding the need to query the 3D model information through the input-output relationship of the implicit 3D model when rendering the 3D model, thus speeding up the model rendering speed. In addition, the server can reduce the model's complexity by filtering spatial points with low density values, further speeding up the model rendering speed.

[0069] In conjunction with the fifth aspect, in some embodiments, the explicit three-dimensional model stores the information in an octree data structure, the octree including multiple nodes, one node corresponding to a spatial point in the subset of spatial points, and one node storing index information used to find the density of the spatial point corresponding to the node, and the coefficient of the spherical harmonic function of the spatial point corresponding to the node.

[0070] Since an octree is a tree-like structure for describing three-dimensional space, using an octree to describe this explicit 3D model is beneficial because the leaf nodes of an octree inherently represent their relative positional relationships with other leaf nodes. Thus, when rendering this explicit 3D model, the device does not need to determine the position of a spatial point in actual physical space based on its coordinates. It only needs to reconstruct the image of the target scene presented on the device based on the relative positional relationships of these spatial points, thereby speeding up the device's model rendering process.

[0071] In conjunction with the fifth aspect, in some embodiments, the explicit 3D model is implemented as a texture map, which includes: an index map and a model parameter map. The pixel values ​​of the index map are used to indicate the index information of nodes, and the pixel values ​​of the model parameter map are used to indicate the density of spatial points or the coefficients of spherical harmonic functions. The first node indicated by a pixel value of the index map is used to find the density of the first spatial point corresponding to the first node indicated by multiple pixel values ​​in the model parameter map, and the coefficients of the spherical harmonic function of the first spatial point.

[0072] In other words, when the server sends an explicit 3D model to the first device, it can directly send a texture map to the first device. The first device can obtain information about the explicit 3D model through the texture map and render the explicit 3D model.

[0073] In conjunction with the fifth aspect, in some embodiments, the camera pose when capturing an image is determined by the server based on the image and the images adjacent to it, wherein any two adjacent images among the plurality of images contain overlapping content in a preset proportion.

[0074] In conjunction with the fifth aspect, in some implementations, after the server acquires multiple images of the target scene, the system further includes: the server determining the camera pose corresponding to each of the multiple images based on the multiple images and prior parameters, wherein the prior parameters include one or more of the following: the position and pose of the camera that captured the image, the position of the object captured in the image, and the depth information of the image.

[0075] The server uses prior parameters to determine the camera pose corresponding to the image, which enables the server to obtain a more accurate camera pose and achieve alignment of the constructed 3D model with the position, angle and scale of the target scene in real 3D space.

[0076] In conjunction with the fifth aspect, in some embodiments, before the server acquires multiple images of the target scene, the system further includes: the first device sending the acquired multiple images of the target scene to the server; or, the first device sending the acquired video of the target scene to the server, wherein the multiple images are multiple frames of the video.

[0077] In other words, the first device can participate in the early data acquisition stage of reconstructing the 3D model. Users can collect images or videos of the target scene themselves to achieve 3D reconstruction of the target scene, enhance the user's sense of participation in 3D reconstruction, and increase the fun of 3D reconstruction.

[0078] In conjunction with the fifth aspect, in some embodiments, before the first device sends multiple images or videos of the target scene to the server, the system further includes: the first device displaying a second user interface containing a first control; the first device detecting a first operation performed on the first control; in response to the first operation, the first device activating a camera and displaying a third user interface, the third user interface including: a first preview frame and a second control, the first preview frame being used to display images captured by the camera in real time; and the first device detecting a second operation performed on the second control, the second operation being used to trigger the first device to capture multiple images or videos of the target scene.

[0079] In a sixth aspect, embodiments of this application provide an electronic device, including a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device causes the electronic device to implement the method described in the first aspect or any embodiment of the first aspect.

[0080] In a seventh aspect, embodiments of this application provide an electronic device, including a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device causes the electronic device to implement the methods described in the second aspect or any embodiment of the second aspect, or the third aspect or any embodiment of the third aspect.

[0081] Eighthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect or any embodiment of the first aspect, the second aspect or any embodiment of the second aspect, or the third aspect or any embodiment of the third aspect.

[0082] Ninthly, embodiments of this application provide a computer program product that, when run on a computer, causes the computer to perform the methods described in the first aspect or any one of the first aspects, the second aspect or any one of the second aspects, or the third aspect or any one of the third aspects. Attached Figure Description

[0083] Figure 1 A schematic diagram of a communication system 1000 provided in an embodiment of this application;

[0084] Figure 2 This application provides a schematic diagram of the structure of an indoor scene according to an embodiment of the present application;

[0085] Figures 3A-3I , Figures 4A-4I Some user interfaces provided for embodiments of this application;

[0086] Figure 5 This is a schematic diagram illustrating a user capturing an indoor scene using the camera of an electronic device, as provided in an embodiment of this application.

[0087] Figure 6 A schematic flowchart illustrating the three-dimensional reconstruction method provided in this application embodiment;

[0088] Figure 7 A schematic diagram of the shooting scene involved when the camera provided in the embodiments of this application captures the target scene from multiple perspectives;

[0089] Figure 8A schematic diagram of the process involved in calculating camera pose using multiple images, provided for an embodiment of this application;

[0090] Figure 9 A schematic diagram illustrating the association between two images and three-dimensional points in space, provided for embodiments of this application;

[0091] Figure 10 A schematic diagram illustrating the process of reconstructing a three-dimensional model based on a neural radiation field model, as provided in the embodiments of this application.

[0092] Figure 11 This is a schematic diagram illustrating the block division of a target scene as provided in an embodiment of this application;

[0093] Figure 12 A schematic diagram of the texture mapping provided in the embodiments of this application;

[0094] Figure 13 A schematic diagram of the hardware structure of the electronic device 300 provided in the embodiments of this application;

[0095] Figure 14 A schematic diagram of the software structure of the electronic device 300 provided in the embodiments of this application;

[0096] Figure 15 This is a schematic diagram of the hardware structure of the electronic device 400 provided in an embodiment of this application. Detailed Implementation

[0097] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0098] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0099] The term "user interface (UI)" used in the following embodiments of this application refers to the medium interface through which an application or operating system interacts and exchanges information with the user. It realizes the conversion between the internal form of information and the form that the user can accept. The user interface is source code written in a specific computer language such as Java or Extensible Markup Language (XML). The interface source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets displayed on the screen of an electronic device.

[0100] To achieve better reconstruction of 3D scenes, this application provides a 3D reconstruction method based on Neural Radiance Fields (NeRF). In this method, firstly, images or videos of the target scene are captured, obtaining multiple images of the target scene. These multiple images, along with the corresponding camera poses, are used as training data to train the NeRF model, resulting in an implicit 3D model of the target scene. Secondly, this implicit 3D model is converted into an explicit 3D model that can be rendered in real-time on a mobile device. Finally, when a new user perspective is detected, the explicit 3D model presented from that new perspective is rendered in real-time. In this way, users can continuously adjust their perspective to view the 3D model of the target scene in the real environment on the electronic device from different angles, achieving a roaming effect within the target scene.

[0101] The explicit 3D model refers to the 3D model of the target scene displayed on the electronic device. Any object existing in the physical world can be represented by a 3D model. Displaying this 3D model through an electronic device allows for a realistic virtual representation of objects in nature. Furthermore, converting the implicit 3D model into an explicit 3D model is for lightweighting and real-time rendering of the 3D model, enabling real-time rendering of the trained neural radiation field model on mobile devices. Specific details regarding the calculation of the camera pose corresponding to the image, as well as descriptions of the implicit and explicit 3D models, can be found in subsequent method embodiments and will not be elaborated upon here.

[0102] This enables the real-time display of 3D models trained using neural radiation fields on mobile devices, lowering the barrier to 3D reconstruction and allowing ordinary consumers to achieve 3D reconstruction of target scenes using mobile devices such as smartphones, tablets, and computers. This expands the application scenarios of 3D reconstruction, truly bringing 3D reconstruction into people's lives and introducing it into the public eye.

[0103] In some embodiments, during the process of an electronic device displaying an explicit 3D model of the target scene, the electronic device can adjust the explicit 3D model according to user operations, such as adding, adjusting, or deleting virtual model materials in the explicit 3D model. In this way, users can view the effects of adding, adjusting, or deleting virtual objects in the target scene. For example, after a user captures an image of their living room and obtains a corresponding explicit 3D model, the user can add virtual furniture, change wallpaper styles, etc., in the explicit 3D model. This allows them to view the effects of placing furniture and changing wallpaper styles in the living room without expending any financial or material resources or changing the layout of the living room, providing a reference for decorating the target scene and offering convenience to the user's life.

[0104] Furthermore, the electronic device can send the adjusted explicit 3D model to other devices, which can then display it. This allows other users to view the virtual scene after adding, adjusting, or deleting virtual objects. Alternatively, other devices can activate their cameras to capture images of the target scene and overlay the added model elements onto the real-time captured images of the target scene. For example, if a virtual potted plant is added next to the sofa in the 3D model of the living room, and the electronic device sends this adjusted model to other devices, then when those devices activate their cameras to capture images of the area near the sofa in the target scene, they can overlay the virtual potted plant onto the image corresponding to the sofa. This allows users to view virtual objects in the target scene through the electronic device, achieving a convergence of the virtual and real worlds. This introduces virtual reality and mixed reality into 3D reconstruction, increasing user interaction with the 3D model and enhancing the fun of 3D reconstruction.

[0105] As can be seen, the 3D reconstruction method provided in this application can display a virtual scene corresponding to a real scene on a mobile device. Users do not need to be physically present in the real scene to understand the scene from multiple different perspectives, achieving a virtual tour effect of the real scene without leaving home. Furthermore, reconstructing the 3D scene through neural radiation fields can quickly obtain a 3D model of the real scene, eliminating the need for manual modeling and saving users time. In addition, the 3D model obtained through neural radiation field training can accurately reproduce the lighting and environment of the real scene, resulting in a realistic virtual scene and improving the accuracy and quality of the reconstructed 3D scene.

[0106] The following is combined with Figure 1 This application introduces a communication system 1000 provided in an embodiment.

[0107] Figure 1 This is a schematic diagram of the structure of the communication system 1000 provided in an embodiment of this application.

[0108] In this embodiment, the communication system 1000 may include a server and one or more electronic devices. These one or more electronic devices are user-facing mobile devices. This embodiment does not limit the number of electronic devices. For example, as shown... Figure 1 As shown, the communication system 1000 may include a server 100, an electronic device 100, and an electronic device 200. Wherein:

[0109] Electronic device 100 can be used to capture images or videos of a target scene and send the images or videos to server 100. In addition, electronic device 100 can also be used to acquire an explicit 3D model of the target scene and render the explicit 3D model, providing a user interface to display the image presented by the explicit 3D model from different perspectives. Furthermore, electronic device 100 can also be used to adjust the explicit 3D model according to user operation and send the adjusted explicit 3D model to server 100 or electronic device 200.

[0110] Server 100 can be used to train a neural radiation field model using images or videos of a target scene to obtain an implicit 3D model corresponding to the target scene. In addition, server 100 can also be used to convert the implicit 3D model into an explicit 3D model that can be rendered in real time by mobile devices, such as electronic devices 100 and 200, and send the explicit 3D model to electronic devices 100 and 200.

[0111] The electronic device 200 can be used to acquire an explicit 3D model of the target scene and an adjusted explicit 3D model, determine the model material added to the adjusted explicit 3D model relative to the explicit 3D model of the target scene (i.e., the explicit 3D model before adjustment), and overlay the added model material at a specified position in the image of the target scene captured by the camera, so that the placement of the model material in the target scene as observed by the user through the image captured by the camera is the same as or similar to the placement of the model material in the adjusted explicit 3D model.

[0112] This application does not limit the communication connection method between the server and one or more electronic devices. Specifically, the communication connection can be a wired connection or a wireless connection. The wireless connection can be a short-range connection such as wireless fidelity (Wi-Fi), Bluetooth, infrared, NFC, or ZigBee, or a long-range connection, including but not limited to long-range connections based on 2G, 3G, 4G, 5G, and subsequent standard protocols of mobile networks. For example, server 100 can wirelessly send an explicit 3D model to electronic device 100.

[0113] Additionally, it should be noted that the server mentioned in the embodiments of this application, such as server 100, can be a single server or a server cluster composed of multiple servers. For example, server 100 can be a server cluster deployed through a distributed architecture, which may include one or more of the following: cloud computing servers, content delivery network (CDN) servers, network time protocol (NTP) servers, domain name system (DNS) servers, etc. The servers can coordinate with each other to jointly complete functions such as computing, data storage, and communication. For example, server 100 may include server A and server B. Server A can be used to train the model and send the trained implicit 3D model to server B, while server B can be used to convert the implicit 3D model into an explicit 3D model. For ease of description, in the embodiments of this application, a single server, a distributed server, and a server cluster are collectively referred to as a server.

[0114] It should be understood that electronic device 100 can also perform the functions of electronic device 200 described above, namely, overlaying and displaying the added model material at a specified location in the image of the target scene captured by the camera. Additionally, electronic device 200 can also perform the corresponding functions of electronic device 100, such as real-time rendering of the explicit 3D model, adjusting the explicit 3D model according to user operations, etc. For details regarding the user interface presented by electronic devices 100 and 200 in implementing the above functions, please refer to the following sections. Figures 3A-3I , Figures 4A-4I And its related descriptions.

[0115] The following describes several application scenarios of the three-dimensional reconstruction method provided in the embodiments of this application:

[0116] 1) Virtual Navigation

[0117] The 3D reconstruction method provided in this application can reconstruct a virtual 3D map of the destination and its surroundings by acquiring multiple images of the destination and its vicinity. When a user arrives near the destination, for road sections with complex terrain or routes and many surrounding buildings, the user can view the virtual 3D map through mobile devices such as mobile phones and tablets, view the virtual 3D map from different angles, and quickly compare the virtual 3D map with the buildings around the user, helping the user quickly find the route from the user's current location to the designated location, thus providing convenience for the user's travel.

[0118] Furthermore, users can adjust the virtual 3D map using electronic devices, such as adding road signs at a certain location on the virtual 3D map. Other users can also view the adjusted virtual 3D map using other electronic devices. When viewing that location on the virtual 3D map, or when viewing that location in the real environment through an image captured by the camera of an electronic device, they can see the road signs left by the user, enhancing the interactivity between the user and virtual reality during the travel process.

[0119] 2) Virtual Tourism

[0120] The three-dimensional reconstruction method provided in this application can reconstruct a virtual three-dimensional model of a real landscape by acquiring multiple images of the real landscape. Users can enjoy the scenery of distant places in a three-dimensional virtual environment without leaving home.

[0121] Furthermore, users can leave "footprints" in the virtual 3D model, enabling interaction with the virtual landscape and providing a novel virtual travel experience. Moreover, adjustments made by users to the virtual landscape during their "tour" will not affect the real-world landscape, thus satisfying users' desire to appreciate the scenery without damaging the real-world environment.

[0122] 3) Virtual decoration

[0123] The three-dimensional reconstruction method provided in this application embodiment can reconstruct a virtual three-dimensional model of the interior by acquiring multiple indoor images. Users can add, adjust, or delete virtual objects in the virtual three-dimensional model to see the effect of adding, adjusting, or deleting objects in the real indoor scene. This allows users to see the effect of the interior scene after decoration without changing the layout of the real indoor scene.

[0124] It should be understood that the above three application scenarios are merely illustrative examples to help readers better understand this solution and do not constitute a limitation on the embodiments of this application. In other embodiments of this application, the three-dimensional reconstruction method may also include other application scenarios, and the embodiments of this application do not limit this.

[0125] The following is combined with Figure 2 , Figures 3A-3I , Figures 4A-4I This application introduces some user interfaces provided in its embodiments.

[0126] in, Figure 2 This is a schematic diagram of an indoor scene provided in an embodiment of this application. Figures 3A-3I , Figures 4A-4I 100 pairs of electronic devices provided in the embodiments of this application Figure 2 The following are some user interfaces involved in the 3D reconstruction of the indoor scene shown.

[0127] Figures 3A-3I An example is shown of an electronic device 100 performing a 3D reconstruction of an indoor scene and implementing some user interfaces involved in navigating in the reconstructed virtual scene.

[0128] Figure 3A An exemplary user interface 10 for an application menu on an electronic device 100 is shown. This user interface 10 can be used to display time, weather, one or more application icons, etc. Specifically, the user interface 10 may include: a status bar 101, calendar and weather icons 102, and one or more application icons 103. Wherein:

[0129] The status bar 101 may include one or more signal strength indicators for mobile communication signals, one or more signal strength indicators for wireless fidelity (WiFi) signals, a battery status indicator, and a time indicator. Calendar and weather icons 102 may be used to indicate the current time and weather type. One or more application icons 103 may be used to display the application icons for one or more applications. Among these, one or more application icons 103 may include a "Modeling" application icon 103A, which may be used to trigger the launch of the "Modeling" application. This "Modeling" application may be used to trigger the electronic device 100 to acquire images or videos of the target scene, obtain and display a 3D model of the target scene, and adjust the 3D model according to user operations.

[0130] It is understood that the names of the "modeling" applications are not limited in the embodiments of this application.

[0131] like Figure 3A As shown, the electronic device 100 can detect user actions, such as clicks, applied to the "Modeling" application icon 103A, and in response to these actions, the electronic device 100 displays... Figure 3B The user interface 20 shown can be a user interface provided by the "modeling" application.

[0132] It is understood that the electronic device 100 can also launch the "modeling" application and display the user interface 20 in other ways. For example, the electronic device 100 can detect the user's voice output "open the 'modeling' application", and in response to the voice command, launch the "modeling" application and display the user interface 20. This application embodiment does not limit the way the electronic device 100 launches the "modeling" application.

[0133] like Figure 3B As shown, the user interface 20 may include: an creation icon 201, a template display area 202, and a menu bar 203. Wherein:

[0134] The creation icon 201 can be used to trigger the electronic device 100 to activate the camera and acquire images or videos needed to reconstruct the 3D model. The template display area 202 can be used to display a display image of one or more 3D models. The menu bar 203 can be used to display multiple menu options. The electronic device 100 can detect user actions on the menu options and, in response to the actions, display different pages provided by the "Modeling" application in the user interface 20. For example, Figure 3B The user interface 20 shown can be the default home page provided by the "Modeling" application after the electronic device 100 launches the "Modeling" application.

[0135] When the electronic device 100 detects a user action on the creation icon 201, in response to the action, the electronic device 100 can activate the camera and display, as shown below. Figure 3C The user interface 30 shown is used to display images captured in real time by the camera.

[0136] like Figure 3C As shown, the user interface 30 may include: a preview box 301, camera mode options 302, a gallery shortcut control 303, a shutter control 304, and a camera flip control 305. Among them:

[0137] The preview frame 301 can be used to display images captured in real time by the camera of the electronic device 100. The electronic device 100 can refresh the displayed content in real time so that the user can preview the image currently captured by the camera. For example, when the user points the camera of the electronic device 100 at the camera... Figure 2 When the indoor scene is shown, Figure 3C The image displayed in preview box 301 contains a portion of the interior scene.

[0138] Camera mode option 302 may display one or more shooting mode options. These shooting mode options may include: video recording mode option 302A and photo taking mode option 302B. When a user operation on a shooting mode option is detected, the electronic device 100 can activate the user-selected shooting mode. (Not limited to...) Figure 3C As shown, camera mode option 302 can also include more or fewer other shooting mode options.

[0139] The gallery shortcut control 303 can be used to open the gallery application. In response to a user action, such as a click, on the gallery shortcut control 303, the electronic device 100 can open the gallery application. This allows users to conveniently view captured photos and videos without having to exit the camera application first and then open the gallery application. The gallery application is a picture management application on electronic devices such as smartphones and tablets, and can also be called a "photo album." This embodiment does not limit the name of the application. Additionally, the electronic device 100 can also display thumbnails of saved images in the gallery shortcut control 303.

[0140] The shutter control 304 can be used to listen for user actions that trigger a photo capture. In response to this action, the electronic device 100 can save the image in the preview box 301 as a picture in a gallery application.

[0141] The camera flip control 305 can be used to listen for user operations that trigger the camera flipping. In response to this operation, the electronic device 100 can flip the camera, for example, to switch the rear camera to the front camera.

[0142] When the electronic device 100 detects a user operation on the shutter control 304, in response to the operation, the electronic device 100 can start the camera to capture data. Figure 2 Information about the indoor scene shown, for example, that electronic device 100 can capture images of... Figure 2 Any one or more of the images or videos of the indoor scene shown will be displayed sequentially. Figure 3D , Figure 3E The user interface 30 shown.

[0143] Compared to Figure 3C The user interface 30 shown, including the gallery shortcut control 303, shutter control 304, and camera flip control 305, can be updated to... Figure 3D and Figure 3E The recording end control 306 is shown. This control 306 can be used to trigger the electronic device 100 to generate a 3D model of the currently captured indoor scene based on any one or more of the captured images or videos. Additionally, the recording end control 306 may include a progress bar indicating the recording time. The total length indicated by the progress bar can be a preset total duration by the electronic device 100, such as 10 seconds. This allows the user to easily view the current recording duration and the interval until the preset total duration, preventing the user from spending excessive time recording video.

[0144] After the user starts recording, the user can hold the electronic device 100, point the camera at the indoor scene they want to reconstruct, and move the camera to capture images of the indoor scene from different perspectives. Simultaneously, the display in the preview frame 301 will follow the user's movement, showing real-time images of the indoor scene captured by the camera from different angles, such as... Figure 3D to Figure 3E The image displayed in preview box 301.

[0145] Optionally, during video recording, the electronic device 100 may also display prompts to remind the user of precautions during the recording process. For example, if the screen in the preview frame 301 changes slowly or not at all, the electronic device 100 may display a prompt to remind the user to increase the speed of movement. If the material of the object being filmed is not conducive to scene reconstruction, the electronic device 100 may display a prompt to remind the user to avoid filming the object. For example, the material of the object could refer to glass, smoke, etc. This embodiment of the application does not limit the content indicated by the prompts.

[0146] Optionally, in addition to recording video and using it to reconstruct the 3D model, the electronic device 100 can also capture multiple images and use these images to reconstruct the 3D model. In this case, the user can hold the phone, point the camera at the indoor scene they want to reconstruct, and continuously capture images while moving the camera. For example, two consecutive images may contain a certain percentage of overlapping content. For instance, there may be 40% overlap between two consecutive images.

[0147] Furthermore, during the image capture process, the electronic device 100 can display prompts to remind the user of important information regarding the photo capture process, including when to capture an image and when to stop capturing the image. For example, while the user is holding the electronic device 100 and moving the camera, the electronic device 100 can continuously calculate the overlap ratio between the content in the currently captured image and the content in the previously captured image. When the overlap ratio is about to fall below a threshold, the device prompts the user to capture an image as soon as possible and save the currently captured image locally. Alternatively, after the overlap ratio falls below the threshold, the device prompts the user to move the camera in the opposite direction to return to the area previously scanned by the camera to capture another image.

[0148] Optionally, in addition to generating a 3D model based on images or videos captured by the electronic device 100, a 3D model can also be generated based on images or videos captured by other devices, such as professional cameras. This application embodiment does not limit the source device of the images or videos used to train the 3D model.

[0149] Additionally, it should be noted that when capturing images or videos, users can ensure that the content in the captured video or all images includes all areas of the scene they need to reconstruct, thus ensuring that the reconstructed scene includes the content of the scene they need to reconstruct.

[0150] When the recording duration of the electronic device 100 reaches the preset total duration, the electronic device 100 can detect the action acting on, such as Figure 3E The user operation of the end recording control 306 shown above allows the electronic device 100 to end video recording and display a message in the user interface 30, as indicated by the operation. Figure 3F The prompt message 307 shown is used to prompt the user whether to generate a 3D model of the real scene based on the video recorded by the user.

[0151] like Figure 3F As shown, the prompt message 307 may include: Rebuild Scene Control 307A, Continue Scan Control 307B. Among them:

[0152] The scene reconstruction control 307A can be used to trigger the electronic device 100 to obtain a 3D model of the currently captured scene. For example, the scene reconstruction control 307A can be used to trigger the electronic device 100 to send the captured image or recorded video to the server, which then trains a neural radiation field model based on the image or video to obtain an implicit 3D model of the real scene. The server then converts the implicit 3D model into an explicit 3D model and returns the explicit 3D model to the electronic device 100.

[0153] The Scan control 307B can be used to trigger the electronic device 100 to continue capturing images or recording videos, for example, by displaying... Figure 3C or Figure 3D The user interface 30 shown.

[0154] Optionally, when the computing power of the electronic device 100 is strong, the electronic device 100 can also generate a three-dimensional model locally. That is, the reconstruction scene control 307A can be used to trigger the electronic device 100 to train a neural radiation field model locally based on images or videos to obtain an implicit three-dimensional model, and then convert the implicit three-dimensional model into an explicit three-dimensional model.

[0155] When electronic device 100 detects an action such as Figure 3F The user operation of the scene reconstruction control 307A shown above, in response to which the electronic device 100 can display on the user interface 30, such as Figure 3G The prompt message 308 shown is used to inform the user that a 3D model of the currently captured indoor scene is being acquired.

[0156] After the electronic device 100 acquires a 3D model of the currently captured indoor scene, the electronic device 100 can display, as follows: Figure 3H The user interface 40 shown is used to display the 3D model.

[0157] like Figure 3H As shown, the user interface 40 may include: a preview box 401, an options bar 402, manipulation controls 403, and an exit control 404. Wherein:

[0158] The preview box 401 is used to display the 3D model from different perspectives. This 3D model is a virtual scene simulated by simulating a real-world scenario. The electronic device 100 can obtain the user's input perspective through the operation of the control 403.

[0159] The options bar 402 displays one or more options. For example, the options bar 402 may include an edit option 402A, a share option 402B, and a save option 402C. The edit option 402A can be used to trigger the electronic device 100 to edit the currently displayed 3D model, adding, adjusting, or deleting virtual objects within the 3D model. The share option 402B can be used to trigger the electronic device 100 to send the currently displayed 3D model to other devices.

[0160] The manipulation control 403 can be used to change the user's viewing angle. The electronic device 100 can detect the operation of sliding in different directions on the manipulation control 403. In response to the operation, the electronic device 100 can update the display content in the preview box 401 according to the sliding operation. For example, when the electronic device 100 detects the left sliding operation on the manipulation control 403, the electronic device 100 adds a screen of the left-side three-dimensional model (not shown) to the left side of the preview box 401 and removes the screen of the three-dimensional model that has been partially shown on the right side, giving the user an indoor roaming effect of moving the viewing angle to the left.

[0161] The exit control 404 can be used to trigger the electronic device 100 to stop displaying the 3D model. For example, when the electronic device 100 detects a user action on the exit control 404, in response to that action, the electronic device 100 can display, as shown below... Figure 3B The user interface 20 shown.

[0162] like Figure 3I As shown, the electronic device 100 can detect an operation performed by sliding the control 403 to the upper right. In response to this operation, the electronic device 100 updates the view of the adjusted 3D model in real time within the preview frame 401. Combined with... Figure 2 ,contrast Figure 3H and Figure 3I It can be seen that, compared to Figure 3H The content displayed in preview box 401 Figure 3I The content displayed in the preview box 401 presents the effect of the user observing the indoor scene after moving to the left and front in a real indoor scene.

[0163] from Figures 3A-3I As can be seen, the electronic device 100 can reconstruct a virtual scene by taking photos or videos of the target scene, and display the virtual scene as presented from the user's input perspective, allowing the user to roam and view the effect of the target scene even without arriving at the target scene.

[0164] Figures 4A-4I Examples are shown of some user interfaces involved when an electronic device 100 edits a reconstructed virtual scene.

[0165] like Figure 4A As shown, electronic device 100 can detect user operations on editing option 402A, and in response to such operations, electronic device 100 can display the following in user interface 40: Figure 4B The editing window 405 shown is used to display a 3D model that can be added to the 3D model of the interior scene displayed in the preview box 401.

[0166] like Figure 4B As shown, the editing window 405 can display one or more 3D models, including virtual objects such as speakers, cabinets, door decorations, vases, etc. These one or more 3D models can originate from the local storage of the electronic device 100, from the cloud, or be generated by the user. When the electronic device 100 detects that the user has selected one of the 3D models, the electronic device 100 can download the 3D model from the cloud to its local storage, or it can originate from other devices. This embodiment of the application does not limit the source of these one or more 3D models.

[0167] like Figure 4C As shown, the editing window 405 may include a first model 405A, which corresponds to a 3D model of a door ornament. When the electronic device 100 detects a drag operation applied to the first model 405A, from its location to the preview frame 401, in response to this operation, the electronic device 100 can add the 3D model of the door ornament to the 3D model of the interior scene and display it as shown. Figure 4D The user interface shown is 40.

[0168] like Figure 4D As shown, compared to Figure 4C The user interface 40 shown has its editing window 405 updated to an adjustment window 407, and a first virtual object 408 added to the preview box 401. Among these:

[0169] The adjustment window 407 may include a pan control 407A, a rotate control 407B, a zoom control 407C, and a return control 407D, wherein the return control 407D can be used to close the adjustment window 407. The first virtual object 408 is a 3D model of the door ornament corresponding to the first model 405A. The user can change the position, orientation, and size of the first virtual object 408 in the 3D model of the indoor scene by using the pan control 407A, rotate control 407B, and zoom control 407C in the adjustment window 407. For example, when the electronic device 100 detects a user operation on the pan control 407A, in response to the operation, the electronic device 100 can add an auxiliary coordinate axis 408A to the first virtual object 408. The auxiliary coordinate axis 408A includes three coordinate axes: X, Y, and Z. The electronic device 100 can detect a user dragging operation on one of the coordinate axes and control the first virtual object 408 to move in the direction of that coordinate axis.

[0170] It should be understood that, in addition to adjusting the first virtual object 408 based on the aforementioned translation control 407A, rotation control 407B, and scaling control 407C, the electronic device 100 can also adjust the position, orientation, and size of the first virtual object 408 in the 3D model of the indoor scene based on the user's direct operation on the first virtual object 408. This application embodiment does not limit the method of adjusting the first virtual object 408.

[0171] For example, after the electronic device 100 adjusts the position, orientation, and size of the first virtual object 408 according to the user's operation, the electronic device 100 can display as follows: Figure 4E The user interface shown is 40.

[0172] like Figure 4E As shown, compared to Figure 4D The first virtual object 408 displayed in the preview box 401 has changed in position, orientation, and size. In addition, the electronic device 100 can also change the viewing angle of the virtual interior scene according to the user's operation on the control 403, and view the effect of the added door decoration in the virtual interior scene from different angles.

[0173] like Figure 4F As shown, preview frame 401 displays the virtual indoor scene from another perspective, and the first virtual object 408 also displays its image from this perspective. Figure 4F As can be seen, after the adjustment of the door ornament, the back of the door ornament is close to the surface of the door, and the first virtual object 408 also presents the same three-dimensional effect as the virtual interior scene, just as if the user adds the door ornament to the door in the real interior scene and views the door decorated with the door ornament from a real perspective.

[0174] Furthermore, users can adjust the display effect of local or all areas of the 3D model obtained from images or videos, including: position, size, color, orientation, etc. This application embodiment does not limit the adjustments involved in the explicit 3D model.

[0175] like Figure 4G As shown, when the electronic device 100 detects a user action, such as a long press, applied to the first area 409 in the preview frame 401, in response to the action, the electronic device 100 can add, for example, to the user interface 40. Figure 4H The adjustment window 410 shown provides the user with further adjustment options, allowing the user to adjust the display effect of the first area 409. The first area is the region of the wall adjacent to the door in the generated 3D model.

[0176] like Figure 4G As shown, the adjustment window 410 may include: a position adjustment option 410A, a size adjustment option 410B, a color adjustment option 410C, and an orientation adjustment option 410D. Specifically, the position adjustment option 410A can be used to adjust the position of the first area 409, the size adjustment option 410B can be used to adjust the size of the first area 409, the color adjustment option 410C can be used to adjust the color of the first area 409, and the orientation adjustment option 410D can be used to adjust the orientation of the first area 409.

[0177] like Figure 4H As shown, when electronic device 100 detects a user operation on color adjustment option 410C, in response to the operation, electronic device 100 can change adjustment window 410 as follows: Figure 4I The color selection window 411 shown can be used to provide users with one or more colors to choose from.

[0178] like Figure 4I As shown, the color selection window 411 can display one or more color options. The color selection window 411 may include a first color option 411A. When the electronic device 100 detects a user operation on the first color option 411A, in response to that operation, the electronic device 100 can change the first area 409 to the color corresponding to the first color option 411A. For example, Figure 4I The dark gray shown. From Figure 4I It can be seen that after the electronic device 100 changes the color of the first area 409, the color of the first area 409 is the same as the color displayed in the first color option 411A.

[0179] For example, the electronic device 100 can change the color of the first region 409 by overlaying a plane of the same size as the first region 409 and of a user-specified color on top of the first region 409.

[0180] from Figures 4A-4I As can be seen, after the electronic device 100 obtains a 3D model of the indoor scene, the electronic device 100 can adjust the 3D model, including changing the 3D model itself, adding, adjusting or deleting new 3D models in the 3D model, thereby increasing the user's operability of the 3D model of the indoor scene.

[0181] Furthermore, the electronic device 100 can send the adjusted 3D model to other devices. These other devices can display the adjusted 3D model and, based on the user's input perspective, show the image presented by the adjusted 3D model from that perspective. Alternatively, other devices can overlay virtual content onto images captured in real-time by a camera; this virtual content represents the changes added to the adjusted 3D model compared to the original 3D model.

[0182] Figure 5 An example illustration shows a user capturing an indoor scene using the camera of an electronic device.

[0183] For example, after acquiring the adjusted 3D model, the electronic device 200 can compare the adjusted 3D model with the virtual content added before the adjustment, such as the original 3D model trained based on images or videos. When the electronic device 200 displays an indoor scene captured by a camera, the virtual content is superimposed on the indoor scene, so that when the user displays the real indoor scene through the screen, the position of the virtual content in the indoor scene is the same as or similar to the position of the virtual content in the virtual scene.

[0184] like Figure 5 As shown, the display screen of electronic device 200 shows images captured by the camera of electronic device 200. Figure 2 The indoor scene shown is an image captured in real time, overlaid with a virtual door decoration and dark gray "wallpaper". The position of the door decoration on the door displayed on the electronic device 200 is consistent with... Figure 4E The door decoration shown in the virtual scene is positioned the same as the door on the door, and the "wallpaper" is positioned the same as the one captured by the electronic device 200 in the image. Figure 4I The position of the first area 409 is the same in the virtual scene shown.

[0185] It should be noted that the electronic device 200 is a device that possesses the adjusted 3D model. It can be the electronic device 100 or other devices different from the electronic device 100. When the electronic device 200 is another device, the electronic device 100 can send the adjusted 3D model to the other device so that the other device can overlay and display the adjusted 3D model on the image captured by the camera, showing the changes added compared to the original 3D model.

[0186] As can be seen, after adjusting the virtual scene corresponding to the target scene—for example, adding a virtual object to a certain location within the virtual scene—when a user views the target scene through a real-time image captured by the electronic device's camera, they can see the virtual object at the same location within the target scene, or near that location. In this way, the changes made by the user in the virtual scene directly affect the image of the target scene captured and displayed by the electronic device, effectively bringing the virtual world into the real world.

[0187] Additionally, it should be noted that the above... Figures 3A-3I ,as well as Figures 4A-4I The three-dimensional model mentioned is an explicit three-dimensional model, which is a three-dimensional model converted from the implicit three-dimensional model obtained by training based on the neural radiation field. For a detailed description of the explicit and implicit three-dimensional models, please refer to the subsequent method flow, which will not be elaborated here.

[0188] The three-dimensional reconstruction method provided in the embodiments of this application will be described in detail below.

[0189] Figure 6 This is a flowchart illustrating the three-dimensional reconstruction method provided in an embodiment of this application.

[0190] like Figure 6 As shown, this method involves multiple devices, including: mobile device 1, mobile device 2, and a server. The mobile devices, such as mobile device 1 and mobile device 2, are user-facing devices capable of updating and displaying a 3D model in real time based on the user's input perspective. For example, mobile device 1 and mobile device 2 can be mobile phones, tablets, laptops, computers, etc. The server is a device for training neural radiation field models to generate 3D models for target scenes. In this embodiment, mobile device 1 can be the aforementioned electronic device 100, mobile device 2 can be the aforementioned electronic device 200, and the server can be the aforementioned server 100. For details regarding the communication networks involved in electronic devices 100, 200, and server 100, please refer to the aforementioned... Figure 1 The relevant description includes the following: The three-dimensional reconstruction method may include:

[0191] S101. Mobile device 1 acquires multiple images for the target scene.

[0192] A target scene refers to a physical space in a real environment. For example, this physical space can be the interior space of a building. The target scene could be a living room scene, an office scene, a library scene, etc.

[0193] These multiple images can be a sequence of consecutive images obtained when the mobile device 1 activates its camera and captures a target scene from different perspectives. Any two consecutive images in this sequence contain a certain proportion of overlapping content. Alternatively, these multiple images can be images sampled from multiple frames of a video recorded by the mobile device 1 using its camera to record a video of the target scene from different perspectives.

[0194] In other words, mobile device 1 can activate its camera to capture images or videos of a target scene, thereby obtaining multiple images of that scene. Specifically, when mobile device 1 captures video of the target scene using its camera, it can sample the video to obtain these multiple images. For example, video processing software can be used to sample the video to obtain these multiple images.

[0195] Optionally, the multiple images may also be images captured by other devices or images sampled from recorded videos. For example, the other devices may include cameras, mobile phones, tablets, etc. This application embodiment does not limit the source devices of these multiple images, nor the number of source devices.

[0196] The three-dimensional reconstruction method can be implemented by an application in this embodiment. For example, the application can be the aforementioned "modeling" application. The name of the application is not limited in this embodiment.

[0197] For example, such as Figure 3B As shown, mobile device 1 can display a user interface 20 of the "modeling" application. This user interface 20 can display a creation icon 201. Mobile device 1 can detect user actions on the creation icon 201. In response to this action, mobile device 1 can activate the camera and display... Figure 3C The user interface 30 shown may display controls for taking photos or recording videos, such as shutter control 304, and preview box 301, which is used to display images captured by the camera in real time. The mobile device 1 can trigger taking photos or recording videos based on the user pointing the camera at the target scene and acting on the control, thereby obtaining multiple images or videos of the target scene.

[0198] In this embodiment, the user interface 20 can also be referred to as the second user interface, the creation icon 201 can also be referred to as the first control, the user operation on the creation icon 201 can also be referred to as the first operation, the user interface 30 can also be referred to as the third user interface, the preview box 301 can also be referred to as the first preview box, the shutter control 304 can also be referred to as the second control, and the operation on the shutter control 304 can also be referred to as the second operation.

[0199] In some embodiments, when the mobile device 1 takes photos or videos of a target scene, the mobile device 1 can output prompts to remind the user of precautions regarding taking photos or videos, such as when to start shooting, when to stop shooting, and how to control the speed at which the user moves the camera. These prompts can manifest as inputting content on the display screen, vibration, or playing audio, etc. This application embodiment does not limit the form in which the prompts are displayed. For a detailed description of the prompts output by the mobile device 1, please refer to the foregoing. Figure 3E The relevant content will not be repeated here.

[0200] In this embodiment of the application, the mobile device 1 may also be referred to as the first device.

[0201] S102. Mobile device 1 sends multiple images to the server.

[0202] Mobile device 1 sends multiple images to the server, and the server receives the multiple images sent by mobile device 1.

[0203] Optionally, when mobile device 1 records a video of the target scene, mobile device 1 can send the video to the server, and the server will receive the video sent by mobile device 1. Additionally, the server can sample multiple images of the target scene from multiple frames of the video.

[0204] In addition, the server can also obtain images or videos of the target scene taken by other devices, and use the images or videos sent by one or more devices to complete the model training process mentioned later.

[0205] For example, such as Figure 3F As shown, after mobile device 1 completes the shooting of the target scene, it can display the following: Figure 3F The user interface 30 shown allows the mobile device 1 to detect user actions performed on the scene reconstruction control 307A within the user interface 30. In response to these actions, the mobile device 1 can send the multiple images to the server, or the mobile device 1 can send a recorded video of the target scene to the server.

[0206] S103. The server calculates the camera pose corresponding to the image.

[0207] Camera pose refers to the position and orientation of a camera in the real environment when capturing a target scene. In other words, the camera pose reflects the position and orientation of the camera when a mobile device takes a photo or records a video frame. One image corresponds to one camera position, and the server can calculate the camera pose for multiple images.

[0208] Figure 7 This diagram illustrates the shooting scenarios involved when a camera captures a target scene from multiple perspectives. For example... Figure 7 As shown, object 100 (i.e. Figure 7 The rectangle in the image represents the target scene to be photographed, and icon 101 indicates the camera pose when the camera captures the target scene. Vertex 'a' represents the location of the camera, and the orientation of icon 101 indicates the camera's facing direction. The camera can capture the target scene from the camera pose shown by icon 101, obtaining an image of the target scene from one of its viewpoints.

[0209] It should be understood that, Figure 7 There can be multiple icons 101 in the image. Only three icons 101 are marked here for explanation. Other similar shapes are also icons 101, but they will not be marked one by one here.

[0210] For the same scene, different camera poses (i.e., different shooting angles) will result in different content of the target scene in the acquired images. By collecting and analyzing the content of the target scene presented from different viewing angles, the camera pose when the images were captured from different perspectives can be calculated.

[0211] Specifically, Figure 8 This diagram illustrates the process involved in calculating camera pose using multiple images. Figure 8 As shown, the server calculates the camera pose of an image, which mainly involves the following steps:

[0212] S201. For any one of these multiple images, extract the feature points in that image.

[0213] Feature points are pixels in an image that are scale-invariant. For example, feature points can be points in an image where the grayscale value changes drastically or points with significant curvature at the image edges. Feature points can reflect the essential features of an image and identify objects within it.

[0214] In some possible cases, feature points in each frame of the image can be extracted based on algorithms such as Scale-invariant feature transform (SIFT) or Speeded-Up Robust Features (SURF). Other algorithms can also be used to extract feature points in each frame of the image, such as Oriented Fast and Rotated BRIEF (ORB) algorithms, etc., but this application does not limit the specific methods used.

[0215] S202. Perform feature point matching on any two adjacent images from these multiple images.

[0216] Two adjacent images can refer to two images taken consecutively, or two consecutive frames extracted from a video. These two adjacent images have a certain proportion of overlapping content, that is, they contain the same feature points. Feature point matching between two images means matching the same feature points in the two images, that is, identifying the common feature points in the two frames.

[0217] S203. Calculate the camera pose using the matched feature points.

[0218] The process of calculating the camera pose is also the process of constructing a feature point map. This feature point map contains the coordinate information of the three-dimensional points of the target scene in real space, as well as the camera pose of each captured image. This feature point map can basically reflect the three-dimensional shape and geometric positional relationships of the target scene.

[0219] Specifically, firstly, the relative camera pose of the two cameras can be calculated using matched feature points. Secondly, the coordinates of the matched feature points in 3D space are determined using the relative camera pose; this 3D point is a spatial point within the target scene in real 3D space. For example, Figure 9 This is a schematic diagram illustrating the association between two images and three-dimensional points in space, provided for an embodiment of this application. For example... Figure 9 As shown, feature point p in image 1 matches feature point p' in image 2. Both feature points correspond to a 3D point p0 in the real 3D space. Finally, the camera pose corresponding to the image is calculated using the 3D point and its corresponding feature point in the image. Furthermore, during the construction of the feature point map, some prior parameters can be used to calibrate the coordinates of the 3D points and the camera pose, achieving alignment of the constructed feature point map with the position, angle, and scale of the target scene in the real 3D space, thus obtaining a more accurate camera pose.

[0220] The prior parameters include, but are not limited to, information such as position, angle, and depth. Position information may include: the position information of mobile device 1 when capturing images or videos of the target scene, or the position information of spatial points in the target scene. For example, mobile device 1 can use a Global Positioning System (GPS) to obtain its position and a distance sensor to obtain the position information of spatial points in the target scene. Depth information may be the depth information of objects in the target scene determined by LiDAR, depth sensors, AI depth estimation, etc., when mobile device 1 captures images or videos of the target scene. Angle information may include the attitude information of mobile device 1 measured by an Inertial Measurement Unit (IMU). In other words, the prior parameters may include one or more of the following: the position and attitude of the camera capturing the image, the position of objects captured in the image, and the depth information of the image, etc.

[0221] In other words, the server can use multiple captured images and prior parameters to determine the camera pose corresponding to each image.

[0222] It is understood that the device for obtaining the prior parameters of the target scene is not limited to mobile device 1, but can also be other professional devices used to specifically measure location information, depth information or angle information, such as depth cameras, total stations, etc. In addition, the method of obtaining prior parameters is not limited to the methods mentioned above. For example, it can also be combined with high-precision maps and use RTK positioning technology to determine location information. This application embodiment does not limit the method of obtaining prior parameters.

[0223] In some possible implementations, the COLMAP tool can be used to estimate the camera pose for each image using the structure from motion (SFM) algorithm.

[0224] It should be understood that by aligning the feature point map with the position, orientation, and scale of the target scene, the server can make the calculated camera pose closer to the actual position and orientation of the camera during shooting. Using this camera pose as training data in step S104 ensures that the trained 3D model is also aligned with the position, orientation, and scale of the target scene. This allows for subsequent adjustments to the explicit 3D model by the mobile device, such as adding model assets to the explicit 3D model. When the mobile device views the target scene through real-time images captured by the camera, the position or angle of the model assets superimposed on the image will have a smaller error compared to the corresponding position or angle of the model assets in the explicit 3D model. For details, please refer to the aforementioned... Figures 4C-4I , Figure 5 And its related descriptions.

[0225] S104. The server uses multiple images and their corresponding camera poses as training data to train the neural radiation field model, thus obtaining a trained implicit 3D model.

[0226] The neural radiation field model is a multilayer perceptron model, which is essentially a polynomial function represented by a neural network. As can be seen from Equation 1, the neural radiation field model F... Θ The input is a 5-dimensional vector (x, y, z, θ, φ), where the first three dimensions X (x, y, z) are the coordinates of a spatial point in the target scene, and the last two dimensions d = (θ, φ) are the viewing direction or light direction, where θ represents the viewing pitch angle and φ represents the viewing horizontal angle. The output is a 4-dimensional vector (R, G, B, σ), where the first three dimensions c = (R, G, B) represent the color of the spatial point, σ represents the spatial density of the spatial point, and σ can be approximated as opacity, with smaller values ​​indicating greater transparency.

[0227] F Θ Formula 1: (X,d)→(c,σ)

[0228] As can be seen from Formula 1, this neural radiation field model can realize the mapping of the coordinates and observation direction of a spatial point to the color and spatial density of that spatial point.

[0229] Figure 10 This is a schematic diagram illustrating the reconstruction of a 3D model based on a neural radiation field model provided in this application embodiment. R1 and R2 are two rays with different directions, and one ray can correspond to the observation angle of the camera at a certain position coordinate. Image 1 shows the target observed through the observation direction of R1 (i.e., Figure 10 Image 1 is the image observed when viewing the target through the viewing direction of R2 (the gray cube in the image). Image 2 is the image observed when viewing the target through the viewing direction of R2. From Figure 10As can be seen, a ray can pass through multiple spatial points in the target scene, that is, a ray includes multiple spatial points. The color and volume density of all spatial points on the ray together constitute the pixel color observed in the observation direction of the ray.

[0230] Specifically, the training process of the neural radiation field model includes:

[0231] 1) Construct model training vectors (x, y, z, θ, φ) for each spatial point based on the coordinates (x, y, z) and the corresponding observation direction (θ, φ) of each spatial point;

[0232] 2) During model training, the model training vector is input into the initialized neural radiation field model to obtain the color and volume density of each spatial point;

[0233] 3) Based on the color and volume density of all spatial points along a ray, estimate the observed pixel color in the observation direction of the ray;

[0234] The pixel color of any ray in the image can be estimated based on Equation 2:

[0235]

[0236] i,j represent the i-th and j-th sampling points, c i σ represents the color of the i-th spatial point on the ray. i δ represents the volume density at the i-th spatial point. i δ j This indicates the sampling interval for spatial points.

[0237] 4) Using the preset loss function, see Formula 3, calculate the output loss of the current neural radiation field model based on the estimated pixel color and the true value of the pixel color represented by the image obtained under the observation direction.

[0238]

[0239] Where L represents the output loss of the current neural radiation field model F(·), C represents the estimated pixel color of the ray r corresponding to a pixel in the i-th image after querying the RGB color and spatial density σ of the spatial point in the current neural radiation field model F(·). (i) (r) is the true value of the pixel color of this pixel as collected by the device, and R is the sampled ray set.

[0240] 5) To reduce output loss, adjust the model parameters Θ of the current neural radiation field model according to the backpropagation gradient to obtain the implicit 3D model of the current target scene.

[0241] In addition, during the training of the neural radiation field model, multiple images of the target scene and the corresponding camera poses are used as training data. The input-output relationship between these training data and the neural radiation field model is as follows: the coordinates of the spatial points in the target scene corresponding to the pixels in the image are determined by the image and the corresponding camera pose. The viewing direction is indicated by the camera position when the image was captured, pointing to the pixels in the image. The pixel value of the observed pixel in the image can be obtained by integrating the color and density of one or more spatial points in the target scene in a viewing direction.

[0242] S105. The server converts the implicit 3D model into an explicit 3D model.

[0243] Since the neural radiation field model obtained through direct training is an implicit 3D model, rendering this implicit 3D model has high computational and spatial complexity, making it unsuitable for direct rendering on mobile devices. Therefore, before sending it to mobile devices, this implicit 3D model needs to be converted into an explicit 3D model that can be rendered in real time by mobile devices.

[0244] Specifically, converting an implicit 3D model into an explicit 3D model includes:

[0245] 1) Use spherical harmonic functions to represent the relationship between the input direction d and the output color c in an implicit 3D model.

[0246] Spherical harmonics (SH) are special functions that are solutions to the Laplace equation for a sphere. A spherical harmonic function can be illustrated by Equation 4:

[0247]

[0248] Where [·] are the basis functions of the spherical harmonic function. Let be the spherical harmonic coefficients of the basis functions. These spherical harmonic coefficients are parameters of the basis functions, and the spherical harmonic functions can be reconstructed by obtaining these spherical harmonic coefficients.

[0249] In this embodiment, a spherical harmonic function can be used to fit the color of a spatial point in a target scene under different viewing directions. Specifically, the server can use a spherical harmonic function to fit the color output by the implicit 3D model of a spatial point in the target scene under different viewing directions.

[0250] In this way, spherical harmonic functions can be used to represent the relationship between input direction and output color in the implicit 3D model. When rendering the model, it is not necessary to input the coordinates of the spatial point and a certain viewing direction into the implicit 3D model to obtain the color of the spatial point in that viewing direction. The spherical harmonic function can be used to represent the color of the spatial point in different viewing directions, avoiding the trouble of querying the color of the spatial point through the implicit 3D model when rendering the 3D model. The query result can be directly represented by the spherical harmonic function, which speeds up the process of obtaining the color of the spatial point when rendering the 3D model.

[0251] 2) Presample the target scene to obtain multiple presampled points corresponding to the target scene.

[0252] Specifically, the server can perform uniform sampling on the target scene to obtain multiple pre-sampling points corresponding to the target scene.

[0253] 3) Determine the density value of each presampling point based on the implicit 3D model.

[0254] Specifically, the server can determine the density value of each presampling point based on the input and output information of each presampling point in the implicit 3D model.

[0255] 4) Filter the presampling points based on their density values.

[0256] Filtering presampled points refers to deleting presampled points with lower density values ​​from among multiple presampled points. For example, removing presampled points with density values ​​less than a threshold τ. w Sampling points with density values ​​greater than the threshold τ are retained. w The sampling points are reduced. This reduces the number of spatial points described by the implicit 3D model, thus achieving a lightweight 3D model.

[0257] 5) Use an octree to store the density values ​​and spherical harmonic coefficients of the filtered pre-sampling points.

[0258] An octree is a tree-like structure that describes three-dimensional space. Each node in an octree represents a volume element of a cube, and each node has eight child nodes. The volume elements represented by the eight child nodes are added together to equal the volume of the parent node.

[0259] After filtering out some sample points of multiple pre-sampled points corresponding to the target scene, these remaining pre-sampled points can be divided into blocks according to the preset block division interval to obtain the spatial octree corresponding to the target scene. Figure 11 An exemplary diagram illustrates the segmentation of a target scene into blocks. Wherein, Figure 11 The gray cube shown represents the target scene.

[0260] In this octree, the leaf nodes store the density values ​​and spherical harmonic coefficients of the presampled points corresponding to those leaf nodes. This octree-based octree data is used to characterize the explicit 3D model.

[0261] It should be understood that the use of an octree to describe the explicit 3D model is due to the fact that the leaf nodes of the octree have the characteristic of representing the relative positional relationship between themselves and other leaf nodes. In this way, when the device renders the explicit 3D model, it does not need to determine the position of the spatial point in the actual physical space based on the coordinates of the spatial point. It can simply restore the image of the target scene presented on the device based on the relative positional relationship of these spatial points.

[0262] In some embodiments, since the density values ​​and spherical harmonic coefficients are large data sets, the leaf nodes of the octree can be used to store index information. This index information can be used to find the density values ​​and spherical harmonic coefficients of the presampled points corresponding to the leaf node. This simplifies the complexity of the octree data.

[0263] Furthermore, the server can store the octree data of the explicit 3D model as texture maps, including indexed maps and model parameter maps, improving the speed of rendering explicit 3D models on mobile devices. Among these, Figure 12 This is a schematic diagram of the texture mapping provided in an embodiment of this application. Figure 12 (a) illustrates a model parameter map. Figure 12 (b) illustrates an indexed texture map. Specifically, the server can treat the index information of all leaf nodes of the octree as pixel values ​​in an image and store this index information as an indexed texture map. Additionally, the server can treat the density values ​​and spherical harmonic coefficients of the presampled points corresponding to the leaf nodes of the octree as pixel values ​​in an image and store these density values ​​and spherical harmonic coefficients as a model parameter texture map. Therefore, when the server sends an explicit 3D model to a mobile device, it can directly send the texture map to the mobile device. The mobile device can obtain information about the explicit 3D model through this texture map and render the explicit 3D model.

[0264] In summary, converting an implicit 3D model into an explicit 3D model involves pre-sampling the target scene, representing the sampled scene using a spherical harmonic function, and storing the information of the explicit 3D model using an octree data structure. This approach gives explicit 3D models at least three advantages over implicit 3D models:

[0265] 1) Explicit 3D models can directly describe the information of spatial points in the target scene, including density and color under different viewing directions. Implicit 3D models only represent the mapping relationship between the coordinates and viewing direction of spatial points and color and density. In actual rendering of 3D models, the coordinates and viewing direction of spatial points need to be input into the implicit 3D model in order to obtain the color and density of the spatial points.

[0266] In practice, explicit 3D models directly describe the information of spatial points in the target scene by using spherical harmonic functions to characterize the color of spatial points and by directly calculating and storing the density of each spatial point.

[0267] 2) Explicit 3D models only describe information about some spatial points in the target scene, making them more lightweight than implicit 3D models.

[0268] In practice, by pre-sampling the target scene and removing spatial points with low density values, the number of spatial points represented by the explicit 3D model is reduced.

[0269] 3) Explicit 3D models describe the coordinates of each spatial point by describing the relative positional relationship between each spatial point.

[0270] In the specific implementation, the information of the explicit 3D model is stored through the octree data results. The relative positional relationship of each spatial point is represented by the relative positional relationship of each leaf node. In this way, when the device renders the explicit 3D model, it does not need to determine the position of the spatial point in the actual physical space based on the coordinates of the spatial point. It can simply restore the image of the target scene presented on the device based on the relative positional relationship of these spatial points.

[0271] It can be seen that converting the implicit 3D model into an explicit 3D model achieves lightweighting and real-time rendering of the model. This avoids forward inference of the implicit 3D model when rendering the 3D model, improves rendering efficiency, shortens the latency of displaying the 3D model on mobile devices, and enhances the user experience.

[0272] S106. The server sends the explicit 3D model to the mobile device 1.

[0273] The server sends an explicit 3D model to mobile device 1, and mobile device 1 receives the explicit 3D model sent by the server.

[0274] Additionally, it should be noted that besides sending the explicit 3D model to mobile device 1, the server can also send the explicit 3D model to other devices, such as mobile device 2. This application embodiment does not limit the devices to which the server sends the model.

[0275] Specifically, sending the explicit 3D model to mobile device 1 means that the server sends a texture map to mobile device 1, whereby the texture map contains information about the explicit 3D model. For a detailed description of the texture map, please refer to the relevant content in step S105 above, which will not be repeated here.

[0276] S107. Mobile device 1 renders the explicit 3D model in real time.

[0277] The real-time rendering of the explicit 3D model by the mobile device 1 refers to the view (e.g., the first view) presented by the explicit 3D model when displayed on the mobile device 1 from a certain viewing angle (e.g., the first viewing angle). This viewing angle can be a default view preset by the mobile device 1, and this embodiment does not limit it.

[0278] For example, mobile device 1 can render the explicit 3D model in real time using a graphics processing unit (GPU).

[0279] Specifically, after the mobile device 1 obtains the texture map representing the explicit 3D model, it can store the texture map in the GPU texture unit. When the mobile device 1 performs shading calculations through the fragment shader in the GPU, it can directly obtain the texture map from the GPU texture unit and complete the real-time rendering of the explicit 3D model based on the texture map, thereby improving the efficiency of the mobile device in rendering the explicit 3D model.

[0280] For any ray of light from a camera directed towards a target scene, the color and spatial density of all spatial points along the path of that ray can be predicted using a neural radiation field model. The color of the corresponding pixel can be obtained by integrating the color and density of the spatial points along that path. Real-time rendering of an explicit 3D model involves integrating the color and density along the paths of multiple rays directed towards the target scene to obtain the colors of multiple pixels, thus constructing an image of the target scene presented on the device from a single viewpoint. Specifically, the color and density along the path of the ray can be used to determine the corresponding spatial point position based on the viewing direction of the ray. The index value corresponding to this spatial point position is obtained through an index map, and the spherical harmonic coefficients and density corresponding to this index value are obtained through a model parameter map. The color function expression for the spatial point is then reconstructed based on the spherical harmonic coefficients, i.e., the spherical harmonic function, and the color under that viewing direction is determined based on the spherical harmonic function.

[0281] For example, see Figure 3G The preview box 401 of the user interface 40 shows the screen presented by the real-time rendered explicit 3D model of the mobile device 1.

[0282] In this embodiment of the application, the user interface 40 may also be referred to as the first user interface.

[0283] It should be noted that the image presented by the explicit 3D model is basically consistent with the image of the target scene directly observed by the user. For any object presented in the explicit 3D model, the user can find a corresponding object in the target scene in the real environment. The difference is that the objects displayed by the mobile device 1 are virtual objects reconstructed by the computer based on multiple images captured, corresponding to the real objects in the target scene. The user can view the object from different perspectives by specifying any angle.

[0284] S108. Mobile device 1 receives an operation from the user to adjust the viewing angle.

[0285] For example, see Figure 3H This operation of adjusting the viewing angle can be a user operation applied to the manipulation control 403. For example, the mobile device 1 receives an operation from the user to adjust the first viewing angle to the second viewing angle.

[0286] S109. The mobile device 1 displays the image presented by the explicit three-dimensional model from the viewing perspective.

[0287] Mobile device 1 can display the view (e.g., the second view) of the explicit 3D model presented from the viewing perspective (e.g., the second viewing perspective).

[0288] Similar to step S108, after changing the viewing angle, mobile device 1 can render an explicit 3D model in real time from that viewing angle. For a detailed description of mobile device 1's real-time rendering of the explicit 3D model, please refer to the relevant content of step S107 above, which will not be repeated here.

[0289] As can be seen, mobile device 1 can receive the user's operation of continuously adjusting the viewing angle, and continuously adjust the image presented by the explicit 3D model under different viewing angles, thereby achieving the effect of immersive roaming of the user in the virtual scene corresponding to the target scene. The user can observe the virtual scene from any angle by controlling the viewing angle.

[0290] It should be understood that when the user does not adjust the viewing angle, the mobile device 1 can always display the explicit 3D model under the viewing angle mentioned in step S107, and in this case, steps S108-S109 are optional steps.

[0291] S110. Mobile device 1 receives an operation from the user to adjust the explicit 3D model.

[0292] This operation can be used to add, modify, or delete model materials in the explicit 3D model. Modifications to model materials can include changing their position, size, placement angle, color, etc. The model materials can be objects in the explicit 3D model, model materials pre-stored on the mobile device 1, or model materials sent to the mobile device 1 by other devices (such as a cloud server). This application embodiment does not limit the source of the model materials.

[0293] For example, see Figures 4A-4C The electronic device 100 detected a user operation on the editing option 402A. Figure 4B The display window 405 contains one or more 3D models. Operations that adjust the explicit 3D models can include actions performed on... Figure 4C The first model 405A shown is dragged from its location to the preview frame 401. Alternatively, see, for example, [link to example]. Figure 4G-Figure 4I The operation of adjusting the explicit 3D model includes Figure 4G The user operation shown applies to the first area 409 in the preview frame 401, and Figure 4H The user operation shown applies to the color adjustment option 410C, and Figure 4I The user action shown applies to the first color option 411A.

[0294] In this embodiment of the application, the editing option 402A can also be referred to as a third control, the first model 405A can also be referred to as a first model icon, and the operation of adjusting the explicit 3D model can also be referred to as a fourth operation.

[0295] S111. Mobile device 1 adjusts and displays the adjusted explicit 3D model.

[0296] In response to the operation of adjusting the explicit 3D model, the mobile device 1 can adjust and display the adjusted explicit 3D model. Specifically, the mobile device 1 can display the view presented by the adjusted explicit 3D model from a certain viewing angle.

[0297] For example, see Figures 4C-4D , Figure 4D The first virtual object 408 shown is the mobile device 1 receiving, as... Figure 4C The virtual object added to the explicit 3D model after the operation shown. See, for example, [link to example]. Figures 4H-4I , Figure 4I The color of the first area 409 shown is follow. Figure 4H The user action shown indicates the changed color.

[0298] Additionally, it should be noted that during the process of adjusting the explicit 3D model on mobile device 1, mobile device 1 needs to determine the placement position, angle, etc. of the adjusted model material in the explicit 3D model based on the user's operation, and then determine the placement position and angle of the model material in the real physical space. This enables the model material to be superimposed at a specified angle and displayed at a specified position in the image when the target scene image captured in real time by the camera is displayed. The specified angle and position correspond to the angle and position of the model material in the explicit 3D model.

[0299] S112. Mobile device 1 sends the adjusted explicit 3D model to the server.

[0300] Mobile device 1 sends the adjusted explicit 3D model to the server, and the server receives the adjusted explicit 3D model sent by mobile device 1.

[0301] In this embodiment, mobile device 1 can send the adjusted explicit 3D model to the server in real time while the user is adjusting the model, or mobile device 1 can send the adjusted explicit 3D model to the server after receiving the user's operation. This application does not limit the timing of when mobile device 1 sends the adjusted explicit 3D model.

[0302] S113. The server sends the adjusted explicit 3D model to the mobile device 2.

[0303] The server can send the adjusted explicit 3D model to mobile device 2, and correspondingly, mobile device 2 can receive the adjusted explicit 3D model sent by the server.

[0304] It is understood that, in addition to forwarding the adjusted explicit 3D model from mobile device 1 via a server, mobile device 1 can also directly send the adjusted explicit 3D model to mobile device 2. Mobile device 2 can be a device that has established a communication connection with mobile device 1, or a device belonging to the same user account or group as mobile device 1. This application embodiment does not limit the method by which mobile device 2 obtains the adjusted explicit 3D model.

[0305] In this embodiment of the application, the mobile device 2 may also be referred to as a second device.

[0306] S114. Mobile device 2 receives a user's command to turn on the camera.

[0307] S115. Mobile device 2 activates its camera to capture images of the target scene.

[0308] After the camera of mobile device 2 is activated, when the user points the camera of mobile device 2 towards the target scene, mobile device 2 can capture images of the target scene.

[0309] S116. The mobile device 2 displays an image captured by the camera and overlays the adjusted explicit 3D model onto the image.

[0310] Specifically, after the mobile device 2 acquires the adjusted explicit 3D model, it can compare the added virtual model in the adjusted explicit 3D model with the original explicit 3D model. This virtual model is then identified as the model that the mobile device 2 can overlay on the image when displaying the image captured in real time by the camera. This virtual model can be a 3D model, a curved surface, a plane, or lines, etc. The original explicit 3D model is the one mentioned in step S107, which is the explicit 3D model trained from multiple images of the target scene.

[0311] Since the explicit 3D model is aligned with the position, orientation, and scale of the target scene, when the mobile device 2 displays the model added to the adjusted explicit 3D model, the position of the model displayed on the image captured in real time by the camera corresponds to its position in the explicit 3D model. For example, if a user places a cube model at point A of the model corresponding to the target scene, then when the user faces the target scene, the cube model will be displayed at point A of the target scene captured by the camera.

[0312] It should be understood that the position of the added model in the target scene displayed in the image may have a certain positional error compared to its position in the corresponding explicit 3D model of the target scene. For example, this positional error may be within 0.5 meters. For instance, if a user adjusts the explicit 3D model of the target scene and places model B at the location of object A in the explicit 3D model, then when the user views the target scene in the real environment through a camera, model B will be superimposed on and displayed within 0.5 meters of object A in the target scene captured by the user. Furthermore, the orientation of the added model in the target scene displayed in the image may have a certain angular error compared to its orientation in the corresponding explicit 3D model of the target scene. For example, this angular error may be within 5°. For instance, if a user adjusts the explicit 3D model of the target scene and places model C in the explicit 3D model, and the orientation of model C is the same as that of an original object D in the explicit 3D model, then when the user views the target scene in the real environment through a camera, the angle at which model C is placed may deviate from the angle at which object D is placed in the user's photograph within 5°.

[0313] For example, Figure 5 The image of the target scene captured by the mobile device 2 is shown, as well as the virtual objects superimposed on the target scene.

[0314] As can be seen, when mobile device 2 is near the target scene, mobile device 2 can turn on the camera and overlay the model added by the user when adjusting the explicit 3D model on the image captured by the camera in real time.

[0315] It should be understood that after acquiring the adjusted explicit 3D model, mobile device 2 can, in addition to overlaying the added model into the image captured by the camera, directly display the adjusted explicit 3D model. In other words, after mobile device 1 adjusts the explicit 3D model, other devices, such as mobile device 2, can also display the adjusted explicit 3D model. This allows users not only to adjust the explicit 3D model but also to share it with other users, expanding the application scenarios of 3D reconstruction and enhancing user engagement in the process.

[0316] Figure 13 A schematic diagram of the hardware structure of electronic device 300 is shown.

[0317] The electronic device 300 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, and / or smart city device. This application embodiment does not impose any special limitations on the specific type of the electronic device. In this application embodiment, the electronic device 300 can be the aforementioned mobile terminal device 1 or mobile terminal device 2.

[0318] Electronic device 300 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, a depth sensor 180N, etc.

[0319] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 300. In other embodiments of this application, the electronic device 300 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0320] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0321] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0322] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0323] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 300. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0324] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, display 194, camera 193, and wireless communication module 160, etc.

[0325] The wireless communication function of electronic device 300 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0326] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0327] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 300. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0328] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194.

[0329] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 300, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, demodulates and filters the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, frequency modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0330] In some embodiments, antenna 1 of electronic device 300 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 300 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0331] Electronic device 300 implements display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0332] The display screen 194 is used to display images, videos, etc. In some embodiments, the electronic device may include one or N display screens 194, where N is a positive integer greater than 1.

[0333] Electronic device 300 can achieve shooting function through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0334] The ISP (Image Signal Processor) processes data fed back from the camera. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing, transforming it into a visible image. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. Furthermore, the ISP can optimize parameters such as exposure and color temperature for the shooting scene.

[0335] Camera 193 is used to capture still images or videos. An object passes through the lens to generate an optical image that is projected onto a photosensitive element. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, the electronic device 300 may include one or N cameras 193, where N is a positive integer greater than 1.

[0336] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the electronic device 300 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0337] Video codecs are used to compress or decompress digital video. Electronic device 300 may support one or more video codecs.

[0338] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0339] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0340] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0341] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0342] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 300. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0343] Electronic device 300 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0344] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Audio module 170 can also be used for encoding and decoding audio signals.

[0345] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. Electronic device 300 can listen to music or make hands-free calls through the speaker 170A.

[0346] The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals.

[0347] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C.

[0348] The 170D headphone jack is used to connect wired headphones.

[0349] The pressure sensor 180A is used to sense pressure signals and can convert pressure signals into electrical signals.

[0350] The gyroscope sensor 180B can be used to determine the motion attitude of the electronic device 300. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the electronic device 300 about three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shake of the electronic device 300, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the electronic device 300 by moving in the opposite direction, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing game scenarios.

[0351] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 300 calculates altitude using the air pressure value measured by the barometric pressure sensor 180C to assist in positioning and navigation.

[0352] The magnetic sensor 180D includes a Hall sensor. The electronic device 300 can use the magnetic sensor 180D to detect the opening and closing of the flip cover.

[0353] The 180E accelerometer can detect the magnitude of acceleration of an electronic device 300 in various directions (typically three axes). When the electronic device 300 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of the electronic device, and can be applied to applications such as screen orientation switching and pedometers.

[0354] A distance sensor 180F is used to measure distance. Electronic device 300 can measure distance via infrared or laser. In some embodiments, during a shooting scene, electronic device 300 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0355] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The electronic device 300 emits infrared light outward through the LED. The electronic device 300 uses the photodiode to detect infrared reflected light from a nearby object. When sufficient reflected light is detected, it can be determined that an object is near the electronic device 300. When insufficient reflected light is detected, the electronic device 300 can determine that no object is near the electronic device 300.

[0356] An ambient light sensor 180L is used to sense the ambient light intensity. Electronic device 300 can adaptively adjust the brightness of display screen 194 according to the sensed ambient light intensity.

[0357] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 300 can utilize the characteristics of the collected fingerprints to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0358] The 180J temperature sensor is used to detect temperature.

[0359] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can then transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194.

[0360] The bone conduction sensor 180M can acquire vibration signals.

[0361] The depth sensor 180N is used to acquire depth information, which is used to characterize the distance between environmental objects and electronic device 300.

[0362] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 300 can receive button input and generate key signal inputs related to user settings and function control of electronic device 300.

[0363] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback.

[0364] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0365] The SIM card interface 195 is used to connect the SIM card.

[0366] When electronic device 300 is mobile device 1:

[0367] The processor 110 can be used to render an explicit 3D model from a certain viewpoint in real time and adjust the displayed 3D model according to the user's operation.

[0368] Internal memory 121 can be used to store explicit 3D models.

[0369] The mobile communication module 150 and the wireless communication module 160 can be used to send images or videos captured on the target scene to the server and receive explicit 3D models sent by the server. The wireless communication module 160 can also use GPS to obtain the location information of the electronic device 300 when it captures the target scene, send the adjusted explicit 3D model to the server, and send the attitude information of the mobile device 1, the location information and depth information of the captured objects in the target scene, and the location information of the electronic device 300 to the server.

[0370] The gyroscope sensor 180B or the accelerometer sensor 180E can be used to acquire the attitude information of the mobile device 1, so that the server can use the attitude information to determine the angle information of the target scene.

[0371] The 180F distance sensor can be used to acquire position information of objects in the target scene.

[0372] The 180N depth sensor can be used to acquire depth information of objects in a target scene.

[0373] Camera 193 can be used to capture images or videos of a target scene.

[0374] The display screen 194 can be used to display the image of the explicit 3D model rendered by the processor 110, as well as one or more user interfaces related to the aforementioned "modeling" application.

[0375] When electronic device 300 is mobile device 2:

[0376] The processor 110 can be used to determine, based on the explicit 3D model before adjustment and the explicit 3D model after adjustment, the model added to the explicit 3D model after adjustment, and to overlay the added model onto the image captured by the camera.

[0377] Internal memory 121 can be used to store explicit 3D models before and after adjustment.

[0378] The mobile communication module 150 and the wireless communication module 160 can be used to acquire explicit 3D models before and after adjustment.

[0379] Camera 193 can be used to capture images of a target scene.

[0380] The display screen 194 can be used to display images captured by the camera, and to overlay additional models on the images.

[0381] For details regarding mobile device 1 and mobile device 2 not mentioned above, please refer to the aforementioned... Figure 6 The relevant descriptions of the flowchart shown will not be repeated here.

[0382] The electronic device can be a portable terminal device running iOS, Android, Microsoft, or other operating systems, such as a mobile phone, tablet computer, or wearable device. It can also be a non-portable terminal device such as a laptop computer or desktop computer with a touch-sensitive surface or touch panel. The software system of the electronic device 300 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered Android system as an example to illustrate the software structure of the electronic device 300.

[0383] Figure 14 This is a software structure block diagram of the electronic device 300 according to an embodiment of the present invention.

[0384] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0385] The application layer can include a series of application packages.

[0386] like Figure 14 As shown, the application package may include applications such as "modeling" app, camera, gallery, calling, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0387] The "Modeling" application provides an interactive interface for users to capture images or videos of a target scene, displays an explicit 3D model of the target scene, adjusts the explicit 3D model based on user actions, and displays the adjusted explicit 3D model. For details regarding the user interface of the "Modeling" application, please refer to the aforementioned documentation. Figures 3A-3I , Figures 4A-4I , Figure 5 The user interface shown and its related description.

[0388] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0389] like Figure 14 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0390] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0391] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0392] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0393] The phone manager is used to provide communication functions for electronic devices 300. For example, it manages call status (including connection and disconnection).

[0394] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0395] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0396] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0397] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0398] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0399] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0400] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0401] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0402] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0403] A 2D graphics engine is a graphics engine for 2D drawing.

[0404] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0405] The following example, using a scene of capturing a photograph, illustrates the workflow of the software and hardware of the electronic device 300.

[0406] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0407] Figure 15 A schematic diagram of the hardware structure of electronic device 400 is shown.

[0408] like Figure 15 As shown, the electronic device 400 may include: one or more processors 201, memory 202, communication interface 203, transmitter 205, receiver 206, coupler 207, and antenna 208. These components may be connected via bus 204 or other means. Figure 15 Taking a bus connection as an example. For instance, the electronic device 400 can be the aforementioned server. Wherein:

[0409] Communication interface 203 can be used by electronic device 400 to communicate with other communication devices, such as electronic device 300. Specifically, communication interface 203 can be a 3G communication interface, a Long Term Evolution (LTE) (4G) communication interface, a 5G communication interface, a WLAN communication interface, a WAN communication interface, etc. Not limited to wireless communication interfaces, electronic device 400 can also be configured with wired communication interface 203 to support wired communication; for example, the backhaul link between electronic device 400 and other servers can be a wired communication connection.

[0410] In some embodiments of this application, transmitter 205 and receiver 206 can be considered as a wireless modem. Transmitter 205 can be used to transmit signals output by processor 201. Receiver 206 can be used to receive signals. In electronic device 400, the number of transmitters 205 and receivers 206 can be one or more. Antenna 208 can be used to convert electromagnetic energy in a transmission line into electromagnetic waves in free space, or to convert electromagnetic waves in free space into electromagnetic energy in a transmission line. Coupler 207 can be used to split a mobile signal into multiple paths and distribute them to multiple receivers 206. Understandably, antenna 208 of electronic device 400 can be implemented as a large-scale antenna array.

[0411] In some embodiments, receiver 206 may be used to receive images or videos of a target scene sent by other devices, such as electronic device 300, and transmitter 205 may be used to send an explicit 3D model or an adjusted explicit 3D model.

[0412] The memory 202 is coupled to the processor 201 and is used to store various software programs and / or multiple sets of instructions. Specifically, the memory 202 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices.

[0413] The memory 202 can store an operating system (hereinafter referred to as the system), such as uCOS, VxWorks, RTLinux and other embedded operating systems.

[0414] In some embodiments, memory 202 can be used to store a neural radiation field model and an explicit 3D model corresponding to the target scene.

[0415] In this embodiment of the application, the processor 201 can be used to read and execute computer-readable instructions.

[0416] In some embodiments, the processor 201 can be used to calculate camera pose based on multiple images, train a neural radiation field model, generate an implicit 3D model, and convert the implicit 3D model into an explicit 3D model.

[0417] It needs to be explained that, Figure 15 The electronic device 400 shown is merely one implementation of the embodiments of this application. In actual applications, the electronic device 400 may include more or fewer components, which is not limited here.

[0418] It should be understood that each step in the above method embodiments can be completed by integrated logic circuits in the processor hardware or by instructions in software form. The method steps disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor.

[0419] This application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program; the processor may be used to invoke the computer program in the memory, causing the electronic device to execute the method executed by electronic device 300 or electronic device 400 in any of the above embodiments.

[0420] This application also provides a chip system including at least one processor for implementing the functions involved in the methods performed by electronic device 300 or electronic device 400 in any of the above embodiments.

[0421] In one possible design, the chip system also includes a memory for storing program instructions and data, which may be located within or outside the processor.

[0422] The chip system can consist of chips or include chips and other discrete components.

[0423] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.

[0424] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.

[0425] For example, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0426] This application also provides a computer program product, which includes a computer program (also referred to as code or instructions) that, when run, causes a computer to execute the method executed by either electronic device 300 or electronic device 400 in any of the above embodiments.

[0427] This application also provides a computer-readable storage medium storing a computer program (also referred to as code or instructions). When the computer program is run, it causes a computer to perform the method executed by either electronic device 300 or electronic device 400 in any of the above embodiments.

[0428] It should be understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0429] Additionally, this application also provides an apparatus. Specifically, the apparatus may be a component or module, and may include one or more processors and a memory connected together. The memory stores a computer program. When the computer program is executed by one or more processors, the apparatus performs the methods described in the above-described method embodiments.

[0430] The apparatus, computer-readable storage medium, computer program product, or chip provided in the embodiments of this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0431] The various embodiments of this application can be combined arbitrarily to achieve different technical effects.

[0432] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0433] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0434] In summary, the above description is merely an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made according to the disclosure of the present invention should be included within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method, wherein the method is applied to a server, characterized in that, The method includes: The server acquires multiple images of the target scene; The server trains a neural radiation field model based on the multiple images and the camera pose when the multiple images were captured, to obtain an implicit 3D model; wherein the input of the implicit 3D model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point; The server converts the implicit 3D model into an explicit 3D model, which describes the density of some spatial points in the target scene and their colors under different viewing directions. The server sends the texture map of the explicit 3D model to the first device; the texture map includes: an index map and a model parameter map, the pixel values ​​of the index map are used to indicate the index information of the nodes of the octree, one node corresponds to a spatial point in the explicit 3D model, the index information of one node is used to find the density of the spatial point corresponding to the node and the coefficient of the spherical harmonic function in the model parameter map, the pixel values ​​of the model parameter map are used to indicate the density of spatial points in the target scene and the coefficient of the spherical harmonic function, the spherical harmonic function is used to represent the relationship between the viewing direction and color represented in the implicit 3D model.

2. The method according to claim 1, characterized in that, The server converts the implicit 3D model into an explicit 3D model, specifically including: The server uses spherical harmonic functions to represent the relationship between the viewing direction and color in the implicit 3D model. Each spherical harmonic function represents the relationship between different viewing directions and colors of a spatial point. The server filters some spatial points in the target scene to retain the spatial points with a density value greater than a preset value. The density value of the spatial points in the target scene is obtained by inputting the coordinates of the spatial points into the implicit 3D model. The server generates an explicit 3D model, which includes the following information: the density of the partial spatial points, and the coefficients of the spherical harmonic functions of the partial spatial points.

3. The method according to claim 1 or 2, characterized in that, The explicit 3D model stores the information in an octree data structure. The octree includes multiple nodes, each node corresponding to one of the spatial points in the partial spatial points. Each node stores index information, which is used to find the density of the spatial point corresponding to the node, and the coefficient of the spherical harmonic function of the spatial point corresponding to the node.

4. The method according to claim 3, characterized in that, The first node indicated by a pixel value of the index map is used to find the density of the first spatial point corresponding to the first node indicated by multiple pixel values ​​in the model parameter map, and the coefficient of the spherical harmonic function of the first spatial point.

5. The method according to claim 1 or 2, characterized in that, The camera pose when taking an image is determined by the server based on the image and the images adjacent to it, wherein any two adjacent images among the multiple images contain overlapping content with a preset ratio.

6. The method according to claim 1 or 2, characterized in that, After the server acquires multiple images of the target scene, the method further includes: The server determines the camera pose corresponding to each of the multiple images based on the multiple images and prior parameters. The prior parameters include one or more of the following: the position and pose of the camera that took the image, the position of the object captured in the image, and the depth information of the image.

7. The method according to claim 1 or 2, characterized in that, The server acquires multiple images for the target scene, specifically including: The server acquires multiple images of the target scene collected by the first device; or, Before the server acquires multiple images of the target scene, the method further includes: The server acquires a video of the target scene captured by the first device, wherein the multiple images are multiple frames from the video.

8. A three-dimensional reconstruction method, characterized in that, The method is applied to a first device, and the method includes: The first device acquires the texture map of the explicit 3D model sent by the server. The explicit 3D model is the model obtained by the server after converting the implicit 3D model. The implicit 3D model is obtained by training a neural radiation field model based on multiple images of the target scene and the camera pose when the multiple images were taken. The input of the implicit 3D model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point. The explicit 3D model describes the density and color of some spatial points in the target scene under different viewing directions. The texture map includes: an index map and a model parameter map. The pixel values ​​of the index map are used to indicate the index information of the nodes of the octree. One node corresponds to a spatial point in the explicit 3D model. The index information of one node is used to find the density and coefficient of the spherical harmonic function of the spatial point corresponding to the node in the model parameter map. The pixel values ​​of the model parameter map are used to indicate the density and coefficient of the spherical harmonic function of the spatial point in the target scene. The spherical harmonic function is used to represent the relationship between the viewing direction and color represented in the implicit 3D model. The first device displays a first user interface, which shows a first view of the explicit three-dimensional model from a first viewing perspective.

9. The method according to claim 8, characterized in that, The pixel value of the pixel in the first image is obtained by integrating the color and density of the spatial point in the first viewing direction described by the explicit 3D model. The first viewing direction is the direction in which the spatial point corresponding to the pixel is observed from the first viewing angle.

10. The method according to claim 8 or 9, characterized in that, The method further includes: The first device detects that the user has adjusted the first viewing angle to the second viewing angle; The first device displays a second view of the explicit 3D model from the second viewing perspective in the first user interface.

11. The method according to claim 8 or 9, characterized in that, The multiple images of the target scene are multiple images of the target scene captured by the first device; Alternatively, the method further includes: the first device sending the acquired video of the target scene to the server, wherein the multiple images of the target scene are multiple frames of the video.

12. The method according to claim 11, characterized in that, Before the first device sends the collected images or videos of the target scene to the server, the method further includes: The first device displays a second user interface that includes the first control; The first device detects a first operation performed on the first control; In response to the first operation, the first device activates the camera and displays a third user interface, the third user interface including: a first preview box and a second control, the first preview box being used to display the image captured by the camera in real time; The first device detects a second operation applied to the second control, the second operation being used to trigger the first device to acquire multiple images or videos of the target scene.

13. The method according to claim 8 or 9, characterized in that, The first user interface further includes: a third control; after the first device displays the first user interface, the method further includes: The first device detects a third operation performed on the third control; In response to the third operation, the first device displays one or more model icons in a third user interface, the one or more model icons including the first model icon; The first device detects a fourth operation applied to the first model icon; In response to the fourth operation, the first device adds a first model corresponding to the first model icon to the first user interface, and from the first viewing perspective, the first device displays a third screen in the first user interface showing the explicit 3D model after the first model has been added.

14. The method according to claim 13, characterized in that, After the first device displays the third screen, the method further includes: The first device sends the explicit 3D model after adding the first model to the second device, so that the second device overlays and displays the first model on the first image of the target scene acquired in real time, and the position of the first model in the target scene displayed in the first image corresponds to the position of the first model in the explicit 3D model after adding the first model.

15. A three-dimensional reconstruction system, the system comprising a server and a first device, characterized in that, The system includes: The server acquires multiple images of the target scene; The server trains a neural radiation field model based on the multiple images and the camera pose when the multiple images were captured, to obtain an implicit 3D model; wherein the input of the implicit 3D model is the coordinates and viewing direction of a spatial point in the target scene, and the output is the color and density of the spatial point; The server converts the implicit 3D model into an explicit 3D model, which describes the density of some spatial points in the target scene and their colors under different viewing directions. The server sends the texture map of the explicit 3D model to the first device; the texture map includes: an index map and a model parameter map, the pixel values ​​of the index map are used to indicate the index information of the nodes of the octree, one node corresponds to a spatial point in the explicit 3D model, the index information of one node is used to find the density of the spatial point corresponding to the node and the coefficient of the spherical harmonic function in the model parameter map, the pixel values ​​of the model parameter map are used to indicate the density of spatial points in the target scene and the coefficient of the spherical harmonic function, the spherical harmonic function is used to represent the relationship between the viewing direction and color represented in the implicit 3D model; The first device displays a first user interface, which displays a first image presented by the explicit three-dimensional model from a first viewing perspective. The pixel value of the pixel in the first image is obtained by integrating the color and density of the spatial point in the first viewing direction described by the explicit three-dimensional model. The first viewing direction is the direction in which the spatial point corresponding to the pixel is observed from the first viewing perspective.

16. An electronic device, characterized in that, The device includes a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 7.

17. An electronic device, characterized in that, The device includes a memory, one or more processors, and one or more programs; when the one or more processors execute the one or more programs, the electronic device causes the electronic device to perform the method as described in any one of claims 8 to 14.

18. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 7, 8 to 14.

19. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7, 8 to 14.

Images