Panoramic dynamic scene NeRF reconstruction and rendering method

Abstract

The invention discloses a panoramic dynamic scene NeRF reconstruction and rendering method, and belongs to the technical field of panoramic NeRF reconstruction. The method comprises the following steps: acquiring a multi-frame panoramic image containing all-dimensional view angle information in a scene under a panoramic view angle; preprocessing each frame of panoramic image; after preprocessing, decomposing a scene contained in the panoramic image into a static part and a dynamic part; respectively modeling the static part and the dynamic part to obtain a static field and a dynamic field; superposing and summing the static field and the dynamic field, and rendering to obtain a neural radiation field; and performing panoramic scene rendering and new view angle synthesis based on the neural radiation field to complete panoramic dynamic scene reconstruction and rendering. Through the combination of the panorama and the dynamic and static separation NeRF technology, the reconstruction and rendering capability of the 360-degree panorama video in a dynamic scene is remarkably improved, and high-quality dynamic expressive force is brought to various virtual, security, traffic and media application scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic NeRF reconstruction, and in particular relates to a panoramic dynamic scene NeRF reconstruction and rendering method, an electronic device and a storage medium. Background Art

[0002] With the development of technologies such as virtual reality (VR) and augmented reality (AR), panoramic scene reconstruction and rendering technology has gradually become a research hotspot in the field of computer vision and graphics. Neural Radiance Field (NeRF) is a 3D scene reconstruction method based on neural networks, which can generate realistic 3D scenes from 2D images. In traditional static scenes, the NeRF method has made significant progress, but there are still many challenges in the processing and rendering of panoramic dynamic scenes.

[0003] Panoramic image capture technology can obtain image information from a 360° perspective, providing a data basis for the reconstruction of panoramic scenes. However, combining panoramic images with dynamic scenes and achieving efficient panoramic dynamic scene reconstruction and rendering through the NeRF method is still a complex technical challenge. Existing methods often perform poorly in rendering quality, computational efficiency, and temporal and spatial consistency when dealing with large-scale perspective changes.

[0004] The existing technology has made progress in the processing of dynamic scenes and panoramic scenes respectively, but the panoramic dynamic scene processing combining the two still faces many technical problems.

[0005] Insufficient spatiotemporal consistency of dynamic scenes. Existing dynamic scene processing methods cannot effectively capture the complex trajectories of moving objects under a panoramic perspective, especially when there are a large number of dynamic objects in the scene, the reconstructed spatiotemporal consistency is poor.

[0006] The reconstruction accuracy of panoramic images is insufficient. Panoramic image reconstruction technology based on NeRF is prone to artifacts and blurring when the perspective changes, especially in dynamic scenes, making it difficult to maintain high-quality output of scene rendering.

[0007] Inaccurate separation of dynamic objects and background. In the prior art, the separation effect of static background and dynamic objects is not ideal, which makes the moving objects in dynamic scenes easily confused with the background, affecting the overall quality of reconstruction.

[0008] High computing resource requirements. The 360° viewing angle data of panoramic scenes is large, and the processing complexity of dynamic scenes is high, resulting in high computing resource and time requirements for existing technologies, which cannot meet the needs of real-time application scenarios. Summary of the invention

[0009] The purpose of the embodiment of the present invention is to provide a panoramic dynamic scene NeRF reconstruction and rendering method, which uses 360° panoramic images to generate scenes, and at the same time decomposes static and dynamic scenes to achieve efficient reconstruction and rendering of panoramic dynamic scenes. The method can ensure the spatiotemporal consistency and rendering quality of dynamic scenes under a wide range of panoramic viewing angles, thereby solving at least one technical problem involved in the background technology.

[0010] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: In a first aspect, an embodiment of the present invention provides a NeRF reconstruction and rendering method for a panoramic dynamic scene, comprising the following steps: Step S1, obtaining a multi-frame panoramic image containing all-round perspective information of a scene under a panoramic perspective; Step S2, preprocessing each frame of the panoramic image; Step S3, after preprocessing, decomposing the scene contained in the panoramic image into a static part and a dynamic part; Step S4, modeling the static part and the dynamic part respectively to obtain a static field and a dynamic field; Step S5, superimposing and summing the static field and the dynamic field, and then rendering to obtain the neural radiation field; Step S6, performing panoramic scene rendering and new perspective synthesis based on the neural radiation field to complete panoramic dynamic scene reconstruction and rendering.

[0011] In a second aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the method described in the first aspect.

[0012] In a third aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present invention provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly improves the reconstruction and rendering capabilities of 360° panoramic videos in dynamic scenes through the innovative combination of panoramic and static-dynamic separation NeRF technologies, bringing high-quality dynamic expression to various virtual, security, transportation and media application scenarios. This method not only ensures the efficiency and spatiotemporal consistency of dynamic scenes, but also improves the real-time rendering efficiency of panoramic videos, providing technical support for real-time processing of complex scenes; 2. In VR / AR scenarios, the present invention can realize a full-scale, high-precision dynamic virtual scene experience through panoramic dynamic mapping, which is particularly suitable for applications that need to display a wide range of dynamic content, such as virtual tourism, online exhibitions, VR experiences of live sports and concerts, etc.; 3. The present invention can be used for dynamic scene understanding and modeling in autonomous driving systems, and can monitor the movement of surrounding vehicles and pedestrians through panoramic dynamic modeling and dynamic object separation, thereby improving the perception and response capabilities of autonomous driving; 4. In film and television and media production, the 360° panoramic dynamic modeling method supports complex dynamic scene reconstruction, saves a lot of manual tracking and post-editing time for dynamic objects, and provides a smooth and natural multi-view switching effect. It can significantly improve the user experience in panoramic video production and immersive image content. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flowchart of a NeRF reconstruction and rendering method for a panoramic dynamic scene provided by an embodiment of the present invention; Figure 2 This is one of the hardware structure diagrams of the electronic device provided by the embodiment of the present invention; Figure 3 This is the second schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0018] The panoramic dynamic scene NeRF reconstruction and rendering method provided by the embodiment of the present invention is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios.

[0019] See also Figure 1 , is a panoramic dynamic scene NeRF reconstruction and rendering method provided by an embodiment of the present invention, comprising the following steps: Step S1, obtaining a multi-frame panoramic image containing all-round perspective information of a scene under a panoramic perspective; Step S2, preprocessing each frame of the panoramic image; Step S3, after preprocessing, decomposing the scene contained in the panoramic image into a static part and a dynamic part; Step S4, modeling the static part and the dynamic part respectively to obtain a static field and a dynamic field; Step S5, superimposing and summing the static field and the dynamic field, and then rendering to obtain the neural radiation field; Step S6, performing panoramic scene rendering and new perspective synthesis based on the neural radiation field to complete panoramic dynamic scene reconstruction and rendering.

[0020] In step S1, a panoramic image is acquired using a 360° camera or a panoramic image capturing device.

[0021] In step S2, the preprocessing specifically includes: Use geometric correction algorithm to correct the panoramic image and restore the true proportion relationship of the panoramic image; The white balance correction and histogram equalization method are used to balance the brightness and color distribution of the panoramic image, which is expressed as follows: ; ; In the formula, I capture Indicates at time t A collection of panoramic images captured at every moment; θ and φis the spherical coordinate angle; , , Indicates time t Panoramic images captured at all times; Indicates white balance correction and histogram equalization methods; represents the geometric correction function; Represents the image after preprocessing.

[0022] In step S3, the scene contained in the panoramic image is decomposed into a static part and a dynamic part, which specifically includes: For slowly moving objects in the scene, the optical flow algorithm is used to decompose the dynamic part, and the area with little or no optical flow change is regarded as the static background, which is expressed by the following formula: ; In the formula, v optical is the speed of pixel movement; ) is the pixel position; Δt is the time interval; Indicates the horizontal offset of the pixel; Indicates the vertical offset of the pixel; I (·) represents image pixels; For objects with fast movement or complex shapes in the scene, use deep learning image segmentation algorithms to decompose the dynamic parts, including: A threshold is set to separate the static background from the dynamic object, where the area with small optical flow change is regarded as the static background, and the area with significant optical flow change is marked as a dynamic object, which is expressed as follows: ; in, is the set threshold; ||v optical || represents the magnitude of optical flow; S scene represents the radiation field of the scene; S static Represents the static part; S dynamic Represents the dynamic part.

[0023] In step S4, the static part is modeled using the NeRF method to obtain a NeRF model, and the NeRF model is trained by optimizing the following loss function: ; In the formula, Indicates light loss; C pred is the image color generated by the NeRF model;C true is the color of the real image; N is the number of pixels in the image; Represents the images generated by the model and the real images.

[0024] In step S4, the dynamic part is modeled using the HyperNeRF method to obtain a HyperNeRF model, which specifically includes: Establish a reference hyperspace to learn the topological possibilities of slices from all input image frames. At any given time, the features of a spatial point are represented by selecting the correct slice in the reference hyperspace to solve the problem of complex topology. During the training process, we search for topological slice level sets between different image frames, learn to generate reference hyperspace, use spatial position coordinates and slice features to generate input, and query the corresponding output, which is expressed as follows: ; ; In the formula, W represents the number of dimensions of the base hyperspace, represents the slice feature vector, represents the color of a point in space, Represents the density of points in space; represents implicit appearance encoding; represents canonical hyperspace; Indicates the viewing direction.

[0025] It should be noted that the neural network can learn various types of topological changes that occur in the details of the scene; at a specific time point, it can find suitable slices in the high-dimensional canonical space for integration.

[0026] In step S5, the static field and the dynamic field are superimposed and summed, which is expressed by the following formula: ; In the formula, is the neural radiation field after superposition and summation; is a static field; For dynamic field.

[0027] In step S6, the panoramic scene rendering specifically includes: The panoramic field is rendered using a conical projection method similar to Tri-MipRF, including: Sampling is performed using a sphere with the field of view angle as a parameter at the sampling point on the ray; For rendering of sampling points, the sampling ray parameters are the inverse function of the projection formula , the equation of the sampling center point and the equation of the sphere radius at the sampling point are: ; ; in, The direction angle is θ The direction unit vector of is the camera focal length; is the pixel disk radius; The independent variable is θ Function of Indicates the distance between the pixel point and the center of the pixel disk; represents the optical center of the camera; Based on the concept of transmittance, the probability of a point in space being terminated is: ; The pixel color observed by a beam of light is the integral of the probability weights of the coordinates of each point in space; therefore, combined with the sampling points obtained above, the color of each pixel can be calculated to obtain the rendering result, which is expressed by the following formula: ; ; In the formula, t n Represents the near plane in space; t f Represents the far plane in space; is the transmittance; s Indicates the distance from point o; Indicates that the ray is along the direction Passing distance t ; Represents the density of points in space; Indicates that the ray is along the direction Passing distance t The color of the particles.

[0028] like Figure 2 As shown, an embodiment of the present invention further provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned panoramic dynamic scene NeRF reconstruction and rendering method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0029] It should be noted that the first electronic device in the embodiment of the present invention includes the mobile electronic device and the non-mobile electronic device mentioned above.

[0030] Figure 3The present invention is a schematic diagram of the hardware structure of an electronic device.

[0031] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0032] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0033] It should be understood that in the embodiment of the present invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static image or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 may be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0034] An embodiment of the present invention also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned panoramic dynamic scene NeRF reconstruction and rendering method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0035] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0036] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned panoramic dynamic scene NeRF reconstruction and rendering method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0037] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0038] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0039] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0040] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned panoramic dynamic scene NeRF reconstruction and rendering method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0041] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0042] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A NeRF reconstruction and rendering method for panoramic dynamic scenes, characterized in that: The following steps are involved: Step S1, obtaining a multi-frame panoramic image containing all-round perspective information of a scene under a panoramic perspective; Step S2, preprocessing each frame of the panoramic image; Step S3, after preprocessing, decomposing the scene contained in the panoramic image into a static part and a dynamic part; Step S4, modeling the static part and the dynamic part respectively to obtain a static field and a dynamic field; Step S5, superimposing and summing the static field and the dynamic field, and then rendering to obtain the neural radiation field; Step S6, performing panoramic scene rendering and new perspective synthesis based on the neural radiation field to complete panoramic dynamic scene reconstruction and rendering.

2. The method according to claim 1, characterized in that In step S1, a panoramic image is acquired using a 360° camera or a panoramic image capturing device.

3. The method according to claim 1, characterized in that In step S2, the preprocessing specifically includes: Use geometric correction algorithm to correct the panoramic image and restore the true proportion relationship of the panoramic image; The white balance correction and histogram equalization method are used to balance the brightness and color distribution of the panoramic image, which is expressed as follows: ； ； In the formula, Indicates at time t A collection of panoramic images captured at every moment; θ and φ is the spherical coordinate angle; , , Indicates time t Panoramic images captured at all times; Indicates white balance correction and histogram equalization methods; represents the geometric correction function; Represents the image after preprocessing.

4. The method according to claim 3, characterized in that: In step S3, the scene contained in the panoramic image is decomposed into a static part and a dynamic part, which specifically includes: For slowly moving objects in the scene, the optical flow algorithm is used to decompose the dynamic part, and the area with little or no optical flow change is regarded as the static background, which is expressed by the following formula: ； In the formula, v optical is the speed of pixel movement; ) is the pixel position; Δt is the time interval; Indicates the horizontal offset of the pixel; Indicates the vertical offset of the pixel; I (·) represents image pixels; For objects with fast movement or complex shapes in the scene, use deep learning image segmentation algorithms to decompose the dynamic parts, including: A threshold is set to separate the static background from the dynamic object, where the area with small optical flow change is regarded as the static background, and the area with significant optical flow change is marked as a dynamic object, which is expressed as follows: ； in, is the set threshold; ||v optical || represents the magnitude of optical flow; S scene represents the radiation field of the scene; S static Represents the static part; S dynamic Represents the dynamic part.

5. The method according to claim 4, characterized in that In step S4, the static part is modeled using the NeRF method to obtain a NeRF model, and the NeRF model is trained by optimizing the following loss function: ； In the formula, Indicates luminosity loss; C pred is the image color generated by the NeRF model; C true is the color of the real image; N is the number of pixels in the image; Represents the images generated by the model and the real images.

6. The method according to claim 5, characterized in that In step S4, the dynamic part is modeled using the HyperNeRF method to obtain a HyperNeRF model, which specifically includes: Establish a reference hyperspace to learn the topological possibilities of slices from all input image frames. At any given time, the features of a spatial point are represented by selecting the correct slice in the reference hyperspace to solve the problem of complex topology. During the training process, we search for topological slice level sets between different image frames, learn to generate reference hyperspace, use spatial position coordinates and slice features to generate input, and query the corresponding output, which is expressed as follows: ； ； In the formula, W represents the number of dimensions of the base hyperspace, represents the slice feature vector, represents the color of a point in space, Represents the density of points in space; represents implicit appearance encoding; represents canonical hyperspace; Indicates the viewing direction.

7. The method according to claim 6, characterized in that In step S5, the static field and the dynamic field are superimposed and summed, which is expressed by the following formula: ； In the formula, is the neural radiation field after superposition and summation; is a static field; For dynamic field.

8. The method according to claim 7, characterized in that In step S6, the panoramic scene rendering specifically includes: The panoramic field is rendered using a conical projection method similar to Tri-MipRF, including: Sampling is performed using a sphere with the field of view angle as a parameter at the sampling point on the ray; For rendering of sampling points, the sampling ray parameters are the inverse function of the projection formula , the equation of the sampling center point and the equation of the sphere radius at the sampling point are: ； ； in, The direction angle is θ The direction unit vector of is the camera focal length; is the pixel disk radius; The independent variable is θ Function of Indicates the distance between the pixel point and the center of the pixel disk; represents the optical center of the camera; Based on the concept of transmittance, the probability of a point in space being terminated is: ; The pixel color observed by a beam of light is the integral of the probability weights of the coordinates of each point in space; therefore, combined with the sampling points obtained above, the color of each pixel is calculated to obtain the rendering result, which is expressed by the following formula: ； ； In the formula, t n Represents the near plane in space; t f Represents the far plane in space; is the transmittance; s Indicates the distance from point o; Indicates that the ray is along the direction Passing distance t ; Represents the density of points in space; Indicates that the ray is along the direction Passing distance t The color of the particles.

Images