A Method for Panoramic Dynamic Scene NeRF Reconstruction and Rendering

Abstract

The present invention discloses a panoramic dynamic scene NeRF reconstruction and rendering method, belonging to the technical field of panoramic NeRF reconstruction. The method includes the following steps: obtaining multiple panoramic images containing omnidirectional view information of the scene from a panoramic perspective; preprocessing each panoramic image; after preprocessing, decomposing the scene contained in the panoramic image into a static part and a dynamic part; respectively modeling the static part and the dynamic part to obtain a static field and a dynamic field; superimposing and summing the static field and the dynamic field, and then performing rendering to obtain a neural radiance field; performing panoramic scene rendering and new view synthesis based on the neural radiance field to complete the reconstruction and rendering of the panoramic dynamic scene. The invention significantly improves the reconstruction and rendering capabilities of 360° panoramic videos in dynamic scenes through the combination of panoramic and static-dynamic separation NeRF technologies, bringing high-quality dynamic expressiveness to various virtual, security, transportation, and media application scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of panoramic NeRF reconstruction, and specifically relates to a method for panoramic dynamic scene NeRF reconstruction and rendering, an electronic device, and a storage medium. Background Art

[0002] With the development of technologies such as virtual reality (VR) and augmented reality (AR), panoramic scene reconstruction and rendering technologies have gradually become research hotspots in the fields of computer vision and graphics. Neural Radiance Field (NeRF) is a neural network-based 3D scene reconstruction method that can generate realistic 3D scenes from 2D images. In traditional static scenes, the NeRF method has made significant progress, but there are still many challenges in the processing and rendering of panoramic dynamic scenes.

[0003] Panoramic image capture technology can obtain image information from a 360° perspective, providing a data basis for the reconstruction of panoramic scenes. However, combining panoramic images with dynamic scenes and achieving efficient panoramic dynamic scene reconstruction and rendering through the NeRF method is still a complex technical problem. Existing methods often perform poorly in terms of rendering quality, computational efficiency, and spatio-temporal consistency when dealing with large-scale perspective changes.

[0004] Existing technologies have made progress in the processing of dynamic scenes and panoramic scenes respectively, but the processing of panoramic dynamic scenes that combines the two still faces many technical problems.

[0005] The spatio-temporal consistency of dynamic scenes is insufficient. Existing dynamic scene processing methods cannot effectively capture the complex trajectories of moving objects from a panoramic perspective. Especially when there are a large number of dynamic objects in the scene, the spatio-temporal consistency of the reconstruction is poor.

[0006] The reconstruction accuracy of panoramic images is insufficient. In the NeRF-based panoramic image reconstruction technology, artifacts and blurring are likely to occur during perspective transformation, especially in dynamic scenes, and it is difficult to maintain high-quality output of scene rendering.

[0007] The separation of dynamic objects from the background is inaccurate. In existing technologies, the separation effect of static backgrounds and dynamic objects is not ideal, resulting in the easy confusion of moving objects in dynamic scenes with the background, affecting the overall quality of the reconstruction.

[0008] High computational resource requirements. The 360° perspective data of panoramic scenes is large, and the processing complexity of dynamic scenes is high, resulting in high requirements for computational resources and time in existing technologies, and unable to meet the needs of real-time application scenarios. Summary of the Invention

[0009] The purpose of the embodiments of the present invention is to provide a panoramic dynamic scene NeRF reconstruction and rendering method, which generates a scene using 360° panoramic images and, by decomposing static and dynamic scenes, realizes efficient reconstruction and rendering of panoramic dynamic scenes. This method can ensure the spatio-temporal consistency and rendering quality of dynamic scenes from a wide range of panoramic perspectives, thereby solving at least one technical problem involved in the background art.

[0010] To solve the above technical problems, the present invention is implemented as follows:

[0011] In a first aspect, embodiments of the present invention provide a panoramic dynamic scene NeRF reconstruction and rendering method, including the following steps:

[0012] Step S1, obtaining multiple panoramic images containing all-round perspective information of the scene from a panoramic perspective;

[0013] Step S2, preprocessing each frame of panoramic image;

[0014] After preprocessing in Step S3, decomposing the scene included in the panoramic image into a static part and a dynamic part;

[0015] Step S4, respectively modeling the static part and the dynamic part to obtain a static field and a dynamic field;

[0016] Step S5, adding and summing the static field and the dynamic field, and then performing rendering to obtain a neural radiance field;

[0017] Step S6, performing panoramic scene rendering and new view synthesis based on the neural radiance field to complete the reconstruction and rendering of the panoramic dynamic scene.

[0018] In a second aspect, embodiments of the present invention provide an electronic device, including:

[0019] At least one processor;

[0020] At least one memory for storing at least one program;

[0021] When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the method described in the first aspect.

[0022] In a third aspect, embodiments of the present invention provide a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, it implements the steps of the method described in the first aspect.

[0023] In a fourth aspect, embodiments of the present invention provide a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] 1. By combining the innovative panoramic and static-dynamic separation NeRF technologies, the present invention significantly improves the reconstruction and rendering capabilities of 360° panoramic videos in dynamic scenarios, bringing high-quality dynamic expressiveness to various virtual, security, transportation, and media application scenarios. This method not only ensures the efficiency and spatio-temporal consistency of dynamic scenarios but also improves the real-time rendering efficiency of panoramic videos, providing technical support for the real-time processing of complex scenarios;

[0026] 2. In VR / AR scenarios, the present invention can achieve an all-round and high-precision dynamic virtual scene experience through panoramic dynamic mapping, especially suitable for applications that need to display a large range of dynamic content, such as virtual tourism, online exhibitions, VR experiences of live sports and concerts, etc.;

[0027] 3. The present invention can be used for dynamic scene understanding and modeling in autonomous driving systems. By panoramic dynamic modeling and dynamic object separation, it monitors the movements of surrounding vehicles and pedestrians, thereby enhancing the perception and reaction capabilities of driverless vehicles;

[0028] 4. In film and media production, the 360° panoramic dynamic modeling method supports the reconstruction of complex dynamic scenarios, saving a large amount of time for manual tracking and post-editing of dynamic objects and providing a smooth and natural multi-perspective switching effect. It can significantly improve the user experience, especially in panoramic video production and immersive video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings, where:

[0030] Figure 1 is a schematic flowchart of a method for panoramic dynamic scene NeRF reconstruction and rendering provided by an embodiment of the present invention;

[0031] Figure 2 is one of the hardware structure schematic diagrams of an electronic device provided by an embodiment of the present invention;

[0032] Figure 3 is the second hardware structure schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and do not limit the number of objects. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / ", generally represents an "or" relationship between the associated objects before and after.

[0035] Next, in conjunction with the accompanying drawings, a panoramic dynamic scene NeRF reconstruction and rendering method provided by the embodiments of the present invention will be described in detail through specific embodiments and their application scenarios.

[0036] Please refer to Figure 1 , which is a panoramic dynamic scene NeRF reconstruction and rendering method provided by the embodiments of the present invention, including the following steps:

[0037] Step S1, obtaining multiple panoramic images containing omnidirectional view information of the scene from a panoramic perspective;

[0038] Step S2, preprocessing each panoramic image;

[0039] After preprocessing in Step S3, decomposing the scene included in the panoramic image into a static part and a dynamic part;

[0040] Step S4, respectively modeling the static part and the dynamic part to obtain a static field and a dynamic field;

[0041] Step S5, superimposing and summing the static field and the dynamic field, and then performing rendering to obtain a neural radiance field;

[0042] Step S6, performing panoramic scene rendering and new view synthesis based on the neural radiance field to complete the reconstruction and rendering of the panoramic dynamic scene.

[0043] In Step S1, a 360° camera or a panoramic image capture device is used to obtain panoramic images.

[0044] In Step S2, the preprocessing specifically includes:

[0045] Use a geometric correction algorithm to correct the panoramic image and restore the true proportional relationship of the panoramic image;

[0046] Adopt white balance correction and histogram equalization methods to balance the brightness and color distribution of the panoramic image, which is expressed by the following formula:

[0047] ;

[0048] ;

[0049] In the formula, I capture represents the set of panoramic images captured at time T ; θ and φ are spherical coordinate angles; , , represent the panoramic image captured at time T ; represents the white balance correction and histogram equalization method; represents the geometric correction function; represents the image after preprocessing.

[0050] In step S3, the scene included in the panoramic image is decomposed into a static part and a dynamic part, specifically including:

[0051] For the slow-moving objects in the scene, use the optical flow algorithm to decompose the dynamic part, and the areas with small or no optical flow changes are regarded as the static background, which is expressed by the following formula:

[0052] ;

[0053] In the formula, v optical is the movement speed of the pixel; ([[]]END]] ) is the pixel position; ΔT is the time interval; represents the horizontal offset of the pixel; represents the vertical offset of the pixel; I (·) represents the image pixel;

[0054] For the objects with faster movement or complex shapes in the scene, use the image segmentation algorithm of deep learning to decompose the dynamic part, specifically including:

[0055] Set a threshold to separate the static background from the dynamic objects. Among them, the areas with small optical flow changes are regarded as the static background, and the areas with significant optical flow changes are marked as dynamic objects, which is expressed by the following formula:

[0056] ;

[0057] Among them, is the set threshold; ||v optical || represents the magnitude of the optical flow; S scene represents the radiation field of the scene; S static represents the static part; S dynamic represents the dynamic part.

[0058] In step S4, the NeRF method is used to model the static part to obtain a NeRF model, and the NeRF model is trained by optimizing the following loss function:

[0059] ;

[0060] In the formula, represents the photometric loss; C pred is the color of the image generated by the NeRF model; C true is the color of the real image; N is the number of pixels in the image; represents the image generated by the model and the real image.

[0061] In step S4, the HyperNeRF method is used to model the dynamic part to obtain a HyperNeRF model, specifically including:

[0062] Set up a reference hyperspace, learn the topological possibilities of slices from all input image frames. At any given time, the features of spatial points are represented by selecting the correct slice in the reference hyperspace to solve the problem of complex topology;

[0063] During the training process, find the topological slice level sets between different image frames, learn to generate the reference hyperspace, use the spatial position coordinates and slice features to generate inputs, query the corresponding outputs, which are represented by the following formula:

[0064] ;

[0065] ;

[0066] In the formula, W represents the number of dimensions of the reference hyperspace, represents the slice feature vector, represents the color of the point in space, represents the density of the point in space; represents the implicit appearance encoding; represents the canonical hyperspace; represents the viewing direction.

[0067] It should be noted that the neural network can learn various types of topological changes occurring in the scene details; at a specific time point, a suitable slice can be found in the high-dimensional canonical space for integration.

[0068] In step S5, the static field and the dynamic field are superimposed and summed, which is expressed by the following formula:

[0069] ;

[0070] In the formula, is the neural radiance field after superimposing and summing; is the static field; is the dynamic field.

[0071] In step S6, the panoramic scene is rendered, specifically including:

[0072] The panoramic field is rendered using the Tri-MipRF conical projection method, specifically including:

[0073] Sampling is performed using a sphere with the field of view angle as a parameter at the sampling points on the ray;

[0074] For the rendering of the sampling points, the sampling ray parameter is the inverse function of the projection formula , and the sampling center point equation and the sphere radius equation at the sampling points are:

[0075] ;

[0076] ;

[0077] Among them, is the unit direction vector with the direction angle of θ ; is the camera focal length; is the pixel disk radius; is a function with the independent variable of θ ; represents the distance between the pixel point and the center of the pixel disk; represents the camera optical center;

[0078] Based on the concept of transmittance, the probability that a point in space is terminated from transmission is ; the pixel color observed by a beam of light is the integral of the coordinate probability weights of each point in space; therefore, combining the obtained sampling points, the color of each pixel point can be calculated to obtain the rendering result, which is expressed by the following formula:

[0079] ;

[0080] ;

[0081] In the formula, t n represents the near plane in space; t f represents the far plane in space; is the transmittance; s represents the distance from point o; represents that the ray is along the direction after a distance of t ; represents the density of points in space; represents that the ray is along the direction after a distance of t the color of the particle at that point.

[0082] As Figure 2 shown, an embodiment of the present invention also provides an electronic device 600. The electronic device 600 includes a processor 601, a memory 602, a program or instruction stored on the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, it implements each process of the above-mentioned panoramic dynamic scene NeRF reconstruction and rendering method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0083] It should be noted that the first electronic device in the embodiment of the present invention includes the above-mentioned mobile electronic device and non-mobile electronic device.

[0084] Figure 3 is a schematic diagram of the hardware structure of an electronic device for implementing an embodiment of the present invention.

[0085] The electronic device 700 includes, but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710 and other components.

[0086] Those skilled in the art can understand that the electronic device 700 may further include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 710 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. Figure 3 The structure of the electronic device shown in

[0087] It should be understood that in the embodiments of the present invention, the input unit 704 may include a Graphics Processing Unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes the image data of static images or videos obtained by an image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. The other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here. The memory 709 may be used to store software programs and various data, including but not limited to application programs and operating systems. The processor 710 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 710.

[0088] The embodiments of the present invention further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above-mentioned embodiments of the panoramic dynamic scene NeRF reconstruction and rendering method, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0089] Among them, the processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes a computer-readable storage medium, such as a computer Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disc, etc.

[0090] The embodiments of the present invention further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run a program or instruction to implement each process of the above-mentioned embodiments of the panoramic dynamic scene NeRF reconstruction and rendering method, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0091] It should be understood that the chip mentioned in the embodiments of the present invention may also be referred to as a system-on-chip, a system chip, a chip system, or a system-on-chip, etc.

[0092] It should be noted that in this text, the term "including", "comprising", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes such element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Additionally, the features described with reference to certain examples may be combined in other examples.

[0093] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described example methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0094] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the purpose of the present invention and the scope protected by the claims, and all of them fall within the protection scope of the present invention.

Claims

1. A panoramic dynamic scene NeRF reconstruction and rendering method, characterized in that Including the following steps: Step S1: Obtain multiple panoramic images containing omnidirectional view information of the scene from a panoramic perspective; Step S2: Preprocess each panoramic image; After preprocessing in Step S3, decompose the scene included in the panoramic image into a static part and a dynamic part, specifically including: For slow-moving objects in the scene, use the optical flow algorithm to decompose the dynamic part, and the areas with small or no optical flow changes are regarded as the static background, which is expressed by the following formula: where v optical is the motion speed of the pixel; (x, y) is the pixel position; T is the time; ΔT is the time interval; Δx represents the pixel horizontal direction offset; Δy represents the pixel vertical direction offset; I(·) represents the image pixel; For objects with fast movement or complex shapes in the scene, use the image segmentation algorithm of deep learning to decompose the dynamic part, specifically including: Set a threshold to separate the static background from the dynamic objects. Among them, the areas with small optical flow changes are regarded as the static background, and the areas with significant optical flow changes are marked as dynamic objects, which is expressed by the following formula: where a is a set threshold value; ||v optical || represents the magnitude of the optical flow; S scene represents the radiation field of the scene; S static represents the static part; S dynamic represents the dynamic part; Step S4: Model the static part and the dynamic part respectively to obtain a static field and a dynamic field; Step S5: Add the static field and the dynamic field together and sum them, and then perform rendering to obtain a neural radiance field; Step S6: Based on the neural radiance field, perform panoramic scene rendering and new view synthesis to complete the reconstruction and rendering of the panoramic dynamic scene.

2. The method according to claim 1, wherein In Step S1, use a 360° camera or a panoramic image capture device to obtain panoramic images.

3. The method according to claim 1, wherein In Step S2, the preprocessing specifically includes: Use a geometric correction algorithm to correct the panoramic image and restore the true proportional relationship of the panoramic image; Adopt white balance correction and histogram equalization methods to balance the brightness and color distribution of the panoramic image, which is expressed by the following formula: I capture (θ, φ, T) = {I1(θ, φ), I2(θ, φ),..., I n (θ, φ)}; I preprocess (θ, φ) = Calibrate(Distort(I capture (θ, φ))); Where, I capture represents the panoramic image set captured at time T; θ and φ are spherical coordinate angles; I1, I2, I n represent the panoramic images captured at time T; Calibrate represents the white balance correction and histogram equalization method; Distort represents the geometric correction function; I preprocess represents the image after preprocessing.

4. The method according to claim 3, wherein In Step S4, use the NeRF method to model the static part to obtain a NeRF model, and train the NeRF model by optimizing the following loss function: In the formula, represents the photometric loss; C pred is the color of the image generated by the NeRF model; C true is the color of the real image; N is the number of pixels in the image; i represents the image generated by the model and the real image.

5. The method according to claim 4, wherein In Step S4, use the HyperNeRF method to model the dynamic part to obtain a HyperNeRF model, specifically including: Establish a reference hyperspace, learn the topological possibilities of slices from all input image frames, and at any given time, the features of spatial points are represented by selecting the correct slice in the reference hyperspace to solve the problem of complex topology; During the training process, find the topological slice level sets between different image frames, learn to generate the reference hyperspace, use the spatial position coordinates and slice features to generate inputs, and query the corresponding outputs, which is expressed by the following formula: F: (x, w, d, ψ i ) → (c, σ); Wherein, W represents the number of dimensions of the reference hyperspace, w represents the slice feature vector, c represents the color of the point in the space, σ represents the density of the point in the space; ψ i represents the implicit appearance encoding; represents the canonical hyperspace; d represents the observation direction.

6. The method according to claim 5, wherein In Step S5, the static field and the dynamic field are added together and summed, which is expressed by the following formula: MLP = MLP S + MLP D ; where MLP is the neural radiance field after superposition and summation; MLP S is the static field; MLP D is the dynamic field.

7. The method according to claim 6, characterized in that, In Step S6, the panoramic scene rendering specifically includes: Use the Tri-MipRF conical projection method to render the panoramic field, specifically including: Sample using a sphere with the field of view angle size as a parameter at the sampling points on the ray; For the rendering of sampling points, the sampling ray parameter is the inverse function of the projection formula The sampling center point equation and the sphere radius equation on the sampling point are as follows: x = o + td θ ; Among them, d θ is the unit direction vector with the direction angle θ; f is the camera focal length; is the pixel disk radius; G(θ) is a function with the independent variable θ; r d represents the distance between the pixel point and the center of the pixel disk; O represents the camera optical center; Based on the concept of transmittance, the probability that a point in space is terminated from transmitting is The pixel color observed by a beam of light is the integral of the coordinate probabilities of each point in space; thus, combining the obtained sampling points, the color of each pixel point is calculated to obtain the rendering result, which is expressed by the following formula: where t n represents the near plane in space; t f represents the far plane in space; is the transmittance; s represents the distance from point o; r(t) represents the ray passing through a distance t along direction d; σ(s) represents the density of points in space; c(r(t), d) represents the color of particles at the point where the ray passes through a distance t along direction d.

Images