Dynamic scene reconstruction methods, devices, equipment, media and products

Abstract

This application discloses a dynamic scene reconstruction method, apparatus, device, medium, and product, relating to the field of computer vision technology. The method includes: acquiring an initial frame image; generating or determining dynamic point clouds and static point clouds based on the initial frame image; performing deformation processing on the dynamic point clouds to obtain deformed dynamic point clouds; and performing hybrid rendering based on the static point clouds and the deformed dynamic point clouds to obtain a target rendered image. By dividing the dynamic point clouds and static point clouds, and avoiding uniform deformation processing of the static point clouds, the computational overhead and storage requirements for deformation can be reduced, thereby improving the efficiency of dynamic scene reconstruction.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to dynamic scene reconstruction methods, apparatus, equipment, media and products. Background Technology

[0002] Dynamic scene reconstruction, as an important research direction in computer vision, computer graphics, and robotics, can recover a three-dimensional environment that changes over time from a series of images or videos. This process involves capturing and analyzing moving objects in the scene, as well as changes in the background, in order to build a 3D model in the computer that can reflect the dynamic characteristics of the real world.

[0003] Existing dynamic scene reconstruction methods typically involve uniform processing of all Gaussian points during the modeling process, which faces challenges of high computational overhead and high storage requirements.

[0004] Therefore, it is necessary to propose a solution to reduce the computational overhead and storage requirements in the dynamic scene reconstruction process.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this application is to provide a method, apparatus, device, medium, and product for dynamic scene reconstruction, which aims to reduce the computational overhead and storage requirements in the dynamic scene reconstruction process.

[0007] To achieve the above objectives, this application provides a dynamic scene reconstruction method, the method comprising:

[0008] Obtain the initial frame image;

[0009] Dynamic and static point clouds are generated or determined based on the initial frame image;

[0010] The dynamic point cloud is deformed to obtain a deformed dynamic point cloud.

[0011] The target rendered image is obtained by blending the static point cloud with the deformed dynamic point cloud.

[0012] In one embodiment, the step of generating or determining dynamic and static point clouds based on the initial frame image includes:

[0013] The initial frame image is initialized to obtain Gaussian points;

[0014] Dynamic values ​​are assigned to each Gaussian point to obtain the dynamic values ​​corresponding to each Gaussian point.

[0015] The dynamic value corresponding to each Gaussian point is compared with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

[0016] In one embodiment, the step of dynamically assigning values ​​to each Gaussian point to obtain the dynamic value corresponding to each Gaussian point includes:

[0017] Select or determine the pixels corresponding to each Gaussian point in each view, and calculate the time variance of each pixel in the time domain;

[0018] Using the time variance as a monitoring signal, backpropagation and iterative optimization are performed on the dynamic values ​​of each Gaussian point to obtain the dynamic values ​​corresponding to each Gaussian point.

[0019] In one embodiment, the step of comparing the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud includes:

[0020] Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold.

[0021] The Gaussian points whose dynamic values ​​are greater than the preset dynamic threshold are classified as the dynamic point cloud.

[0022] Gaussian points whose dynamic values ​​are less than or equal to the preset dynamic threshold are classified as static point clouds.

[0023] In one embodiment, the step of deforming the dynamic point cloud to obtain a deformed dynamic point cloud includes:

[0024] Construct a canonical space based on the initial frame image;

[0025] In the specified space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash encoding features;

[0026] The deformed dynamic point cloud is obtained by combining the multi-resolution hash-encoded features using a multi-layer perception model.

[0027] In one embodiment, the step of performing hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image includes:

[0028] The static point cloud and the deformed dynamic point cloud are mixed to obtain mixed Gaussian points;

[0029] Calculate the importance of each Gaussian point in the mixture of Gaussian points;

[0030] The Gaussian points whose importance is less than a preset importance threshold are pruned to obtain the target Gaussian points;

[0031] The target Gaussian point is rendered to obtain the target rendered image.

[0032] Furthermore, to achieve the above objectives, this application also proposes a dynamic scene reconstruction device, which includes:

[0033] The acquisition module is used to acquire the initial frame image;

[0034] The classification module is used to generate or determine dynamic point clouds and static point clouds based on the initial frame image;

[0035] The deformation module is used to deform the dynamic point cloud to obtain the deformed dynamic point cloud.

[0036] The rendering module is used to perform hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image.

[0037] In addition, to achieve the above objectives, this application also proposes a dynamic scene reconstruction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dynamic scene reconstruction method as described above.

[0038] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the dynamic scene reconstruction method described above.

[0039] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the dynamic scene reconstruction method described above.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] By acquiring an initial frame image; generating or determining dynamic and static point clouds based on the initial frame image; performing deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud; and performing hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image, the process of dividing the dynamic and static point clouds avoids uniformly deforming the static point cloud, thereby reducing the computational overhead and storage requirements for deformation and improving the efficiency of dynamic scene reconstruction. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating an embodiment of the dynamic scene reconstruction method of this application.

[0045] Figure 2 This is a flowchart illustrating Embodiment 2 of the dynamic scene reconstruction method of this application;

[0046] Figure 3 This is a schematic diagram illustrating the overall method principle according to the second embodiment;

[0047] Figure 4 This is a schematic diagram showing the perspective breakdown according to the second embodiment;

[0048] Figure 5 This is a flowchart illustrating Embodiment 3 of the dynamic scene reconstruction method of this application;

[0049] Figure 6 This is a schematic diagram illustrating the application effects of different methods according to the third embodiment;

[0050] Figure 7 This is a schematic diagram of the module structure of the dynamic scene reconstruction device according to an embodiment of this application;

[0051] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the dynamic scene reconstruction method in this application embodiment.

[0052] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0054] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of this application embodiment is as follows: acquiring an initial frame image; generating or determining dynamic point clouds and static point clouds based on the initial frame image; performing deformation processing on the dynamic point clouds to obtain deformed dynamic point clouds; and performing hybrid rendering based on the static point clouds and the deformed dynamic point clouds to obtain a target rendered image. By dividing the dynamic point clouds and static point clouds, the deformation processing of static point clouds is avoided, which can reduce the computational overhead and storage requirements required for deformation, thereby improving the efficiency of dynamic scene reconstruction.

[0056] In this embodiment, for ease of description, the dynamic scene reconstruction device will be used as the execution subject in the following description.

[0057] Currently, dynamic scene reconstruction is crucial for immersive imaging, driving the development of various multimedia technologies such as VR, AR, and the metaverse. Rendering dynamic scenes from arbitrary time, location, and viewpoint is essential for enhancing the user experience of multimedia products such as free-viewpoint video and bullet-time effects.

[0058] Existing technologies for dynamic scene reconstruction mainly include 4DGS, 3DGStream, Real-Time GS, and Space-Time GS. 4DGS introduces a neural voxel encoder to model deformation relationships that change over time. 3DGStream utilizes a compact neural transform buffer to efficiently model the translation and rotation of 3D Gaussians, significantly reducing training time and storage requirements per frame. It also features an adaptive 3DG addition strategy to manage new objects in dynamic scenes. Real-Time GS optimizes a set of 4D primitives, treating spacetime as a whole, parameterizing it as an anisotropic ellipse to capture geometry and appearance. Furthermore, Space-Time GS enhances standard 3D Gaussians with temporal opacity and motion / rotation parameters, effectively capturing both static and dynamic elements to simulate dynamic deformation, and uses neural features for splatted rendering, effectively modeling view- and time-dependent appearance. While these methods achieve higher quality results with faster rendering times, they still face challenges such as long training times and high storage requirements. A potential limitation of these methods is the uniform treatment of all Gaussian points during the modeling process.

[0059] Considering that static points (such as static points in the background area) constitute most of the scene in practical applications, and these points exhibit minimal or no deformation, thus eliminating the need for complex dynamic modeling, this application provides a solution by dividing the scene into static points and dynamic points and modeling them separately. This proposes a method that simultaneously achieves fast convergence, compact storage, and real-time high-quality rendering. This strategy can significantly reduce computational overhead and storage requirements.

[0060] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or dynamic scene reconstruction device capable of performing the above functions. The following description uses a dynamic scene reconstruction device as an example to illustrate this embodiment and the subsequent embodiments.

[0061] Based on this, embodiments of this application provide a dynamic scene reconstruction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the dynamic scene reconstruction method of this application.

[0062] In this embodiment, the dynamic scene reconstruction method includes steps S10 to S40:

[0063] Step S10: Obtain the initial frame image;

[0064] Specifically, in this embodiment of the application, dynamic scene reconstruction is performed based on the acquired initial frame image, wherein the initial frame image contains several frame images, which can be a series of images or individual frames in a video.

[0065] Step S20: Generate or determine dynamic point cloud and static point cloud based on the initial frame image;

[0066] Furthermore, after obtaining the initial frame image, dynamic point clouds and static point clouds can be generated or determined based on the initial frame image.

[0067] Optionally, by initializing the initial frame image, Gaussian points can be obtained. Then, dynamic values ​​can be assigned to each Gaussian point using a dynamic-static decoupling method. Based on the dynamic values ​​corresponding to each Gaussian point, each Gaussian point can be divided into dynamic point clouds and static point clouds, thereby avoiding redundant calculations related to static Gaussian points.

[0068] Step S30: Perform deformation processing on the dynamic point cloud to obtain the deformed dynamic point cloud;

[0069] Furthermore, after generating or determining the dynamic point cloud and static point cloud based on the initial frame image, the dynamic point cloud can be deformed to obtain the deformed dynamic point cloud.

[0070] Optionally, a normalized space is constructed based on the acquired initial frame image. In the normalized space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash-encoded features. The multi-resolution hash-encoded features are combined by a multi-layer perception model to obtain the deformed dynamic point cloud.

[0071] Optionally, the preset hash encoder and multilayer perceptron model need to be trained. The specific training approach includes: taking the dynamic point cloud in the normal space (i.e., the point cloud constructed in the initial frame, the first frame) and time t (i.e., the t-th frame) as input, and then obtaining the deformation at time t through hash encoding and multilayer perceptron. Adding this deformation to the dynamic point cloud can obtain the dynamic point cloud corresponding to the image of frame t. It should be noted that the dynamic point cloud obtained at this time is actually incorrect because the hash encoder and multilayer perceptron model have not been trained correctly. Therefore, it is necessary to render this point cloud to obtain the rendered image at time t, calculate the loss function with the real image of frame t, and perform backpropagation and iterative optimization to train the hash encoder and multilayer perceptron model.

[0072] Step S40: Perform hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image.

[0073] Furthermore, the dynamic point cloud is deformed to obtain a deformed dynamic point cloud, which can then be used for blending rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image.

[0074] Optionally, the step of performing hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image includes:

[0075] The static point cloud and the deformed dynamic point cloud are mixed to obtain mixed Gaussian points;

[0076] Calculate the importance of each Gaussian point in the mixture of Gaussian points;

[0077] The Gaussian points whose importance is less than a preset importance threshold are pruned to obtain the target Gaussian points;

[0078] The target Gaussian point is rendered to obtain the target rendered image.

[0079] Optionally, in the original 3DGS, there is an opacity reset operation where the opacity of all points is periodically reduced. After the pruning phase, points with low opacity are removed, which is beneficial for static scenes. However, this method is not suitable for dynamic scenes because it leads to excessive coupling between the canonical and deformable spaces, resulting in low rendering quality outside the training viewpoint. To alleviate the severe coupling between the canonical and deformable fields, the opacity reset operation is eliminated in this embodiment. To address the resulting float, a temporal importance pruning method is employed in this embodiment. This involves calculating the importance of each Gaussian point at each timestamp to each training viewpoint. Integrals contributing below a certain threshold can be pruned, effectively reducing the float problem. For a Gaussian point g... i Importance wi The calculation is as follows:

[0080]

[0081] Where I represents an image from all training views, p represents a pixel in image I, and α i (t) is the product of the transparency of the Gaussian point at time t and the 2D Gaussian probability.

[0082] In this embodiment, the preset importance threshold is set to 0.02, that is, when the importance satisfies w i When the value is less than 0.02, it is trimmed in this embodiment of the application. Using this method, artifacts suspended in the air that are not observed in the training view can be effectively removed.

[0083] This embodiment, through the above scheme, specifically obtains an initial frame image; generates or determines dynamic and static point clouds based on the initial frame image; performs deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud; and performs hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image. By dividing the dynamic and static point clouds, and avoiding uniform deformation processing of the static point cloud, the computational overhead and storage requirements for deformation can be reduced, thereby improving the efficiency of dynamic scene reconstruction.

[0084] Based on the first embodiment of this application, a second embodiment of this application is proposed. In the second embodiment of this application, the contents that are the same as or similar to those in the first embodiment can be referred to the above description, and will not be repeated hereafter.

[0085] Based on this, please refer to Figure 2 Step S20 also includes steps S201 to S203:

[0086] S201: Initialize the initial frame image to obtain Gaussian points;

[0087] S202: Perform dynamic value allocation on each Gaussian point to obtain the dynamic value corresponding to each Gaussian point;

[0088] S203: Compare the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

[0089] Reference Figure 3 , Figure 3 This is a schematic diagram of the overall method principle according to the second embodiment, such as... Figure 3As shown in the embodiment of this application, an initial frame image is used to obtain a well-initialized point cloud. Then, the point cloud is divided into dynamic and static categories using the trained dynamic value d. A 4D hash encoder and a multilayer perceptron decoder are used to deform the dynamic points. For static points, the conventional approach is followed in this embodiment. Finally, the deformed dynamic points and static points are blended for rendering.

[0090] Optionally, an initial sparse point cloud is first obtained using the traditional SfM method, and then the initial sparse point cloud is trained using the first frame image to obtain a complete point cloud corresponding to the first frame image. The space constructed by this point cloud is referred to as the canonical space in this embodiment of the application.

[0091] Optionally, the step of dynamically assigning values ​​to each Gaussian point to obtain the dynamic value corresponding to each Gaussian point includes:

[0092] Select or determine the pixels corresponding to each Gaussian point in each view, and calculate the time variance of each pixel in the time domain;

[0093] Using the time variance as a monitoring signal, backpropagation and iterative optimization are performed on the dynamic values ​​of each Gaussian point to obtain the dynamic values ​​corresponding to each Gaussian point.

[0094] Optionally, the step of comparing the dynamic values ​​corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud includes:

[0095] Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold.

[0096] The Gaussian points whose dynamic values ​​are greater than the preset dynamic threshold are classified as the dynamic point cloud.

[0097] Gaussian points whose dynamic values ​​are less than or equal to the preset dynamic threshold are classified as static point clouds.

[0098] Optionally, this application provides a dynamic-static decoupling method. This method avoids redundant calculations related to static Gaussian points. Therefore, the time-varying model can focus on learning dynamic components, accelerating convergence, and improving rendering quality. To achieve precise control over each Gaussian point for finer-grained operation and global sharing, this application introduces a dynamic parameter d (ranging from negative infinity to positive infinity) for each Gaussian point. This parameter represents the dynamic value of the Gaussian point; a higher d indicates more significant motion. To accurately assign the dynamic value d to each Gaussian point, this application utilizes the pixel-level temporal variance in the training video as a supervision signal. The specific formula is as follows:

[0099]

[0100] Here, T represents the entire time domain. To distinguish between dynamic and static pixels, in this embodiment, the temporal variance S(p) of each pixel p over the entire time domain T is calculated, and then S(p) is binarized to D(p) using a threshold γ = 0.02. This provides pixel-level supervision as follows: if S(p) > γ, then D(p) = 1, indicating that the pixel is dynamic. All Gaussian points intersecting with dynamic pixels will receive a positive dynamic value during backpropagation, which decreases with depth and occlusion by other Gaussian points. Otherwise, if S(p) ≤ γ and D(p) = 0, the pixel is considered static. Accordingly, the associated Gaussian points will receive a negative dynamic value.

[0101] Optionally, the dynamic values ​​are rendered into a dynamic value rendering image using alpha blending technology, and then the pixel values ​​are controlled between [0,1] using the Sigmoid activation function. This allows the loss to be calculated with the mask after binarization of the temporal variance, and backpropagation is performed to iteratively optimize the dynamic values. Optimization can be stopped after 3000 rounds, and the dynamic value d at this point can be taken as the final result.

[0102] Reference Figure 4 , Figure 4 This is a schematic diagram showing the exploded view according to the second embodiment, as follows: Figure 4 As shown, in this embodiment, the dynamic value d of each Gaussian point is rendered onto the camera plane to obtain a dynamic image. The dynamic value of a single pixel p in the dynamic image... The calculation formula is as follows:

[0103]

[0104] in, Let α represent the estimated dynamic value of a pixel, where α is the product of the Gaussian pixel's transparency and the probability of a 2D Gaussian pixel. By applying the sigmoid function, the dynamic value is optimized to span from negative infinity to positive infinity, thus enabling finer differentiation of the degree of dynamics. This transforms dynamic value estimation into a binary classification problem.

[0105] Therefore, optimizing the dynamic value of each Gaussian point can be achieved by minimizing the binary cross-entropy loss:

[0106]

[0107] By minimizing this loss function, the dynamic values ​​d of the Gaussian points pre-trained using the first frame are effectively optimized. The entire optimization process is highly efficient, typically completed within one minute. Finally, Gaussian points with dynamic values ​​greater than a preset dynamic threshold ζ (e.g., set to 7.0) are classified as dynamic; otherwise, they are classified as static.

[0108] It should be noted that the method in the embodiments of this application can be adapted even in the presence of obstruction. For example... Figure 4 As shown, although the dynamic pixels in view 1 (marked in orange) assign positive dynamic values ​​to Gaussian points that should be classified as static (marked in blue), the results of other views (such as view 2 and view 3) cause these Gaussian points to receive larger negative dynamic values, thus ensuring their correct classification overall.

[0109] This embodiment, through the above scheme, specifically obtains Gaussian points by initializing the initial frame image; dynamically assigns values ​​to each Gaussian point to obtain the corresponding dynamic value; and compares the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud. By using a dynamic-static decoupling method, the accuracy of the classification of dynamic and static point clouds is ensured, thereby avoiding redundant calculations related to static Gaussian points, reducing the computational overhead and storage requirements for deformation, and thus improving the efficiency of dynamic scene reconstruction.

[0110] Based on any of the foregoing embodiments of this application, a third embodiment of this application is proposed. In this third embodiment, any content that is the same as or similar to any of the foregoing embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 Step S30 also includes steps S301 to S303:

[0111] S301: Construct a canonical space based on the initial frame image;

[0112] S302: In the specified space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash encoding features;

[0113] S303: The multi-resolution hash-encoded features are combined using a multi-layer perceptual model to obtain the deformed dynamic point cloud.

[0114] Optionally, this application proposes a high-efficiency spatiotemporal structure encoder and a 4D multi-resolution hash function. 4d A Gaussian deformation decoder φ is used to predict the deformation of each 3D Gaussian. First, in this embodiment, a canonical space is constructed using the first frame image from all training angles. Subsequently, the static-dynamic segmentation method described in the foregoing embodiments is used to obtain static and dynamic point clouds.

[0115] Optionally, the preset hash encoder and multilayer perceptron model need to be trained. The specific training approach includes: taking the dynamic point cloud in the normal space (i.e., the point cloud constructed in the initial frame, the first frame) and time t (i.e., the t-th frame) as input, and then obtaining the deformation at time t through hash encoding and multilayer perceptron. Adding this deformation to the dynamic point cloud can obtain the dynamic point cloud corresponding to the image of frame t. It should be noted that the dynamic point cloud obtained at this time is actually incorrect because the hash encoder and multilayer perceptron model have not been trained correctly. Therefore, it is necessary to render this point cloud to obtain the rendered image at time t, calculate the loss function with the real image of frame t, and perform backpropagation and iterative optimization to train the hash encoder and multilayer perceptron model.

[0116] Optionally, in the embodiments of this application, 4D multi-resolution hashing h is utilized. 4d Encoding is performed to efficiently model the temporal information of dynamic 3D Gaussian points by normalizing the point cloud to a hash grid range. Each resolution voxel grid is mapped to a hash table storing F-dimensional learnable feature vectors. For a given 4D dynamic Gaussian point (μ,t)∈R... 4d Its hash code at resolution l is represented as: h 4d (μ,t;l)∈R LF It is calculated through linear interpolation of the feature vectors associated with the eight corners of the surrounding grid. Therefore, its multi-resolution hash encoding features are as follows:

[0117]

[0118] Here, L represents the number of resolution levels, typically set to 16. Following this, a small MLP combines all features to produce f. d =φ d (f h Using 4D hashing 4d Hash tables offer several advantages as encoders: compactness, O(1) query complexity, and the ability to effectively integrate global and local information across multiple resolutions. It's worth noting that hash tables can compactly store dynamic information due to the separation of static and dynamic data. In contrast, grid-based methods may result in significant space waste because the proportion of truly dynamic information that needs to be stored is relatively small.

[0119] Reference Figure 6 , Figure 6 This is a schematic diagram illustrating the application effects of different methods according to the third embodiment, such as... Figure 6 As shown, for time modeling, this embodiment employs a deformation field method, using a compact multi-resolution 4DHash as the deformer, which maps dynamic Gaussian points from the normal space to a deformation space with arbitrary timestamps. It is worth noting that, as... Figure 6As shown in (c), temporal modeling is specifically applied to dynamic points, while static points are treated as time-invariant, significantly reducing computational requirements. The reduced number of dynamic points allows 4DHash to focus on deformation, thereby accelerating convergence and improving rendering quality. Finally, the static and dynamic rendering results are combined in this embodiment to generate the final output. This method also addresses the blurring issue caused by static elements interfering with time-aware multi-resolution 4DHash. The method in this embodiment achieves state-of-the-art performance in terms of training and rendering speed, storage efficiency, and rendering quality. Furthermore, experimental results demonstrate that the method in this embodiment remains effective even when dealing with large-scale motion.

[0120] This embodiment, through the above scheme, specifically constructs a normalized space based on the initial frame image; in the normalized space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash encoded features; the multi-resolution hash encoded features are combined by a multi-layer perceptron model to obtain the deformed dynamic point cloud. By introducing a compact multi-resolution 4D hash encoder, which occupies as little as 8MB, it can effectively model the spatiotemporal domain. This method can not only improve rendering quality and accelerate training, but also ensure efficient and compact storage.

[0121] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the dynamic scene reconstruction method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.

[0122] This application also provides a dynamic scene reconstruction device; please refer to... Figure 7 The dynamic scene reconstruction device includes:

[0123] The acquisition module is used to acquire the initial frame image;

[0124] The classification module is used to generate or determine dynamic point clouds and static point clouds based on the initial frame image;

[0125] The deformation module is used to deform the dynamic point cloud to obtain the deformed dynamic point cloud.

[0126] The rendering module is used to perform hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image.

[0127] The dynamic scene reconstruction apparatus provided in this application, employing the dynamic scene reconstruction method in the above embodiments, can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the dynamic scene reconstruction apparatus provided in this application are the same as those of the dynamic scene reconstruction method provided in the above embodiments, and other technical features in the dynamic scene reconstruction apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0128] This application provides a dynamic scene reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the dynamic scene reconstruction method in the first embodiment described above.

[0129] The following is for reference. Figure 8 This document illustrates a structural schematic diagram of a dynamic scene reconstruction device suitable for implementing embodiments of this application. The dynamic scene reconstruction device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The dynamic scene reconstruction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0130] like Figure 8As shown, the dynamic scene reconstruction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the dynamic scene reconstruction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the dynamic scene reconstruction device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows dynamic scene reconstruction devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0131] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0132] The dynamic scene reconstruction device provided in this application, employing the dynamic scene reconstruction method in the above embodiments, can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the dynamic scene reconstruction device provided in this application are the same as those of the dynamic scene reconstruction method provided in the above embodiments, and other technical features of the dynamic scene reconstruction device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0133] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0135] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the dynamic scene reconstruction method in the above embodiments.

[0136] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0137] The aforementioned computer-readable storage medium may be included in the dynamic scene reconstruction device; or it may exist independently and not be assembled into the dynamic scene reconstruction device.

[0138] The aforementioned computer-readable storage medium carries one or more programs. When the aforementioned one or more programs are executed by the dynamic scene reconstruction device, the dynamic scene reconstruction device: acquires an initial frame image; generates or determines dynamic point clouds and static point clouds based on the initial frame image; performs deformation processing on the dynamic point clouds to obtain deformed dynamic point clouds; and performs hybrid rendering based on the static point clouds and the deformed dynamic point clouds to obtain a target rendered image. By dividing the dynamic point clouds and static point clouds, and avoiding uniform deformation processing of the static point clouds, the computational overhead and storage requirements for deformation can be reduced, thereby improving the efficiency of dynamic scene reconstruction.

[0139] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0142] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described dynamic scene reconstruction method, and is capable of solving the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the dynamic scene reconstruction method provided in the above embodiments, and will not be repeated here.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the dynamic scene reconstruction method described above.

[0144] The computer program product provided in this application can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the dynamic scene reconstruction method provided in the above embodiments, and will not be repeated here.

[0145] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A dynamic scene reconstruction method, characterized in that, The method includes: Obtain the initial frame image; Dynamic and static point clouds are generated or determined based on the initial frame image; The dynamic point cloud is deformed to obtain a deformed dynamic point cloud. The target rendered image is obtained by performing a hybrid rendering of the static point cloud and the deformed dynamic point cloud. The step of performing hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image includes: The static point cloud and the deformed dynamic point cloud are mixed to obtain mixed Gaussian points; Calculate the importance of each Gaussian point in the mixture of Gaussian points; The Gaussian points whose importance is less than a preset importance threshold are pruned to obtain the target Gaussian points; The target Gaussian point is rendered to obtain the target rendered image.

2. The method as described in claim 1, characterized in that, The steps of generating or determining dynamic and static point clouds based on the initial frame image include: The initial frame image is initialized to obtain Gaussian points; Dynamic values ​​are assigned to each Gaussian point to obtain the dynamic values ​​corresponding to each Gaussian point. The dynamic value corresponding to each Gaussian point is compared with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

3. The method as described in claim 2, characterized in that, The step of dynamically assigning values ​​to each Gaussian point to obtain the dynamic value corresponding to each Gaussian point includes: Select or determine the pixels corresponding to each Gaussian point in each view, and calculate the time variance of each pixel in the time domain; Using the time variance as a monitoring signal, backpropagation and iterative optimization are performed on the dynamic values ​​of each Gaussian point to obtain the dynamic values ​​corresponding to each Gaussian point.

4. The method as described in claim 2, characterized in that, The step of comparing the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud includes: Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold. The Gaussian points whose dynamic values ​​are greater than the preset dynamic threshold are classified as the dynamic point cloud. Gaussian points whose dynamic values ​​are less than or equal to the preset dynamic threshold are classified as static point clouds.

5. The method as described in claim 1, characterized in that, The step of deforming the dynamic point cloud to obtain the deformed dynamic point cloud includes: Construct a canonical space based on the initial frame image; In the specified space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash encoding features; The deformed dynamic point cloud is obtained by combining the multi-resolution hash-encoded features using a multi-layer perception model.

6. A dynamic scene reconstruction device, characterized in that, The device includes: The acquisition module is used to acquire the initial frame image; The classification module is used to generate or determine dynamic point clouds and static point clouds based on the initial frame image; The deformation module is used to deform the dynamic point cloud to obtain the deformed dynamic point cloud. The rendering module is used to perform hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain the target rendered image; The process of blending and rendering the static point cloud with the deformed dynamic point cloud to obtain the target rendered image includes: The static point cloud and the deformed dynamic point cloud are mixed to obtain mixed Gaussian points; Calculate the importance of each Gaussian point in the mixture of Gaussian points; The Gaussian points whose importance is less than a preset importance threshold are pruned to obtain the target Gaussian points; The target Gaussian point is rendered to obtain the target rendered image.

7. A dynamic scene reconstruction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the dynamic scene reconstruction method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the dynamic scene reconstruction method as described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the dynamic scene reconstruction method as described in any one of claims 1 to 5.

Images