Dynamic scene reconstruction method and device, equipment, medium and product

Abstract

The invention discloses a dynamic scene reconstruction method and device, equipment, a medium and a product, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an initial frame image; generating or determining a dynamic point cloud and a static point cloud based on the initial frame image; performing deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud; and performing hybrid rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image, and by dividing the dynamic point cloud and the static point cloud, avoiding unified deformation processing of the static point cloud, so that the calculation overhead and storage requirements required by deformation can be reduced, and the efficiency of dynamic scene reconstruction is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to methods, devices, equipment, media and products for dynamic scene reconstruction. Background Art

[0002] Dynamic scene reconstruction is one of the important research directions in computer vision, computer graphics, and robotics. It can restore a three-dimensional environment that changes over time from a series of images or videos. This process involves capturing and analyzing moving objects in the scene and changes in the background in order to build a 3D model in the computer that can reflect the dynamic characteristics of the real world.

[0003] The dynamic scene reconstruction methods in the prior art usually uniformly process all Gaussian points during the modeling process, which faces the challenges of high computational overhead and high storage requirements.

[0004] Therefore, it is necessary to propose a solution to reduce the computational overhead and storage requirements during dynamic scene reconstruction.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a dynamic scene reconstruction method, device, equipment, medium and product, aiming to reduce the computing overhead and storage requirements in the dynamic scene reconstruction process.

[0007] To achieve the above object, the present application provides a dynamic scene reconstruction method, the method comprising:

[0008] Get the initial frame image;

[0009] Generate or determine a dynamic point cloud and a static point cloud based on the initial frame image;

[0010] Performing deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud;

[0011] A target rendered image is obtained by performing mixed rendering based on the static point cloud and the deformed dynamic point cloud.

[0012] In one embodiment, the step of generating or determining a dynamic point cloud and a static point cloud based on the initial frame image includes:

[0013] Initializing the initial frame image to obtain Gaussian points;

[0014] Performing dynamic value assignment on each Gaussian point to obtain a dynamic value corresponding to each Gaussian point;

[0015] The dynamic value corresponding to each Gaussian point is compared with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

[0016] In one embodiment, the step of performing dynamic value assignment on each Gaussian point to obtain a dynamic value corresponding to each Gaussian point includes:

[0017] Selecting or determining pixels corresponding to the Gaussian points in each view, and calculating the time variance of the pixels in the time domain;

[0018] The time variance is used as a supervisory signal, and the dynamic values ​​of the Gaussian points are back-propagated and iteratively optimized to obtain the dynamic values ​​corresponding to the Gaussian points.

[0019] In one embodiment, the step of comparing the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud includes:

[0020] Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold;

[0021] Dividing the Gauss points whose corresponding dynamic values ​​are greater than the preset dynamic threshold among the Gauss points into the dynamic point cloud;

[0022] The Gauss points whose corresponding dynamic values ​​are less than or equal to the preset dynamic threshold value are divided into the static point cloud.

[0023] In one embodiment, the step of deforming the dynamic point cloud to obtain a deformed dynamic point cloud includes:

[0024] Constructing a standard space according to the initial frame image;

[0025] In the canonical space, the dynamic point cloud is encoded by a preset hash encoder to obtain a multi-resolution hash coding feature;

[0026] The multi-resolution hash coding features are combined through a multi-layer perception model to obtain the deformed dynamic point cloud.

[0027] In one embodiment, the step of performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image includes:

[0028] Mixing the static point cloud with the deformed dynamic point cloud to obtain mixed Gaussian points;

[0029] Calculating the importance of each Gaussian point in the mixed Gaussian point;

[0030] Pruning the Gaussian points whose importance is less than a preset importance threshold to obtain target Gaussian points;

[0031] The target Gaussian points are rendered to obtain the target rendered image.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a dynamic scene reconstruction device, the dynamic scene reconstruction device comprising:

[0033] An acquisition module, used for acquiring an initial frame image;

[0034] A classification module, used for generating or determining a dynamic point cloud and a static point cloud based on the initial frame image;

[0035] A deformation module, used for deforming the dynamic point cloud to obtain a deformed dynamic point cloud;

[0036] The rendering module is used to perform mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a dynamic scene reconstruction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the dynamic scene reconstruction method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dynamic scene reconstruction method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the dynamic scene reconstruction method described above are implemented.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] By acquiring an initial frame image; generating or determining a dynamic point cloud and a static point cloud based on the initial frame image; deforming the dynamic point cloud to obtain a deformed dynamic point cloud; performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image, by dividing the dynamic point cloud and the static point cloud, avoiding uniform deformation processing of the static point cloud, the computational overhead and storage requirements required for the deformation can be reduced, thereby improving the efficiency of dynamic scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a flow chart provided for the first embodiment of the dynamic scene reconstruction method of the present application;

[0045] Figure 2 A schematic diagram of a flow chart provided for Embodiment 2 of the dynamic scene reconstruction method of the present application;

[0046] Figure 3 is a schematic diagram of the overall method principle according to the second embodiment;

[0047] Figure 4 is a perspective decomposition schematic diagram according to the second embodiment;

[0048] Figure 5 A flowchart of the third embodiment of the dynamic scene reconstruction method of the present application is provided;

[0049] Figure 6 It is a schematic diagram of the application effect of different methods according to the third embodiment;

[0050] Figure 7 This is a schematic diagram of the module structure of the dynamic scene reconstruction device according to an embodiment of the present application;

[0051] Figure 8 Schematic diagram of the device structure of the hardware operating environment involved in the dynamic scene reconstruction method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: by acquiring an initial frame image; generating or determining a dynamic point cloud and a static point cloud based on the initial frame image; deforming the dynamic point cloud to obtain a deformed dynamic point cloud; performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image. By dividing the dynamic point cloud and the static point cloud, the unified deformation processing of the static point cloud is avoided, which can reduce the computational overhead and storage requirements required for the deformation, thereby improving the efficiency of dynamic scene reconstruction.

[0056] In this embodiment, for the convenience of description, the following description is made by taking the dynamic scene reconstruction device as the execution subject.

[0057] Currently, dynamic scene reconstruction is crucial for immersive imaging, driving the development of multiple multimedia technologies such as VR, AR, and the Metaverse. Rendering dynamic scenes from arbitrary time, position, and viewpoint is essential for improving the user experience of multimedia products such as free viewpoint video and bullet time effects.

[0058] In the prior art, dynamic scene reconstruction methods mainly include 4DGS, 3DGStream, Real-Time GS, and Space-TimeGS. Among them, 4DGS introduces a neural voxel encoder to model deformation relationships that change over time; 3DGStream uses a compact neural transformation cache to effectively model the translation and rotation of 3D Gaussians, thereby significantly reducing the training time and storage requirements for each frame. It also has an adaptive 3DG addition strategy to manage new objects in dynamic scenes; Real-Time GS considers space and time as a whole by optimizing a set of 4D primitives, parameterized as anisotropic ellipses that capture geometry and appearance. In addition, Space-TimeGS enhances the standard 3D Gaussian through temporal opacity and motion / rotation parameters, effectively captures static and dynamic elements to simulate dynamic deformation, and uses neural features for splatted rendering, thereby effectively modeling view and time-related appearance. Although these methods achieve higher quality results with faster rendering times, they still face the challenges of long training time and high storage requirements. A potential limitation of the above methods is the uniform treatment of all Gaussian points during the modeling process.

[0059] Considering that in practical applications static points (such as static points in the background area) constitute most of the scene, these points exhibit minimal or no deformation, and therefore do not require complex dynamic modeling, the present application provides a solution by dividing the scene into static points and dynamic points and modeling them separately, proposing a method that simultaneously achieves fast convergence, compact storage, and real-time high-quality rendering. This strategy can significantly reduce computational overhead and storage requirements.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a dynamic scene reconstruction device, etc. The following takes the dynamic scene reconstruction device as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides a dynamic scene reconstruction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the dynamic scene reconstruction method of the present application.

[0062] In this embodiment, the dynamic scene reconstruction method includes steps S10 to S40:

[0063] Step S10, obtaining an initial frame image;

[0064] Specifically, in the embodiment of the present application, dynamic scene reconstruction is performed based on the acquired initial frame image, wherein the initial frame image includes several frame images, and the several frame images can be a series of images or each frame image in a video.

[0065] Step S20, generating or determining a dynamic point cloud and a static point cloud based on the initial frame image;

[0066] Furthermore, after the initial frame image is acquired, a dynamic point cloud and a static point cloud may be generated or determined based on the initial frame image.

[0067] Optionally, each Gaussian point can be obtained by initializing the initial frame image, and then dynamic values ​​are assigned to each Gaussian point through the dynamic-static decoupling method. According to the dynamic values ​​corresponding to each Gaussian point, each Gaussian point can be divided into a dynamic point cloud and a static point cloud, thereby avoiding redundant calculations related to static Gaussian points.

[0068] Step S30, performing deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud;

[0069] Furthermore, after the dynamic point cloud and the static point cloud are generated or determined based on the initial frame image, the dynamic point cloud may be deformed to obtain a deformed dynamic point cloud.

[0070] Optionally, a standard space is constructed based on the acquired initial frame image, in which the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash coding features; the multi-resolution hash coding features are combined by a multi-layer perception model to obtain a deformed dynamic point cloud.

[0071] Optionally, the preset hash encoder and multi-layer perception model need to be trained. The specific training ideas include: taking the dynamic point cloud in the standard space (that is, the initial frame, the point cloud constructed in the first frame) and time t (that is, the tth frame) as input, and then obtaining the deformation at time t through hash coding and multi-layer perception machine. The dynamic point cloud plus this deformation can obtain the dynamic point cloud corresponding to the t-frame image. It should be noted that the dynamic point cloud obtained at this time is actually incorrect, because the hash encoder and the multi-layer perception model have not been correctly trained. Therefore, it is necessary to render this point cloud to obtain the rendered image at time t, and calculate the loss function with the real t-frame image, perform back propagation and iterative optimization, thereby training the hash encoder and multi-layer perception model.

[0072] Step S40: performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image.

[0073] Furthermore, after the dynamic point cloud is deformed to obtain the deformed dynamic point cloud, mixed rendering may be performed based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image.

[0074] Optionally, the step of performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image includes:

[0075] Mixing the static point cloud with the deformed dynamic point cloud to obtain mixed Gaussian points;

[0076] Calculating the importance of each Gaussian point in the mixed Gaussian point;

[0077] Pruning the Gaussian points whose importance is less than a preset importance threshold to obtain target Gaussian points;

[0078] The target Gaussian points are rendered to obtain the target rendered image.

[0079] Optionally, in the original 3DGS, there is a reset opacity operation, in which the opacity of all points is periodically reduced. After the pruning stage, points with lower opacity are deleted, which is beneficial for static scenes. However, this approach is not suitable for dynamic scenes because it can lead to excessive coupling between the canonical space and the deformed space, resulting in lower rendering quality outside the training viewpoints. In order to alleviate the severe coupling between the canonical field and the deformed field, the reset opacity operation is eliminated in the embodiment of the present application. In order to deal with the resulting floating objects, a temporal importance pruning method is used in the embodiment of the present application. This involves calculating the importance of each Gaussian point at each timestamp for each training viewpoint. Integrals whose contributions are below a certain threshold can be cut, effectively reducing the floating problem. For the Gaussian point g i , importance wi The calculation is as follows:

[0080]

[0081] where I represents images from all training views, p represents a pixel in image I, and α i (t) is the product of the transparency of the Gaussian point at time t and the 2D Gaussian probability.

[0082] In the embodiment of the present application, the preset importance threshold is 0.02, that is, when the importance satisfies w i When θ is less than 0.02, it is trimmed in the embodiment of the present application. Using this method, the artifacts suspended in the air that are not observed in the training view can be effectively removed.

[0083] This embodiment adopts the above scheme, specifically by acquiring an initial frame image; generating or determining a dynamic point cloud and a static point cloud based on the initial frame image; deforming the dynamic point cloud to obtain a deformed dynamic point cloud; performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image, by dividing the dynamic point cloud and the static point cloud to avoid uniformly deforming the static point cloud, the computational overhead and storage requirements required for the deformation can be reduced, thereby improving the efficiency of dynamic scene reconstruction.

[0084] Based on the first embodiment of the present application, a second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the above-mentioned embodiment 1 can be referred to the above introduction and will not be repeated later.

[0085] On this basis, please refer to Figure 2 , step S20 also includes steps S201 to S203:

[0086] S201: Initializing the initial frame image to obtain Gaussian points;

[0087] S202: performing dynamic value assignment on each Gaussian point to obtain a dynamic value corresponding to each Gaussian point;

[0088] S203: Compare the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

[0089] Reference Figure 3 , Figure 3 FIG. 1 is a schematic diagram showing the overall method principle according to the second embodiment. Figure 3As shown, in the embodiment of the present application, the initial frame image is used to obtain a well-initialized point cloud. Then, the point cloud is divided into two categories, dynamic and static, through the dynamic value d obtained by training. The dynamic points are deformed using a 4D hash encoder and a multi-layer perceptron decoder. For static points, the embodiment of the present application follows the usual practice. Finally, the deformed dynamic points are mixed with the static points for rendering.

[0090] Optionally, an initial sparse point cloud is first obtained through the traditional SfM method, and then the initial sparse point cloud is trained using the first frame image to obtain a complete point cloud corresponding to the first frame image. The space constructed by this point cloud is referred to as the standard space in the embodiment of the present application.

[0091] Optionally, the step of performing dynamic value assignment on each Gaussian point to obtain a dynamic value corresponding to each Gaussian point includes:

[0092] Selecting or determining pixels corresponding to the Gaussian points in each view, and calculating the time variance of the pixels in the time domain;

[0093] The time variance is used as a supervisory signal, and the dynamic values ​​of the Gaussian points are back-propagated and iteratively optimized to obtain the dynamic values ​​corresponding to the Gaussian points.

[0094] Optionally, the step of comparing the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud includes:

[0095] Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold;

[0096] Dividing the Gauss points whose corresponding dynamic values ​​are greater than the preset dynamic threshold among the Gauss points into the dynamic point cloud;

[0097] The Gauss points whose corresponding dynamic values ​​are less than or equal to the preset dynamic threshold value are divided into the static point cloud.

[0098] Optionally, a dynamic-static decoupling method is provided in an embodiment of the present application. This method can avoid redundant calculations associated with static Gaussian points. Therefore, the time-varying model can focus on learning dynamic components, accelerating convergence and improving rendering quality. In order to achieve precise control of each Gaussian point to achieve more sophisticated operations and global sharing, a dynamic parameter d (ranging from negative infinity to positive infinity) is introduced for each Gaussian point in an embodiment of the present application. This parameter represents the dynamic value of the Gaussian point, and the higher d is, the more significant the movement. In order to accurately assign the dynamic value d to each Gaussian point, the pixel-level temporal variance in the training video is used as a supervisory signal in an embodiment of the present application. The specific formula is as follows:

[0099]

[0100] Wherein, T represents the entire time domain. In order to distinguish between dynamic and static pixels, the temporal variance S(p) of each pixel p in the entire time domain T is calculated in an embodiment of the present application, and then S(p) is binarized into D(p) using a threshold value γ=0.02. This provides pixel-level supervision as follows: If S(p)>γ, then D(p)=1, indicating that the pixel is dynamic. All Gaussian points that intersect with dynamic pixels will receive positive dynamic values ​​during back propagation, which decreases with depth and occlusion of other Gaussian points. Otherwise, if S(p)≤γ, D(p)=0, the pixel is considered to be static. Accordingly, the relevant Gaussian points will receive negative dynamic values.

[0101] Optionally, the dynamic value is rendered into a dynamic value rendering image through alpha blending technology, and then the pixel value is controlled between [0,1] through the Sigmoid activation function, so that the loss can be calculated with the mask after the time variance is binarized, and the dynamic value is back-propagated and iteratively optimized. When the number of optimization rounds reaches 3000, the optimization can be stopped and the dynamic value d at this time is taken as the final result.

[0102] Reference Figure 4 , Figure 4 FIG. 1 is a perspective decomposition diagram according to the second embodiment. Figure 4 As shown, in the embodiment of the present application, the dynamic value d of each Gaussian point is rendered onto the camera plane to obtain a dynamic image. The dynamic value of a single pixel p in the dynamic image The calculation formula is as follows:

[0103]

[0104] in, represents the estimated dynamic value of the pixel, and α is the probability product of the transparency of the Gaussian point and the 2D Gaussian. By applying the sigmoid function, the dynamic value is optimized to span from negative infinity to positive infinity, so that the degree of dynamics can be distinguished more finely. This transforms the dynamic value estimation into a binary classification problem.

[0105] Therefore, optimizing the dynamic value of each Gaussian point can be achieved by minimizing the binary cross entropy loss:

[0106]

[0107] By minimizing this loss function, the dynamic value d of the Gaussian points pre-trained using the first frame is effectively optimized. The entire optimization process is very efficient and is usually completed within 1 minute. Finally, Gaussian points whose dynamic values ​​are greater than the preset dynamic threshold ζ (for example, set to 7.0) are classified as dynamic; otherwise, they are classified as static.

[0108] It should be noted that even if there is occlusion, the method in the embodiment of the present application can also adapt. Figure 4 As shown, although the dynamic pixels in view 1 (marked in orange) assign positive dynamic values ​​to Gaussian points that should be classified as static (marked in blue), the presence of other views (such as view 2 and view 3) causes these Gaussian points to receive larger negative dynamic values, thereby ensuring their overall correct classification.

[0109] This embodiment adopts the above scheme, specifically by initializing the initial frame image to obtain each Gaussian point; assigning dynamic values ​​to the Gaussian points to obtain the dynamic values ​​corresponding to each Gaussian point; comparing the dynamic values ​​corresponding to the Gaussian points with the preset dynamic thresholds to determine the dynamic point cloud and the static point cloud, and through the dynamic-static decoupling method, ensure the accuracy of the classification of the dynamic point cloud and the static point cloud, thereby avoiding redundant calculations related to static Gaussian points, reducing the computational overhead and storage requirements required for deformation, and thus improving the efficiency of dynamic scene reconstruction.

[0110] Based on any of the foregoing embodiments of the present application, a third embodiment of the present application is proposed. In the third embodiment of the present application, the same or similar contents as any of the foregoing embodiments can be referred to the above introduction and will not be described in detail later. Figure 5 , step S30 also includes steps S301 to S303:

[0111] S301: constructing a standard space according to the initial frame image;

[0112] S302: In the standard space, the dynamic point cloud is encoded by a preset hash encoder to obtain a multi-resolution hash coding feature;

[0113] S303: Combining the multi-resolution hash coding features through a multi-layer perception model to obtain the deformed dynamic point cloud.

[0114] Optionally, in the embodiment of the present application, an efficient spatiotemporal structure encoder, a 4D multi-resolution hash 4d and Gaussian deformation decoder φ, used to predict the deformation of each 3D Gaussian. First, the first frame image of all training angles is used to construct a canonical space in the embodiment of the present application. Subsequently, the static-dynamic segmentation method described in the above embodiment is used to obtain a static point cloud and a dynamic point cloud.

[0115] Optionally, the preset hash encoder and multi-layer perception model need to be trained. The specific training ideas include: taking the dynamic point cloud in the standard space (that is, the initial frame, the point cloud constructed in the first frame) and time t (that is, the tth frame) as input, and then obtaining the deformation at time t through hash coding and multi-layer perception machine. The dynamic point cloud plus this deformation can obtain the dynamic point cloud corresponding to the t-frame image. It should be noted that the dynamic point cloud obtained at this time is actually incorrect, because the hash encoder and the multi-layer perception model have not been correctly trained. Therefore, it is necessary to render this point cloud to obtain the rendered image at time t, and calculate the loss function with the real t-frame image, perform back propagation and iterative optimization, thereby training the hash encoder and multi-layer perception model.

[0116] Optionally, in the embodiment of the present application, 4D multi-resolution hashing h is used 4d Encoding is performed to efficiently model the temporal information of dynamic 3D Gaussian points by normalizing the point cloud into a hash grid range. Each voxel grid of resolution is mapped to a hash table storing F-dimensional learnable feature vectors. For a given 4D dynamic Gaussian point (μ, t) ∈ R 4d , its hash code at resolution l is expressed as: h 4d (μ,t;l)∈R LF , is calculated by linear interpolation of the feature vectors associated with the eight corners of the surrounding grid. Therefore, its multi-resolution hash encoding features are as follows:

[0117]

[0118] Where L represents the number of resolution levels, which is usually set to 16. After this, a small MLP combines all features to produce f d =φ d (f h ). Using 4D hash h 4d As an encoder, it has several advantages: compactness, O(1) query complexity, and multi-resolution methods effectively integrate global and local information. It should be noted that due to the separation of dynamic and static information, the hash table can compactly store dynamic information. In contrast, grid-based methods may lead to serious space waste because the proportion of truly dynamic information that needs to be stored is relatively small.

[0119] Reference Figure 6 , Figure 6 FIG. 1 is a schematic diagram showing the effects of applying different methods according to the third embodiment. Figure 6 As shown, for temporal modeling, the deformation field method is adopted in the embodiment of the present application, and a compact multi-resolution 4DHash is used as a deformer, which maps dynamic Gaussian points from the standard space to the deformation space of arbitrary timestamps. It is worth noting that, as Figure 6As shown in (c), temporal modeling is applied specifically to dynamic points, while static points are treated as time-invariant, significantly reducing computational requirements. The reduction in the number of dynamic points enables 4DHash to focus on deformation, thereby accelerating convergence and improving rendering quality. Finally, in an embodiment of the present application, the static and dynamic rendering results are combined to generate the final output. This approach also solves the blurring problem caused by static elements interfering with the time-aware multi-resolution 4DHash. The method in the embodiment of the present application achieves advanced performance in terms of training and rendering speed, storage efficiency, and rendering quality. In addition, experimental results show that the method in the embodiment of the present application remains effective even in cases involving large motions.

[0120] This embodiment adopts the above scheme, specifically by constructing a canonical space according to the initial frame image; in the canonical space, the dynamic point cloud is encoded by a preset hash encoder to obtain multi-resolution hash coding features; the multi-resolution hash coding features are combined by a multi-layer perception model to obtain the deformed dynamic point cloud. By introducing a compact multi-resolution 4D hash encoder, which occupies a space as small as 8MB, the spatiotemporal domain can be effectively modeled. This method can not only improve rendering quality and accelerate training, but also ensure efficient and compact storage.

[0121] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the dynamic scene reconstruction method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0122] This application also provides a dynamic scene reconstruction device, please refer to Figure 7 , the dynamic scene reconstruction device comprises:

[0123] An acquisition module, used for acquiring an initial frame image;

[0124] A classification module, used for generating or determining a dynamic point cloud and a static point cloud based on the initial frame image;

[0125] A deformation module, used for deforming the dynamic point cloud to obtain a deformed dynamic point cloud;

[0126] The rendering module is used to perform mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image.

[0127] The dynamic scene reconstruction device provided by the present application adopts the dynamic scene reconstruction method in the above embodiment, which can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the dynamic scene reconstruction device provided by the present application are the same as the beneficial effects of the dynamic scene reconstruction method provided by the above embodiment, and other technical features in the dynamic scene reconstruction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0128] The present application provides a dynamic scene reconstruction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the dynamic scene reconstruction method in the above-mentioned embodiment one.

[0129] Reference below Figure 8 , which shows a schematic diagram of the structure of a dynamic scene reconstruction device suitable for implementing the embodiment of the present application. The dynamic scene reconstruction device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The dynamic scene reconstruction device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 8As shown, the dynamic scene reconstruction device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the dynamic scene reconstruction device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the dynamic scene reconstruction device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a dynamic scene reconstruction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0131] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0132] The dynamic scene reconstruction device provided by the present application adopts the dynamic scene reconstruction method in the above embodiment to solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the dynamic scene reconstruction device provided by the present application are the same as the beneficial effects of the dynamic scene reconstruction method provided by the above embodiment, and the other technical features in the dynamic scene reconstruction device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0133] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0135] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the dynamic scene reconstruction method in the above-mentioned embodiment.

[0136] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0137] The computer-readable storage medium may be included in the dynamic scene reconstruction device; or may exist independently without being assembled into the dynamic scene reconstruction device.

[0138] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the dynamic scene reconstruction device, the dynamic scene reconstruction device: obtains an initial frame image; generates or determines a dynamic point cloud and a static point cloud based on the initial frame image; deforms the dynamic point cloud to obtain a deformed dynamic point cloud; performs mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image. By dividing the dynamic point cloud and the static point cloud, the static point cloud can be avoided from being uniformly deformed, which can reduce the computational overhead and storage requirements required for the deformation, thereby improving the efficiency of dynamic scene reconstruction.

[0139] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0142] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned dynamic scene reconstruction method, and can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the dynamic scene reconstruction method provided in the above-mentioned embodiment, and will not be repeated here.

[0143] The present application also provides a computer program product, including a computer program, which implements the steps of the dynamic scene reconstruction method as described above when executed by a processor.

[0144] The computer program product provided by the present application can solve the technical problem of dynamic scene reconstruction. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the dynamic scene reconstruction method provided by the above embodiment, which will not be repeated here.

[0145] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A dynamic scene reconstruction method, characterized in that: The method comprises: Get the initial frame image; Generate or determine a dynamic point cloud and a static point cloud based on the initial frame image; Performing deformation processing on the dynamic point cloud to obtain a deformed dynamic point cloud; A target rendered image is obtained by performing mixed rendering based on the static point cloud and the deformed dynamic point cloud.

2. The method according to claim 1, characterized in that The step of generating or determining a dynamic point cloud and a static point cloud based on the initial frame image comprises: Initializing the initial frame image to obtain Gaussian points; Performing dynamic value assignment on each Gaussian point to obtain a dynamic value corresponding to each Gaussian point; The dynamic value corresponding to each Gaussian point is compared with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud.

3. The method according to claim 2, characterized in that The step of allocating dynamic values ​​to the Gaussian points to obtain dynamic values ​​corresponding to the Gaussian points comprises: Selecting or determining pixels corresponding to the Gaussian points in each view, and calculating the time variance of the pixels in the time domain; The time variance is used as a supervisory signal, and the dynamic values ​​of the Gaussian points are back-propagated and iteratively optimized to obtain the dynamic values ​​corresponding to the Gaussian points.

4. The method according to claim 2, characterized in that The step of comparing the dynamic value corresponding to each Gaussian point with a preset dynamic threshold to determine the dynamic point cloud and the static point cloud comprises: Determine whether the dynamic value corresponding to each Gaussian point is greater than the preset dynamic threshold; Dividing the Gauss points whose corresponding dynamic values ​​are greater than the preset dynamic threshold among the Gauss points into the dynamic point cloud; The Gauss points whose corresponding dynamic values ​​are less than or equal to the preset dynamic threshold value are divided into the static point cloud.

5. The method according to claim 1, characterized in that The step of deforming the dynamic point cloud to obtain a deformed dynamic point cloud comprises: Constructing a standard space according to the initial frame image; In the canonical space, the dynamic point cloud is encoded by a preset hash encoder to obtain a multi-resolution hash coding feature; The multi-resolution hash coding features are combined through a multi-layer perception model to obtain the deformed dynamic point cloud.

6. The method according to claim 1, characterized in that The step of performing mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image comprises: Mixing the static point cloud with the deformed dynamic point cloud to obtain mixed Gaussian points; Calculating the importance of each Gaussian point in the mixed Gaussian point; Pruning the Gaussian points whose importance is less than a preset importance threshold to obtain target Gaussian points; The target Gaussian points are rendered to obtain the target rendered image.

7. A dynamic scene reconstruction device, characterized in that: The device comprises: An acquisition module, used for acquiring an initial frame image; A classification module, used for generating or determining a dynamic point cloud and a static point cloud based on the initial frame image; A deformation module, used for deforming the dynamic point cloud to obtain a deformed dynamic point cloud; The rendering module is used to perform mixed rendering based on the static point cloud and the deformed dynamic point cloud to obtain a target rendered image.

8. A dynamic scene reconstruction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the dynamic scene reconstruction method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the dynamic scene reconstruction method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the dynamic scene reconstruction method according to any one of claims 1 to 6 are implemented.

Images