Three-dimensional dynamic scene rendering method, device, equipment, storage medium and program product

Abstract

The present application discloses a three-dimensional dynamic scene rendering method, device, equipment, storage medium and program product, which relates to the field of three-dimensional reconstruction technology. The method includes: receiving a multi-point video stream of a target initial scene, the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with several cameras; reconstructing the three-dimensional scene according to the multi-point video stream to obtain an initial scene representation; learning scene motion changes based on the initial scene representation to obtain scene motion change parameters; sending the scene motion change parameters to the client in the form of a data stream, and the client rendering and generating a scene view according to the scene motion change parameters. The method realizes the three-dimensional dynamic scene reconstruction that can be streamed by free viewpoint video, solves the problem that the prior art free viewpoint video cannot realize the three-dimensional dynamic scene reconstruction that can be streamed, and the rendering speed is slow, and improves the rendering speed of the free viewpoint video.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional dynamic scene rendering method, device, equipment, storage medium and program product. Background Art

[0002] Driven by the increasing demand for virtual reality and augmented reality, as well as the pursuit of a more immersive viewing experience, Free-viewpoint Video (FVV) technology is based on multi-view image capture and 3D reconstruction technology. It obtains image data from multiple cameras, reconstructs the 3D model of the scene through an algorithm, and then renders and generates videos that can be viewed from any perspective.

[0003] However, Gaussian splashing is currently the most advanced 3D reconstruction technology. It uses Gaussian ellipsoids as primitives to explicitly represent scenes. Although it has high-quality rendering capabilities, most existing technologies are offline learning methods and require the collection of complete videos before reconstruction. They cannot be transmitted to the client through the network for real-time processing as conveniently as streaming media. They require large amounts of calculations and slow rendering speeds.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide a three-dimensional dynamic scene rendering method, device, equipment, storage medium and program product, aiming to solve the technical problems that the existing technology of free viewpoint video cannot achieve streaming three-dimensional dynamic scene reconstruction and has slow rendering speed.

[0006] To achieve the above objectives, the present application proposes a three-dimensional dynamic scene rendering method, the method comprising:

[0007] Receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives by a plurality of cameras;

[0008] Reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation;

[0009] Performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters;

[0010] The scene motion change parameters are sent to the client in the form of a data stream, and the client renders and generates a scene view according to the scene motion change parameters.

[0011] In one embodiment, the step of reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation includes:

[0012] Reading a first multi-view frame in the multi-point video stream;

[0013] Initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene;

[0014] Based on the three-dimensional Gaussian splashing and noise smoothing mechanism, a densification operation is performed on the initial scene three-dimensional Gaussian to obtain an initial scene representation.

[0015] In one embodiment, the step of initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene includes:

[0016] Estimate the position information of the multi-point video streams corresponding to the plurality of cameras by using a motion structure recovery algorithm;

[0017] Based on the pose information and the first frame multi-view, construct a sparse point cloud of the target initial scene, wherein the sparse point cloud includes at least position and color;

[0018] The Gaussian primitives of the target initial scene are initialized according to the position and color of the sparse point cloud to obtain a three-dimensional Gaussian of the initial scene.

[0019] In one embodiment, the step of performing a densification operation on the initial scene three-dimensional Gaussian based on a three-dimensional Gaussian splashing and noise smoothing mechanism to obtain an initial scene representation comprises:

[0020] Calculate the spatial position gradient of the three-dimensional Gaussian of the initial scene;

[0021] If the spatial position gradient exceeds a preset threshold, a new scene 3D Gaussian is generated based on the initial scene 3D Gaussian based on a 3D Gaussian splash and noise smoothing mechanism;

[0022] An initial scene representation is obtained according to the initial scene three-dimensional Gaussian and the newly added scene three-dimensional Gaussian.

[0023] In one embodiment, before the step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters, the step further includes:

[0024] Dividing the target initial scene into a plurality of scene areas of preset sizes, and dividing the plurality of scene areas into a plurality of hierarchical sub-areas respectively;

[0025] The step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters comprises:

[0026] Based on the initial scene representation and the plurality of hierarchical sub-regions, scene motion change learning is performed according to the multi-point video stream to obtain hierarchical region motion change parameters;

[0027] A hierarchical motion superposition is performed according to the hierarchical region motion change parameters to obtain scene motion change parameters, wherein the scene motion change parameters at least include one or more three-dimensional Gaussian parameters of position change parameters and rotation change parameters.

[0028] In one embodiment, the step of performing scene motion change learning based on the initial scene representation and the plurality of hierarchical sub-regions according to the multi-point video stream to obtain hierarchical region motion change parameters includes:

[0029] Acquire a second frame multi-view corresponding to a next frame of the first frame multi-view in the multi-point video stream;

[0030] Based on the initial scene representation, performing cross-frame dynamic change learning according to the first frame multi-view and the second frame multi-view to obtain a cross-frame dynamic change parameter;

[0031] According to the plurality of hierarchical sub-regions and the cross-frame dynamic change parameter, a hierarchical region motion change parameter is obtained.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a three-dimensional dynamic scene rendering device, the three-dimensional dynamic scene rendering device comprising:

[0033] A receiving module, used for receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with a plurality of cameras;

[0034] A reconstruction module, used to reconstruct a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation;

[0035] A learning module, configured to perform scene motion change learning based on the initial scene representation to obtain scene motion change parameters;

[0036] The rendering module is used to send the scene motion change parameters to the client in the form of a data stream, and the client generates a scene view by rendering according to the scene motion change parameters.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a three-dimensional dynamic scene rendering device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of three-dimensional dynamic scene rendering as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of three-dimensional dynamic scene rendering as described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of three-dimensional dynamic scene rendering as described above are implemented.

[0040] The present application provides a three-dimensional dynamic scene rendering, which first receives a multi-point video stream obtained by continuously shooting a target initial scene from multiple perspectives through several cameras to obtain video stream data shot at different perspectives; performs three-dimensional scene reconstruction based on the multi-point video stream to obtain an initial scene representation, and performs three-dimensional scene processing before scene rendering with the scene representation obtained through the three-dimensional scene reconstruction; performs scene motion change learning based on the initial scene representation to obtain scene motion change parameters to learn the influence of multi-point video streams on three-dimensional scene motion changes; sends the scene motion change parameters to the client for scene rendering to generate scene views for users to watch from different perspectives in the client. The present application realizes three-dimensional scene reconstruction that can be streamed with free viewpoint video, solves the problem that the prior art free viewpoint video cannot realize three-dimensional dynamic scene reconstruction that can be streamed, and the rendering speed is slow, and improves the rendering speed of free viewpoint video. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of the process flow provided for the first embodiment of the three-dimensional dynamic scene rendering of the present application;

[0044] Figure 2 A schematic diagram of a process flow provided for the second embodiment of the three-dimensional dynamic scene rendering of the present application;

[0045] Figure 3 A brief flowchart diagram of the third embodiment of the three-dimensional dynamic scene rendering of the present application;

[0046] Figure 4A reconstruction process framework and schematic diagram of the main module designs provided for the third embodiment of the three-dimensional dynamic scene rendering of the present application;

[0047] Figure 5 A histogram comparing the number of Gaussian primitives in the initial scene and the quality of new perspective synthesis provided in the third embodiment of the three-dimensional dynamic scene rendering of the present application;

[0048] Figure 6 This is a schematic diagram of the module structure of the three-dimensional dynamic scene rendering device according to an embodiment of the present application;

[0049] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the three-dimensional dynamic scene rendering in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solution of the embodiment of the present application is: receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with several cameras; reconstructing a three-dimensional scene based on the multi-point video stream to obtain an initial scene representation; performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters; sending the scene motion change parameters to a client in the form of a data stream, and the client rendering and generating a scene view according to the scene motion change parameters.

[0054] Technical terms involved in this application:

[0055] Free viewpoint video: a video technology that allows users to choose from multiple angles when watching. Usually multiple cameras are used to shoot the same scene. Users can freely switch perspectives through the interface to get a more immersive viewing experience.

[0056] 3D Gaussian Splatting: is a method for explicit radiation field representation. It uses 3D Gaussian function as a flexible and efficient representation to store the radiation information of the scene in a discrete spatial structure. It combines the advantages of implicit and explicit radiation fields and aims to achieve high-quality rendering with faster training and real-time performance.

[0057] 3D Gaussian Primitive: A mathematical model commonly used in computer graphics, image processing, and physical simulation to represent and generate objects and effects in three-dimensional space. It is particularly effective in dealing with phenomena such as blur, scattering, and lighting.

[0058] Alpha blending method: is a technology used in image processing and computer graphics, mainly used to achieve image transparency and synthesis effects, creating visual effects by controlling the transparency (ie alpha value) between different images or layers.

[0059] In the prior art, traditional methods use principles such as image interpolation and multi-view stereo geometry to synthesize new viewpoints, but they do not work well when dealing with occlusions, and unnatural textures are prone to occur when interpolating views between densely captured viewpoints. Therefore, these methods face challenges in dealing with the complex geometry and changing appearance of real-world scenes, and are also limited in capturing the details and dynamic changes of the actual environment.

[0060] With the development of computer vision and graphics, reconstructing free viewpoint video using Neural Radiance Fields (NeRF) has become an emerging method. NeRF uses neural networks to learn the mapping between 3D points in space and their corresponding radiation colors and opacity, and achieves high-fidelity visual effects through volume rendering technology. However, NeRF's volume rendering framework requires sampling a large number of points in space, which is computationally intensive and slow to render, limiting its feasibility in real-time applications.

[0061] Gaussian splashing is currently the most advanced 3D reconstruction technology. It also uses a neural rendering framework similar to NeRF. However, since it directly uses Gaussian ellipsoids as primitives to explicitly represent the scene, it avoids dense sampling in space, so the training is more efficient and high-quality rendering can be performed in real time. Gaussian splashing technology is naturally used to reconstruct free-viewpoint videos, but most of these methods are offline learning methods. They need to collect the complete video before reconstruction, and they cannot be easily transmitted to the client through the network for real-time rendering like streaming media. Recently, there are also methods that use online methods to reconstruct dynamic scenes, but because neural networks are used to implicitly model scene motion, the storage cost is high and the convergence speed is slow.

[0062] The present application discloses a method for online reconstruction of free-viewpoint video that can be streamed based on Gaussian splashing. It fully utilizes the advantages of Gaussian splashing in training and rendering speed, achieves reconstruction and storage efficiency that significantly surpasses existing free-viewpoint video reconstruction methods, explicitly represents scenes with Gaussian basis elements, has efficient training and can achieve real-time rendering.

[0063] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a three-dimensional dynamic scene rendering device, etc. The following takes a three-dimensional dynamic scene rendering device as an example to illustrate this embodiment and the following embodiments.

[0064] Based on this, the embodiment of the present application provides a three-dimensional dynamic scene rendering, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the three-dimensional dynamic scene rendering of the present application.

[0065] In this embodiment, the three-dimensional dynamic scene rendering includes steps S10 to S40:

[0066] Step S10, receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with a plurality of cameras;

[0067] It should be noted that multi-point video stream refers to real-time video data streams from multiple cameras, that is, image sequences captured simultaneously by multiple cameras arranged at different positions and angles of the scene. The target initial scene refers to the specific environment or object to be reconstructed.

[0068] It can be understood that the multi-point video streams continuously shot by multiple cameras arranged at different angles in the target initial scene are obtained to facilitate the subsequent three-dimensional scene reconstruction.

[0069] Step S20, reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation;

[0070] It should be noted that 3D scene reconstruction refers to generating a corresponding 3D model using input multi-view images. The initial scene representation refers to the data structure of the 3D model, which includes the geometric information and texture information of the scene.

[0071] It can be understood that the three-dimensional scene model is reconstructed using multi-point video streams to generate a three-dimensional model of the target initial scene, thereby creating an accurate initial scene representation for training and rendering for the target initial scene.

[0072] Step S30, performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters;

[0073] It should be noted that the scene motion change refers to the process of dynamic transformation of the three-dimensional Gaussian in the initial scene representation. The scene motion change parameter refers to the characteristic parameter describing the motion and change of the three-dimensional Gaussian in the scene.

[0074] It can be understood that in order to understand the motion characteristics of objects in the scene and generate a scene view that can be viewed by users from any perspective, the movement and change laws of objects in the scene are identified and learned through analysis of the initial scene representation, and the parameters of the scene motion change are obtained.

[0075] Step S40: sending the scene motion change parameters to the client in the form of a data stream, and the client generates a scene view by rendering according to the scene motion change parameters.

[0076] It should be noted that the data stream refers to the parameter information stream including the motion change parameters, corresponding to the received video stream, and can perform real-time processing of the video stream when the video stream is received, and generate and return the corresponding data stream transmitted to the client for rendering. The client refers to the system or device that performs the final rendering and display of the scene.

[0077] It can be understood that the learned scene motion change parameters are transmitted to the client in the form of a data stream, and the client can render the scene in real time, so that the user can watch the dynamic scene view (i.e., free viewpoint video) with arbitrary perspective switch on the client, thereby realizing the complete process of reconstruction, transmission, and rendering of the streamable three-dimensional dynamic scene.

[0078] In a feasible implementation manner, the step of reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation includes:

[0079] Step S201, reading the first frame of multi-view in the multi-point video stream;

[0080] It is understandable that reading the first video frame in the multi-point video stream corresponds to images from multiple perspectives, ensuring temporal synchronization so as to form a comprehensive three-dimensional model of the scene using images of the scene from different perspectives.

[0081] Step S202, initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene;

[0082] It should be noted that 3D Gaussian sputtering uses a structure similar to that of an explicit radiation field, using a large number of 3D Gaussian bodies as primitives of the scene. By optimizing the parameters of each primitive, an accurate scene description is finally completed. Then, the sputtering method is used to project it onto a plane and render to complete the synthesis of the new view. Each Gaussian primitive contains the center position , rotation q, scale s, opacity α, and spherical harmonics h.

[0083] It can be understood that the target initial scene uses a three-dimensional Gaussian as a primitive for constructing a three-dimensional model of the scene, and the parameters of the Gaussian primitives of the target initial scene are initialized according to the first frame multi-view to obtain an initial scene three-dimensional Gaussian.

[0084] Step S203: Based on the three-dimensional Gaussian splashing and noise smoothing mechanism, a densification operation is performed on the initial scene three-dimensional Gaussian to obtain an initial scene representation.

[0085] It can be understood that the Gaussian splashing technology is used to expand the influence range of the scene primitives, and the noise smoothing algorithm is used to add a suitable three-dimensional Gaussian to the initial scene three-dimensional Gaussian to complete the online reconstruction of the three-dimensional scene, ensuring the smooth transition of the three-dimensional Gaussian of the scene three-dimensional model, thereby generating a more detailed and compact three-dimensional model as the initial scene representation.

[0086] In this implementation, more scene information is captured by multi-perspective data fusion, and the initial scene three-dimensional Gaussian is densified using Gaussian splashing and noise smoothing techniques, which can effectively improve the quality of the initial scene three-dimensional Gaussian, making the final output three-dimensional model more continuous and realistic, and providing a more compact initial scene representation for subsequent three-dimensional scene motion change learning.

[0087] In a feasible implementation manner, the step of initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene includes:

[0088] Step S2021, estimating the position information of the multi-point video stream corresponding to the plurality of cameras by using a motion structure recovery algorithm;

[0089] It should be noted that Structure from Motion (SFM) is a method to recover the camera pose and 3D scene structure by analyzing a series of static images. The pose information refers to the position and orientation of the camera in 3D space, which determines how to recover the 3D structure from the 2D image.

[0090] It can be understood that through the principle of multi-view geometry and the relationship between cameras, the position and posture (pose information) of each camera in the three-dimensional space of the target initial scene can be calculated to obtain the camera's position and direction information, providing a basis for subsequent three-dimensional scene construction.

[0091] Step S2022, constructing a sparse point cloud of the target initial scene based on the pose information and the first frame multi-view, wherein the sparse point cloud includes at least position and color;

[0092] It should be noted that sparse point cloud refers to a data set consisting of a small but critical number of 3D points, each of which contains position and color information, used to represent the geometric structure of the scene. Position and color refer to the spatial coordinates (position) and color information of each point in the point cloud.

[0093] It can be understood that by using the known camera pose information and the first frame of multi-view images, the three-dimensional coordinates of the feature points in the scene are calculated through stereo vision technology (such as stereo matching or triangulation) to construct a sparse point cloud.

[0094] Step S2023, initializing the Gaussian primitives of the target initial scene according to the position and color of the sparse point cloud to obtain a three-dimensional Gaussian of the initial scene.

[0095] It should be noted that Gaussian primitives refer to a mathematical model used to represent points in three-dimensional space, which is parameterized by mean, covariance, and opacity and is used to construct a continuous representation of the scene.

[0096] It can be understood that the position and color information in the sparse point cloud is used to initialize the Gaussian primitives of the target initial scene to obtain the initial scene 3D Gaussian, which will be used to construct a complete 3D Gaussian representation of the scene, providing a basis for subsequent rendering and refinement.

[0097] In this implementation, the camera's pose information is first estimated through a motion structure recovery algorithm to provide accurate camera parameters for subsequent 3D reconstruction. Then, a sparse point cloud is constructed using the pose information and multi-view images through stereo matching or triangulation technology. These sparse point cloud data are used to initialize Gaussian primitives to construct a 3D Gaussian representation of the initial scene. Finally, the initial scene's 3D Gaussian is densified through Gaussian splashing and noise smoothing mechanisms to further improve the details and accuracy of the scene representation. This improves the accuracy and efficiency of 3D scene reconstruction, and also enhances the continuity and details of the scene, making the reconstruction result more realistic and accurate, and suitable for real-time rendering and interaction.

[0098] In a feasible implementation manner, the step of performing a densification operation on the initial scene three-dimensional Gaussian based on the three-dimensional Gaussian splashing and noise smoothing mechanism to obtain the initial scene representation includes:

[0099] Step S2031, calculating the spatial position gradient of the three-dimensional Gaussian of the initial scene;

[0100] It should be noted that the spatial position gradient describes the degree of change of the Gaussian primitive in the three-dimensional space of the scene, and is used to determine whether a new Gaussian primitive needs to be generated to improve the densification of the scene.

[0101] It is understandable that in order to obtain a more compact three-dimensional scene representation, it is necessary to calculate the spatial position gradient of the three-dimensional Gaussian of the initial scene and understand the density changes of the Gaussian basis elements in different directions in the scene space.

[0102] Step S2032, if the spatial position gradient exceeds a preset threshold, generating a newly added scene 3D Gaussian according to the initial scene 3D Gaussian based on a 3D Gaussian splash and noise smoothing mechanism;

[0103] It is understandable that when the spatial position gradient exceeds the preset threshold, it indicates that the current Gaussian primitives are not sufficient to accurately describe the details of the scene, and it is necessary to create a smooth transition between the newly generated Gaussian primitives and the existing Gaussian primitives through a noise smoothing mechanism, and generate new Gaussian primitives to smooth the transition and avoid the discontinuity of the scene Gaussian primitives during subsequent learning of scene motion changes.

[0104] Step S2033, obtaining an initial scene representation according to the initial scene 3D Gaussian and the newly added scene 3D Gaussian.

[0105] It can be understood that in order to determine the state of the scene before the motion change occurs during subsequent scene motion change learning, the Gaussian primitives of the initial scene are combined with the newly generated Gaussian primitives in the scene to form a complete initial scene representation.

[0106] In this embodiment, by calculating the spatial position gradient of the three-dimensional Gaussian of the initial scene, it is determined which areas need a more detailed description. When the gradient exceeds a preset threshold, a three-dimensional Gaussian splashing technique is used to generate new Gaussian primitives in these areas, and a noise smoothing mechanism is introduced to ensure a smooth transition between new and old Gaussian primitives. Finally, the initial Gaussian primitives and the newly added Gaussian primitives are merged to form a complete initial scene representation, which improves the density of the scene and maintains the continuity and realism of the scene, making the reconstructed three-dimensional scene more accurate and detailed, and providing a smoother and more realistic scene reconstruction solution.

[0107] This embodiment provides a three-dimensional dynamic scene rendering, an online reconstruction method for streaming free viewpoint video based on Gaussian splashing, which fully utilizes the advantages of Gaussian splashing in training and rendering speed, and achieves reconstruction and storage efficiency that greatly exceeds the existing free viewpoint video reconstruction methods. It explicitly represents the scene with Gaussian basis elements, has efficient training and can achieve real-time rendering.

[0108] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 , before the step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters, the step further includes:

[0109] Step S301, dividing the target initial scene into a plurality of scene areas of preset sizes, and dividing the plurality of scene areas into a plurality of hierarchical sub-areas respectively;

[0110] It should be noted that the scene region refers to a subset of the scene divided into different regions for processing, which is used for localized analysis and processing. The hierarchical sub-region refers to the region where the scene region is further subdivided, which is used to capture the motion changes of the scene at different levels.

[0111] It is understandable that in order to more effectively learn and analyze the motion changes of the scene, the entire scene is divided into multiple manageable areas, and these areas are hierarchically divided into several smaller hierarchical sub-areas to capture motion details at different levels.

[0112] The step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters comprises:

[0113] Step S302, based on the initial scene representation and the plurality of hierarchical sub-regions, scene motion change learning is performed according to the multi-point video stream to obtain hierarchical region motion change parameters;

[0114] It should be noted that scene motion change learning learns the motion pattern of the scene by analyzing the scene changes in the video stream. The hierarchical region motion change parameters describe the parameters of the motion change of each hierarchical sub-region, such as position, rotation, etc.

[0115] It can be understood that by combining Gaussian splashing and the principle of local consistency of scene motion, using the initial scene representation and hierarchical sub-region division, the scene motion in the multi-point video stream is analyzed, and the motion changes of each sub-region are learned, so as to obtain the parameters describing these motions.

[0116] Step S303 : performing hierarchical motion superposition according to the hierarchical region motion change parameters to obtain scene motion change parameters, wherein the scene motion change parameters at least include one or more three-dimensional Gaussian parameters of position change parameters and rotation change parameters.

[0117] It should be noted that hierarchical motion superposition refers to combining motion change parameters of different levels of regions to reflect the motion of the entire scene. The scene motion change parameters describe the parameters of the entire scene motion change, including at least one or more three-dimensional Gaussian parameters of position change parameters and rotation change parameters.

[0118] It can be understood that in order to reflect the overall motion change of the scene, the motion change parameters of sub-regions at different levels are superimposed to obtain the motion change parameters of the entire scene.

[0119] In this embodiment, the scene is first divided into scene areas of preset sizes, and further divided into hierarchical sub-areas, so that the motion details of the scene can be captured at different levels. Then, the scene motion change learning is performed based on the multi-point video stream using the initial scene representation and the division of hierarchical sub-areas to obtain the motion change parameters of each hierarchical sub-area. Finally, the motion change parameters of different hierarchical sub-areas are combined through hierarchical motion superposition to obtain the motion change parameters of the entire scene. This improves the accuracy of understanding and describing the scene motion changes, achieves more realistic and dynamic scene rendering, and can effectively solve the problems of motion blur, discontinuity, and difficulty in capturing complex motion patterns that may occur in traditional scene analysis, providing strong technical support for real-time rendering and interaction of dynamic scenes.

[0120] In a feasible implementation manner, the step of performing scene motion change learning based on the initial scene representation and the plurality of hierarchical sub-regions according to the multi-point video stream to obtain hierarchical region motion change parameters includes:

[0121] Step S3021, obtaining a second multi-view frame corresponding to a next frame of a first multi-view frame in the multi-point video stream;

[0122] It can be understood that two consecutive frames of multi-view images are extracted from a multi-point video stream of a video sequence of the same scene captured by multiple cameras to analyze the dynamic changes of the target initial scene between the two frames.

[0123] Step S3022, based on the initial scene representation, performing cross-frame dynamic change learning according to the first frame multi-view and the second frame multi-view to obtain a cross-frame dynamic change parameter;

[0124] It should be noted that cross-frame dynamic change learning refers to analyzing the difference between two consecutive frames of images to learn the motion changes of the scene. The cross-frame dynamic change parameters describe the parameters of the motion changes of the scene between two consecutive frames, such as displacement, rotation, etc.

[0125] It can be understood that by using the initial scene representation and two consecutive frames of multi-view images, the changes of each three-dimensional Gaussian in the scene between the two frames are analyzed, their motion patterns and laws are learned, and the change parameters describing these motions are obtained.

[0126] Step S3023: Obtain a hierarchical region motion change parameter according to the plurality of hierarchical sub-regions and the cross-frame dynamic change parameter.

[0127] It can be understood that the cross-frame dynamic change parameters of each three-dimensional Gaussian in the scene are applied to sub-regions at different levels to obtain the motion change parameters of the sub-regions at each level.

[0128] Furthermore, after the step of obtaining the initial scene representation according to the initial scene 3D Gaussian and the newly added scene 3D Gaussian, the method further includes:

[0129] Step S401: Send the initial scene representation to the client in the form of a data stream, and the client generates a scene view by rendering based on the scene motion change parameters and the initial scene representation.

[0130] It can be understood that the three-dimensional representation of the scene, including information such as the initial scene three-dimensional Gaussian and the newly added scene three-dimensional Gaussian, is transmitted to the client in real time in the form of a data stream through the network. After the client receives the data stream representing the initial scene, it uses these data and the scene motion change parameters.

[0131] In this implementation, a dynamic scene view is generated through streaming and graphics rendering, which ensures real-time updating and transmission of scene data. It also allows users to interact with the scene in real time on different devices and experience realistic three-dimensional dynamic scenes. It solves the problems of data transmission delay, untimely scene updates and poor user interactivity in traditional scene rendering, and provides an efficient, real-time and highly interactive three-dimensional dynamic scene rendering solution.

[0132] In this embodiment, two consecutive multi-view frames in a multi-point video stream are first obtained, and then based on the initial scene representation, cross-frame dynamic change learning is used to analyze and learn the changes of the scene between different frames, and the cross-frame dynamic change parameters describing the scene motion are obtained. Finally, these parameters are applied to sub-regions at different levels to obtain detailed motion change parameters for each sub-region, thereby synthesizing motion change parameters for regions at different levels. By capturing and analyzing the dynamic changes of the scene at different levels, the accuracy of understanding and describing the scene motion changes is improved, providing strong technical support for real-time rendering and interaction of three-dimensional dynamic scenes.

[0133] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the first and second embodiments mentioned above can be referred to the above introduction and will not be repeated later.

[0134] For example, in order to help understand the implementation process of the three-dimensional dynamic scene rendering obtained by combining the first embodiment and the second embodiment, please refer to Figure 3 and Figure 4 , Figure 3 A brief flowchart of the third embodiment of the three-dimensional dynamic scene rendering of this application is provided. Figure 4 A reconstruction process framework and a schematic diagram of the design of each main module are provided for the third embodiment of 3D dynamic scene rendering of this application.

[0135] Reconstructs efficient free-viewpoint video from multi-point video streams in an online learning manner. The operations provided include learning initial scene representation, learning scene motion changes, continuous dynamic improvement, and real-time transmission rendering. Specifically:

[0136] Step S1, learn the initial scene representation:

[0137] The scene is represented by a 3D Gaussian ellipsoid based on Gaussian splashing technology. Each Gaussian primitive contains the center position. , rotation q, scale s, opacity α, and spherical harmonics h, these Gaussian primitives are projected onto a 2D plane and finally rendered into an image using the alpha blending method.

[0138] like Figure 4 As shown in step (a) of the initial 3D Gaussian primitive representation learning, we first obtain multiple views of the first frame from multi-point videos captured by multiple cameras, then use the SfM algorithm to estimate the pose information of these cameras and generate some sparse point clouds. The positions and colors of these points are used to initialize the Gaussian primitives of the scene. Subsequently, the properties of the Gaussian primitives are optimized by the gradient descent algorithm based on the loss between the rendered view and the real view. After every 100 optimization steps, a densification operation is performed to copy a new Gaussian from the Gaussian primitive with a larger accumulated gradient. The two are performed alternately for a total of 15,000 steps of training.

[0139] In order to solve the problem of unlimited growth of the number of Gaussians during the learning process, the densification is set to terminate at the 5000th step. Adding tiny Gaussian noise to the center of the Gaussian during the entire optimization process to reconstruct a compact initial scene representation can alleviate the Gaussian splitting speed and prevent overfitting to sparse training views. Finally, a compact and high-quality 3D Gaussian representation of the initial scene is reconstructed, laying the foundation for subsequent online learning. Gaussian noise:

[0140]

[0141] Step S2, learning scene motion changes:

[0142] In order to learn efficiently, we start from the second frame and quickly optimize based on the scene representation of the previous frame. A more efficient way is to only learn the motion changes of the scene from the previous frame to the current frame. However, learning a set of motion parameters for each Gaussian will take up more storage space and the optimization complexity is relatively high.

[0143] According to the principle of local consistency, the scene is first divided into regions of the same size. The motion of all Gaussian primitives in each region is set to the same value, which can effectively reduce the motion parameters. The region where each Gaussian is located can be calculated by the following formula:

[0144]

[0145] like Figure 4 As shown in step (b) of hierarchical coherent motion in Fig. 1, since the distribution of Gaussian primitives in space is not uniform, a considerable part of the region does not contain Gaussian primitives, so the motion parameters are further reduced. Considering that the Gaussian motion in the region may also have slight differences, the large region is further divided into small sub-regions, and the motion parameters of each sub-region are learned separately. The motion of each Gaussian is finally obtained by superimposing the motion of the regions at different levels where it is located:

[0146]

[0147] Where L represents the number of levels of movement, and is the position change and rotation change of the lth layer, and is the comprehensive position change and rotation change of Gaussian.

[0148] This hierarchical consistent motion mechanism can capture the Gaussian primitive motion in the scene from coarse to fine, more detailed and accurate representation of cross-frame dynamic changes, and maintain consistent and coherent motion between adjacent regions. By sharing motion parameters between multiple Gaussian primitives, the required amount of calculation and the number of parameters that need to be optimized are reduced, which helps the learning process converge faster and more reliably, and improves the efficiency of real-time reconstruction.

[0149] Step S3, continuous dynamic improvement:

[0150] like Figure 4 As shown in the continuous refinement of step (c) in , after applying the learned motion changes, a 3D Gaussian representation that is more aligned with the latest scene can be obtained. However, the motion mechanism mainly focuses on moderate adjustments to the position and rotation of the previous Gaussian primitives, which may not be able to capture all the details of the scene changes or quickly adapt to significant scene content updates. This gap can be strategically bridged by adding new Gaussian primitives in appropriate areas.

[0151] Specifically, during motion learning, some regions accumulate significant gradients, indicating that there is a large difference between the learned scene and the actual scene. According to existing methods, Gaussian primitives are cloned in these regions where the gradient exceeds a preset threshold, and the newly cloned Gaussian primitives are then further optimized and density-controlled to better align with the latest scene.

[0152] Since dynamic scenes in the real world usually exhibit coherent changes, newly added Gaussian primitives are often still relevant in subsequent frames and cannot be simply discarded. Therefore, these Gaussian primitives are retained and integrated into the initial scene representation as the basis for learning the next frame. However, continuously incorporating new Gaussian primitives in each frame may lead to an asymptotic increase in the number of Gaussian primitives, which may slow down rendering, delay convergence, and extend training time. Therefore, an equal amount of low-opacity Gaussian primitives that have the least impact on the overall visual integrity of the scene are selectively removed to maintain the compactness and efficiency of the 3D Gaussian representation without affecting the fidelity of the scene. At the same time, this continuous optimization is able to address and enhance areas of the 3D Gaussian representation that may be underrepresented or undermodeled in the early learning stages.

[0153] Usually, the framework works in a sequential process frame by frame. To further improve the efficiency and responsiveness of practical applications, the method of multi-frame parallel training based on video coding technology is used for motion change learning, where the reference frame (such as the I frame in H.264 / AVC) is used to predict multiple subsequent frames, making full use of temporal redundancy and reducing computational cost. Considering that the difference between consecutive frames is usually small, the first The 3D Gaussian representation of the frame is used as a reference and learned After processing these frames, The 3D Gaussian representation of the frame becomes the learning follow-up This parallel training strategy significantly reduces the average waiting time cost per frame and enhances the real-time processing capability and practical application capability of the method.

[0154] Step S4, real-time transmission rendering:

[0155] The free viewpoint video reconstructed by this method can be transmitted like streaming media, and the client can freely render the specified viewpoint without waiting for all video frames to be processed.

[0156] Specifically, after the initial scene reconstruction is completed, it can be transmitted to the client immediately. After receiving it, the user can immediately select the specified perspective for rendering. Subsequently, the scene motion parameters learned for each frame and the newly added Gaussian primitives are also transmitted to the client in the form of data streams. The user can update the 3D Gaussian representation of the scene and render the latest scene view every time a frame of data is received. Since the online reconstruction framework training process is very efficient, 3D Gaussian splash can render more than 200 frames of view per second, so a complete process of reconstruction, transmission, and rendering can be achieved.

[0157] Experimental results: Experiments were conducted on two widely used datasets, N3DV and Meet Room, which contain a wide variety of indoor scenes. Four evaluation indicators were used to compare the proposed method with other methods:

[0158] 1) Use peak signal-to-noise ratio (PSNR, unit: decibel) to evaluate the accuracy of reconstruction;

[0159] 2) Use the average training time per frame (Train, unit: seconds) to evaluate the reconstruction efficiency;

[0160] 3) Use frame rate (FPS, frames processed per second) to evaluate rendering speed;

[0161] 4) The average parameter size represented per frame (Storage, unit: MB) is used to evaluate storage and transmission efficiency.

[0162] The experiments involved in this embodiment are all run on the NVIDIA RTX 4090 graphics card. The value is set to 0.01, the training steps for motion learning and continuous dynamic improvement are set to 100 steps, the training steps for the initial frame are set to 15000, and the other settings are the same as the standard Gaussian splash method. The following results are obtained:

[0163]

[0164] As shown in the table above, the method of this embodiment is compared with the most advanced free viewpoint video online reconstruction methods StreamRF and 3DGStream. Among them, 3DGS is used as the baseline method, and the standard 3DGS method is used to train from scratch frame by frame. The HiCoM method of this embodiment remains competitive in terms of free viewpoint video accuracy (PSNR), and significantly surpasses the two comparison methods in terms of training speed. Compared with 3DGStream, the average learning time per frame (including the initial frame) on the two data sets is reduced by more than 17%. HiCoM-P4 is the process of the HiCoM method using four threads for multi-frame parallel training. The 3D Gaussian representation of the first frame will become a reference for learning the subsequent 4 frames for training, which has a higher training speed than the HiCoM method.

[0165] In more complex scenarios, the learning speed is increased by more than 20%, mainly due to the faster convergence speed of the HiCoM method of this embodiment. Both HiCoM and 3DGStream achieve a rendering speed of more than 200 fps, demonstrating real-time rendering capabilities.

[0166] In terms of storage and transmission efficiency, the HiCoM method of this embodiment is significantly better than the two comparison methods, and the average storage space requirement for each frame after the initial frame is less than 10% of the comparison method. This is mainly due to two factors: first, the motion parameters of the HiCoM method of this embodiment are shared among multiple Gaussian primitives in a local area; second, the inherent uneven distribution of Gaussian primitives makes most areas empty, thereby further reducing the required motion parameters. Since the amount of information that needs to be transmitted is reduced, it is more advantageous in three-dimensional dynamic scenes that need to be transmitted in the form of streaming media.

[0167] In this embodiment, the most advanced and representative free viewpoint video online reconstruction technology 3DGStream is used for comparison with the HiCoM method proposed in this embodiment. The 3DGStream method also uses an online method to reconstruct dynamic scenes and uses a neural network to implicitly model scene motion. Compared with the HiCoM method in this embodiment, which explicitly represents the scene with Gaussian primitives, the comparison is made using two evaluation indicators, the number of Gaussian primitives and the quality of new perspective synthesis. The comparison results are as follows: Figure 5 As shown, Figure 5 A bar chart comparing the number of Gaussian primitives and the quality of new view synthesis in the initial scene provided for the third embodiment of the three-dimensional dynamic scene rendering of this application. 3DGStream uses the standard Gaussian splash method to reconstruct the initial scene, and the HiCoM method of this embodiment additionally uses the designed noise perturbation smoothing mechanism, which significantly reduces the number of Gaussian primitives in the three representative scenes of the two data sets (i.e., Coffee Martini, Flame Steak, and Discussion), and significantly improves the quality of new view synthesis. Therefore, compared with 3DGStream, the HiCoM method of this embodiment has a faster rendering speed and higher new view synthesis quality.

[0168] Since Gaussian splashing has the capabilities of efficient training and real-time rendering, this embodiment focuses on using these advantages to reconstruct efficiently transmitted free-viewpoint videos from multi-point video streams in an online learning manner. First, multiple views of the first frame from different perspectives are obtained from the multi-point video stream, and the SfM technology is used to estimate the camera's pose information and generate an initial point cloud. The initial representation of the scene is learned using the optimization strategy of Gaussian splashing. In order to make the initial scene representation more compact and lightweight, a noise perturbation smoothing mechanism is proposed to slow down the speed of Gaussian splitting, which to a certain extent also alleviates the problem of overfitting to sparse training views. Then, multiple views of the second frame are obtained from the multi-point video stream, and optimization is continued based on the initial scene. This embodiment designs two mechanisms to achieve efficient learning of the current scene representation from the initial scene representation:

[0169] 1) Hierarchical consistent motion, used to quickly learn the motion changes of the scene from the previous frame to the current frame;

[0170] 2) Continuous dynamic improvement, which is used to learn the dramatic update of scene content from the previous frame to the current frame, supplement the scene details and keep the scene representation compact. Each subsequent frame is learned quickly and efficiently based on the previous frame in a similar way to the second frame.

[0171] In addition, this embodiment also proposes a parallel training mechanism, which can simultaneously learn continuous multi-frame scenes, further improving the real-time processing capability in practical applications.

[0172] During the reconstruction process, the initial scene representation can be first transmitted to the client, and then the learned motion change parameters and newly added Gaussian primitives are transmitted to the client for rendering in each frame, realizing the complete process of reconstruction, transmission and rendering.

[0173] In this embodiment, the online reconstruction method of free-viewpoint video that can be streamed based on Gaussian splashing fully utilizes the advantages of Gaussian splashing in training and rendering speed, achieves reconstruction and storage efficiency that greatly exceeds the existing free-viewpoint video reconstruction methods, explicitly represents the scene with Gaussian basis elements, has efficient training and can achieve real-time rendering.

[0174] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the three-dimensional dynamic scene rendering of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0175] This application also provides a three-dimensional dynamic scene rendering device, please refer to Figure 6 , the three-dimensional dynamic scene rendering device comprises:

[0176] A receiving module 10 is used to receive a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with a plurality of cameras;

[0177] A reconstruction module 20, configured to reconstruct a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation;

[0178] A learning module 30, configured to perform scene motion change learning based on the initial scene representation to obtain scene motion change parameters;

[0179] The rendering module 40 is used to send the scene motion change parameters to the client in the form of a data stream, and the client generates a scene view by rendering according to the scene motion change parameters.

[0180] Optionally, the reconstruction module 20 is further configured to:

[0181] Reading a first multi-view frame in the multi-point video stream;

[0182] Initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene;

[0183] Based on the three-dimensional Gaussian splashing and noise smoothing mechanism, a densification operation is performed on the initial scene three-dimensional Gaussian to obtain an initial scene representation.

[0184] Optionally, the reconstruction module 20 is further configured to:

[0185] Estimate the position information of the multi-point video streams corresponding to the plurality of cameras by using a motion structure recovery algorithm;

[0186] Based on the pose information and the first frame multi-view, construct a sparse point cloud of the target initial scene, wherein the sparse point cloud includes at least position and color;

[0187] The Gaussian primitives of the target initial scene are initialized according to the position and color of the sparse point cloud to obtain a three-dimensional Gaussian of the initial scene.

[0188] Optionally, the reconstruction module 20 is further configured to:

[0189] Calculate the spatial position gradient of the three-dimensional Gaussian of the initial scene;

[0190] If the spatial position gradient exceeds a preset threshold, a new scene 3D Gaussian is generated based on the initial scene 3D Gaussian based on a 3D Gaussian splash and noise smoothing mechanism;

[0191] An initial scene representation is obtained according to the initial scene three-dimensional Gaussian and the newly added scene three-dimensional Gaussian.

[0192] Optionally, the learning module 30 is further used for:

[0193] Dividing the target initial scene into a plurality of scene areas of preset sizes, and dividing the plurality of scene areas into a plurality of hierarchical sub-areas respectively;

[0194] The step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters comprises:

[0195] Based on the initial scene representation and the plurality of hierarchical sub-regions, scene motion change learning is performed according to the multi-point video stream to obtain hierarchical region motion change parameters;

[0196] A hierarchical motion superposition is performed according to the hierarchical region motion change parameters to obtain scene motion change parameters, wherein the scene motion change parameters at least include one or more three-dimensional Gaussian parameters of position change parameters and rotation change parameters.

[0197] Optionally, the learning module 30 is further used for:

[0198] Acquire a second frame multi-view corresponding to a next frame of the first frame multi-view in the multi-point video stream;

[0199] Based on the initial scene representation, performing cross-frame dynamic change learning according to the first frame multi-view and the second frame multi-view to obtain a cross-frame dynamic change parameter;

[0200] According to the plurality of hierarchical sub-regions and the cross-frame dynamic change parameter, a hierarchical region motion change parameter is obtained.

[0201] The 3D dynamic scene rendering device provided by the present application adopts the 3D dynamic scene rendering in the above-mentioned embodiment, which can solve the technical problems that the free viewpoint video in the prior art cannot realize the reconstruction of the 3D dynamic scene that can be streamed and the rendering speed is slow. Compared with the prior art, the beneficial effects of the 3D dynamic scene rendering device provided by the present application are the same as the beneficial effects of the 3D dynamic scene rendering provided by the above-mentioned embodiment, and the other technical features in the 3D dynamic scene rendering device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0202] The present application provides a three-dimensional dynamic scene rendering device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the three-dimensional dynamic scene rendering in the above-mentioned first embodiment.

[0203] Reference below Figure 7 , which shows a schematic diagram of the structure of a 3D dynamic scene rendering device suitable for implementing the embodiment of the present application. The 3D dynamic scene rendering device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7The three-dimensional dynamic scene rendering device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0204] like Figure 7 As shown, the three-dimensional dynamic scene rendering device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the three-dimensional dynamic scene rendering device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the three-dimensional dynamic scene rendering device to communicate with other devices wirelessly or by wire to exchange data. Although the three-dimensional dynamic scene rendering device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0205] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0206] The 3D dynamic scene rendering device provided by the present application adopts the 3D dynamic scene rendering in the above embodiment, which can solve the technical problems that the free viewpoint video in the prior art cannot realize the reconstruction of the 3D dynamic scene that can be streamed and the rendering speed is slow. Compared with the prior art, the beneficial effects of the 3D dynamic scene rendering device provided by the present application are the same as the beneficial effects of the 3D dynamic scene rendering provided by the above embodiment, and the other technical features in the 3D dynamic scene rendering device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0207] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0208] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0209] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the three-dimensional dynamic scene rendering in the above-mentioned embodiment.

[0210] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0211] The computer-readable storage medium may be included in the three-dimensional dynamic scene rendering device; or may exist independently without being assembled into the three-dimensional dynamic scene rendering device.

[0212] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a three-dimensional dynamic scene rendering device, the three-dimensional dynamic scene rendering device: receives a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with a plurality of cameras; reconstructs the three-dimensional scene according to the multi-point video stream to obtain an initial scene representation; performs scene motion change learning based on the initial scene representation to obtain scene motion change parameters; sends the scene motion change parameters to a client in the form of a data stream, and the client renders and generates a scene view according to the scene motion change parameters.

[0213] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0214] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0215] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0216] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned three-dimensional dynamic scene rendering, and can solve the technical problems that the free viewpoint video of the prior art cannot achieve the reconstruction of three-dimensional dynamic scenes that can be streamed, and the rendering speed is slow. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the three-dimensional dynamic scene rendering provided by the above-mentioned embodiments, and will not be repeated here.

[0217] The present application also provides a computer program product, including a computer program, which implements the above-mentioned steps of three-dimensional dynamic scene rendering when executed by a processor.

[0218] The computer program product provided by the present application can solve the technical problems that the free viewpoint video in the prior art cannot realize the reconstruction of the three-dimensional dynamic scene that can be streamed and the rendering speed is slow. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the three-dimensional dynamic scene rendering provided by the above-mentioned embodiment, which will not be described in detail here.

[0219] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A three-dimensional dynamic scene rendering method, characterized in that: The method comprises: Receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives by a plurality of cameras; Reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation; Performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters; Wherein, before the step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters, the step further includes: Dividing the target initial scene into a plurality of scene areas of preset sizes, and dividing the plurality of scene areas into a plurality of hierarchical sub-areas respectively; The step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters comprises: Based on the initial scene representation and the plurality of hierarchical sub-regions, scene motion change learning is performed according to the multi-point video stream to obtain hierarchical region motion change parameters; Performing hierarchical motion superposition according to the hierarchical region motion change parameters to obtain scene motion change parameters, wherein the scene motion change parameters at least include position change parameters and / or rotation change parameters; The scene motion change parameters are sent to the client in the form of a data stream, and the client renders and generates a scene view according to the scene motion change parameters.

2. The method according to claim 1, characterized in that: The step of reconstructing a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation comprises: Reading a first multi-view frame in the multi-point video stream; Initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain a three-dimensional Gaussian of the initial scene; Based on the three-dimensional Gaussian splashing and noise smoothing mechanism, a densification operation is performed on the initial scene three-dimensional Gaussian to obtain an initial scene representation.

3. The method according to claim 2, characterized in that: The step of initializing the Gaussian primitives of the target initial scene according to the first frame multi-view to obtain the three-dimensional Gaussian of the initial scene includes: Estimate the position information of the multi-point video streams corresponding to the plurality of cameras by using a motion structure recovery algorithm; Based on the pose information and the first frame multi-view, construct a sparse point cloud of the target initial scene, wherein the sparse point cloud includes at least position and color; The Gaussian primitives of the target initial scene are initialized according to the position and color of the sparse point cloud to obtain a three-dimensional Gaussian of the initial scene.

4. The method according to claim 2, characterized in that: The step of performing a densification operation on the initial scene three-dimensional Gaussian based on the three-dimensional Gaussian splashing and noise smoothing mechanism to obtain the initial scene representation comprises: Calculate the spatial position gradient of the three-dimensional Gaussian of the initial scene; If the spatial position gradient exceeds a preset threshold, a new scene 3D Gaussian is generated based on the initial scene 3D Gaussian based on a 3D Gaussian splash and noise smoothing mechanism; An initial scene representation is obtained according to the initial scene three-dimensional Gaussian and the newly added scene three-dimensional Gaussian.

5. The method according to claim 4, characterized in that: The step of performing scene motion change learning based on the initial scene representation and the plurality of hierarchical sub-regions according to the multi-point video stream to obtain hierarchical region motion change parameters includes: Acquire a second frame multi-view corresponding to a next frame of the first frame multi-view in the multi-point video stream; Based on the initial scene representation, performing cross-frame dynamic change learning according to the first frame multi-view and the second frame multi-view to obtain a cross-frame dynamic change parameter; According to the plurality of hierarchical sub-regions and the cross-frame dynamic change parameter, a hierarchical region motion change parameter is obtained.

6. A three-dimensional dynamic scene rendering device, characterized in that: The device comprises: A receiving module, used for receiving a multi-point video stream of a target initial scene, wherein the multi-point video stream is obtained by continuously shooting the target initial scene from multiple perspectives with a plurality of cameras; A reconstruction module, used to reconstruct a three-dimensional scene according to the multi-point video stream to obtain an initial scene representation; A learning module, configured to perform scene motion change learning based on the initial scene representation to obtain scene motion change parameters; Wherein, the learning module is also used for: Dividing the target initial scene into a plurality of scene areas of preset sizes, and dividing the plurality of scene areas into a plurality of hierarchical sub-areas respectively; The step of performing scene motion change learning based on the initial scene representation to obtain scene motion change parameters comprises: Based on the initial scene representation and the plurality of hierarchical sub-regions, scene motion change learning is performed according to the multi-point video stream to obtain hierarchical region motion change parameters; Performing hierarchical motion superposition according to the hierarchical region motion change parameters to obtain scene motion change parameters, wherein the scene motion change parameters at least include position change parameters and / or rotation change parameters; The rendering module is used to send the scene motion change parameters to the client in the form of a data stream, and the client generates a scene view by rendering according to the scene motion change parameters.

7. A three-dimensional dynamic scene rendering device, characterized in that: The device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the three-dimensional dynamic scene rendering method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the three-dimensional dynamic scene rendering method according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional dynamic scene rendering method according to any one of claims 1 to 5 are implemented.

Images