Dynamic scene streaming reconstruction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a dynamic scene streaming reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision. Constructing a three-dimensional Gaussian model by using the initial video frame of each view angle, and dividing Gaussian primitives into an activity set and a frozen set based on historical video frames in the optimization process of the current video frame; determining a sudden motion mask based on the current motion mask of the current video frame, obtaining a reference rendering image corresponding to the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, and calculating a motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, if the motion loss value indicates that part of Gaussian primitives in the frozen set are active primitives, dividing the active primitives into an active set; and performing an optimization process at the current moment at least based on the updated activity set. The Gaussian primitives corresponding to the dynamic region and the static region are accurately separated, the inter-frame flicker phenomenon is effectively inhibited, and the stability and the reconstruction quality of streaming reconstruction are improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to methods, apparatus, devices and storage media for dynamic scene streaming reconstruction. Background Technology

[0002] Free-viewpoint video (FVV) synthesis of dynamic scenes aims to generate realistic video content that can be viewed freely from any angle by using video sequences captured synchronously by multiple fixed cameras. With the rapid development of applications such as virtual reality (VR), augmented reality (AR), and short video live streaming, high-quality, low-latency free-viewpoint synthesis of dynamic scenes shows broad application prospects in immersive entertainment, sports broadcasting, and remote teaching.

[0003] In related technologies, the streaming free-viewpoint synthesis method achieves this by updating the attributes of all 3D Gaussian primitives during each frame optimization process. However, this method has an unavoidable drawback: static areas that should maintain stable colors (such as walls and ceilings) exhibit noticeable flickering. To address this, some schemes have attempted to introduce optical flow during the synthesis process to distinguish between moving and stationary areas; however, optical flow itself is susceptible to noise interference, which can introduce additional misjudgments in streaming free-viewpoint synthesis, further exacerbating the flickering problem. Summary of the Invention

[0004] The main objective of this application is to propose a dynamic scene streaming reconstruction method, apparatus, device, and storage medium to improve the accuracy of dynamic scene streaming reconstruction and reduce flickering.

[0005] To achieve the above objectives, a first aspect of this application proposes a dynamic scene streaming reconstruction method, comprising: Acquire video sequences of the target scene captured from multiple perspectives, each video sequence including an initial video frame captured at an initial time and other video frames captured at other times; A three-dimensional Gaussian model is constructed using the initial video frame from each of the aforementioned viewpoints, the three-dimensional Gaussian model comprising multiple Gaussian elements; Select each of the other video frames as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification state label of each Gaussian element based on the historical video frame, and divide the Gaussian element into an active set and a frozen set based on the classification state label. Obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some of the Gaussian primitives in the frozen set are active primitives, assign the active primitives to the active set; The optimization process at the current moment is based at least on the updated set of activities.

[0006] In some embodiments, the attribute parameters of the Gaussian elements include at least position parameters, the classification state markers include stationary markers and active markers, and the step of obtaining the classification state marker for each Gaussian element based on the historical video frames includes: For each Gaussian element, the viewpoint is selected one by one. From the current video frame to the historical video frame, the angular displacement corresponding to the Gaussian element between the historical video frame and the adjacent historical video frame is obtained based on the position parameters. The projected area of ​​the Gaussian element under the viewpoint is obtained. The normalized angular displacement is calculated based on the angular displacement and the projected area. The average angular displacement corresponding to the Gaussian element is calculated based on all the normalized angular displacements. If the average angular displacement is less than a preset threshold, a stationary marker indicating that the Gaussian element is stationary in the historical video frame is generated; otherwise, an active marker is generated.

[0007] In some embodiments, dividing the Gaussian elements into an active set and a frozen set based on the classification state label includes: Obtain the classification state label of each Gaussian element in all the historical video frames; If all the classification state labels are static labels, the Gaussian element is assigned to the frozen set; otherwise, it is assigned to the active set.

[0008] In some embodiments, obtaining the current motion mask of the current video frame and determining a sudden motion mask based on the motion mask includes: For each viewpoint, the standard deviation of each pixel position in the time dimension is calculated based on the corresponding current video frame and at least one historical video frame. After Gaussian smoothing, the current motion mask of the current video frame under the viewpoint is obtained. Obtain the motion mask corresponding to each historical video frame as the historical motion mask, and invert the historical motion mask to obtain the intermediate mask. Calculate the intersection of all the intermediate masks to obtain the sudden motion region. Obtain the sudden motion mask based on the current motion mask and the sudden motion region.

[0009] In some embodiments, calculating the motion loss value based on the current video frame, the reference rendered image, and the sudden motion mask includes: Calculate the image pixel loss value and structural loss value based on the current video frame and the reference rendered image, and calculate the color loss value based on the image pixel loss value and the structural loss value; The motion loss value is obtained by multiplying the color loss value by the sudden motion mask based on the pixel position.

[0010] In some embodiments, the step of assigning the active primitives to the active set if the motion loss value indicates that a portion of the Gaussian primitives in the frozen set are active primitives includes: The gradient data of the Gaussian primitives related to the generation of the reference rendered image is obtained using the motion loss value. If the gradient data is greater than a preset gradient value and the Gaussian primitive is located in the frozen set, the Gaussian primitive is defined as the active primitive, and the active primitive is moved from the frozen set to the active set.

[0011] In some embodiments, the optimization process for the current moment, based at least on the updated set of activities, includes: The attribute prediction network is used to learn the attributes of the three-dimensional Gaussian model corresponding to the previous time step to obtain the residual change of the Gaussian element in each activity set at the current time step. The attributes of the Gaussian elements in the activity set are updated based on the residual change. The three-dimensional Gaussian model corresponding to the current time step is obtained based on the frozen set and the updated activity set. The image is rendered using the three-dimensional Gaussian model to obtain the rendered image corresponding to the current video frame. An optimized loss value is calculated based on the rendered image and the current video frame. The parameters of the attribute prediction network are updated based on the optimized loss value.

[0012] In some embodiments, constructing a 3D Gaussian model using the initial video frames from each of the said viewpoints includes: A three-dimensional Gaussian model is constructed using the initial video frame from each of the aforementioned viewpoints, and the initial rendered image corresponding to the initial video frame is generated using the three-dimensional Gaussian model. An initial loss value is calculated based on the initial video frame and the initial rendered image. Based on the initial loss value, the attributes of each Gaussian element in the 3D Gaussian model, the color correction matrix corresponding to each viewpoint, and the bias vector are adjusted.

[0013] To achieve the above objectives, a second aspect of this application provides a dynamic scene streaming reconstruction apparatus, comprising: Video frame acquisition module: used to acquire video sequences of the target scene from multiple perspectives, each video sequence including an initial video frame acquired at the initial moment and other video frames acquired at other moments; Initial model building module: used to build a three-dimensional Gaussian model using the initial video frame of each viewpoint, the three-dimensional Gaussian model including multiple Gaussian elements; Primitive initial classification module: used to select the other video frames one by one as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification status label of each Gaussian primitive based on the historical video frame, and divide the Gaussian primitive into an active set and a frozen set based on the classification status label. The primitive activation classification module is used to obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some of the Gaussian primitives in the frozen set are active primitives, the active primitives are assigned to the active set. Optimization module: Used to perform the optimization process at the current moment, at least based on the updated set of activities.

[0014] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a storage medium that stores a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0016] The dynamic scene streaming reconstruction method, apparatus, device, and storage medium proposed in this application acquire video sequences of the target scene from multiple perspectives, each video sequence including an initial video frame acquired at an initial moment and other video frames acquired at other moments; construct a three-dimensional Gaussian model using the initial video frames of each perspective, the three-dimensional Gaussian model including multiple Gaussian primitives; select other video frames one by one as the current video frame, acquire at least one historical video frame corresponding to the current video frame, acquire the classification state label of each Gaussian primitive based on the historical video frame, and classify the Gaussian primitives into an active set and a frozen set based on the classification state label; acquire the current motion mask of the current video frame, determine a sudden motion mask based on the current motion mask, acquire the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image, and the sudden motion mask, and if the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, classify the active primitives into the active set; and perform an optimization process for the current moment based at least on the updated active set. This application first constructs a 3D Gaussian model using initial video frames from multiple perspectives. When processing subsequent video frames, it labels each Gaussian element with a classification state marker from historical video frames, dividing the dynamic object's active set into a frozen set corresponding to the static region. Simultaneously, it determines a sudden motion mask by combining the motion mask of the current video frame, accurately capturing targets with sudden movements in the scene. This avoids modeling omissions or positional deviations caused by rapid movement of dynamic targets, achieving refined separation of dynamic and static regions. Since flickering is caused by attribute changes in static regions during each frame's optimization process, leading to temporal inconsistencies, this application accurately separates the Gaussian elements corresponding to dynamic and static regions. During optimization, the active set is adjusted in real-time for the motion trajectory and shape changes of dynamic targets, ensuring smooth movement of dynamic objects in consecutive frames. The frozen set maintains unchanged attributes, preserving the stability of the static region and avoiding flickering caused by frequent, meaningless updates. Therefore, it effectively suppresses inter-frame flickering, improving the stability and reconstruction quality of dynamic scene streaming reconstruction. Attached Figure Description

[0017] Figure 1 This is a flowchart of the dynamic scene streaming reconstruction method provided in the embodiments of this application.

[0018] Figure 2 This is a flowchart of constructing a three-dimensional Gaussian model using the initial video frames from each viewpoint, provided in an embodiment of this application.

[0019] Figure 3 This is a flowchart of obtaining the classification state label of each Gaussian element based on historical video frames, provided in an embodiment of this application.

[0020] Figure 4This is a schematic diagram illustrating the use of a prediction module to obtain Gaussian elements for classification state labeling, as provided in an embodiment of this application.

[0021] Figure 5 This is a flowchart illustrating how Gaussian elements are divided into active and frozen sets based on classification state labels, as provided in an embodiment of this application.

[0022] Figure 6 This is a flowchart provided in an embodiment of the present application for obtaining the current motion mask of the current video frame and determining a sudden motion mask based on the motion mask.

[0023] Figure 7 This is a flowchart provided in an embodiment of the present application for calculating motion loss values ​​based on the current video frame, a reference rendered image, and a sudden motion mask.

[0024] Figure 8 This is a schematic diagram of updating Gaussian elements using the observation module in an embodiment of this application.

[0025] Figure 9 This is a flowchart illustrating the optimization process for the current moment based at least on the updated set of activities, as provided in the embodiments of this application.

[0026] Figure 10 This is a schematic diagram of the overall process of the dynamic scene streaming reconstruction method provided in the embodiments of this application.

[0027] Figure 11 This is a structural block diagram of a dynamic scene streaming reconstruction device provided in another embodiment of this application.

[0028] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0030] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0032] First, let's analyze some of the terms used in this application: Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0033] Free-viewpoint video (FVV) synthesis of dynamic scenes aims to generate realistic video content that can be viewed freely from any angle by using video sequences captured synchronously by multiple fixed cameras. With the rapid development of applications such as virtual reality (VR), augmented reality (AR), and short video live streaming, high-quality, low-latency free-viewpoint synthesis of dynamic scenes shows broad application prospects in immersive entertainment, sports broadcasting, and remote teaching.

[0034] In recent years, 3D Gaussian Splatting (3DGS), as a differentiable volumetric rendering method, has achieved high-quality static scene reconstruction and real-time free-view rendering under the condition of inputting multi-view images, thanks to its explicit Gaussian primitive representation and differentiable rasterizer. Due to the advantages of 3DGS in reconstruction accuracy and rendering speed, more and more research is beginning to explore its application in dynamic scene free-view compositing. Among them, the dynamic scene free-view compositing method using streaming processing is becoming increasingly widely used in real-world applications because it can support real-time transmission and arbitrary length video sequence input.

[0035] In related technologies, the streaming free-viewpoint synthesis method achieves this by updating the attributes of all 3D Gaussian primitives during each frame optimization process. However, this method has an unavoidable drawback: static areas that should maintain stable colors (such as walls and ceilings) exhibit noticeable flickering. To address this, some schemes have attempted to introduce optical flow during the synthesis process to distinguish between moving and stationary areas; however, optical flow itself is susceptible to noise interference, which can introduce additional misjudgments in streaming free-viewpoint synthesis, further exacerbating the flickering problem.

[0036] Based on this, embodiments of this application provide a dynamic scene streaming reconstruction method, apparatus, device, and storage medium. First, a 3D Gaussian model is constructed using initial video frames from multiple perspectives. When processing subsequent video frames, a classification state label is used to annotate each Gaussian element with historical video frames, dividing the dynamic object's active set into a frozen set corresponding to the static region. Simultaneously, a sudden motion mask is determined by combining the motion mask of the current video frame, accurately capturing targets with sudden movements in the scene. This avoids modeling omissions or positional deviations caused by excessively rapid movement of dynamic targets, achieving refined separation of dynamic and static regions. Since flickering is caused by attribute changes in static regions during each frame's optimization process, leading to temporal inconsistencies, embodiments of this application accurately separate the Gaussian elements corresponding to dynamic and static regions. During optimization, the active set is adjusted in real-time for the motion trajectory and morphological changes of dynamic targets, ensuring smooth movement of dynamic objects in consecutive frames. The frozen set maintains unchanged attributes, preserving the stability of the static region and avoiding flickering caused by frequent, meaningless updates to the static region. Therefore, it can effectively suppress inter-frame flickering, improving the stability and reconstruction quality of dynamic scene streaming reconstruction.

[0037] This application provides a dynamic scene streaming reconstruction method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the dynamic scene streaming reconstruction method in this application embodiment is described.

[0038] This application's embodiments can acquire and process relevant data based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0039] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] The dynamic scene streaming reconstruction method provided in this application relates to the field of computer vision technology. This method can be applied to a terminal, a server, or a computer program running on either the terminal or the server. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a client supporting dynamic scene streaming reconstruction, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin. The terminal communicates with the server via a network. The dynamic scene streaming reconstruction method can be executed by the terminal or the server, or by the terminal and the server working together.

[0041] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and server can connect via Bluetooth, Universal Serial Bus (USB), or a network, etc., and this embodiment does not impose any limitations.

[0042] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0043] The following describes the dynamic scene streaming reconstruction method in the embodiments of this application.

[0044] Figure 1 This is an optional flowchart of the dynamic scene streaming reconstruction method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps 110 to 150. It is also understood that this embodiment... Figure 1 The order of steps 110 to 150 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0045] Step 110: Obtain video sequences of the target scene from multiple perspectives.

[0046] In one embodiment, the target scene can be a dynamic scene, and acquisition devices can be deployed from different perspectives to acquire images of the target scene at multiple times. At this time, the acquisition device corresponding to each perspective acquires a video sequence, and each video sequence includes a consistent number of video frames, wherein the video frames include the initial video frame acquired at the initial time and other video frames acquired at other times.

[0047] Step 120: Construct a 3D Gaussian model using the initial video frames from each viewpoint.

[0048] In one embodiment, reference is made to Figure 2 , Figure 2 This is a flowchart of constructing a 3D Gaussian model using initial video frames from each viewpoint, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 210: Construct a 3D Gaussian model using the initial video frames from each viewpoint, and use the 3D Gaussian model to generate the initial rendered image corresponding to the initial video frame.

[0049] In one embodiment, the initial video frames acquired at the initial moment of each viewpoint are input into COLMAP for processing. Key features in the initial video frames, such as pixel color, depth information, and object edges, are extracted. Then, through stereo matching algorithms, such as feature point matching and epipolar constraints, the three-dimensional coordinates of corresponding points under different viewpoints are calculated to establish a sparse three-dimensional point cloud of the target scene.

[0050] Next, based on these 3D point cloud data, a 3D Gaussian model is constructed using clustering and parameter estimation methods. The 3D Gaussian model includes multiple Gaussian primitives. Specifically, the sparse 3D point cloud is divided into multiple subsets using the expectation-maximization algorithm. Each subset corresponds to a Gaussian primitive. The center of the primitive represents the spatial center position of the region, while the axis length and direction of the primitive reflect the distribution range and spatial orientation of the points within the region.

[0051] For example, in a dynamic indoor scene, the initial video frame might contain static objects such as tables and chairs. After obtaining the 3D point clouds of these objects through the process described above, the 3D Gaussian model will generate multiple Gaussian primitives: the primitives in the table region might be flat, with a small variance in their covariance matrix in the vertical direction, and their center located at a certain height above the ground; the primitives in the chair region might be cylindrical, with a certain variance in both the horizontal and vertical directions, and maintaining a specific spatial distance from the table. These primitives together constitute a compact representation of the target scene.

[0052] With a 3D Gaussian model, predictions are made for each viewpoint using this model. The 2D image under a specific viewpoint is derived through the probability distribution in 3D space. The core is mapping the 3D features of Gaussian primitives into pixel information that conforms to the projection rules of the viewpoint. First, a projection matrix from 3D space to the 2D image plane is established based on the camera parameters corresponding to each viewpoint, such as focal length, pixel size, position, and pose. Then, each Gaussian primitive in the 3D Gaussian model is traversed, and the projection region of the primitive under the current viewpoint is calculated based on its mean and covariance matrix. The 3D boundary points of the primitive are converted into 2D image coordinates according to the projection matrix, determining the pixel range occupied by the Gaussian primitive in the corresponding initial rendered image. Simultaneously, based on the attributes of the Gaussian primitives, such as position parameters, transparency, scaling parameters, rotation matrix, and spherical harmonic color coefficients, visual information such as color and brightness is assigned to the corresponding pixels, resulting in the corresponding initial rendered image.

[0053] Step 220: Calculate the initial loss value based on the initial video frame and the initial rendered image, and adjust the parameters of each Gaussian primitive in the 3D Gaussian model, the color correction matrix and the bias vector corresponding to each viewpoint according to the initial loss value.

[0054] In one embodiment, for each viewpoint, the initial rendered image needs to be as close as possible to the corresponding initial video frame; therefore, an initial loss value is calculated based on the initial video frame and the initial rendered image. The initial loss value... Represented as:

[0055] in, Indicates the initial video frame. This represents the initial rendered image. This represents the weighting coefficient, which is set according to the actual situation. This represents the loss value calculated for each pixel in the image. This indicates the calculation of the D-SSIM loss value.

[0056] Next, the specific values ​​of the attributes of each Gaussian primitive are adjusted according to the initial loss value so that the initial rendered image is as close as possible to the corresponding initial video frame.

[0057] In one embodiment, to eliminate the inconsistency in viewing angle caused by different exposures of various acquisition devices in the target scene, a dedicated color correction matrix is ​​also optimized for each acquisition device simultaneously. With bias vector The rendered image, such as the initial rendered image, undergoes photometric consistency correction and exposure effects are eliminated, thus enabling the 3D Gaussian model to converge faster. Therefore, based on the initial loss value, the parameters of each Gaussian element, the color correction matrix for each viewpoint, and the bias vector in the 3D Gaussian model are adjusted. This results in an initial 3D Gaussian model, where each Gaussian element has corresponding attributes.

[0058] Step 130: Select other video frames one by one as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification state label of each Gaussian element based on the historical video frame, and divide the Gaussian elements into the active set and the frozen set based on the classification state label.

[0059] In one embodiment, for video frames other than the initial video frame, each video frame is selected sequentially as the current video frame, and the corresponding acquisition time is called the current time. The K video frames preceding the current video frame (including the initial video frame) are selected as its corresponding historical video frames. If the number of historical video frames is less than K, the actual number of video frames is selected as historical video frames. The parameter K is set according to the actual situation.

[0060] Next, dynamic and static regions are divided based on historical video frames. (Refer to...) Figure 3 , Figure 3This is a flowchart of obtaining the classification state label of each Gaussian element based on historical video frames, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 310: For each Gaussian element, select the viewpoint one by one, from the current video frame to the historical video frame, obtain the angular displacement corresponding to the Gaussian element between the historical video frame and the adjacent historical video frame based on the position parameters, and obtain the projected area of ​​the Gaussian element under the viewpoint. Calculate the normalized angular displacement based on the angular displacement and the projected area, and calculate the average angular displacement corresponding to the Gaussian element based on all normalized angular displacements.

[0061] In one embodiment, reference is made to Figure 4 , Figure 4 This is a schematic diagram illustrating the use of a prediction module to obtain Gaussian elements for classification state labeling, as provided in an embodiment of this application. The diagram uses historical video frames. For example, the historical moments corresponding to the historical video frames here have already undergone the optimization process of the 3D Gaussian module. A Gaussian activity classifier can be used to classify the Gaussian primitives to determine whether the motion of each Gaussian primitive in the historical video frames is "active" or "stationary".

[0062] Specifically, each Gaussian lattice is input into a Gaussian activity classifier, and viewpoints are selected one by one, from the current video frame to the historical video frames. Based on position parameters, the angular displacement corresponding to the Gaussian lattice between the current video frame and the historical video frames is calculated. For example, for historical video frames... Its position parameters are expressed as Its preceding adjacent historical video frame is Its position parameters are expressed as At this point, selecting the i-th viewpoint, for the j-th Gaussian element, based on the corresponding position parameters, the preceding adjacent historical video frame is obtained as follows: and historical video frames Between these points, the angular displacement corresponding to the Gaussian element at the i-th viewpoint is expressed as: Next, retrieve the historical video frames. The projected area of ​​the Gaussian element at the i-th viewpoint Then, the normalized angular displacement is calculated based on the angular displacement and the projected area, and the average angular displacement corresponding to the Gaussian element is calculated based on all the normalized angular displacements.

[0063] The average angular displacement is expressed as:

[0064] Where R represents the number of viewpoints.

[0065] Step 320: If the average angular displacement is less than a preset threshold, generate a stationary marker indicating that the Gaussian element is stationary in the historical video frame; otherwise, generate an active marker.

[0066] In one embodiment, classification status label It is a binary parameter, including a static marker and an active marker, if the following conditions are met:

[0067] Then, a still marker indicating that the Gaussian element is stationary in the historical video frame is generated, represented as: Otherwise, an activity flag is generated, indicating that... .

[0068] This application utilizes two-dimensional angular displacement, rather than three-dimensional spatial displacement, as the criterion for determining the state of the Gaussian element. This effectively avoids misjudgments caused by viewpoint jitter. Angular displacement is a change relative to the optical center of the acquisition device, such as a camera, and is not sensitive to minute camera-related movements, such as rotation and translation. Therefore, it more stably reflects the motion state of the Gaussian element itself. Furthermore, by combining observation results from multiple viewpoints and reducing the probability of false detections from a single viewpoint through averaging, the accuracy of the judgment results can be improved.

[0069] Next, in one embodiment, refer to Figure 5 , Figure 5 This is a flowchart illustrating the process of dividing Gaussian elements into an active set and a frozen set based on classification state labels, as provided in this application embodiment. The flowchart specifically includes the following steps: Step 510: Obtain the classification state label for each Gaussian element across all historical video frames.

[0070] In one embodiment, reference is made to Figure 4 The results obtained from the Gaussian activity classifier are input into the frozen candidate predictor for historical video frames. , … , Determine the classification state label corresponding to each Gaussian unit. , … , At this point, we can obtain the changes in the classification state label of each Gaussian unit in the time dimension.

[0071] Step 520: If all classification state labels are static labels, assign Gaussian elements to the frozen set; otherwise, assign them to the active set.

[0072] In one embodiment, the frozen candidate predictor reads the classification state label sequence of each Gaussian unit over K historical video frames { , ,…, , If a Gaussian element is stationary in all historical video frames, it is added to the frozen set; otherwise, it is assigned to the active set.

[0073] Therefore, the partitioning process can be represented as:

[0074] in, Indicates an indicator function, if If the condition is met, it means that the 3D Gaussian element needs to be assigned to the frozen set and will not participate in the attribute update of the current video frame; otherwise, it will be assigned to the active set. This strategy is based on the assumption that "a continuously static region is likely to remain static in the next frame as well," which can significantly reduce flicker in static regions.

[0075] Step 140: Obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the 3D Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, assign the active primitives to the active set.

[0076] In one embodiment, although the Gaussian elements can be initially divided into active and frozen sets based on the continuous static information of historical Gaussian elements, there may be objects that suddenly start moving in the current video frame. Therefore, it is necessary to update the Gaussian elements corresponding to these moving objects.

[0077] In one embodiment, reference is made to Figure 6 , Figure 6 This application provides a flowchart for obtaining the current motion mask of the current video frame and determining a sudden motion mask based on the motion mask, which specifically includes the following steps: Step 610: For each viewpoint, calculate the standard deviation of each pixel position in the time dimension based on the corresponding current video frame and at least one historical video frame, perform Gaussian smoothing, and obtain the current motion mask of the current video frame in the viewpoint.

[0078] In one embodiment, taking the i-th viewpoint as an example, M historical video frames corresponding to the current video frame are selected to form a time-series frame group containing "historical video frames + current video frame". Here, M can be less than or equal to K, depending on the actual needs.

[0079] Next, the standard deviation of each pixel location over time is calculated. Specifically, the pixel values ​​at the same pixel coordinates in the frame group are extracted in chronological order to obtain the time series value of that pixel. Here, the pixel value can be a grayscale value or the luminance component of the RGB channel to avoid interference from the color channel. Then, the dispersion of the sequence is calculated according to the standard deviation formula: the larger the standard deviation, the more drastic the brightness change of the pixel in a short period of time, which is likely to correspond to a dynamic target, such as the pixel brightness of a moving vehicle fluctuating rapidly with its position; the smaller the standard deviation, the more stable the pixel brightness, which is likely to belong to a static background, such as the pixel brightness of a road surface or building remains basically unchanged.

[0080] Then, the calculated pixel standard deviation matrix is ​​Gaussian smoothed. Since the original standard deviation matrix may contain noise, such as slight fluctuations in illumination or false high standard deviation pixels caused by sensor noise, a Gaussian filter is used for convolution to smooth the standard deviation values ​​of local pixels. The core of Gaussian smoothing is to make the standard deviation of each pixel influenced by its neighboring pixels, with the weight decreasing in a Gaussian distribution as distance increases. For example, a high standard deviation pixel caused by noise will be "pulled down" by surrounding low standard deviation background pixels, thus eliminating isolated noise points. Meanwhile, consecutive high standard deviation pixels in the real dynamic region will retain higher values ​​after smoothing, forming a complete dynamic region block. This improves the accuracy of subsequent motion masks and avoids false detections caused by noise interference.

[0081] Finally, the current motion mask is generated based on the smoothed standard deviation matrix. An adaptive threshold can be arbitrarily set, and the smoothed pixel standard deviation is compared with the threshold: if the pixel standard deviation is greater than the threshold, it is determined to be a "moving pixel" and marked as foreground in the mask; if it is less than or equal to the threshold, it is determined to be a "static pixel" and marked as background. The final result is the binarized current motion mask of the current video frame under the current viewpoint.

[0082] Step 620: Obtain the motion mask corresponding to each historical video frame as the historical motion mask, and invert the historical motion mask to obtain the intermediate mask. Calculate the intersection of all intermediate masks to obtain the sudden motion region. Obtain the sudden motion mask based on the current motion mask and the sudden motion region.

[0083] In one embodiment, for all K historical video frames, their corresponding motion masks are obtained as historical motion masks, where the motion mask is the current motion mask obtained when the historical video frame is the current video frame. All historical motion masks are inverted to obtain corresponding intermediate masks, and the intersection of all intermediate masks is calculated to obtain the sudden motion region. The sudden motion mask is then obtained based on the current motion mask and the sudden motion region.

[0084] The sudden motion mask obtained by the above process This indicates motion that occurs only in the current video frame and has not appeared in any previous video frames, represented as:

[0085] in, This represents logical NOT, which is the negation operation. This represents the intersection operation. Indicates a mask for historical movements. This indicates the region of sudden movement in the calculation. This indicates the current motion mask.

[0086] In one embodiment, the reference rendering image corresponding to the current video frame is obtained based on the three-dimensional Gaussian model corresponding to the previous time step. The three-dimensional Gaussian model of the previous time step has been optimized, so it is directly used for image rendering, and the reference rendering image is obtained using a rasterizer. During this process, gradient updates are turned off, meaning that only image generation is performed. However, the generated reference rendered image will not affect the relevant parameters of the 3D Gaussian model, such as the attribute values ​​of Gaussian primitives.

[0087] In one embodiment, reference is made to Figure 7 , Figure 7 This is a flowchart illustrating the calculation of motion loss values ​​based on the current video frame, a reference rendered image, and a sudden motion mask, provided in an embodiment of this application. The flowchart specifically includes the following steps: Step 710: Calculate the image pixel loss value and structure loss value based on the current video frame and the reference rendered image, and calculate the color loss value based on the image pixel loss value and structure loss value.

[0088] In one embodiment, referring to the calculation process of the initial loss value, firstly, the image pixel loss value is calculated based on the current video frame and the reference rendered image. This is used to directly measure the deviation between the current video frame and the reference rendered image in the color value of a single pixel. Secondly, the structural loss value is calculated. Then, the color loss value is calculated based on the image pixel loss value and the structural loss value. , represented as:

[0089] in, This represents the image pixel loss value, which is the L1 loss. Indicates the structural loss value. Indicates the current video frame. Indicates a reference rendered image. This represents the weighting parameter, which should be set according to the actual situation.

[0090] Step 720: Multiply the color loss value by the sudden motion mask based on pixel position to obtain the motion loss value.

[0091] In one embodiment, motion loss value The goal is to ensure that the color loss value only applies to motion-dependent pixel locations, expressed as:

[0092] in, The purpose of multiplying pixel by pixel is to include color loss in the total loss only when the pixel belongs to the moving area; the loss in the static area is masked.

[0093] Next, the motion loss value is used to evaluate whether each 3D Gaussian primitive involved in the rendering process is related to the rendering of the motion region. If the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, the active primitives are assigned to the active set. Specifically, the motion loss value is used to obtain the gradient data of the Gaussian primitives related to the generation of the reference rendered image. If the gradient data is greater than a preset gradient value and the Gaussian primitive is located in the frozen set, the Gaussian primitive is defined as an active primitive, and the active primitive is moved from the frozen set to the active set.

[0094] The process of determining the activity primitive is expressed as follows:

[0095] in, Represents the gradient data of Gaussian elements. This represents the preset gradient value. Based on the gradient data, Gaussian primitives that were originally assigned to the frozen set in the prediction module are reactivated. A spatial clustering algorithm can also be performed on the activated Gaussian primitives, and at least one related Gaussian primitive is also made an active primitive to ensure regional consistency of the moving object. Finally, the active primitives are transferred to the active set. Meanwhile, the frozen set is represented as .

[0096] In one embodiment, reference is made to Figure 8 , Figure 8 This is a schematic diagram illustrating how the observation module updates Gaussian elements in an embodiment of this application. First, based on the video sequence, M historical video frames are selected within the video frame window to form a time-series frame group. Since each historical video frame has a cached motion mask, the corresponding historical motion mask can be obtained. Next, the current motion mask corresponding to the current video frame is calculated based on the time-series frame group. Then, the sudden motion mask is obtained by using the historical motion mask. Then, based on the current video frame, the reference rendered image, and the sudden motion mask, the motion loss value is calculated. According to the motion loss value Obtain the gradient data of Gaussian elements related to the generation of the reference rendered image. If the gradient data is greater than a preset gradient value... Furthermore, since the Gaussian primitive is located in the frozen set, the Gaussian primitive is defined as the active primitive, and the active primitive is moved from the frozen set to the active set.

[0097] At this point, the precise set of activities corresponding to the current video frame is obtained. and frozen collection Based on this, the optimization process of the three-dimensional Gaussian model corresponding to the current video frame is carried out.

[0098] Step 150: Perform the optimization process for the current moment based at least on the updated set of activities.

[0099] In one embodiment, reference is made to Figure 9 , Figure 9 This is a flowchart illustrating the optimization process for the current moment based at least on the updated set of activities, as provided in this application embodiment. The process specifically includes the following steps: Step 910: Use the attribute prediction network to learn the attributes of the 3D Gaussian model corresponding to the previous time step, obtain the residual change of Gaussian elements in each activity set at the current time step, update the attributes of Gaussian elements in the activity set based on the residual change, and obtain the 3D Gaussian model corresponding to the current time step based on the frozen set and the updated activity set.

[0100] In one embodiment, when dynamically rendering the target scene using a 3D Gaussian model, an attribute prediction network is also constructed. By inputting the 3D Gaussian model into the attribute prediction network, prediction results for the attribute values ​​of each Gaussian unit can be obtained. The attribute prediction network can be constructed using a hash-encoded grid and a two-layer neural network MLP; this embodiment does not limit this approach.

[0101] In one embodiment, since the properties of a three-dimensional Gaussian element include position parameters ,transparency Scaling parameters Rotation matrix and spherical harmonic color coefficients Therefore, by using an attribute prediction network to learn the attributes of the 3D Gaussian model corresponding to the previous time step, the residual change of each Gaussian element in the current activity set is obtained, expressed as:

[0102]

[0103]

[0104] in, This represents the Gaussian elements in the active set of the 3D Gaussian model corresponding to the previous moment. Indicates an attribute prediction network. This represents the change in residuals. This indicates the updated attribute.

[0105] Step 920: Use a 3D Gaussian model to render the image to obtain the rendered image corresponding to the current video frame. Calculate the optimized loss value based on the rendered image and the current video frame. Update the parameters of the attribute prediction network based on the optimized loss value.

[0106] In one embodiment, the attributes of Gaussian elements in the active set are updated using residual changes. The updated active set, along with the frozen set of Gaussian elements from the previous time step without updated parameters, participate in the rendering process to obtain the rendered image corresponding to the current video frame. Then, following the initial loss value calculation process, an optimized loss value is calculated based on the rendered image and the current video frame. The parameters of the attribute prediction network are updated based on the optimized loss value, completing the optimization process of the 3D Gaussian model corresponding to the current video frame at the current time step.

[0107] Understandably, during the optimization of the initial video frames, the attributes of each Gaussian element in the 3D Gaussian model, as well as the color correction matrix and bias vector of the acquisition device, are optimized. However, during the optimization of non-initial video frames, the optimization loss value is used to optimize the attribute prediction network at each time step, instead of directly optimizing all attributes of each 3D Gaussian element. The attributes can be predicted by the optimized attribute prediction network.

[0108] In one embodiment, additional caching is required during model streaming training, including caching of the active states of Gaussian elements corresponding to K historical video frames, caching of historical motion masks corresponding to historical video frames, and image caching of M historical video frames, for the prediction and observation coordination mechanism of the current frame. Additionally, for the initial video frame, all attributes of all 3D Gaussian elements, as well as the color correction matrix and bias vector of each acquisition device, need to be stored and transmitted. For each subsequent video frame, a binary mask recording the changed Gaussian elements is stored and transmitted to represent the frozen set and active set in that video frame. The active set... The Gaussian element is defined as 1, while the frozen set is... The Gaussian element is defined as 0, and the active set also needs to be stored. The residual change of each Gaussian element.

[0109] In one embodiment, reference is made to Figure 10 , Figure 10This is a schematic diagram of the overall process of the dynamic scene streaming reconstruction method provided in this application embodiment. After acquiring the video sequence, a 3D Gaussian model is generated using the initial video frame corresponding to the initial time T=0. Then, during the optimization process of subsequent video frames, the Gaussian primitives contained in the frozen set and the active set are determined in the prediction module. Then, the possible Gaussian primitives located in the frozen module are activated by the observation module, turning them into active primitives and transferring them to the active set. Finally, the 3D Gaussian model corresponding to the current time is obtained based on the frozen set and the updated active set. Image rendering is performed using the 3D Gaussian model to obtain the rendered image corresponding to the current video frame. The optimization loss value is calculated based on the rendered image and the current video frame, and the parameters of the attribute prediction network are updated based on the optimization loss value.

[0110] Related technologies require optical flow assistance to identify the motion mask of each video frame and utilize maximum weight Gaussian unit indexing and morphological convex hulling operations to separate complete moving objects from static regions. This method requires time-consuming optical flow estimation, which is inaccurate in some cases and easily introduces optical flow noise, causing additional flicker in some frames and reducing rendering quality and temporal consistency. In contrast, the embodiments of this application are based on three-dimensional Gaussian units and combine a predictive and observational collaborative mechanism to distinguish and optimize static and moving regions in dynamic scenes. It only relies on historical cache information during model optimization, without relying on pre-trained models, and does not increase training complexity. It can achieve stable, low-flicker streaming free-viewpoint video synthesis, without time-consuming optical flow operations and without introducing additional optical flow noise. It can achieve higher temporal consistency and rendering quality, and can generate high-fidelity, low-flicker free-viewpoint videos. It is suitable for dynamic video scenarios such as online live streaming, motion broadcasting, and stage live streaming.

[0111] The technical solution provided in this application involves acquiring video sequences of a target scene from multiple perspectives, each video sequence including an initial video frame acquired at an initial moment and other video frames acquired at other moments; constructing a three-dimensional Gaussian model using the initial video frames from each perspective, the three-dimensional Gaussian model including multiple Gaussian primitives; selecting other video frames one by one as the current video frame, acquiring at least one historical video frame corresponding to the current video frame, acquiring a classification state label for each Gaussian primitive based on the historical video frame, and classifying the Gaussian primitives into an active set and a frozen set based on the classification state label; acquiring the current motion mask of the current video frame, determining a sudden motion mask based on the current motion mask, acquiring the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculating a motion loss value based on the current video frame, the reference rendering image, and the sudden motion mask, and if the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, classifying the active primitives into the active set; and performing an optimization process for the current moment based at least on the updated active set. This application first constructs a 3D Gaussian model using initial video frames from multiple perspectives. When processing subsequent video frames, it labels each Gaussian element with a classification state marker from historical video frames, dividing the dynamic object's active set into a frozen set corresponding to the static region. Simultaneously, it determines a sudden motion mask by combining the motion mask of the current video frame, accurately capturing targets with sudden movements in the scene. This avoids modeling omissions or positional deviations caused by rapid movement of dynamic targets, achieving refined separation of dynamic and static regions. Since flickering is caused by attribute changes in static regions during each frame's optimization process, leading to temporal inconsistencies, this application accurately separates the Gaussian elements corresponding to dynamic and static regions. During optimization, the active set is adjusted in real-time for the motion trajectory and shape changes of dynamic targets, ensuring smooth movement of dynamic objects in consecutive frames. The frozen set maintains unchanged attributes, preserving the stability of the static region and avoiding flickering caused by frequent, meaningless updates. Therefore, it effectively suppresses inter-frame flickering, improving the stability and reconstruction quality of dynamic scene streaming reconstruction.

[0112] This application also provides a dynamic scene streaming reconstruction apparatus, which can implement the above-described dynamic scene streaming reconstruction method, as described above. Figure 11 The device includes: Video frame acquisition module 1110: used to acquire video sequences of the target scene from multiple perspectives, each video sequence including the initial video frame acquired at the initial moment and other video frames acquired at other moments.

[0113] Initial Model Building Module 1120: Used to build a 3D Gaussian model using the initial video frames from each viewpoint. The 3D Gaussian model includes multiple Gaussian primitives.

[0114] Primitive initial classification module 1130: used to select other video frames one by one as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification status label of each Gaussian primitive based on the historical video frame, and divide the Gaussian primitive into the active set and the frozen set based on the classification status label.

[0115] Primitive activation classification module 1140: used to obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, the active primitives are assigned to the active set.

[0116] Optimization module 1150: Used for optimizing the current time step based at least on the updated set of activities.

[0117] The specific implementation of the dynamic scene streaming reconstruction device in this embodiment is basically the same as the specific implementation of the dynamic scene streaming reconstruction method described above, and will not be repeated here.

[0118] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in a memory, and the processor executes the at least one program to implement the dynamic scene streaming reconstruction method described above. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0119] Please see Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1201 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called and executed by the processor 1201 using the dynamic scene streaming reconstruction method of the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0120] This application embodiment also provides a storage medium that stores a computer program. When the computer program is executed by a processor, it implements the above-described dynamic scene streaming reconstruction method.

[0121] Memory, as a non-transitory storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The dynamic scene streaming reconstruction method, apparatus, device, and storage medium proposed in this application acquire video sequences of the target scene from multiple perspectives, each video sequence including an initial video frame acquired at an initial moment and other video frames acquired at other moments; construct a three-dimensional Gaussian model using the initial video frames of each perspective, the three-dimensional Gaussian model including multiple Gaussian primitives; select other video frames one by one as the current video frame, acquire at least one historical video frame corresponding to the current video frame, acquire the classification state label of each Gaussian primitive based on the historical video frame, and classify the Gaussian primitives into an active set and a frozen set based on the classification state label; acquire the current motion mask of the current video frame, determine a sudden motion mask based on the current motion mask, acquire the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image, and the sudden motion mask, and if the motion loss value indicates that some Gaussian primitives in the frozen set are active primitives, classify the active primitives into the active set; and perform an optimization process for the current moment based at least on the updated active set. This application first constructs a 3D Gaussian model using initial video frames from multiple perspectives. When processing subsequent video frames, it labels each Gaussian element with a classification state marker from historical video frames, dividing the dynamic object's active set into a frozen set corresponding to the static region. Simultaneously, it determines a sudden motion mask by combining the motion mask of the current video frame, accurately capturing targets with sudden movements in the scene. This avoids modeling omissions or positional deviations caused by rapid movement of dynamic targets, achieving refined separation of dynamic and static regions. Since flickering is caused by attribute changes in static regions during each frame's optimization process, leading to temporal inconsistencies, this application accurately separates the Gaussian elements corresponding to dynamic and static regions. During optimization, the active set is adjusted in real-time for the motion trajectory and shape changes of dynamic targets, ensuring smooth movement of dynamic objects in consecutive frames. The frozen set maintains unchanged attributes, preserving the stability of the static region and avoiding flickering caused by frequent, meaningless updates. Therefore, it effectively suppresses inter-frame flickering, improving the stability and reconstruction quality of dynamic scene streaming reconstruction.

[0123] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0124] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0127] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0128] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0130] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A dynamic scene streaming reconstruction method, characterized in that, include: Acquire video sequences of the target scene captured from multiple perspectives, each video sequence including an initial video frame captured at an initial time and other video frames captured at other times; A three-dimensional Gaussian model is constructed using the initial video frame from each of the aforementioned viewpoints, the three-dimensional Gaussian model comprising multiple Gaussian elements; Select each of the other video frames as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification state label of each Gaussian element based on the historical video frame, and divide the Gaussian element into an active set and a frozen set based on the classification state label. Obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some of the Gaussian primitives in the frozen set are active primitives, assign the active primitives to the active set; The optimization process at the current moment is based at least on the updated set of activities.

2. The dynamic scene streaming reconstruction method according to claim 1, characterized in that, The attribute parameters of the Gaussian elements include at least position parameters, and the classification state markers include stationary markers and active markers. Obtaining the classification state marker for each Gaussian element based on the historical video frames includes: For each Gaussian element, the viewpoint is selected one by one. From the current video frame to the historical video frame, the angular displacement corresponding to the Gaussian element between the historical video frame and the adjacent historical video frame is obtained based on the position parameters. The projected area of ​​the Gaussian element under the viewpoint is obtained. The normalized angular displacement is calculated based on the angular displacement and the projected area. The average angular displacement corresponding to the Gaussian element is calculated based on all the normalized angular displacements. If the average angular displacement is less than a preset threshold, a stationary marker indicating that the Gaussian element is stationary in the historical video frame is generated; otherwise, an active marker is generated.

3. The dynamic scene streaming reconstruction method according to claim 2, characterized in that, The step of dividing the Gaussian elements into an active set and a frozen set based on the classification state label includes: Obtain the classification state label of each Gaussian element in all the historical video frames; If all the classification state labels are static labels, the Gaussian element is assigned to the frozen set; otherwise, it is assigned to the active set.

4. The dynamic scene streaming reconstruction method according to claim 1, characterized in that, The step of obtaining the current motion mask of the current video frame and determining a sudden motion mask based on the motion mask includes: For each viewpoint, the standard deviation of each pixel position in the time dimension is calculated based on the corresponding current video frame and at least one historical video frame. After Gaussian smoothing, the current motion mask of the current video frame under the viewpoint is obtained. Obtain the motion mask corresponding to each historical video frame as the historical motion mask, and invert the historical motion mask to obtain the intermediate mask. Calculate the intersection of all the intermediate masks to obtain the sudden motion region. Obtain the sudden motion mask based on the current motion mask and the sudden motion region.

5. The dynamic scene streaming reconstruction method according to claim 1, characterized in that, The calculation of motion loss values ​​based on the current video frame, the reference rendered image, and the sudden motion mask includes: Calculate the image pixel loss value and structural loss value based on the current video frame and the reference rendered image, and calculate the color loss value based on the image pixel loss value and the structural loss value; The motion loss value is obtained by multiplying the color loss value by the sudden motion mask based on the pixel position.

6. The dynamic scene streaming reconstruction method according to claim 1, characterized in that, If the motion loss value indicates that a portion of the Gaussian primitives in the frozen set are active primitives, then assigning the active primitives to the active set includes: The gradient data of the Gaussian primitives related to the generation of the reference rendered image is obtained using the motion loss value. If the gradient data is greater than a preset gradient value and the Gaussian primitive is located in the frozen set, the Gaussian primitive is defined as the active primitive, and the active primitive is moved from the frozen set to the active set.

7. The dynamic scene streaming reconstruction method according to claim 1, characterized in that, The optimization process for the current moment, based at least on the updated set of activities, includes: The attribute prediction network is used to learn the attributes of the three-dimensional Gaussian model corresponding to the previous time step to obtain the residual change of the Gaussian element in each activity set at the current time step. The attributes of the Gaussian elements in the activity set are updated based on the residual change. The three-dimensional Gaussian model corresponding to the current time step is obtained based on the frozen set and the updated activity set. The image is rendered using the three-dimensional Gaussian model to obtain the rendered image corresponding to the current video frame. An optimized loss value is calculated based on the rendered image and the current video frame. The parameters of the attribute prediction network are updated based on the optimized loss value.

8. The dynamic scene streaming reconstruction method according to any one of claims 1 to 7, characterized in that, The construction of a 3D Gaussian model using the initial video frames from each of the aforementioned viewpoints includes: A three-dimensional Gaussian model is constructed using the initial video frame from each of the aforementioned viewpoints, and the initial rendered image corresponding to the initial video frame is generated using the three-dimensional Gaussian model. An initial loss value is calculated based on the initial video frame and the initial rendered image. Based on the initial loss value, the attributes of each Gaussian element in the 3D Gaussian model, the color correction matrix corresponding to each viewpoint, and the bias vector are adjusted.

9. A dynamic scene streaming reconstruction device, characterized in that, include: Video frame acquisition module: used to acquire video sequences of the target scene from multiple perspectives, each video sequence including an initial video frame acquired at the initial moment and other video frames acquired at other moments; Initial model building module: used to build a three-dimensional Gaussian model using the initial video frame of each viewpoint, the three-dimensional Gaussian model including multiple Gaussian elements; Primitive initial classification module: used to select the other video frames one by one as the current video frame, obtain at least one historical video frame corresponding to the current video frame, obtain the classification status label of each Gaussian primitive based on the historical video frame, and divide the Gaussian primitive into an active set and a frozen set based on the classification status label. The primitive activation classification module is used to obtain the current motion mask of the current video frame, determine the sudden motion mask based on the current motion mask, obtain the corresponding reference rendering image of the current video frame based on the three-dimensional Gaussian model corresponding to the previous moment, calculate the motion loss value based on the current video frame, the reference rendering image and the sudden motion mask, and if the motion loss value indicates that some of the Gaussian primitives in the frozen set are active primitives, the active primitives are assigned to the active set. Optimization module: Used to perform the optimization process at the current moment, at least based on the updated set of activities.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the dynamic scene streaming reconstruction method according to any one of claims 1 to 8.

11. A storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dynamic scene streaming reconstruction method according to any one of claims 1 to 8.

Images