Method and electronic device for dynamic reconstruction of a video

Abstract

The application provides a method and an electronic device for dynamically reconstructing a video, which comprises the following steps: removing a foreground region from a video sequence to obtain a new video sequence with only a background region, then using the new video sequence to obtain background Gaussian sphere target attributes, using the background Gaussian sphere target attributes to obtain foreground Gaussian sphere target attributes of the original video sequence, using the foreground Gaussian sphere target attributes and the background Gaussian sphere target attributes of each frame of image to perform sputtering rendering to obtain a new rendering image, and further obtaining a reconstructed video. According to the scheme, the foreground of the image is removed first, then the background Gaussian sphere attributes are obtained, and then the foreground Gaussian sphere attributes are obtained based on the background Gaussian sphere attributes, so that even if the background changes greatly, the background Gaussian sphere attributes of each frame of image can be obtained in time, and the reconstructed scheme can better perform three-dimensional reconstruction on videos in static scenes and dynamic scenes.

Description

Technical Field

[0001] This application relates to the field of dynamic video reconstruction technology, and more particularly to a method and electronic device for dynamic video reconstruction. Background Technology

[0002] With technological advancements, 3D reconstruction and new perspective generation are no longer limited to reconstructing single images; they can now also dynamically reconstruct video sequences. However, traditional 3D reconstruction schemes often rely on volumetric rendering, such as methods based on implicit neural radiance fields (NeRF). These methods require extensive and complex training of neural network models, suffer from low rendering efficiency, and are limited by image resolution. To address these shortcomings, Gaussian splitting, an explicit representation method, has emerged, offering advantages such as higher computational efficiency and faster rendering speed compared to traditional implicit representation methods.

[0003] However, the Gaussian sputtering method requires prior acquisition of camera pose and point cloud information using a structure-from-motion (SfM) method. This introduces a new problem: the Gaussian sputtering method is only suitable for static scenes, i.e., scenes with fixed or minimally changing backgrounds, and cannot adapt to dynamic scenes with large background changes. For example, a video of Xiao Jia making tea by the window is an example of a static scene; another example is a video of Xiao Yi skateboarding in a park, which is an example of a dynamic scene. For the video of Xiao Jia making tea, the SfM method can be used to obtain camera pose and point cloud information, and then the Gaussian sputtering method can be used for 3D reconstruction. However, for the video of Xiao Yi skateboarding, the SfM method cannot be used to obtain camera pose and point cloud information, and therefore the Gaussian sputtering method cannot be used for 3D reconstruction.

[0004] Therefore, how to better perform 3D reconstruction, especially 3D reconstruction of videos in dynamic scenes, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides a method and electronic device for dynamically reconstructing videos, which can better perform 3D reconstruction, especially for 3D reconstruction of videos in dynamic scenes.

[0006] In a first aspect, a method for dynamically reconstructing video is provided. This method includes: acquiring a first video sequence, where each frame of the first video sequence includes a foreground region and a background region, the foreground region representing the image region where the target object is located in each frame, and the background region representing other regions in each frame besides the foreground region; removing the foreground region from each frame of the first video sequence to obtain a second video sequence; using the second video sequence to obtain the background Gaussian sphere target attribute of each frame of the first video sequence; using the background Gaussian sphere target attribute of each frame of the first video sequence to obtain the foreground Gaussian sphere target attribute of each frame; performing sputter rendering on each frame of the first video sequence using the background Gaussian sphere target attribute and the foreground Gaussian sphere target attribute to obtain the target rendered image of that frame; and outputting the target reconstructed video corresponding to the first video sequence after obtaining the target rendered images of all images in the first video sequence.

[0007] In this technical solution, the foreground of the image is removed first, then the background Gaussian sphere properties are obtained, and finally the foreground Gaussian sphere properties are obtained based on the background Gaussian sphere properties. This allows for timely and accurate determination of the background Gaussian sphere properties for each frame, even with significant background changes. This enables the reconstruction scheme to perform better 3D reconstruction of videos in both static and dynamic scenes. Furthermore, this scheme eliminates the need for camera pose estimation and point cloud extraction using the SfM method, making it more suitable for dynamic scenes and not limited to static scenes.

[0008] It should be understood that Gaussian sputtering methods were developed to address the high training costs and low rendering efficiency of traditional volumetric rendering methods like NeRF. However, Gaussian sputtering requires camera pose estimation and point cloud extraction, and currently, camera pose estimation is only possible with methods like SfM, which are only suitable for static scenes. To address this issue, this application proposes a novel 3D reconstruction method. This method first removes the foreground from the video, retaining only the background. Then, it first calculates the Gaussian sphere properties of the background, and then calculates the Gaussian sphere properties of the foreground based on these properties. Finally, it performs sputtering rendering to achieve 3D reconstruction of the video. This approach eliminates the need for camera pose estimation and point cloud extraction using the SfM method, making it more suitable for dynamic scenes, not just static ones.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, when obtaining the background Gaussian sphere target attributes of each frame in the first video sequence using the second video sequence, the following operations may be included: For each frame in the second video sequence, the following operations are performed: The target camera pose of the frame is obtained by minimizing the image-level loss between the rendered image and the frame; the initial background Gaussian sphere attributes of the frame are transformed using the target camera pose to obtain the background Gaussian sphere target attributes of the frame. These background Gaussian sphere target attributes are the background Gaussian sphere target attributes of the image corresponding to that frame in the first video sequence. In this implementation, the target camera pose is obtained by minimizing the image set loss between the rendered image and the real image, and then the subsequent steps required by the Gaussian sputtering method are performed to obtain the background Gaussian sphere target attributes.

[0010] In one example, when obtaining the target camera pose of a frame image by minimizing the image-level loss between the rendered image and the frame image, the process may include: transforming the initial properties of the background Gaussian sphere of the frame image using the i-th adjusted camera pose of the frame image to obtain the i-th adjusted background Gaussian sphere properties of the frame image, where i is a positive integer; performing sputter rendering using the i-th adjusted background Gaussian sphere properties of the frame image to obtain the (i+1)-th rendered image of the frame image, where i is a positive integer; when the (i+1)-th rendered image of the frame image... When the image-level loss between the rendered image and the frame image meets a preset condition, the camera pose adjusted in the i-th iteration is determined as the target camera pose. Alternatively, when the image-level loss between the (i+1)-th rendered image and the frame image does not meet the preset condition, the camera pose adjusted in the i-th iteration is adjusted based on the principle of reducing the image-level loss between the (i+1)-th rendered image and the frame image to obtain the (i+1)-th adjusted camera pose. i+1 is then assigned to i, and the above steps are repeated until the target camera pose is obtained. In this example, through multiple iterative calculations and adjustments to the camera pose based on the calculation results, a new image-level loss is obtained until the image-level loss reaches the preset condition, at which point the target camera pose is output.

[0011] In one example, the initial value of the camera pose for this frame is the target camera pose from the previous frame. When calculating the target camera pose, the estimated camera pose is gradually adjusted based on the principle of minimizing image-level loss. This results in a new estimated camera pose, reducing the image-level loss calculated using this new estimate. The image-level loss continues until it falls within a preset range (close to a certain minimum range), at which point the output camera pose is the target camera pose. Therefore, the initial value of the camera pose can be the target camera pose obtained from the previous frame.

[0012] It should be noted that for the first frame image, since the initial attributes of the background Gaussian sphere are the same as the target attributes of the background Gaussian sphere, and a depth map has already been obtained through depth estimation, there is no need to estimate the camera pose. It is only necessary to obtain the target camera pose based on the depth map.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the initial properties of the background Gaussian sphere in the t-th frame of the second video sequence are obtained using the target properties of the background Gaussian sphere in the (t-1)-th frame and the target camera pose in the (t-1)-th frame, where t is an integer greater than 1. In this implementation, a method for obtaining the initial properties of the background Gaussian sphere is provided.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the background Gaussian sphere target attribute of the first frame image in the second video sequence is obtained by performing the following steps: performing depth estimation on the first frame image to obtain a depth map of the first frame image; using the depth map to obtain a rendered image of the first frame image; and obtaining the background Gaussian sphere target attribute of the first frame image by minimizing the image-level loss between the rendered image of the first frame image and the first frame image.

[0015] Since the first frame does not contain the previous frame, the background Gaussian sphere target attributes and target camera pose of the previous frame cannot be obtained. Therefore, the initial attributes of the background Gaussian sphere in the first frame are its background Gaussian sphere target attributes, which are obtained by acquiring the depth map and using the depth map.

[0016] It should also be understood that in this application, depth estimation is performed only on the first frame, not on all images. Furthermore, depth estimation can be performed on the first frame of either the second or first video sequence. Alternatively, depth estimation can be performed on any frame in the video sequence to obtain its depth map and the initial value of the background Gaussian sphere property. While this method is feasible, it is not suitable for frames preceding the selected frame. Therefore, the first frame is selected to obtain the depth map and the initial value of the background Gaussian sphere property. Subsequently, the initial values ​​of the background Gaussian sphere property for the second, third, and all subsequent frames can be calculated sequentially.

[0017] In conjunction with the first aspect, in certain implementations of the first aspect, when obtaining the foreground Gaussian sphere target attributes of each frame in the first video sequence using the background Gaussian sphere target attributes, the implementation may include: performing sputter rendering using the background Gaussian sphere target attributes and the initial foreground Gaussian sphere attributes of that frame to obtain the rendered image of that frame; and obtaining the foreground Gaussian sphere target attributes of that frame by minimizing the image-level loss between the rendered image of each frame in the first video sequence and the frame itself. In this implementation, with the background Gaussian sphere target attributes and the initial foreground Gaussian sphere attributes, the entire image can be rendered, and then the image-level loss is calculated and minimized to obtain the foreground Gaussian sphere target attributes.

[0018] As can be seen, in the above implementation, the initial properties of the foreground Gaussian sphere do not need to be calculated individually for each frame. That is, the initial properties of the foreground Gaussian sphere are the same for all images. This is because the foreground is the same target object in the same video sequence, and the initial value of the foreground Gaussian sphere properties only needs to be obtained once. Moreover, the initial properties of the foreground Gaussian sphere will be "corrected" by minimizing the image-level loss for each frame. That is, the parameters will be adjusted to obtain the target properties of the foreground Gaussian sphere for each frame.

[0019] It is also possible to achieve this by calculating the initial properties of the foreground Gaussian sphere for each frame of the image, but the computational efficiency is low and may even lead to the inability to complete the calculation in time, affecting the execution of subsequent steps.

[0020] In one example, the initial properties of the foreground Gaussian sphere can be obtained by performing the following steps: Depth estimation is performed on the first frame of the first video sequence to obtain a depth map of the first frame; based on the depth map of the first frame, the initial properties of the foreground Gaussian sphere are obtained. This example illustrates a method for obtaining the initial properties of the foreground Gaussian sphere. In this example, the initial values ​​of the position parameters (attributes) in the initial properties of the foreground Gaussian sphere are first obtained based on the depth map, and then other attributes (including rotation vectors, spherical harmonic coefficients, and other attributes) are randomly initialized to obtain the initial values ​​of each attribute in the initial properties of the foreground Gaussian sphere.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the first video sequence is a single-view video sequence, and the target reconstructed video is a new-view video sequence with a different viewpoint than the first video sequence. In this implementation, the solution of this application is applied to the generation of the new viewpoint, enabling the generation of a new video from a different viewpoint than the original video.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, the first video sequence is a video sequence in a dynamic scene. In this implementation, the solution of this application is applied to video reconstruction in dynamic scenes, thereby solving the technical problem that traditional solutions cannot be used for reconstruction in dynamic scenes using Gaussian sputtering. It should be understood that the solution of this application can achieve better 3D reconstruction for both static and dynamic scenes, while the above implementation emphasizes the case of dynamic scenes. That is to say, the solution of the embodiments of this application is very suitable for dynamic scenes, but it can also be applied to static scenes. When applied to static scenes, since the foreground Gaussian sphere properties and background Gaussian sphere properties are determined frame by frame, instead of performing only one camera pose estimation and point cloud extraction, the quality of the reconstructed video can still be improved to a certain extent compared to the traditional solution of SfM + Gaussian sputtering.

[0023] In a second aspect, an apparatus for dynamically reconstructing video is provided, the apparatus comprising units consisting of software and / or hardware for performing any of the methods of the first aspect.

[0024] Thirdly, an electronic device is provided, comprising: one or more processors, and a memory; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to enable the electronic device to implement any of the methods of the first aspect.

[0025] Fourthly, a chip system is provided, the chip system being applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to enable the electronic device to implement any of the methods of the first aspect.

[0026] Optionally, the chip system also includes a memory electrically connected to the processor.

[0027] Optionally, the chip system may also include a communication interface.

[0028] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium including instructions that, when executed on an electronic device, enable the electronic device to implement any of the methods of the first aspect.

[0029] In a sixth aspect, a computer program product is provided, comprising a computer program that, when executed by an electronic device, can implement any of the methods of the first aspect. Attached Figure Description

[0030] Figure 1This is an example of generating new perspectives or dynamically reconstructing a static scene.

[0031] Figure 2 This is an example of a video in a dynamic scene.

[0032] Figure 3 This is a schematic flowchart illustrating a method for dynamically reconstructing video according to an embodiment of this application.

[0033] Figure 4 These are examples of the first video sequence and the second video sequence in the embodiments of this application.

[0034] Figure 5 This is a schematic diagram illustrating the execution process of obtaining the background Gaussian sphere target attributes according to an embodiment of this application.

[0035] Figure 6 This is a schematic diagram illustrating the execution process of obtaining the foreground Gaussian sphere target attributes according to an embodiment of this application.

[0036] Figure 7 This is a schematic diagram of the structure of a device for dynamically reconstructing video according to an embodiment of this application.

[0037] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0038] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0039] Figure 1 This is an example of generating new perspectives or dynamically reconstructing scenes within a static context. For example... Figure 1 As shown, the static scene depicts Xiaojia giving a speech at a podium, with a static blackboard as the background; the only dynamic element is Xiaojia's actions. Therefore, in Figure 1 In video A, the background of each frame remains unchanged, and the shooting angle is also fixed, always a distant shot facing Xiaojia and the blackboard.

[0040] A new perspective was generated from video A to produce video B. The angle of video B is different from that of video A, and a close-up of Xiao Jia appears, while only a part of the blackboard is shown. This close-up is not actually captured in the video, but is generated through the new perspective.

[0041] By dynamically reconstructing video A, video C can be obtained. Each frame of video C is basically the same as that of video A.

[0042] In such static scenes, since it is only necessary to obtain the camera pose and point cloud information once, and subsequent changes are only minor, it can be understood that the difference between each two adjacent frames is small. Therefore, the SfM method can be used to obtain the camera pose and point cloud information, and then Gaussian sputtering can be used for 3D reconstruction to achieve new perspective generation or video reconstruction.

[0043] Figure 2 This is an example of a video in a dynamic scene. For example... Figure 2 As shown, Video D is a video of Xiao Yi cycling outdoors. The background in Video D (the part other than Xiao Yi) is constantly changing, for example... Figure 2 In a frame of video D shown in (a), there are three trees and two vehicles in the background: a bus and a car. Figure 2 In another frame of video D shown in (b), the background... Figure 2 In the image (a), only two and a half of the three trees are visible, and the relative positions of the two cars in the background are also relatively smaller. Figure 2 The two cars shown in (a) have changed, while Figure 2 In another frame of video D shown in (c), the background... Figure 2 In diagram (b), only one and a half trees are shown; the bus and car have disappeared, and a nearby truck has appeared. Figure 2 In another frame of video D shown in (d), the background... Figure 2 The half-tree shown in (c) is relatively less, and the relative position of the truck has also changed.

[0044] for Figure 2 The video in the dynamic scene shown has a constantly changing background, which can be understood as a situation where there are large differences between two adjacent frames. Therefore, it is impossible to use the SfM method to obtain camera pose and point cloud information. For example, the commonly used colmap method (a very common method in SfM) can be used to obtain camera pose and point cloud information, which makes it impossible to use the Gaussian sputtering method for 3D reconstruction.

[0045] It should be understood that Gaussian sputtering methods were developed to address the high training costs and low rendering efficiency of traditional volumetric rendering methods like NeRF. However, Gaussian sputtering requires camera pose estimation and point cloud extraction, and currently, camera pose estimation is only possible with methods like SfM, which are only suitable for static scenes. To address this issue, this application proposes a novel 3D reconstruction method. This method first removes the foreground from the video, retaining only the background. Then, it first calculates the Gaussian sphere properties of the background, and then calculates the Gaussian sphere properties of the foreground based on these properties. Finally, it performs sputtering rendering to achieve 3D reconstruction of the video. This approach eliminates the need for camera pose estimation and point cloud extraction using the SfM method, making it more suitable for dynamic scenes, not just static ones. Let's assume the solution proposed in this application is applied... Figure 1 In video A, the speaker, Xiao Jia, in each frame of video A is the target object, the area where the speaker is located is the foreground area, and the other areas outside the speaker are the background areas. Suppose the solution of this application is applied... Figure 2 In video D, the cyclist Xiao Yi and the bicycle in each frame of video D are the target objects, the area where the cyclist and the bicycle are located is the foreground area, and the other areas outside the cyclist and the bicycle are the background areas.

[0046] Figure 3 This is a schematic flowchart illustrating a method for dynamically reconstructing video according to an embodiment of this application. The following is a description of... Figure 3 The steps shown will be explained.

[0047] S301, Obtain the first video sequence.

[0048] Each frame in the first video sequence includes a foreground region and a background region. The foreground region represents the image area where the target object is located in each frame, and the background region represents the other areas in each frame besides the foreground region.

[0049] It should be understood that the target object can be a person, an animal, or a moving vehicle, etc. There can be one or more targets, and the types of targets can be different, for example... Figure 1 Xiao Jia in the middle Figure 2 Both Xiao Yi and Xiao Yi's bicycle can be examples of the target object. Figure 2 This can be seen as an example of multiple target objects.

[0050] Figure 1 Videos A, B, and C, and Figure 2 Video D in the video, Figure 4 The video sequences shown in (a)-(c) can all be used as the first video sequence.

[0051] The first video sequence can be obtained by using electronic devices such as mobile phones, tablets, cameras, or webcams, or by reading it from a storage unit that stores the first video sequence, or by obtaining it from a network through a communication interface. There are no limitations in the embodiments of this application.

[0052] In one implementation, the first video sequence is a single-view video sequence, and the target reconstructed video is a new viewpoint video sequence with a different viewpoint than the first video sequence. In this implementation, the solution of this application is applied to the generation of the new viewpoint, enabling the generation of a new video from a different viewpoint than the original video.

[0053] In one implementation, the first video sequence is a video sequence from a dynamic scene. In this implementation, the solution of this application is applied to video reconstruction in dynamic scenes, thereby solving the technical problem that traditional methods cannot be used for reconstruction in dynamic scenes using Gaussian sputtering. It should be understood that the solution of this application can achieve better 3D reconstruction for both static and dynamic scenes, but the above implementation emphasizes the dynamic scene scenario. That is, the solution of this application embodiment is very suitable for dynamic scenes, but it can also be applied to static scenes. When applied to static scenes, because the foreground and background Gaussian sphere properties are determined frame by frame, instead of performing only one camera pose estimation and point cloud extraction, the quality of the reconstructed video can still be improved to some extent compared to the traditional SfM+Gaussian sputtering solution.

[0054] S302. Remove the foreground region of each frame in the first video sequence to obtain the second video sequence.

[0055] The target object can be segmented from each frame of the image using image segmentation, and then the foreground region can be removed. Alternatively, the target object can be identified first using object recognition, and then the region containing the target object can be removed.

[0056] In the second video sequence, each frame contains only the background and no foreground; this can be understood as the foreground area being removed. As mentioned above, Figure 4 The video sequences shown in (a)-(c) can be considered as examples of the first video sequence. Figure 4 The video sequences shown in (d)-(f) can be considered as examples of the second video sequence. Figure 4 In the middle (d)-(f) images, the area where the target character is located, i.e., the foreground area, has been removed. This should be understood as... Figure 4 The value of i can be 1-N. The video sequence has a total of N frames, where N is an integer greater than 1.

[0057] S303. Using the second video sequence, obtain the background Gaussian sphere target attributes of each frame in the first video sequence.

[0058] In other words, the background Gaussian sphere attributes of each frame in the second video sequence are calculated frame by frame. These calculated background Gaussian sphere attributes then serve as the background Gaussian sphere target attributes of the corresponding image in the first video sequence. Combined with... Figure 4 In other words, that is to utilize Figure 4 The background Gaussian sphere property obtained from the first frame image shown in (d) is... Figure 4 The background Gaussian sphere target attributes of the first frame image shown in (a) are utilized. Figure 4 The background Gaussian sphere property obtained from the i-th frame image shown in (e) is... Figure 4 The background Gaussian sphere target attributes of the i-th frame image shown in (b) are utilized; Figure 4 The background Gaussian sphere property obtained from the Nth frame image shown in (f) is... Figure 4 The background Gaussian sphere target attributes of the first frame image shown in (N).

[0059] It should be noted that the Gaussian sphere property is a necessary parameter in the Gaussian sputtering method. The Gaussian sphere property can include the following parameters: position (x), rotation vector (r), scaling factor (s), spherical harmonic coefficient (c), and opacity (α). During Gaussian sputtering, each point in the point cloud of the image is expanded into a sphere based on the Gaussian sphere property to complete the rendering of the image.

[0060] In one implementation, step S303 may include: for each frame in the second video sequence, performing the following operations: obtaining the target camera pose of the frame by minimizing the image-level loss between the rendered image and the frame; transforming the initial properties of the background Gaussian sphere of the frame using the target camera pose of the frame to obtain the target properties of the background Gaussian sphere of the frame, which are the target properties of the background Gaussian sphere of the image corresponding to that frame in the first video sequence. In this implementation, the target camera pose is obtained by minimizing the image set loss between the rendered image and the real image, and then the subsequent steps required by the Gaussian sputtering method are performed to obtain the target properties of the background Gaussian sphere.

[0061] Minimizing image-level loss can be achieved through a single-step computation or through multiple iterations. Single-step computation is faster, but it's important to understand that this approach is only suitable if the computation of image-level loss is based on the assumption that it involves the hyperparameter equations. For example, a single-step least-squares method can be used to minimize the image-level loss and obtain the target camera pose. Iterative methods can employ various suitable iterative techniques, such as gradient iteration or least-squares iteration, without limitation. It should also be noted that in the scenario described in this application, minimizing the image-level loss does not require a large number of iterations; only a few iterations are needed to quickly converge to the optimum. This is why this application can quickly obtain the target camera pose by minimizing the image-level loss.

[0062] In one example, the calculation method involving multiple iterations may include: transforming the initial properties of the background Gaussian sphere of the frame image using the camera pose adjusted for the i-th time to obtain the background Gaussian sphere properties adjusted for the i-th time, where i is a positive integer; performing sputter rendering using the background Gaussian sphere properties adjusted for the i-th time to obtain the (i+1)-th rendered image of the frame image, where i is a positive integer; determining the camera pose adjusted for the i-th time as the target camera pose when the image-level loss between the (i+1)-th rendered image of the frame image and the frame image meets a preset condition; or, when the image-level loss between the (i+1)-th rendered image of the frame image and the frame image does not meet the preset condition, adjusting the camera pose adjusted for the i-th time based on the principle of reducing the image-level loss between the (i+1)-th rendered image of the frame image and the frame image, obtaining the (i+1)-th adjusted camera pose, and assigning i+1 to i to repeat the above steps until the target camera pose is obtained. In this example, the camera pose is adjusted based on the calculation results through multiple iterations to obtain a new image-level loss. The target camera pose is then output after the image-level loss reaches the preset conditions.

[0063] In one example, the camera pose of the (t+1)th frame satisfies the following formula: T t+1 =argminL photo (Render(T t ●G t ), I t+1 );

[0064] Where argmin represents the inverse function to be minimized, that is, the value of the parameter on the left side of the formula that minimizes the value on the right side of the formula. photoLet G represent the image-level loss, Render(.) represent the renderer, G represent the Gaussian sphere properties, T represent the camera pose, I represent the image, and "●" represent the dot product. From the above formula, it can be seen that the initial background Gaussian sphere properties G of the (t+1)th frame image are first obtained. t+1 =T t ●G t That is, the initial attribute G of the background Gaussian sphere of the (t+1)th frame image. t+1 The target pose is equal to the dot product of the background Gaussian sphere target attributes in frame t and the target camera pose in frame t. After sputter rendering by the renderer, the rendered image of frame t+1 is obtained, which can be understood as an estimated image of the real image. The image-level loss is calculated between this rendered image and the image of frame t+1. Then, the image-level loss is minimized, and its inverse function is obtained to get the value of T that minimizes the image-level loss, which is the target camera pose of frame t+1.

[0065] In the above formula, the initial properties of the Gaussian sphere are first fixed to obtain the camera pose. After obtaining the target camera pose, the initial properties of the Gaussian sphere are transformed using the target camera pose to obtain the background Gaussian sphere target properties of the current frame image.

[0066] It should also be understood that the rendered image obtained above still does not include the foreground area, only the background area, because the second video sequence does not have a foreground area.

[0067] It should be noted that there is a general formula in the Gaussian sputtering method: the Gaussian sphere property of the current frame image is equal to the dot product of the Gaussian sphere property of the previous frame image and the camera pose of the previous frame image, which can be expressed as G. N =T N-1 ●G N-1 This application obtains the rendered image of the current frame based on this general formula, then calculates the image-level loss and obtains the target camera pose by minimizing it.

[0068] In one example, the initial value of the camera pose for this frame is the target camera pose from the previous frame. When calculating the target camera pose, the estimated camera pose is gradually adjusted based on the principle of minimizing image-level loss. This results in a new estimated camera pose, reducing the image-level loss calculated using this new estimate. The image-level loss continues until it falls within a preset range (close to a certain minimum range), at which point the output camera pose is the target camera pose. Therefore, the initial value of the camera pose can be the target camera pose obtained from the previous frame.

[0069] It should be noted that for the first frame image, since the initial attributes of the background Gaussian sphere are the same as the target attributes of the background Gaussian sphere, and a depth map has already been obtained through depth estimation, there is no need to estimate the camera pose. It is only necessary to obtain the target camera pose based on the depth map.

[0070] In one implementation, the initial properties of the background Gaussian sphere in the t-th frame of the second video sequence are obtained using the target properties of the background Gaussian sphere in the (t-1)-th frame and the target camera pose in the (t-1)-th frame, where t is an integer greater than 1. This implementation provides a method for obtaining the initial properties of the background Gaussian sphere.

[0071] In one example, the initial property G of the background Gaussian sphere of the image in frame t. t Satisfying the following formula: G t =G t-1 ●T t-1 Where G represents the Gaussian sphere property, T represents the camera pose, and "●" represents the dot product. The G of the t-th frame image... t G equal to the image of frame t-1 t-1 and T t-1 The dot product.

[0072] In one implementation, the background Gaussian sphere target attribute of the first frame image in the second video sequence is obtained by performing the following steps: performing depth estimation on the first frame image to obtain a depth map of the first frame image; using the depth map, obtaining a rendered image of the first frame image; and obtaining the background Gaussian sphere target attribute of the first frame image by minimizing the image-level loss between the rendered image and the first frame image. Since the first frame image does not have a previous frame image, the background Gaussian sphere target attribute and target camera pose of the previous frame image cannot be obtained. Therefore, the initial background Gaussian sphere attribute of the first frame image is its background Gaussian sphere target attribute, which is obtained by acquiring and using the depth map.

[0073] It should be noted that this application does not limit the specific method of depth estimation, as long as the depth of the image can be estimated.

[0074] It should also be understood that in this application, depth estimation is performed only on the first frame, not on all images. Furthermore, depth estimation can be performed on the first frame of either the second or first video sequence. Alternatively, depth estimation can be performed on any frame in the video sequence to obtain its depth map and the initial value of the background Gaussian sphere property. While this method is feasible, it is not suitable for frames preceding the selected frame. Therefore, the first frame is selected to obtain the depth map and the initial value of the background Gaussian sphere property. Subsequently, the initial values ​​of the background Gaussian sphere property for the second, third, and all subsequent frames can be calculated sequentially.

[0075] It should be understood that step S303 is a step of processing the second video sequence, so the frame image in step S303 refers to the image frame in the second video sequence, and the Gaussian sphere attribute refers to the background Gaussian sphere attribute.

[0076] It should also be noted that when calculating the image-level loss, since the difference between the first and second video sequences is that the foreground region has been removed from the latter, the image-level loss for this part is fixed. Therefore, for a certain frame in the second video sequence, the image-level loss can be calculated by comparing its rendered image with the corresponding image in the first video sequence, or by comparing its rendered image with the corresponding image in the second video sequence. This does not affect minimizing the image-level loss; the two methods will only result in different minimum values.

[0077] S304. Using the background Gaussian sphere target attributes of each frame in the first video sequence, obtain the foreground Gaussian sphere target attributes of each frame in the first video sequence.

[0078] After obtaining the background Gaussian sphere target attributes, the estimated value of the foreground Gaussian sphere attributes can be estimated based on the initial values ​​of the background Gaussian sphere target attributes and the foreground Gaussian sphere attributes. Then, the rendered image of the entire image can be obtained based on the estimated values ​​of the background Gaussian sphere target attributes and the foreground Gaussian sphere attributes. The target attributes of the foreground Gaussian sphere can be deduced by minimizing the image-level loss between the rendered image and the actual image.

[0079] In one implementation, step S304 may include: performing sputter rendering using the background Gaussian sphere target attributes and the foreground Gaussian sphere initial attributes of the frame image to obtain a rendered image of the frame image; and obtaining the foreground Gaussian sphere target attributes of the frame image by minimizing the image-level loss between the rendered images of each frame image in the first video sequence and the frame image itself. In this implementation, the entire image can be rendered using the background Gaussian sphere target attributes and the foreground Gaussian sphere initial attributes, and then the image-level loss is calculated and minimized to obtain the foreground Gaussian sphere target attributes.

[0080] As can be seen, in the above implementation, the initial properties of the foreground Gaussian sphere do not need to be calculated individually for each frame. That is, the initial properties of the foreground Gaussian sphere are the same for all images. This is because the foreground is the same target object in the same video sequence, and the initial value of the foreground Gaussian sphere properties only needs to be obtained once. Moreover, the initial properties of the foreground Gaussian sphere will be "corrected" by minimizing the image-level loss for each frame. That is, the parameters will be adjusted to obtain the target properties of the foreground Gaussian sphere for each frame.

[0081] It is also possible to achieve this by calculating the initial properties of the foreground Gaussian sphere for each frame of the image, but the computational efficiency is low and may even lead to the inability to complete the calculation in time, affecting the execution of subsequent steps.

[0082] In one example, the initial properties of the foreground Gaussian sphere can be obtained by performing the following steps: Depth estimation is performed on the first frame of the first video sequence to obtain a depth map of the first frame; based on the depth map of the first frame, the initial properties of the foreground Gaussian sphere are obtained. This example illustrates a method for obtaining the initial properties of the foreground Gaussian sphere. In this example, the initial values ​​of the position parameters (attributes) in the initial properties of the foreground Gaussian sphere are first obtained based on the depth map, and then other attributes (including rotation vectors, spherical harmonic coefficients, and other attributes) are randomly initialized to obtain the initial values ​​of each attribute in the initial properties of the foreground Gaussian sphere.

[0083] Since depth estimation of the first frame image is required here, and step S303 also involves depth estimation of the first frame image, in actual use of the solution of this application, depth estimation can be performed only once to obtain the depth map. Then, the steps of obtaining the initial values ​​of the foreground Gaussian sphere attributes and the initial values ​​of the background Gaussian sphere attributes can be performed in parallel to improve processing efficiency.

[0084] S305. For each frame of the first video sequence, perform sputter rendering using the background Gaussian sphere target attributes and the foreground Gaussian sphere target attributes of the frame to obtain the target rendered image of the frame.

[0085] In other words, the target rendered image of each frame in the first video sequence is obtained frame by frame.

[0086] S306. After obtaining the target rendered images of all images in the first video sequence, output the target reconstructed video corresponding to the first video sequence.

[0087] In other words, once all images have obtained their corresponding target rendered images, these target rendered images constitute the target reconstructed video.

[0088] Figure 3 The method described primarily involves first removing the foreground of the image, then calculating the background Gaussian sphere properties, and finally calculating the foreground Gaussian sphere properties based on these background properties. This approach ensures that even with significant background variations, accurate background Gaussian sphere properties can be obtained for each frame, enabling better 3D reconstruction of both static and dynamic video scenes. This method eliminates the need for SfM (Simultaneous Matrix Modeling) methods for camera pose estimation and point cloud extraction, making it more suitable for dynamic scenes and not limited to static ones.

[0089] Figure 5 This is a schematic diagram illustrating the execution process of obtaining the background Gaussian sphere target attributes according to an embodiment of this application. Figure 5 This can be seen as an example of step S303. For example... Figure 5 As shown, for video sequences (d)-(f) containing only background regions, which is the second video sequence, the target camera pose of the i-th frame is obtained.

[0090] For the first frame image, a depth map of the first frame image is obtained after depth estimation. Then, based on this depth map, the initial values ​​of the Gaussian sphere properties are obtained by Gaussian sputtering rendering and minimizing image-level loss, denoted here as G1. It should also be understood that this depth map can also be obtained by depth estimation of the complete image of the frame image (without removing the foreground region), without any limitation. When the latter is used, the resulting depth map can also be used to subsequently obtain the foreground Gaussian sphere target properties.

[0091] For the second frame and all subsequent frames, referred to here as the i-th frame, where i is greater than 1 and less than or equal to N, and N is the total number of frames in the video sequence. For the i-th frame, we first obtain the Gaussian sphere attribute #1, which is the target attribute of the background Gaussian sphere attribute of the (i-1)-th frame. In other words, Gaussian sphere attribute #1 is the target attribute of the background Gaussian sphere attribute of the previous frame. The camera pose of the previous frame is obtained as the initial value of the camera pose of the current frame. The two are multiplied together, that is, the initial value of the camera pose of the current frame is multiplied by the Gaussian sphere attribute #1 to obtain the Gaussian sphere attribute #2, which is the initial background Gaussian sphere attribute of the current frame. The Gaussian sphere attribute #2 is used for sputter rendering to obtain a rendered image. The foreground region in this rendered image is not present. Then, the image-level loss between the rendered image and the i-th frame image is calculated. The estimated value of the camera pose is obtained by minimizing this image-level loss, which is the inverted camera pose Ti in the figure. Then, the Gaussian sphere attribute #1 is multiplied again to obtain a new Gaussian sphere attribute #2, a new rendered image, and a new inverted estimated value of the camera pose. This process is repeated until the image-level loss is minimized. The output camera pose is the target camera pose.

[0092] It should be noted that if a one-step optimization algorithm is used to minimize the image-level loss, the above steps end after the first inversion, and the obtained estimated camera pose is the target camera pose. The product of the target camera pose and Gaussian sphere attribute #1 is the background Gaussian sphere target attribute. If a multi-iteration optimization algorithm is used to minimize the image-level loss, the above steps are executed repeatedly until the image-level loss converges to meet the preset conditions, at which point the target camera pose is output, and the background Gaussian sphere target attribute is obtained based on the target camera pose.

[0093] It should also be understood that the above steps can also be performed by minimizing the image-level loss between the rendered image and the complete image of that frame (the image without the foreground region removed), which will not be elaborated further.

[0094] Figure 6 This is a schematic diagram illustrating the execution process of obtaining the foreground Gaussian sphere target attributes according to an embodiment of this application. Figure 6 This can be seen as an example of step S304.

[0095] For the complete image video sequence (a)-(c), which is the first video sequence, the foreground Gaussian sphere target attributes are obtained by combining the background Gaussian sphere target attributes that have been obtained previously.

[0096] After performing depth estimation on the first frame image to obtain a depth map, the foreground depth value is obtained by combining the foreground map and the depth map. Based on this foreground depth value, Gaussian sphere attribute #3 is obtained. Gaussian sphere attribute #3 is the initial attribute of the foreground Gaussian sphere, which can be represented as G. f1 .

[0097] For any frame image, i.e. the i-th frame image, where i is 1-N and N is the total number of frames, obtain its Gaussian sphere attribute #2, which is the background Gaussian sphere target attribute of the frame image. Use the foreground Gaussian sphere initial attribute (Gaussian sphere attribute #3) common to all images and the background Gaussian sphere target attribute (Gaussian sphere attribute #2) of the frame image to perform sputter rendering to obtain the rendered image. Calculate the image-level loss between the rendered image and the frame image, and obtain the foreground Gaussian sphere target attribute by minimizing the image-level loss.

[0098] It should be understood that Figure 6 The step of performing depth estimation on the first frame image to obtain the depth map until Gaussian sphere attribute #3 is obtained only needs to be performed once; all subsequent images will use Gaussian sphere attribute #3. For the i-th frame image, Gaussian sphere attribute #2 is also fixed, as it has already been calculated previously; for example, it can be used... Figure 5 The method shown yields a result that, in order to distinguish it from the foreground Gaussian sphere, can be represented as G. bi G bThis refers to the background Gaussian sphere, where i is the image frame number.

[0099] When minimizing image-level loss, if a one-step optimization algorithm is used, the above steps end after obtaining Gaussian sphere attribute #4 after the first inversion. The obtained Gaussian sphere attribute #4 is the foreground Gaussian sphere target attribute. If a multi-iteration optimization algorithm is used, the above steps are executed repeatedly until the image-level loss converges to meet the preset conditions. The output Gaussian sphere attribute #4 is then the foreground Gaussian sphere target attribute. That is, after obtaining Gaussian sphere attribute #4 for the first time, Gaussian sphere attribute #4 and Gaussian sphere attribute #2 are sputtered to obtain a new rendered image. The image-level loss between the new rendered image and the current frame image is calculated, and the new image-level loss is minimized to obtain a new Gaussian sphere attribute #4. The above steps are repeated until the image-level loss converges to meet the preset conditions.

[0100] The methods of the embodiments of this application have been described above with reference to the accompanying drawings. It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially, these steps are not necessarily executed in the order shown in the figures. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the steps or stages of other steps. The apparatus of the embodiments of this application will now be described with reference to the accompanying drawings.

[0101] Figure 7 This is a schematic diagram of the structure of a device for dynamically reconstructing video according to an embodiment of this application. Figure 7 As shown, the device 2000 includes an acquisition unit 2001 and a processing unit 2002. The device 2000 can be integrated into electronic devices such as mobile phones, tablets, laptops, computers, and in-vehicle systems.

[0102] The device 2000 can be used to execute any of the above-described methods for dynamically reconstructing video. For example, the acquisition unit 2001 can be used to execute step S301, and the processing unit 2002 can be used to execute steps S302-S306. The processing unit 2002 can also be used to execute... Figure 5 and Figure 6 The execution process.

[0103] In one implementation, the device 2000 may further include a storage unit for storing relevant data. This storage unit may be integrated into any of the aforementioned units, or it may be a unit independent of all of the aforementioned units.

[0104] Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Figure 8 As shown, the electronic device 3000 includes: at least one processor 3001 ( Figure 8 (Only one is shown) a processor, a memory 3002, and a computer program 3003 stored in the memory 3002 and executable on the at least one processor 3001, wherein the processor 3001 executes the computer program 3003 to implement the steps of any of the above methods.

[0105] Those skilled in the art will understand that Figure 8 This is merely an example of an electronic device and does not constitute a limitation on electronic devices. In practice, electronic devices may include more or fewer components than those shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output modules, network access modules, etc.

[0106] Processor 3001 may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network process unit (NPU), other general-purpose processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. Different processing units may be independent devices or integrated into one or more processors.

[0107] The controller can serve as the nerve center and command center of the electronic device 3000. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0108] In some embodiments, memory 3002 may be an internal storage unit of electronic device 3000, such as a hard disk or memory of electronic device 3000. In other embodiments, memory 3002 may be an external storage device of electronic device 3000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on electronic device 3000. Optionally, memory 3002 may include both internal and external storage units of electronic device 3000. Memory 3002 is used to store operating system, application programs, bootloaders, data, and other programs, such as program code of computer programs. Memory 3002 may also be used to temporarily store data that has been output or will be output.

[0109] The processor 3001 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 3001 is a cache memory. This memory can store instructions or data that the processor 3001 has just used or is recurring. If the processor 3001 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 3001, and thus improves the efficiency of the system.

[0110] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] This application also provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the steps in any of the above methods.

[0113] This application also provides a chip system applied to an electronic device. The chip system includes one or more processors, which invoke computer instructions to cause the electronic device to perform the steps in any of the methods described above. Optionally, the chip system further includes a memory electrically connected to the processor. Optionally, the chip system may also include a communication interface.

[0114] This application also provides a computer-readable storage medium storing instructions that, when executed by an electronic device, can implement any of the methods described above. This computer-readable medium may include at least: any entity or device capable of carrying computer program code (instructions) to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0115] This application also provides a computer program product, which includes a computer program that, when executed by an electronic device, can implement any of the above-described methods. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0117] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0119] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0120] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0121] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0122] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0123] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0124] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for dynamically reconstructing video, characterized in that, include: A first video sequence is obtained, wherein each frame of the first video sequence includes a foreground region and a background region, the foreground region being used to represent the image region where the target object is located in each frame, and the background region being used to represent other regions in each frame besides the foreground region; The foreground region of each frame in the first video sequence is removed to obtain the second video sequence; For each frame in the second video sequence, the following operations are performed: A rendered image is generated based on the initial background Gaussian sphere properties and camera pose of that frame; the initial background Gaussian sphere properties of the t-th frame in the second video sequence are obtained using the target background Gaussian sphere properties and the target camera pose of the (t-1)-th frame, where t is an integer greater than 1; the initial background Gaussian sphere properties of the first frame in the second video sequence are obtained based on the depth map of that first frame; the image-level loss between the rendered image and the frame is minimized to obtain the target camera pose of that frame; the initial background Gaussian sphere properties of the frame are transformed using the target camera pose to obtain the target background Gaussian sphere properties of that frame; wherein, the target background Gaussian sphere properties of each frame in the second video sequence are used as the target background Gaussian sphere properties of the corresponding frame in the first video sequence. The background Gaussian sphere target attribute of the first frame image in the second video sequence is obtained by performing the following steps: performing depth estimation on the first frame image to obtain a depth map of the first frame image; using the depth map to obtain a rendered image of the first frame image; and obtaining the background Gaussian sphere target attribute of the first frame image by minimizing the image-level loss between the rendered image of the first frame image and the first frame image. Sputter rendering is performed on the background Gaussian sphere target attributes and the foreground Gaussian sphere initial attributes of each frame in the first video sequence to obtain a rendered image. The foreground Gaussian sphere target attributes of the frame are obtained by minimizing the image-level loss between the rendered image and the frame image. For each frame in the first video sequence, sputter rendering is performed on the background Gaussian sphere target attributes and the foreground Gaussian sphere target attributes of the frame image to obtain the target rendered image of the frame image. The foreground Gaussian sphere initial attributes are obtained based on the depth map of the first frame image of the first video sequence. After obtaining the target rendered images of all images in the first video sequence, the target reconstructed video corresponding to the first video sequence is output.

2. The method according to claim 1, characterized in that, The step of obtaining the target camera pose of the frame image by minimizing the image-level loss between the rendered image and the frame image includes: The initial properties of the background Gaussian sphere of the image frame are transformed using the camera pose after the i-th adjustment of the image frame to obtain the background Gaussian sphere properties after the i-th adjustment of the image frame, where i is a positive integer; Splash rendering is performed using the i-th adjusted background Gaussian sphere property of the frame image to obtain the (i+1)-th rendered image of the frame image, where i is a positive integer. When the image-level loss between the (i+1)th rendered image of the frame and the current frame image meets a preset condition, the camera pose adjusted in the i-th iteration is determined as the target camera pose; or... When the image-level loss between the (i+1)th rendered image of the frame does not meet the preset conditions, the camera pose after the i-th adjustment is adjusted based on the principle of reducing the image-level loss between the (i+1)th rendered image of the frame and the frame, and the camera pose after the i-th adjustment is obtained. Then, i+1 is assigned to i and the above steps are repeated until the target camera pose is obtained.

3. The method according to claim 2, characterized in that, The initial value of the camera pose for this frame is the target camera pose of the previous frame.

4. The method according to claim 1, characterized in that, The initial properties of the foreground Gaussian sphere are obtained by performing the following steps: Depth estimation is performed on the first frame image of the first video sequence to obtain the depth map of the first frame image of the first video sequence; The initial properties of the foreground Gaussian sphere are obtained based on the depth map of the first frame of the first video sequence.

5. The method according to claim 1, characterized in that, The first video sequence is a single-view video sequence, and the target reconstruction video is a new-view video sequence with a different viewpoint than the first video sequence.

6. The method according to any one of claims 1 to 5, characterized in that, The first video sequence is a video sequence in a dynamic scene.

7. The method according to any one of claims 1 to 5, characterized in that, The properties of both the background and foreground Gaussian spheres include: position, rotation vector, scaling factor, spherical harmonic coefficient, and opacity.

8. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

Images