Foreground object segmentation and reconstruction method, device, equipment and storage medium

Abstract

The embodiments of the present application provide foreground target segmentation and reconstruction methods, devices, equipment and storage media, which relate to the field of computer vision technology. The method directly inputs the video frame into the target segmentation model to generate a foreground target segmentation mask, associates the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data and inputs the space-time Gaussian splash model for data processing, divides the input data into time slices, obtains the slice Gaussian point cloud corresponding to each time slice, obtains the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtains the rendering image corresponding to each perspective based on multiple two-dimensional Gaussian surfaces. The foreground target rendering sequence at the processing moment is obtained based on the rendering image corresponding to each perspective, and the foreground target reconstruction result is generated based on the foreground target rendering sequence at all acquisition moments. While reducing the dependence on external prior information, the stability of the timing can also be guaranteed, and the accuracy of the rendering results can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, apparatus, device and storage medium for foreground object segmentation and reconstruction. Background Art

[0002] In today's era where the digital world and reality intersect, the demand for photorealistic dynamic scene representation and rendering is increasing. In this process, to ensure that texture details are accurately presented and the scene's realism is fully expressed, high-quality modeling of objects with complex motion in dynamic scenes is required.

[0003] Related technologies for segmenting and reconstructing three-dimensional objects in complex dynamic environments typically rely on separating the foreground and background of the scene. However, due to the inherent characteristics of dynamic scenes, such as flexible object movements, diverse appearances, complex environments, and the susceptibility of targets to occlusion, there is inherent ambiguity when converting two-dimensional image sequences to three-dimensional space. Furthermore, the complex, large-scale, non-rigid deformations of the human body can also cause ambiguity in the reconstruction results. Therefore, during the model segmentation and reconstruction process, a large amount of prior knowledge is required to assist the reconstruction process, which complicates the entire segmentation and reconstruction process. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a foreground target segmentation and reconstruction method, device, equipment and storage medium to reduce the operational complexity of foreground target segmentation and reconstruction in complex dynamic scenes and improve the efficiency of segmentation and reconstruction.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a foreground object segmentation and reconstruction method, which is performed by a trained spatiotemporal Gaussian splatter model. The method includes:

[0006] At multiple acquisition moments, initial acquisition data corresponding to multiple perspectives are respectively acquired, wherein the initial acquisition data includes acquisition data corresponding to each of the acquisition moments, and the acquisition data includes video frames and initial Gaussian point clouds;

[0007] Inputting all the video frames into the target segmentation model for target selection, generating a foreground target segmentation mask corresponding to each video frame, and associating the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data;

[0008] The acquisition moments are used as processing moments one by one. At the processing moments, the input data corresponding to all viewpoints are input into the spatiotemporal Gaussian splatter model for data processing, the input data are time-sliced ​​to obtain a slice Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and a rendered image corresponding to each viewpoint is obtained based on the multiple two-dimensional Gaussian surfaces;

[0009] A foreground object rendering sequence at the processing moment is obtained according to the rendered image corresponding to each of the viewing angles, and a foreground object reconstruction result is generated according to the foreground object rendering sequences at all the acquisition moments.

[0010] In some embodiments, inputting all the video frames into the target segmentation model for target selection and generating a foreground target segmentation mask corresponding to each video frame includes:

[0011] Obtaining prompt texts corresponding to all the video frames;

[0012] The video frame and the prompt text are input into a target segmentation model for target selection to obtain at least one foreground target, and the at least one foreground target is superimposed to obtain the foreground target segmentation mask corresponding to the video frame.

[0013] In some embodiments, dividing the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice includes:

[0014] Acquire a time slice, and obtain an initial slice point cloud based on the time slice and the initial Gaussian point cloud;

[0015] Using the foreground object segmentation mask, a mask operation is performed on each of the initial slice point clouds to obtain the corresponding slice Gaussian point cloud;

[0016] A corresponding depth value is determined according to a relationship between a center position of the slice Gaussian point cloud and a camera position of the viewing angle, and the slice Gaussian point clouds are sorted based on a magnitude order of the depth values.

[0017] In some embodiments, obtaining a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud includes:

[0018] Obtaining the center position, color parameters, opacity parameters, rotation matrix, and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud;

[0019] Selecting a specific coordinate axis, and adjusting scale information of the initial scaling matrix based on the specific coordinate axis to obtain a target scaling matrix;

[0020] At least the center position, the color parameter, the opacity parameter, the rotation matrix and the target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

[0021] In some embodiments, obtaining a rendered image corresponding to each viewing angle according to the plurality of two-dimensional Gaussian surfaces includes:

[0022] For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces;

[0023] The rendered image corresponding to the viewing angle is obtained based on the pixel rendering color of the two-dimensional Gaussian surface.

[0024] In some embodiments, for each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces includes:

[0025] Obtaining a mixing weight of the two-dimensional Gaussian surface;

[0026] Obtaining the marginal probability density and conditional probability density corresponding to the pixel point, and obtaining a weighted contribution value according to the product of the marginal probability density, the conditional probability density, and the mixing weight;

[0027] Based on the order of the depth values, obtaining the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtaining a cumulative weight value according to the weighted contribution values;

[0028] The pixel rendering color is obtained according to the product of the weighted contribution value, the color parameter and the accumulated weight value.

[0029] In some embodiments, obtaining the mixing weight of the two-dimensional Gaussian surface includes:

[0030] Obtaining the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud;

[0031] Obtaining a camera transformation matrix and a local affine transformation matrix corresponding to the viewing angle;

[0032] Calculating a two-dimensional covariance according to the camera transformation matrix, the local affine transformation matrix, and the three-dimensional covariance, and performing a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density;

[0033] The mixing weight is obtained according to the two-dimensional probability density and the corresponding opacity parameter.

[0034] In some embodiments, the training process of the spatiotemporal Gaussian splash model includes at least:

[0035] Acquire a plurality of training samples, wherein the training samples include at least a training mask, a training Gaussian point cloud, and a label image for each of the viewing angles corresponding to each training acquisition moment;

[0036] Inputting the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering to obtain a training rendered image, wherein the training rendered image corresponds to a two-dimensional Gaussian training surface;

[0037] During the rendering process, generating a depth consistency loss value based on the mixing weight and the training depth value of each of the two-dimensional Gaussian training surfaces;

[0038] A monocular normal regularization loss value is calculated based on the normal vectors corresponding to the training rendered image and the label image respectively;

[0039] For two consecutive training acquisition moments, a time domain regularization loss value is calculated based on the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing term loss value is calculated based on the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface;

[0040] Obtaining a pixel loss value and a structure loss value according to the training rendered image and the label image, and calculating a photometric difference loss value according to the pixel loss value and the structure loss value;

[0041] Selecting at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness term loss value as a target loss value, and obtaining a total loss value according to the target loss value and the photometric difference loss value;

[0042] The model parameters of the spatiotemporal Gaussian splash model are adjusted according to the total loss value until an iteration termination condition is reached, thereby obtaining the trained spatiotemporal Gaussian splash model.

[0043] In some embodiments, generating a depth consistency loss value based on a mixing weight and a training depth value of each of the two-dimensional Gaussian training surfaces includes:

[0044] For the two-dimensional Gaussian training surface, based on the order of training depth values, for all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all the previous two-dimensional Gaussian surfaces, obtain a first intermediate value according to the difference between the first intermediate value and the training mixing weight, accumulate the first intermediate values ​​and multiply the resultant value by the training mixing weight of the two-dimensional Gaussian training surface to obtain a total training mixing weight, select a maximum value from the total training mixing weights to obtain a maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value;

[0045] Obtaining a training weighted contribution value and a training depth value of the two-dimensional Gaussian training surface, calculating a first product of the training weighted contribution value and the training depth value, and calculating a second product of the training weighted contribution value and the maximum training depth value;

[0046] Accumulating the first products to obtain a second intermediate value, and accumulating the second products to obtain a third intermediate value, obtaining a difference between the second intermediate value and the third intermediate value, and calculating the product of the maximum mixing weight and the difference to obtain the depth consistency loss value.

[0047] In some embodiments, calculating the monocular normal regularization loss value based on the normal vectors corresponding to the training rendered image and the label image respectively includes:

[0048] Obtaining the corresponding training slice Gaussian point cloud during the rendering process, and obtaining the monocular estimated normal vector corresponding to the training slice Gaussian point cloud;

[0049] generating a pseudo normal vector according to the training rendered image;

[0050] The monocular normal regularization loss value is calculated according to the monocular estimated normal vector and the pseudo normal vector.

[0051] In some embodiments, for two consecutive training acquisition moments, calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface includes:

[0052] At the current training acquisition moment, a preset number of adjacent Gaussian points corresponding to the central Gaussian point at the center position of the two-dimensional Gaussian training surface are obtained, a current color parameter, a current opacity parameter, and a current target scaling value of the central Gaussian point are obtained from the surface parameters, and a total training mixture weight of the two-dimensional Gaussian training surface is obtained;

[0053] Obtaining the reference color parameter, reference opacity parameter, and reference target scaling value of the adjacent Gaussian point from the surface parameters corresponding to the viewing angle at the previous training acquisition moment;

[0054] Calculating a norm of a color difference between the current color parameter and the reference color parameter, a norm of an opacity difference between the current opacity parameter and the reference opacity parameter, and a norm of a scaling difference between the current target scaling value and the reference target scaling value;

[0055] The time domain regularization loss value is obtained by multiplying the total weight of the training mixture with the norm of the color difference, the norm of the opacity difference, and the norm of the scaling difference. The three products corresponding to each of the adjacent Gaussian points are accumulated.

[0056] In some embodiments, calculating the motion smoothness loss value based on the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface includes:

[0057] At the current training acquisition moment, obtaining the current position parameter and the current rotation parameter of the central Gaussian point from the surface parameters, and obtaining the current reference position parameter of the adjacent Gaussian point;

[0058] Obtaining the reference position parameters and reference rotation parameters of the adjacent Gaussian points, and the center reference position parameters corresponding to the center Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment;

[0059] For each of the adjacent Gaussian points, obtaining a fourth intermediate value according to the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, obtaining a fifth intermediate value according to the difference between the reference position parameter and the center reference position parameter, and obtaining a sixth intermediate value according to the difference between the current reference position parameter and the current position parameter;

[0060] Converting the fourth intermediate value into a rotated intermediate matrix, subtracting the sixth intermediate value from the product of the fifth intermediate value and the rotated intermediate matrix to obtain a seventh intermediate value, and multiplying the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain an eighth intermediate value;

[0061] The eighth intermediate values ​​corresponding to all adjacent Gaussian points are accumulated to obtain the motion loss term corresponding to the current two-dimensional Gaussian training surface training, and all the motion loss terms are accumulated to obtain the motion smoothing term loss value at the current training acquisition moment.

[0062] To achieve the above objectives, a second aspect of an embodiment of the present application provides a foreground object segmentation and reconstruction device, which is performed by a trained spatiotemporal Gaussian splatter model, and includes:

[0063] Acquisition module: used to obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, the initial acquisition data including acquisition data corresponding to each acquisition moment, the acquisition data including video frames and initial Gaussian point clouds;

[0064] Mask calculation module: used to input all the video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data;

[0065] Rendering module: used to use the acquisition moments as processing moments one by one, input the input data corresponding to all viewpoints into the spatiotemporal Gaussian splatter model for data processing at the processing moments, divide the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice, obtain a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain a rendered image corresponding to each viewpoint based on the multiple two-dimensional Gaussian surfaces;

[0066] Reconstruction module: used for obtaining a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each of the viewing angles, and generating a foreground target reconstruction result according to the foreground target rendering sequences at all the acquisition moments.

[0067] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0068] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0069] The foreground target segmentation and reconstruction method, apparatus, device and storage medium proposed in the embodiments of the present application obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, input all video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data. At the processing moment, the input data corresponding to all perspectives are input into the spatiotemporal Gaussian splatter model for data processing, the input data is time-sliced ​​to obtain a sliced ​​Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained, and a rendered image corresponding to each perspective is obtained based on the multiple two-dimensional Gaussian surfaces. A foreground target rendering sequence at the processing moment is obtained based on the rendered image corresponding to each perspective, and a foreground target reconstruction result is generated based on the foreground target rendering sequence at all acquisition moments. In the embodiments of the present application, the video frames are directly input into the target segmentation model to generate a foreground target segmentation mask, without the need for manual pre-setting of too much prior knowledge regarding the specific shape, position, etc. of the target. Furthermore, the spatiotemporal Gaussian splash model divides the input data into time slices based on its inherent temporal dimension, eliminating the need for prior knowledge such as the specific change patterns of each target at different moments. This further reduces reliance on external prior information while ensuring temporal stability. Furthermore, after time slicing, the spatiotemporal Gaussian splash model can track the dynamic changes of complex geometric shapes across different time slices. During the rendered image generation phase, incorporating a two-dimensional Gaussian surface accurately and adaptively fits the object's shape and surface details, thereby improving reconstruction accuracy. Furthermore, by analyzing the motion trajectory and morphological changes of objects within time slices from different viewpoints, the spatiotemporal Gaussian splash model leverages the temporal coherence and spatial correlation of the data to adjust for potential inconsistencies across different viewpoints. This ensures that the resulting rendered images from each viewpoint are more consistent in key elements such as the object's position and shape. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is a flowchart of the foreground target segmentation and reconstruction method provided in an embodiment of the present application.

[0071] Figure 2 Schematic diagram of a video frame provided in an embodiment of the present application.

[0072] Figure 3 This is a flowchart provided by an embodiment of the present application for inputting all video frames into a target segmentation model for target selection, and generating a foreground target segmentation mask corresponding to each video frame.

[0073] Figure 4 A schematic diagram of the process of generating a foreground target mask provided in an embodiment of the present application.

[0074] Figure 5 Schematic diagram of input data provided in an embodiment of the present application.

[0075] Figure 6 This is a flowchart provided by an embodiment of the present application for time slicing input data to obtain a slice Gaussian point cloud corresponding to each time slice.

[0076] Figure 7 Schematic diagram of a sliced ​​Gaussian point cloud provided in an embodiment of the present application.

[0077] Figure 8 This is a flowchart of obtaining the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud provided in an embodiment of the present application.

[0078] Figure 9 This is a flowchart of obtaining the corresponding pixel rendering color for each pixel point in a two-dimensional Gaussian surface according to multiple two-dimensional Gaussian surfaces provided by an embodiment of the present application.

[0079] Figure 10 This is a flowchart of obtaining the mixing weights of a two-dimensional Gaussian surface provided in an embodiment of the present application.

[0080] Figure 11 Schematic diagram of a rendered image provided in an embodiment of the present application.

[0081] Figure 12 This is a schematic diagram of the foreground target reconstruction result provided in an embodiment of the present application.

[0082] Figure 13 This is a flowchart of the training process of the spatiotemporal Gaussian splash model provided in an embodiment of the present application.

[0083] Figure 14 This is a flowchart of generating a depth consistency loss value based on the mixing weight and training depth value of each two-dimensional Gaussian training surface provided by an embodiment of the present application.

[0084] Figure 15 This is a flowchart of an embodiment of the present application for calculating a monocular normal regularization loss value based on the normal vectors corresponding to the training rendered image and the label image.

[0085] Figure 16 This is a flowchart of calculating the time domain regularization loss value based on the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments provided by an embodiment of the present application.

[0086] Figure 17 This is a flowchart of calculating the motion smoothness loss value based on the corresponding position parameters and rotation parameters of a two-dimensional Gaussian training surface provided in an embodiment of the present application.

[0087] Figure 18 Schematic diagram of the application effect of different target loss values ​​in the embodiment of the present application.

[0088] Figure 19 This is a schematic diagram of the overall process of the foreground object segmentation and reconstruction method provided in an embodiment of the present application.

[0089] Figure 20 This is a structural block diagram of a foreground object segmentation and reconstruction device provided by another embodiment of the present application.

[0090] Figure 21 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0092] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0094] First, let’s analyze some of the terms used in this application:

[0095] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0096] In today's era where the digital world and reality intersect, the demand for photorealistic dynamic scene representation and rendering is increasing. In this process, to ensure that texture details are accurately presented and the scene's realism is fully expressed, high-quality modeling of objects with complex motion in dynamic scenes is required.

[0097] In related technologies, when segmenting and reconstructing 3D objects in complex dynamic environments, there are two approaches: unified modeling of foreground and background, and separation of foreground and background within the scene. In the first approach, semantic information is integrated into the network model, enabling fine-grained scene understanding by introducing semantic modeling or feature encoding to support visual features that describe semantics. However, further separation of the foreground and background is required. This separation requires the input of large, manually labeled datasets or 3D point clouds acquired through precise scanning to generate relatively accurate foreground segmentation masks. However, labeling such datasets is costly and limited in scope, resulting in less accurate reconstruction. In the second approach, due to the inherent characteristics of dynamic scenes, such as flexible object movements, diverse appearances, complex environments, and prone to occlusion, there is inherent ambiguity when converting 2D image sequences to 3D space. Furthermore, the complex, large-scale, non-rigid deformations of the human body can also lead to ambiguity in the reconstruction results. Therefore, the segmentation and reconstruction process requires the introduction of a significant amount of prior knowledge to assist in the reconstruction process, complicating the entire process.

[0098] Based on this, embodiments of the present application provide a foreground object segmentation and reconstruction method, apparatus, device, and storage medium. These methods directly input video frames into an object segmentation model to generate foreground object segmentation masks, eliminating the need for pre-setting excessive prior knowledge regarding the specific morphology and position of the objects. Furthermore, a spatiotemporal Gaussian splatter model is used to time-slice the input data based on its temporal characteristics, eliminating the need for prior knowledge of the specific change patterns of each object at different moments. This further reduces reliance on external prior information while ensuring temporal stability. After time slicing, the spatiotemporal Gaussian splatter model can track the dynamic changes of complex geometric shapes across different time slices. During the rendered image generation phase, the model, combined with a two-dimensional Gaussian surface, accurately and adaptively fits the object's shape and surface details, thereby improving reconstruction accuracy. Furthermore, the spatiotemporal Gaussian splatter model analyzes the object's motion trajectory and morphological changes within the time slices from different viewpoints. Leveraging the temporal coherence and spatial correlation of the data, the model adjusts for any inconsistencies that may arise between different viewpoints, ensuring that the resulting rendered images from each viewpoint are more consistent in key elements such as the object's position and shape.

[0099] The embodiments of the present application provide a foreground object segmentation and reconstruction method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the foreground object segmentation and reconstruction method in the embodiments of the present application is described.

[0100] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0101] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0102] The foreground target segmentation and reconstruction method provided in the embodiment of the present application relates to the field of computer vision technology. The foreground target segmentation and reconstruction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports foreground target segmentation and reconstruction, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server through a network. The foreground target segmentation and reconstruction method can be executed by the terminal or the server, or by the terminal and the server in collaboration.

[0103] In some embodiments, the terminal may be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the foreground target segmentation and reconstruction method of this embodiment to provide related services to enhance the driving experience. The server may be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; or a service node in a blockchain system, wherein each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The terminal and the server may be connected via Bluetooth, Universal Serial Bus (USB), or a network, etc., which is not limited in this embodiment.

[0104] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0105] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0106] The following describes the foreground object segmentation and reconstruction method in the embodiment of the present application.

[0107] Figure 1 This is an optional flowchart of the foreground object segmentation and reconstruction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It is also understood that this embodiment is Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0108] Step 110: At multiple acquisition moments, respectively obtain initial acquisition data corresponding to multiple perspectives.

[0109] In one embodiment, for the target scene, the image acquisition devices can be deployed in a 360-degree panoramic surround layout to achieve synchronous acquisition. This surround layout can cover the target scene in all directions, ensuring that no angle information is missed, so that data of the target scene can be acquired from all directions. Each image acquisition device corresponds to a specific viewing angle, similar to observing the same object from different positions, and the pictures seen at different positions are different. At each viewing angle, the image acquisition device is used to continuously shoot and acquire the corresponding video sequence, and the video sequence contains multiple video frames recorded at different acquisition times. For example, the image acquisition device can use a ZCAME2 camera, the number of viewing angles is set to 60, and the resolution of the video sequence is 1920×1080. If 10 acquisition moments are set, then all 60 cameras can capture a video sequence containing 10 video frames.

[0110] In one embodiment, referring to Figure 2 , Figure 2 Schematic diagram of a video frame provided in an embodiment of the present application. Figure 2Assume there are eight viewpoints, each capturing a video sequence. Each video sequence contains a consistent number of video frames, and the same capture time has corresponding video frames at different viewpoints. In the figure, t1, t2, …, tn represent the n capture times.

[0111] In one embodiment, in addition to capturing video sequences, the image acquisition device can also acquire sparse point cloud data. In this case, all data acquired by the image acquisition device is collectively referred to as initial acquisition data. For a single viewpoint, the initial acquisition data consists of the video sequence and sparse point cloud data captured by the corresponding image acquisition device. The initial acquisition data includes the acquisition data corresponding to each acquisition moment, and the acquisition data consists of the video frame and the initial Gaussian point cloud at the corresponding acquisition moment.

[0112] Next, the camera system corresponding to the image acquisition device is calibrated using COLMAP to obtain the camera's intrinsic and extrinsic parameters. Intrinsic parameters primarily encompass the camera's focal length, principal point position, and lens distortion parameters, reflecting the camera's internal imaging characteristics. Extrinsic parameters, such as the camera's rotation angle and translation vector, relate to the camera's position and posture in space and are used to determine the camera's spatial relationship with the captured scene. After COLMAP calibration, the camera's intrinsic and extrinsic parameters can be used to calibrate the sparse point cloud data, thereby obtaining the initial Gaussian point cloud corresponding to each acquisition moment. Specifically, at each acquisition moment, based on the core information provided by the camera's intrinsic and extrinsic parameters, such as imaging characteristics and spatial position relationships, the sparse point cloud data is transformed and adjusted accordingly to ensure that it matches the camera's viewing angle and imaging characteristics, ultimately obtaining the initial Gaussian point cloud corresponding to each acquisition moment.

[0113] Step 120: All video frames are input into the target segmentation model for target selection, a foreground target segmentation mask corresponding to each video frame is generated, and the foreground target segmentation mask and the corresponding initial Gaussian point cloud are associated as input data.

[0114] In one embodiment, firstly, a video frame of each viewpoint is obtained, and all the video frames are input into the target segmentation model for target selection, and a foreground target segmentation mask corresponding to each video frame is generated. Figure 3 , Figure 3 This is a flowchart provided by an embodiment of the present application for inputting all video frames into a target segmentation model for target selection, and generating a foreground target segmentation mask corresponding to each video frame, which specifically includes the following steps:

[0115] Step 310: Obtain prompt texts corresponding to all video frames.

[0116] In one embodiment, the prompt text is used to explicitly inform the target segmentation model what kind of target features need to be segmented. Since the purpose of the embodiment of the present application is to segment dynamic targets, it is necessary to ensure the consistency of the extraction of foreground targets. Since foreground targets often have relatively uniform feature expressions under different viewing angles, for example, when shooting an activity scene containing a human subject, no matter from which viewing angle you observe, the human as a foreground target, its general outline, shape and other key features are similar, so for all video frames, this embodiment uses the same prompt text to perform foreground target segmentation. In other words, when facing the video frames collected from various viewing angles, there is no need to set different prompt texts for each viewing angle or each video frame. The extraction of targets in all video frames can be completed by relying on the same prompt text. It is understandable that the prompt text is also consistent at other processing moments.

[0117] Step 320: Input the video frame and the prompt text into the target segmentation model to select the target, obtain at least one foreground target, superimpose the at least one foreground target, and obtain a foreground target segmentation mask corresponding to the video frame.

[0118] In one embodiment, the prompt text can be entered in batches, with each time a target is extracted for segmentation, thereby obtaining the corresponding foreground target. All foreground targets are then superimposed, and after the superposition is complete, an edge dilation operation is performed to obtain the final overall target. This overall target is then binarized to generate a foreground target segmentation mask for each video frame. Alternatively, the prompt text can be entered all at once, directly extracting all foreground targets as the overall target.

[0119] Reference Figure 4 , Figure 4 This is a schematic diagram of the foreground object mask generation process provided in an embodiment of the present application. In this example, the prompt text can be set to "person," "knife," and "beef," respectively. After three extraction operations, the corresponding foreground objects are obtained. After superposition and dilation, the overall object is obtained, and the foreground object segmentation mask is generated. Alternatively, "person + knife + beef" can be used as the prompt text to directly generate an overall object containing these three objects.

[0120] In one embodiment, the target segmentation model uses the GroundedSAM2 model, which can accurately process various video frames based on given relevant prompt information. The model can be used to generate a foreground target segmentation mask of interest for each video frame. In the embodiment of the present application, the corresponding video sequence can be directly input into the target segmentation model according to different perspectives to obtain the foreground target segmentation mask of the video frame. For example, if the video sequence contains multiple pedestrians and vehicles, and the prompt text is set to: "Extract pedestrians and vehicles in the picture", then for each video frame, the GroundedSAM2 model will identify the foreground targets that meet the requirements in the video frame based on the input prompt text, that is, the pedestrian and vehicle parts, and then obtain the foreground target segmentation mask corresponding to each video frame.

[0121] In one embodiment, through the above process, each video frame corresponds to a foreground object segmentation mask. Considering that the video frame also contains the corresponding initial Gaussian point cloud, it is necessary to associate the foreground object segmentation mask with the corresponding initial Gaussian point cloud and use it as input data. Figure 5 , Figure 5 This is a schematic diagram of the input data provided by the embodiment of the present application. Figure 4 For the scene, for each acquisition moment, there is corresponding input data from view 1 to view 8, and each input data includes the corresponding foreground target segmentation mask and initial Gaussian point cloud.

[0122] Step 130: Take each acquisition moment as a processing moment. For each processing moment, input the input data corresponding to all viewpoints into the spatiotemporal Gaussian splatter model for data processing. Divide the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice, obtain a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain a rendered image corresponding to each viewpoint based on multiple two-dimensional Gaussian surfaces.

[0123] In one embodiment, the segmentation and reconstruction process needs to be performed for each acquisition moment, so each acquisition moment is sequentially set as a processing moment. The following takes the first processing moment as an example to illustrate the specific process of segmentation and reconstruction. The execution process of the remaining processing moments is the same.

[0124] In one embodiment, at each processing moment, the input data corresponding to all viewing angles are input into the spatiotemporal Gaussian splash model for data processing to obtain a rendered image for each viewing angle at that processing moment. Figure 5 ,When the reference time t1 is used as the processing time, the input data corresponding to all ,viewpoints at the reference time t1 need to be input into the ,spacetime Gaussian splash model.

[0125] The following describes the detailed processing process of the spatiotemporal Gaussian splash model by taking the processing time t as an example.

[0126] In one embodiment, referring to Figure 6 , Figure 6 This is a flowchart of an embodiment of the present application for time slicing input data to obtain a slice Gaussian point cloud corresponding to each time slice, specifically including the following steps:

[0127] Step 610: Obtain a time slice, and obtain an initial slice point cloud based on the time slice and the initial Gaussian point cloud.

[0128] In one embodiment, the spatiotemporal Gaussian splash model is a four-dimensional Gaussian splash model. Compared to the three-dimensional Gaussian splash model, the four-dimensional Gaussian splash model considers the influence of time. Therefore, the embodiment of the present application uses the four-dimensional Gaussian splash model to simultaneously consider time and space information. Specifically, for each three-dimensional initial Gaussian point cloud corresponding to the processing time t, time information is added therein. Here, the time information is represented as a time slice μ t , μ t is a very small time frame, used to assume that all movements can be approximated as linear movements within a small time frame.

[0129] The spatiotemporal Gaussian splash model adds time information to the three-dimensional Gaussian model to transform it into a four-dimensional Gaussian model. Assuming that the three-dimensional center position of the initial Gaussian point cloud is μ 3D =(μ x ,μ y ,μ z ) T , where the three-dimensional center position represents the mean vector corresponding to the three-dimensional coordinates of each point in the initial Gaussian point cloud. At this time, adding a time slice to the three-dimensional center position can obtain the four-dimensional center position μ of the corresponding four-dimensional Gaussian function 4D =(μ x ,μ y ,μ z ,μ t ), in constructing the four-dimensional covariance matrix Σ 4D After that, the four-dimensional Gaussian function G can be expressed as 4D (x):

[0130]

[0131] Among them, x represents the input Gaussian point, S 4D represents the four-dimensional scaling matrix, R 4D Represents the four-dimensional rotation matrix, U represents the first decomposition matrix, V represents the second decomposition matrix, and W represents the world-to-camera camera transformation matrix. The first decomposition matrix, the second decomposition matrix, the world-to-camera camera transformation matrix, and the time slice can all be adjusted during training.

[0132] For the processing time t, an initial slice point cloud is obtained based on the time slice and the initial Gaussian point cloud, wherein the initial slice point cloud is a three-dimensional Gaussian function.

[0133] Step 620: Use the foreground object segmentation mask to perform a mask operation on each initial slice point cloud to obtain the corresponding slice Gaussian point cloud.

[0134] In one embodiment, the foreground target segmentation mask clearly defines the area range of the foreground target in the video frame. The initial slice point cloud is a representation of the target scene at the corresponding acquisition time and viewing angle in the form of a point cloud, which covers multiple data points in the target scene. Therefore, based on the foreground target area marked by the foreground target segmentation mask, the corresponding point cloud data can be screened out in the initial slice point cloud as a slice Gaussian point cloud, and the range of its Gaussian points is limited according to the foreground target segmentation mask. Specifically, the three-dimensional probability density of the slice Gaussian point cloud can be obtained based on the three-dimensional probability density of the initial slice point cloud, expressed as:

[0135]

[0136] Among them, σ(t) represents the temporal opacity of the sliced ​​Gaussian point cloud corresponding to the processing time t, μ 3D (t) represents the three-dimensional center position that changes with time after the linear motion term is introduced into the three-dimensional center position, Σ 3d Represents the three-dimensional covariance of the sliced ​​Gaussian point cloud. The three-dimensional covariance is used to describe the distribution shape and direction of the sliced ​​Gaussian point cloud in three-dimensional space, which can be obtained based on the initial Gaussian point cloud.

[0137] For example, the indoor scene represented by the initial slice point cloud contains a variety of household items. If the foreground target segmentation mask marks the tables and chairs as the foreground target area, then with the help of the mask operation, the point cloud data related to the tables and chairs can be extracted from the initial slice point cloud, while the point cloud data belonging to the background part (such as walls, floors, etc.) is eliminated. After such masking operations, the corresponding slice Gaussian point cloud can be obtained. Compared with the initial slice point cloud, the slice Gaussian point cloud focuses more on the data related to the foreground target, and it more accurately presents the key characteristics of the foreground target in space, such as the position and shape, in the form of Gaussian distribution.

[0138] Step 630: Determine the corresponding depth value according to the relationship between the center position of the sliced ​​Gaussian point cloud and the camera position of the viewing angle, and sort the sliced ​​Gaussian point cloud based on the order of the depth values.

[0139] In one embodiment, the relationship between the center position of the sliced ​​Gaussian point cloud and the camera position corresponding to the viewing angle corresponds to a depth value, which can reflect the depth of the foreground target at that viewing angle. Specifically, the depth value is determined by the straight-line distance between the center position and the camera position that captures the viewing angle. The smaller the straight-line distance, the smaller the depth value. A small depth value means that the foreground target is relatively closer to the camera at that viewing angle, and vice versa. Next, based on the order of the depth values ​​determined in this way, the sliced ​​Gaussian point cloud is sorted, thereby sorting out the front and back hierarchical relationships of each foreground target in space in an orderly manner. Figure 7 , Figure 7 Schematic diagram of a sliced ​​Gaussian point cloud provided in an embodiment of the present application. The figure is illustrated with processing time t, where T i =t-μ t , T i+1 =t+1-μ t , T i-1 =t-1-μ t , and then sort the sliced ​​Gaussian point clouds of different perspectives according to the depth value.

[0140] Next, since the sliced ​​Gaussian point cloud is a three-dimensional Gaussian ellipsoid, each sliced ​​Gaussian point cloud can be further represented as a two-dimensional Gaussian surface by controlling the size of the z-scale direction of the rotation matrix of the three-dimensional Gaussian ellipsoid corresponding to the processing time t. Figure 8 , Figure 8 This is a flowchart of obtaining the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud provided by an embodiment of the present application, which specifically includes the following steps:

[0141] Step 810: Obtain the center position, color parameters, opacity parameters, initial rotation matrix, and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud.

[0142] In one embodiment, since there are multiple slice Gaussian point clouds corresponding to the processing time t, the center position corresponding to the k-th slice Gaussian point cloud after sorting is obtained. Color parameters Opacity parameter Initial rotation matrix and the initial scaling matrix

[0143] In one embodiment, it is assumed that the distribution of illumination in a target scene can be approximated using spherical harmonics. Spherical harmonics, as a function form widely used in fields such as mathematics and physics, can effectively mathematically describe physical quantities in space that have certain distribution patterns, such as illumination. For example, in an indoor target scene, the intensity, direction, and spatial distribution of light incident from different directions can be reasonably simulated and approximated using spherical harmonics. Therefore, the present embodiment calculates color parameters based on spherical harmonics. For a two-dimensional Gaussian surface, its color calculation is typically related to illumination. The illumination information can be parameterized using spherical harmonics, which are set according to the actual situation. A two-dimensional Gaussian surface is a data structure that carries scene-related information, and its color rendering is largely affected by illumination. Different illumination conditions will cause the surface to exhibit different color effects. Based on this assumption, spherical harmonics can be used to parameterize illumination information. This means that the complex distribution characteristics of illumination in a scene are converted into corresponding parameter values ​​in the spherical harmonics, such as the spherical harmonic coefficients representing illumination or the spherical harmonic coefficients representing reflectivity. Subsequently, an integral calculation is performed on the color of the pixel points on the two-dimensional Gaussian surface according to the spherical harmonic function until the color parameters are determined. The specific parameters of the spherical harmonic function can be trained during the training process.

[0144] In addition, the position, direction, and shape of the two-dimensional Gaussian surface in space (determined by parameters such as its center position and covariance matrix) will affect the reception of light. For example, if the normal vector of the two-dimensional Gaussian surface (which can be obtained by parametrically derivatizing the Gaussian surface) is at a different angle to the direction of illumination, the intensity of the light received will also be different. Therefore, the color parameters calculated above can also be adjusted based on the geometric information of the two-dimensional Gaussian surface. For example, for diffuse reflection, the color intensity is scaled according to the cosine value of the angle between the normal vector and the illumination direction.

[0145] Step 820: Select a specific coordinate axis, and based on the specific coordinate axis, adjust the scale information of the initial scaling matrix to obtain a target scaling matrix.

[0146] In one embodiment, the specific coordinate axis is the z-axis, so based on the z-axis, the z direction in the initial scaling matrix is ​​changed to zero to obtain the target scaling matrix

[0147] Step 830: At least the center position, the color parameter, the opacity parameter, the rotation matrix, and the target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

[0148] In one embodiment, the parameters of the k-th two-dimensional Gaussian surface at time t are processed Expressed as:

[0149]

[0150] According to the above process, the two-dimensional Gaussian surface corresponding to each three-dimensional slice Gaussian point cloud is obtained. Combining the flexible optimization process of the three-dimensional Gaussian and the surface alignment characteristics of the two-dimensional Gaussian surface, the quality of the reconstructed geometry is significantly improved.

[0151] In one embodiment, a two-dimensional Gaussian surface is a surface structure represented by a specific mathematical function. It can describe characteristics such as the distribution of data points in a foreground object within a two-dimensional space. Pixels on this surface carry information about the corresponding data points. Therefore, after obtaining a two-dimensional Gaussian surface, a rendered image corresponding to each viewing angle is generated based on multiple two-dimensional Gaussian surfaces.

[0152] The specific process is as follows: for each pixel on the 2D Gaussian surface, the corresponding pixel rendering color is determined based on multiple 2D Gaussian surfaces. Based on this, the rendered image corresponding to the viewing angle is further obtained based on the pixel rendering color of the 2D Gaussian surface. Because each viewing angle has its own unique observation range and angle, the rendering colors determined by each pixel on the 2D Gaussian surface in the above steps are arranged and combined according to the corresponding spatial position relationship and image construction rules to construct a complete rendered image corresponding to that viewing angle.

[0153] The detailed rendering process is described below.

[0154] In one embodiment, referring to Figure 9 , Figure 9 This is a flowchart of obtaining the corresponding pixel rendering color for each pixel point in a two-dimensional Gaussian surface according to multiple two-dimensional Gaussian surfaces provided by an embodiment of the present application, specifically including the following steps:

[0155] Step 910: Obtain the mixing weight of the two-dimensional Gaussian surface.

[0156] In one embodiment, referring to Figure 10 , Figure 10 This is a flowchart of obtaining the mixing weight of a two-dimensional Gaussian surface provided by an embodiment of the present application, which specifically includes the following steps:

[0157] Step 1010: Obtain the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud.

[0158] In one embodiment, the three-dimensional covariance is Σ 3D , the three-dimensional probability density is:

[0159]

[0160] Step 1020: Obtain the camera transformation matrix and local affine transformation matrix corresponding to the viewing angle.

[0161] Step 1030: Calculate a two-dimensional covariance based on the camera transformation matrix, the local affine transformation matrix, and the three-dimensional covariance, and perform a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density.

[0162] In one embodiment, to render an image, the world-to-camera transformation matrix W is first used to transform the two-dimensional Gaussian surface from the world coordinate system to the camera coordinate system, and then project it onto the image plane using the local affine transformation matrix J. This process will yield a two-dimensional covariance. Next, a two-dimensional projection of the three-dimensional probability density is performed based on the two-dimensional covariance to obtain the two-dimensional probability density G′. 3D (x, t), this projection process can be obtained by using the transformation projection method using the covariance matrix in the related technology.

[0163] The two-dimensional covariance of the k-th two-dimensional Gaussian surface is expressed as:

[0164] Σ′ k =JWΣ 3D W T J T

[0165] Step 1040: Obtain a blending weight according to the two-dimensional probability density and the corresponding opacity parameter.

[0166] In one embodiment, it is assumed that the pixel point on the image plane is u, u k represents the center position of the k-th two-dimensional Gaussian surface, so the mixing weight of the k-th two-dimensional Gaussian surface is expressed as:

[0167] α k =G′(u;u k ,Σ′ k )σ k

[0168] Among them, α k represents the mixing weight, σ k Represents the opacity parameter of the k-th two-dimensional Gaussian surface, that is, the opacity parameter corresponding to the processing time t It is used to measure the "visibility" of this two-dimensional Gaussian surface in the final rendering result or the weight of its influence on the rendered image, where the opacity parameter is a weighted parameter calculated based on the opacity value of the pixel point. It can be understood that since the two-dimensional Gaussian surface is a surface with distribution characteristics, it presents a "hill"-like distribution form in space, composed of a large number of pixels. These pixels carry different information and together depict the overall characteristics of the surface. Therefore, the mixing weight can be in matrix form, and for each pixel on the two-dimensional Gaussian surface, there is a corresponding mixing weight value.

[0169] In addition, the coordinates of the pixel point u and the center position u of the k-th two-dimensional Gaussian surface are k and the two-dimensional covariance Σ′ k Substituting this two-dimensional probability density into the equation yields the corresponding probability density value. Therefore, the blending weight is the product of the probability density value of the two-dimensional Gaussian surface at the pixel point and the opacity of the two-dimensional Gaussian surface. This is used to determine the weight proportion that the k-th two-dimensional Gaussian surface should possess when multiple two-dimensional Gaussian surfaces are involved.

[0170] Step 920: Obtain the marginal probability density and conditional probability density corresponding to the pixel point, and obtain a weighted contribution value according to the product of the marginal probability density, the conditional probability density and the mixing weight.

[0171] In one embodiment, for a pixel u, its marginal probability density is p k (t), is the marginal probability density of time, belongs to the three-dimensional Gaussian distribution, and describes the probability distribution of the k-th two-dimensional Gaussian surface at time t. The conditional probability density is expressed as p k (u|t) is a one-dimensional Gaussian distribution that describes the probability distribution of the k-th two-dimensional Gaussian surface at pixel u at a given time t.

[0172] In one embodiment, the weighted contribution value is expressed as:

[0173] p k (t)p k (u|t)*α k

[0174] Step 930: Based on the order of the depth values, obtain the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtain the cumulative weight value according to the weighted contribution values.

[0175] In one embodiment, for the k-th two-dimensional Gaussian surface, the previous two-dimensional Gaussian surface is the two-dimensional Gaussian surface j, and the value of j ranges from 1 to k-1. In this case, the weighted contribution value of the two-dimensional Gaussian surface j is expressed as:

[0176] p j (t)p j (u|t)α j

[0177] The cumulative weight value is expressed as:

[0178]

[0179] The above accumulated weight values ​​are used to adjust the contribution of each two-dimensional Gaussian surface to the final color, ensuring that the sum of the contributions of all two-dimensional Gaussian surfaces is 1.

[0180] Step 940: Obtain the pixel rendering color according to the product of the reference contribution value, the color parameter and the accumulated weight value.

[0181] In one embodiment, the reference contribution value, the color parameter, and the accumulated weight value are multiplied and then accumulated to obtain the pixel rendering color. The pixel rendering color is expressed as:

[0182]

[0183] Among them, c(u,t) represents the pixel rendering color corresponding to the pixel u at the processing time t, N represents the number of two-dimensional Gaussian surfaces, and c k Represents the color parameters of the k-th two-dimensional Gaussian surface.

[0184] In one embodiment, after obtaining the pixel rendering colors of a two-dimensional Gaussian surface according to the above process, a rendered image corresponding to each viewpoint can be generated based on these pixel rendering colors. Specifically, these pixel rendering colors are like tiny "color units" that together construct the subsequent rendered image corresponding to the corresponding viewpoint. For each viewpoint, there is a specific observation range and angle. The pixel rendering colors determined on the two-dimensional Gaussian surface need to be reasonably arranged and integrated according to their position in space and the spatial relationship corresponding to the viewpoint.

[0185] For example, if the target scene is an image of an indoor scene containing multiple chairs and tables, different objects in the two-dimensional Gaussian surface corresponding to a certain viewpoint will have different pixel rendering colors. For example, the pixel rendering color corresponding to the table may be a woody brown hue, while the pixel rendering color corresponding to the chair may be a matching color. When generating the rendered image, the pixel rendering colors corresponding to these different objects can be orderly combined according to their spatial layout relationship at that viewpoint, ultimately constructing a complete rendered image corresponding to that viewpoint.

[0186] In one embodiment, referring to Figure 11 , Figure 11 Schematic diagram of the rendered image provided by the embodiment of the present application. Figure 11 It can be seen from the figure that the foreground object segmentation and reconstruction method provided in the embodiment of the present application can completely extract the foreground object and perform a suitable rendering process on it to obtain the corresponding rendered image.

[0187] Step 140: Obtain a foreground object rendering sequence at the processing moment according to the rendered image corresponding to each viewing angle, and generate a foreground object reconstruction result according to the foreground object rendering sequences at all acquisition moments.

[0188] In one embodiment, the above process generates corresponding rendered images for each video frame from each perspective at the processing moment. Thus, for each acquisition moment, it contains a foreground target rendering sequence consisting of rendered images from different perspectives. In this case, after obtaining the foreground target rendering sequence at each acquisition moment, the foreground target reconstruction result can be generated. Furthermore, the sliced ​​Gaussian point cloud obtained based on the foreground target segmentation mask can be associated with the corresponding rendered image. When displayed, the corresponding three-dimensional Gaussian points are displayed according to the color of each pixel in the rendered image, and the foreground target reconstruction result is displayed as a three-dimensional Gaussian point.

[0189] It is understood that after obtaining the foreground object reconstruction results, the display operation can be performed according to actual needs. For example, the display can be based on different acquisition times, showing the rendered images corresponding to different perspectives at a certain acquisition time; or the display can be based on different perspectives, showing the rendered images corresponding to a certain perspective at different acquisition times.

[0190] In a specific embodiment, referring to Figure 12 , Figure 12 This is a schematic diagram of the foreground target reconstruction result provided by the embodiment of this application. Figure 12 In the figure, the rendered images of a certain perspective of the foreground target at different acquisition times can be selected according to actual needs. It can be seen that each rendered image is a foreground target rendered based on the foreground target segmentation mask.

[0191] The following describes the training process of the spatiotemporal Gaussian splash model in an embodiment of the present application.

[0192] In one embodiment, referring to Figure 13 , Figure 13 Flowchart of the training process of the spatiotemporal Gaussian splatter model provided in the embodiment of the present application, including at least the following steps:

[0193] Step 1310: Obtain multiple training samples.

[0194] In one embodiment, the training samples include at least a training mask, a training Gaussian point cloud, and a label image for each viewpoint corresponding to each training acquisition moment. The training mask is obtained in the same manner as the foreground object segmentation mask, the training Gaussian point cloud is obtained in the same manner as the initial Gaussian point cloud, and the label image corresponds to the rendered image.

[0195] Step 1320: Input the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering processing to obtain a training rendered image.

[0196] In one embodiment, the rendering process using the spatiotemporal Gaussian splatter model is similar to the inference process described above and will not be further elaborated here. The calculation process for training rendered images is identical to that for rendering images. Furthermore, during the rendering process, each training rendered image corresponds to a two-dimensional Gaussian training surface, and the two-dimensional Gaussian training surface and the two-dimensional Gaussian surface are calculated in the same manner.

[0197] Step 1330: During the rendering process, generate a depth consistency loss value based on the blending weight and the training depth value of each two-dimensional Gaussian training surface.

[0198] In one embodiment, in the process of obtaining the two-dimensional Gaussian training surface, the scale of the scaling matrix of the sliced ​​Gaussian point cloud in the Z-axis direction is directly set to 0. After completing this operation, when calculating the derivative related to the local Z-axis from the covariance matrix, the result will be zero due to the change in the scale. In the subsequent optimization process, the photometric loss itself loses the ability to exert an influence on the local Z-axis. Photometric loss generally plays an important role in measuring the differences in photometric features of images or data. When it cannot affect the local Z-axis, it may cause the characteristics related to the local Z-axis to be unable to be reasonably corrected and improved by relying on photometric loss during the optimization and adjustment of the data, thereby affecting the overall data accuracy and model training effect.

[0199] To this end, the present invention employs a self-supervised depth consistency loss that requires the local Z axis to approximate the normal calculated from the depth map generated by Gaussian point rendering. The normal in the depth map reflects the directional characteristics of the object's surface at different locations, and requiring the local Z axis to approximate this normal actually constrains and guides the local Z axis using the spatial directional information contained in the depth map's normal.

[0200] In one embodiment, referring to Figure 14 , Figure 14 This is a flowchart of generating a depth consistency loss value based on the mixing weight and training depth value of each two-dimensional Gaussian training surface provided by an embodiment of the present application, specifically including the following steps:

[0201] Step 1410: For the two-dimensional Gaussian training surface, based on the order of the training depth values, for all two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all previous two-dimensional Gaussian surfaces, obtain a first intermediate value based on the difference between one and the training mixing weight, accumulate the first intermediate value and multiply it by the training mixing weight of the two-dimensional Gaussian training surface to obtain the total training mixing weight, select the maximum value from the total training mixing weight to obtain the maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value.

[0202] In one embodiment, each two-dimensional Gaussian training surface is obtained from a three-dimensional Gaussian point cloud, so the training depth value corresponding to each three-dimensional Gaussian point cloud is calculated according to the above depth value calculation process, and reference is made to Figure 7 As shown in the figure, the two-dimensional Gaussian training surfaces are sorted according to the order of the training depth values. For the i-th two-dimensional Gaussian training surface, the training mixture weights of all previous two-dimensional Gaussian surfaces are expressed as: α j ′, where j ranges from 1 to i-1.

[0203] For the j-th two-dimensional Gaussian surface, the first intermediate value is obtained according to the difference between one and the training mixture weight. The first intermediate value is expressed as:

[0204] (1-α j ′)

[0205] Then, the first intermediate value is accumulated and multiplied by the training mixture weight of the two-dimensional Gaussian training surface to obtain the total training mixture weight, the total training mixture weight ω j Expressed as:

[0206]

[0207] Among them, m represents the m-th two-dimensional Gaussian surface corresponding to the j-th previous two-dimensional Gaussian surface, α m Represents the training mixture weights of the m-th two-dimensional Gaussian surface.

[0208] Next, from training the total mixture weight ω j Select the maximum value from the equation to get the maximum mixing weight. Assuming that the maximum value is the j′th two-dimensional Gaussian surface, the maximum mixing weight ω j′ , expressed as:

[0209] j′=argmax(ω j′ )

[0210] Then obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value, and the maximum training depth value is expressed as z j′ .

[0211] Step 1420: Obtain the training weighted contribution value and the training depth value of the two-dimensional Gaussian training surface, calculate a first product of the training weighted contribution value and the training depth value, and calculate a second product of the training weighted contribution value and the maximum training depth value.

[0212] In one embodiment, the first product is expressed as:

[0213] ω i z i

[0214] The second product is expressed as:

[0215] ω i z j′

[0216] Step 1430: Accumulate the first product to obtain the second intermediate value, and accumulate the second product to obtain the third intermediate value, obtain the difference between the second intermediate value and the third intermediate value, and calculate the product of the maximum mixing weight and the difference to obtain the depth consistency loss value.

[0217] In one embodiment, the second intermediate value is expressed as:

[0218] ∑ i ω i z i

[0219] The third intermediate value is expressed as:

[0220] ∑ i ω i z j′

[0221] Depth consistency loss value Expressed as:

[0222]

[0223] Step 1340: Calculate the monocular normal regularization loss value based on the normal vectors corresponding to the training rendered image and the label image respectively.

[0224] In one embodiment, a pseudo-normal map estimated from the depth gradient of the rendered image can be used to supervise the training process. However, since there will be noise in the depth map obtained from the rendered image, especially in complex scenes, the scene contains many different types of objects, complex lighting conditions, and diverse spatial structures. These factors will make the noise in the rendered depth map more obvious. The presence of this noise will cause deviations in the pseudo-normal map estimated based on the depth gradient, and then produce artifacts in subsequent applications. The appearance of artifacts will destroy the quality of the final generated image, causing some abnormal parts in the image that are inconsistent with the actual situation and visually abrupt. Therefore, the embodiment of the present application uses a monocular normal regularization loss value calculated from the normal vectors corresponding to the existing training rendered image and the label image to perform normal constraints.

[0225] In one embodiment, referring to Figure 15 , Figure 15 This is a flowchart of calculating the monocular normal regularization loss value based on the normal vectors corresponding to the training rendered image and the label image provided by the embodiment of the present application, which specifically includes the following steps:

[0226] Step 1510: Obtain the corresponding training slice Gaussian point cloud during the rendering process, and obtain the monocular estimated normal vector corresponding to the training slice Gaussian point cloud.

[0227] In one embodiment, during the rendering process, each training rendered image has a corresponding training slice Gaussian point cloud, and a monocular estimated normal vector corresponding to the training slice Gaussian point cloud is obtained.

[0228] Step 1520: Generate a pseudo normal vector based on the training rendered image.

[0229] In one embodiment, a depth map corresponding to the training rendered image is obtained. The depth map reflects the depth position of the scene objects corresponding to each pixel point in the training rendered image in space, that is, the distance between each pixel point and the observation point of the corresponding perspective (such as the location of the camera).

[0230] Step 1530: Calculate the monocular normal regularization loss value based on the monocular estimated normal vector and the pseudo normal vector.

[0231] In one embodiment, the monocular normal regularization loss value is the L1 loss value, expressed as:

[0232]

[0233] in, represents the monocular estimated normal vector, and N represents the pseudo normal vector.

[0234] Step 1350: For two consecutive training acquisition moments, a time domain regularization loss value is calculated based on the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing loss value is calculated based on the corresponding position parameters and rotation parameters of the two-dimensional Gaussian training surface.

[0235] In one embodiment, in order to reduce temporal jitter while maintaining the quality of geometric surface rendering, temporal smoothing regularization is introduced to finely balance the updates of multi-view geometric priors and Gaussian properties, thereby maintaining spatiotemporal consistency. Figure 16 , Figure 16 This is a flowchart of calculating the time domain regularization loss value based on the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments provided by an embodiment of the present application, specifically including the following steps:

[0236] Step 1610: At the current training acquisition moment, obtain a preset number of adjacent Gaussian points corresponding to the central Gaussian point where the center position of the two-dimensional Gaussian training surface is located, obtain the current color parameter, current opacity parameter and current target scaling value of the central Gaussian point from the surface parameters, and obtain the total training mixture weight of the two-dimensional Gaussian training surface.

[0237] In one embodiment, with respect to one of the two-dimensional Gaussian training surfaces, it presents a specific distribution pattern in space, and the central Gaussian point at the corresponding central position is usually in a key position, and the Gaussian points adjacent to it together constitute the "associated area" around it. Based on this, the embodiment of the present application determines a preset number, such as 5, 10, etc., and selects some Gaussian points that are close to the central Gaussian point from the entire two-dimensional Gaussian training surface as adjacent Gaussian points based on a certain distance metric or spatial distribution rule. These adjacent Gaussian points help to analyze the characteristics of the local area where the central Gaussian point is located and its relationship with the surrounding environment.

[0238] In one embodiment, assuming that the current training acquisition time is t, the current color parameter c of the central Gaussian point is obtained from the surface parameters of the corresponding two-dimensional Gaussian training surface. i,t , current opacity parameter σ i,t and the current target scaling value r i,t , where the target scaling value is the target scaling matrix of the i-th two-dimensional Gaussian training surface at the training acquisition time t. At the same time, the total training mixture weight ω of the two-dimensional Gaussian training surface is obtained i,t .

[0239] Step 1620: Obtain reference color parameters, reference opacity parameters, and reference target scaling values ​​of adjacent Gaussian points from the surface parameters of the corresponding viewing angle at the previous training acquisition moment.

[0240] In one embodiment, the previous training acquisition time is t-1, at which time the reference color parameter c near the Gaussian point is obtained. i,t-1 , reference opacity parameter σ i,t-1 and the reference target scaling value r i,t-1 .

[0241] Step 1630: Calculate the norm of the color difference between the current color parameter and the reference color parameter, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter, and the norm of the scaling difference between the current target scaling value and the reference target scaling value.

[0242] In one embodiment, the norm of the color difference between the current color parameter and the reference color parameter is: k = c, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter is k=σ, the norm of the scaling difference between the current target scaling value and the reference target scaling value is k=r.

[0243] Step 1640: Multiply the total weight of the training mixture by the norm of the color difference, the norm of the opacity difference, and the norm of the scaling difference to obtain three products, accumulate the three products corresponding to each adjacent Gaussian point, and obtain the time domain regularization loss value.

[0244] In one embodiment, the time domain regularization loss value Expressed as:

[0245]

[0246] It can be understood that since the time domain regularization loss value contains color parameters, the specific parameters of the spherical harmonic function can be adjusted through the loss value.

[0247] In one embodiment, to address unnatural artifacts caused by drastic changes in object motion, a motion smoothing term is introduced for fine-grained Gaussian motion properties. The motion smoothing term generates as rigid a local deformation as possible in the space-time domain to constrain consistent four-dimensional Gaussian motion. Figure 17 , Figure 17 This is a flowchart of calculating the motion smoothness loss value based on the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface provided in an embodiment of the present application, which specifically includes the following steps:

[0248] Step 1710: At the current training acquisition moment, obtain the current position parameters and current rotation parameters of the central Gaussian point from the surface parameters, and obtain the current reference position parameters of the adjacent Gaussian points.

[0249] In one embodiment, as described above, at the training acquisition time t, the coordinates of the center position of the central Gaussian point are obtained as the current position parameter p i,t , and obtain the current rotation parameter q according to the rotation matrix of the two-dimensional Gaussian training surface i,t And, in this way, the current reference position parameter p of the adjacent Gaussian point is obtained. j,t .

[0250] Step 1720: Obtain reference position parameters and reference rotation parameters of adjacent Gaussian points, as well as central reference position parameters corresponding to the central Gaussian point, from the surface parameters of the corresponding viewing angle at the previous training acquisition moment.

[0251] In one embodiment, for the previous training acquisition time t-1, the reference position parameter p of the adjacent Gaussian point is obtained. j,t-1 and the reference rotation parameter q j,t-1 , and the central reference position parameter p corresponding to the central Gaussian point i,t-1 .

[0252] Step 1730: For each adjacent Gaussian point, obtain the fourth intermediate value based on the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, obtain the fifth intermediate value based on the difference between the reference position parameter and the center reference position parameter, and obtain the sixth intermediate value based on the difference between the current reference position parameter and the current position parameter.

[0253] In one embodiment, the fourth intermediate value, the fifth intermediate value, and the sixth intermediate value are respectively expressed as:

[0254]

[0255] p j,t-1 -p i,t-1

[0256] p j,t -p i,t

[0257] Step 1740: Convert the fourth intermediate value into a rotated intermediate matrix, subtract the sixth intermediate value from the product of the fifth intermediate value and the rotated intermediate matrix to obtain a seventh intermediate value, and multiply the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain an eighth intermediate value.

[0258] In one embodiment, the rotation intermediate matrix is ​​obtained according to the SO3 operation, which converts the quaternion into a rotation matrix. Therefore, the seventh intermediate value is expressed as:

[0259]

[0260] The eighth intermediate value is expressed as:

[0261]

[0262] Step 1750: Accumulate the eighth intermediate values ​​corresponding to all adjacent Gaussian points to obtain the motion loss item corresponding to the current two-dimensional Gaussian training surface training, and accumulate all motion loss items to obtain the motion smoothing item loss value at the current training acquisition moment.

[0263] In one embodiment, the motion loss term is expressed as:

[0264]

[0265] Motion smoothness loss value Expressed as:

[0266]

[0267] Step 1360: Obtain pixel loss values ​​and structural loss values ​​based on the training rendered image and the label image, and calculate a photometric difference loss value based on the pixel loss values ​​and the structural loss values.

[0268] In one embodiment, the pixel loss value is calculated based on the mean absolute error, and the structural loss value is calculated based on the D-SSIM calculation method. The D-SSIM calculation method is used to evaluate the degree of structural similarity between two images. It comprehensively considers multiple dimensions such as image brightness, contrast, and structural information to measure the similarity between the images in terms of overall structural features.

[0269] Therefore, the photometric difference loss value Expressed as:

[0270]

[0271] Where I′ represents the training rendered image, I represents the label image, λ1 represents the preset weight, L1(I′,I) represents the pixel loss value, and L DSSIM (I′,I) represents the structural loss value.

[0272] In one embodiment, considering that some regularization terms may introduce background artifacts when only dynamic foreground is modeled, the embodiment of the present application also adds an additional mask loss value to minimize the training mask M during the training process to the input foreground mask label M * middle.

[0273] Specifically, the pixel point in the training mask M is recorded as r, and for the training acquisition time t, the pixel point corresponding to the pixel point r on the i-th two-dimensional Gaussian training surface is recorded as u ir , for the pixel points on the two-dimensional Gaussian training surface, obtain the corresponding mixing weight value α(u ir ), therefore, the weight value of the total training mixture weight corresponding to the pixel point is calculated, and after accumulating the values ​​corresponding to each two-dimensional Gaussian training surface, the training mask parameter value of the pixel point r of the training mask M is obtained.

[0274] Among them, the training mask parameter value M(r) is expressed as:

[0275]

[0276] Among them, u jr Indicates the pixel u on the two-dimensional Gaussian training surface before the i-th two-dimensional Gaussian training surface ir The corresponding pixel point.

[0277] Therefore, the mask loss value Expressed as:

[0278]

[0279] Among them, M * (r) represents the pixel r in the foreground mask label M *The pixel values ​​in .

[0280] It can be understood that the target segmentation model can be jointly trained by introducing the mask loss value.

[0281] Step 1370: Select at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness loss value as the target loss value, and obtain the total loss value based on the target loss value and the photometric difference loss value.

[0282] In one embodiment, the depth consistency loss value is given in the above process Monocular normal regularization loss value Mask loss value Time domain regularization loss value and motion smoothness loss Next, at least one of them is selected as the target loss value, and then the target loss value and the luminosity difference loss value are calculated. Get the total loss value.

[0283] Assuming all of the above are selected as target loss values, the total loss value obtained is Expressed as:

[0284]

[0285] Among them, α, β, δ, η, Indicates the weights corresponding to different target loss values, which can be set according to actual needs.

[0286] In one embodiment, referring to Figure 18 , Figure 18 Schematic diagram of the application effect of different target loss values ​​in the embodiment of the present application. Figure 18 The figure shows the depth consistency loss value selected when "the person playing football and the football" are selected as the foreground target from the input video frame. Monocular normal regularization loss value Mask loss Schematic diagram of the segmentation obtained as a geometric consistency constraint, selecting the time domain regularization loss value and motion smoothness loss A schematic diagram of the segmentation achieved with spatiotemporal consistency constraints, and a schematic diagram of the segmentation achieved with collaborative constraints using all of the above as target loss values, is shown. The training rendered image is binarized and compared to the training mask. As can be seen, whether using geometric consistency constraints, spatiotemporal consistency constraints, or all of the collaborative constraints, good segmentation results are achieved, with the training mask and foreground mask labels being relatively close.

[0287] Step 1380: Adjust the model parameters of the spatiotemporal Gaussian splash model according to the total loss value until the iteration termination condition is reached to obtain a trained spatiotemporal Gaussian splash model.

[0288] In one embodiment, the total loss value encompasses multiple aspects of loss. These different types of losses each reflect the degree of deviation between the model output and the expected result from a corresponding perspective. Subsequently, the model parameters of the spatiotemporal Gaussian splash model are adjusted based on the total loss value. For example, the gradient information corresponding to each parameter is calculated based on the total loss value. The model parameters are then updated in the opposite direction of the gradient, causing the spatiotemporal Gaussian splash model parameters to be adjusted in a direction that reduces the total loss value. This adjustment process continues, and after each adjustment, the total loss value is recalculated to observe whether the model is improving in a more optimal direction. This continues until an iteration termination criterion is met. This criterion can typically be set when the total loss value falls below a pre-set threshold, indicating that the model performance has reached an acceptable accuracy range, or when a certain number of iterations has been reached to prevent overtraining. When the iteration termination criterion is met, it indicates that the spatiotemporal Gaussian splash model has been fully trained and optimized, and the resulting spatiotemporal Gaussian splash model is considered fully trained.

[0289] The foreground target segmentation and reconstruction method provided by the embodiment of the present application realizes accurate and high-quality segmentation and appearance reconstruction of the selected target object in four-dimensional space. It uses a two-dimensional plane Gaussian to characterize the geometric shape of the target object of interest, and optimizes the Gaussian surface with the help of geometric consistency constraints such as depth consistency loss value, monocular normal regularization loss value, mask loss value, time domain regularization loss value and motion smoothness term loss value, thereby obtaining an accurate geometric surface. At the same time, under the input condition of multi-viewpoint synchronized video, fine-grained object segmentation is achieved by relying on the structure, texture and geometric consistency between each view, and the continuity constraints of time and space are used to enhance geometric details and continuity. In addition, a prompt automatic video object segmentation model is introduced, which supports text prompts as input and can carry out selection and segmentation optimization for any category of foreground target objects.

[0290] In one embodiment, referring to Figure 19 , Figure 19 This is a schematic diagram of the overall process of the foreground object segmentation and reconstruction method provided in an embodiment of the present application.

[0291] At multiple acquisition moments, multiple initial acquisition data corresponding to multiple viewpoints are obtained. All video frames are input into the target segmentation model for target selection. The video frames and prompt text are input into the target segmentation model for target selection to obtain at least one foreground target. The at least one foreground target is superimposed to obtain a foreground target segmentation mask corresponding to the video frame. The foreground target segmentation mask is associated with the corresponding initial Gaussian point cloud as input data.

[0292] For each processing moment, the input data corresponding to all viewpoints are input into the spatiotemporal Gaussian splatter model for data processing. The input data is divided into time slices based on the foreground target segmentation mask to obtain the slice Gaussian point cloud corresponding to each time slice. Then, based on the center position, color parameters, opacity parameters, rotation matrix and target scaling matrix as surface parameters, the color parameters and opacity parameters are appearance parameters, and the center position, rotation matrix and target scaling matrix are spatial parameters, the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and the foreground target segmentation mask and the two-dimensional Gaussian surface are combined for rendering to obtain the rendered image corresponding to each viewpoint. Finally, the foreground target rendering sequence of the processing moment is obtained according to the rendered image corresponding to each viewpoint, and the foreground target reconstruction result is generated according to the foreground target rendering sequence of all acquisition moments. The foreground target reconstruction result is illustrated in the figure in the form of three-dimensional Gaussian points.

[0293] In addition, the figure also illustrates the training process, which uses at least the depth consistency loss value, the monocular normal regularization loss value, the mask loss value, and the photometric difference loss value to obtain the total loss value for model training.

[0294] The technical solution provided by the embodiments of the present application obtains initial acquisition data corresponding to multiple viewpoints at multiple acquisition moments, then inputs all video frames into a target segmentation model for target selection. A foreground target segmentation mask is generated for each video frame, and the foreground target segmentation mask is associated with the corresponding initial Gaussian point cloud as input data. At the processing moment, the input data corresponding to all viewpoints is input into a spatiotemporal Gaussian splatter model for data processing. The input data is temporally sliced ​​to obtain a sliced ​​Gaussian point cloud corresponding to each time slice, and a two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained. A rendered image corresponding to each viewpoint is obtained based on the multiple two-dimensional Gaussian surfaces. A foreground target rendering sequence at the processing moment is obtained based on the rendered image corresponding to each viewpoint, and a foreground target reconstruction result is generated based on the foreground target rendering sequence at all acquisition moments. The embodiments of the present application directly input video frames into the target segmentation model to generate foreground target segmentation masks, eliminating the need for pre-setting excessive prior knowledge regarding the specific morphology and position of the target. Furthermore, the spatiotemporal Gaussian splatter model uses the temporal dimensionality of the input data to temporally slice the input data, eliminating the need for prior knowledge such as the specific change patterns of each target at different moments. In this way, while further reducing the dependence on external prior information, the stability of the time series can also be guaranteed. Moreover, after completing the time slicing division, the spatiotemporal Gaussian splash model can track the dynamic changes of complex geometric shapes in different time slices. In the stage of generating rendered images, the combination of two-dimensional Gaussian surfaces can accurately and adaptively fit the shape and surface details of the object, thereby improving the reconstruction accuracy. Furthermore, the spatiotemporal Gaussian splash model analyzes the motion trajectory and morphological changes of objects in time slices from different perspectives, and uses the temporal coherence and spatial correlation in the data to adjust the inconsistencies that may occur from different perspectives, so that the final rendered images corresponding to each perspective are more consistent in key elements such as the position and shape of the object.

[0295] The embodiment of the present application also provides a foreground object segmentation and reconstruction device, which can implement the foreground object segmentation and reconstruction method mentioned above, referring to Figure 20 , the device comprises:

[0296] Acquisition module 2010: used to obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, respectively. The initial acquisition data includes acquisition data corresponding to each acquisition moment, and the acquisition data includes video frames and initial Gaussian point clouds.

[0297] Mask calculation module 2020: used to input all video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data.

[0298] Rendering module 2030: used to use the acquisition moments as processing moments one by one, input the input data corresponding to all viewpoints into the space-time Gaussian splash model for data processing, divide the input data into time slices, obtain the slice Gaussian point cloud corresponding to each time slice, obtain the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain the rendering image corresponding to each viewpoint based on multiple two-dimensional Gaussian surfaces.

[0299] Reconstruction module 2040: used to obtain a foreground target rendering sequence at a processing moment according to the rendered image corresponding to each viewing angle, and generate a foreground target reconstruction result according to the foreground target rendering sequence at all acquisition moments.

[0300] The specific implementation of the foreground object segmentation and reconstruction device of this embodiment is basically the same as the specific implementation of the foreground object segmentation and reconstruction method described above, and will not be repeated here.

[0301] An embodiment of the present application further provides an electronic device, including:

[0302] at least one memory;

[0303] at least one processor;

[0304] at least one program;

[0305] The program is stored in the memory, and the processor executes the at least one program to implement the foreground object segmentation and reconstruction method of the present application. The electronic device can be any smart terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0306] See also Figure 21 , Figure 21 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0307] The processor 2101 can be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0308] The memory 2102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 2102 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 2102 and are called by the processor 2101 to execute the foreground object segmentation and reconstruction method of the embodiments of this application.

[0309] Input / output interface 2103, used to implement information input and output;

[0310] Communication interface 2104, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0311] Bus 2105 , which transmits information between various components of the device (e.g., processor 2101 , memory 2102 , input / output interface 2103 , and communication interface 2104 );

[0312] The processor 2101 , the memory 2102 , the input / output interface 2103 and the communication interface 2104 are connected to each other in communication within the device via the bus 2105 .

[0313] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program. When the computer program is executed by a processor, the foreground target segmentation and reconstruction method described above is implemented.

[0314] The memory, as a non-transient storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0315] The foreground target segmentation and reconstruction method, apparatus, device and storage medium proposed in the embodiments of the present application obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, input all video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data. At the processing moment, the input data corresponding to all perspectives are input into the spatiotemporal Gaussian splatter model for data processing, the input data is time-sliced ​​to obtain a sliced ​​Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained, and a rendered image corresponding to each perspective is obtained based on the multiple two-dimensional Gaussian surfaces. A foreground target rendering sequence at the processing moment is obtained based on the rendered image corresponding to each perspective, and a foreground target reconstruction result is generated based on the foreground target rendering sequence at all acquisition moments. In the embodiments of the present application, the video frames are directly input into the target segmentation model to generate a foreground target segmentation mask, without the need for manual pre-setting of too much prior knowledge regarding the specific shape, position, etc. of the target. Furthermore, the spatiotemporal Gaussian splash model divides the input data into time slices based on its inherent temporal dimension, eliminating the need for prior knowledge such as the specific change patterns of each target at different moments. This further reduces reliance on external prior information while ensuring temporal stability. Furthermore, after time slicing, the spatiotemporal Gaussian splash model can track the dynamic changes of complex geometric shapes across different time slices. During the rendered image generation phase, incorporating a two-dimensional Gaussian surface accurately and adaptively fits the object's shape and surface details, thereby improving reconstruction accuracy. Furthermore, by analyzing the motion trajectory and morphological changes of objects within time slices from different viewpoints, the spatiotemporal Gaussian splash model leverages the temporal coherence and spatial correlation of the data to adjust for potential inconsistencies across different viewpoints. This ensures that the resulting rendered images from each viewpoint are more consistent in key elements such as the object's position and shape.

[0316] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0317] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0318] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0319] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0320] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0321] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0322] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0323] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0324] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0325] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0326] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A foreground object segmentation and reconstruction method, characterized in that: Executed by a trained spatiotemporal Gaussian splash model, the method includes: At multiple acquisition moments, initial acquisition data corresponding to multiple perspectives are respectively acquired, wherein the initial acquisition data includes acquisition data corresponding to each of the acquisition moments, and the acquisition data includes video frames and initial Gaussian point clouds; Inputting all the video frames into the target segmentation model for target selection, generating a foreground target segmentation mask corresponding to each video frame, and associating the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data; The acquisition moments are used as processing moments one by one. At the processing moments, the input data corresponding to all viewpoints are input into the spatiotemporal Gaussian splatter model for data processing, the input data are time-sliced ​​to obtain a slice Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and a rendered image corresponding to each viewpoint is obtained based on the multiple two-dimensional Gaussian surfaces; A foreground object rendering sequence at the processing moment is obtained according to the rendered image corresponding to each of the viewing angles, and a foreground object reconstruction result is generated according to the foreground object rendering sequences at all the acquisition moments.

2. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The step of inputting all the video frames into the target segmentation model for target selection and generating a foreground target segmentation mask corresponding to each video frame includes: Obtaining prompt texts corresponding to all the video frames; The video frame and the prompt text are input into a target segmentation model for target selection to obtain at least one foreground target, and the at least one foreground target is superimposed to obtain the foreground target segmentation mask corresponding to the video frame.

3. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The step of dividing the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice includes: Acquire a time slice, and obtain an initial slice point cloud based on the time slice and the initial Gaussian point cloud; Using the foreground object segmentation mask, a mask operation is performed on each of the initial slice point clouds to obtain the corresponding slice Gaussian point cloud; A corresponding depth value is determined according to a relationship between a center position of the slice Gaussian point cloud and a camera position of the viewing angle, and the slice Gaussian point clouds are sorted based on a magnitude order of the depth values.

4. The foreground object segmentation and reconstruction method according to claim 3, characterized in that: The obtaining of the two-dimensional Gaussian surface corresponding to each of the sliced ​​Gaussian point clouds comprises: Obtaining the center position, color parameters, opacity parameters, rotation matrix, and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud; Selecting a specific coordinate axis, and adjusting scale information of the initial scaling matrix based on the specific coordinate axis to obtain a target scaling matrix; At least the center position, the color parameter, the opacity parameter, the rotation matrix and the target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

5. The foreground object segmentation and reconstruction method according to claim 4, characterized in that: Obtaining a rendered image corresponding to each viewing angle according to the plurality of two-dimensional Gaussian surfaces includes: For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces; The rendered image corresponding to the viewing angle is obtained based on the pixel rendering color of the two-dimensional Gaussian surface.

6. The foreground object segmentation and reconstruction method according to claim 5, characterized in that: For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces includes: Obtaining a mixing weight of the two-dimensional Gaussian surface; Obtaining the marginal probability density and conditional probability density corresponding to the pixel point, and obtaining a weighted contribution value according to the product of the marginal probability density, the conditional probability density, and the mixing weight; Based on the order of the depth values, obtaining the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtaining a cumulative weight value according to the weighted contribution values; The pixel rendering color is obtained according to the product of the weighted contribution value, the color parameter and the accumulated weight value.

7. The foreground object segmentation and reconstruction method according to claim 6, characterized in that: The obtaining of the mixing weight of the two-dimensional Gaussian surface includes: Obtaining the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud; Obtaining a camera transformation matrix and a local affine transformation matrix corresponding to the viewing angle; Calculating a two-dimensional covariance according to the camera transformation matrix, the local affine transformation matrix, and the three-dimensional covariance, and performing a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density; The mixing weight is obtained according to the two-dimensional probability density and the corresponding opacity parameter.

8. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The training process of the spatiotemporal Gaussian splash model at least includes: Acquire a plurality of training samples, wherein the training samples include at least a training mask, a training Gaussian point cloud, and a label image for each of the viewing angles corresponding to each training acquisition moment; Inputting the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering to obtain a training rendered image, wherein the training rendered image corresponds to a two-dimensional Gaussian training surface; During the rendering process, generating a depth consistency loss value based on the mixing weight and the training depth value of each of the two-dimensional Gaussian training surfaces; A monocular normal regularization loss value is calculated based on the normal vectors corresponding to the training rendered image and the label image respectively; For two consecutive training acquisition moments, a time domain regularization loss value is calculated according to the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing term loss value is calculated according to the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface; Obtaining a pixel loss value and a structure loss value according to the training rendered image and the label image, and calculating a photometric difference loss value according to the pixel loss value and the structure loss value; Select at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness loss value as a target loss value, and obtain a total loss value according to the target loss value and the photometric difference loss value; The model parameters of the spatiotemporal Gaussian splash model are adjusted according to the total loss value until an iteration termination condition is reached, thereby obtaining the trained spatiotemporal Gaussian splash model.

9. The foreground object segmentation and reconstruction method according to claim 8, characterized in that: Generating a depth consistency loss value based on a mixing weight and a training depth value of each of the two-dimensional Gaussian training surfaces includes: For the two-dimensional Gaussian training surface, based on the order of training depth values, for all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all the previous two-dimensional Gaussian surfaces, obtain a first intermediate value according to the difference between the first intermediate value and the training mixing weight, accumulate the first intermediate values ​​and multiply the resultant value by the training mixing weight of the two-dimensional Gaussian training surface to obtain a total training mixing weight, select a maximum value from the total training mixing weights to obtain a maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value; Obtaining a training weighted contribution value and a training depth value of the two-dimensional Gaussian training surface, calculating a first product of the training weighted contribution value and the training depth value, and calculating a second product of the training weighted contribution value and the maximum training depth value; Accumulating the first products to obtain a second intermediate value, and accumulating the second products to obtain a third intermediate value, obtaining a difference between the second intermediate value and the third intermediate value, and calculating the product of the maximum mixing weight and the difference to obtain the depth consistency loss value.

10. The foreground object segmentation and reconstruction method according to claim 8, characterized in that: The monocular normal regularization loss value is calculated according to the normal vectors corresponding to the training rendered image and the label image, including: Obtaining the corresponding training slice Gaussian point cloud during the rendering process, and obtaining the monocular estimated normal vector corresponding to the training slice Gaussian point cloud; generating a pseudo normal vector according to the training rendered image; The monocular normal regularization loss value is calculated according to the monocular estimated normal vector and the pseudo normal vector.

11. The foreground object segmentation and reconstruction method according to claim 9, characterized in that: The step of calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments includes: At the current training acquisition moment, a preset number of adjacent Gaussian points corresponding to the central Gaussian point at the center position of the two-dimensional Gaussian training surface are obtained, a current color parameter, a current opacity parameter, and a current target scaling value of the central Gaussian point are obtained from the surface parameters, and a total training mixture weight of the two-dimensional Gaussian training surface is obtained; Obtaining the reference color parameter, reference opacity parameter, and reference target scaling value of the adjacent Gaussian point from the surface parameters corresponding to the viewing angle at the previous training acquisition moment; Calculating a norm of a color difference between the current color parameter and the reference color parameter, a norm of an opacity difference between the current opacity parameter and the reference opacity parameter, and a norm of a scaling difference between the current target scaling value and the reference target scaling value; The time domain regularization loss value is obtained by multiplying the total weight of the training mixture with the norm of the color difference, the norm of the opacity difference, and the norm of the scaling difference. The three products corresponding to each of the adjacent Gaussian points are accumulated.

12. The foreground object segmentation and reconstruction method according to claim 11, characterized in that: The motion smoothness loss value is calculated based on the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface, including: At the current training acquisition moment, obtaining the current position parameter and the current rotation parameter of the central Gaussian point from the surface parameters, and obtaining the current reference position parameter of the adjacent Gaussian point; Obtaining the reference position parameters and reference rotation parameters of the adjacent Gaussian points, and the center reference position parameters corresponding to the center Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment; For each of the adjacent Gaussian points, obtaining a fourth intermediate value according to the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, obtaining a fifth intermediate value according to the difference between the reference position parameter and the center reference position parameter, and obtaining a sixth intermediate value according to the difference between the current reference position parameter and the current position parameter; Converting the fourth intermediate value into a rotated intermediate matrix, subtracting the sixth intermediate value from the product of the fifth intermediate value and the rotated intermediate matrix to obtain a seventh intermediate value, and multiplying the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain an eighth intermediate value; The eighth intermediate values ​​corresponding to all adjacent Gaussian points are accumulated to obtain the motion loss term corresponding to the current two-dimensional Gaussian training surface training, and all the motion loss terms are accumulated to obtain the motion smoothing term loss value at the current training acquisition moment.

13. A foreground object segmentation and reconstruction device, characterized in that: Executed by a trained spatiotemporal Gaussian splash model, the apparatus comprises: Acquisition module: used to obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, the initial acquisition data including acquisition data corresponding to each acquisition moment, the acquisition data including video frames and initial Gaussian point clouds; Mask calculation module: used to input all the video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data; Rendering module: used to use the acquisition moments as processing moments one by one, input the input data corresponding to all viewpoints into the spatiotemporal Gaussian splatter model for data processing at the processing moments, divide the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice, obtain a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain a rendered image corresponding to each viewpoint based on the multiple two-dimensional Gaussian surfaces; Reconstruction module: used for obtaining a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each of the viewing angles, and generating a foreground target reconstruction result according to the foreground target rendering sequences at all the acquisition moments.

14. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the foreground object segmentation and reconstruction method according to any one of claims 1 to 12 when executing the computer program.

15. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the foreground object segmentation and reconstruction method according to any one of claims 1 to 12 is implemented.

Images