Foreground target segmentation and reconstruction method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a foreground target segmentation and reconstruction method and device, equipment and a storage medium, and relates to the technical field of computer vision. The method comprises the following steps: directly inputting a video frame into a target segmentation model so as to generate a foreground target segmentation mask, associating the foreground target segmentation mask with a corresponding initial Gaussian point cloud as input data, inputting the input data into a space-time Gaussian splash model for data processing, and performing time slice division on the input data so as to obtain a time slice division result; and obtaining a slice Gaussian point cloud corresponding to each time slice, obtaining a two-dimensional Gaussian curved surface corresponding to each slice Gaussian point cloud, and obtaining a rendered image corresponding to each visual angle according to the plurality of two-dimensional Gaussian curved surfaces. And obtaining a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each visual angle, and generating a foreground target reconstruction result according to the foreground target rendering sequences at all acquisition moments. While the dependence on external prior information is reduced, the stability of a time sequence can be ensured, and the accuracy of a rendering result is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a foreground target segmentation and reconstruction method, device, equipment and storage medium. Background Art

[0002] In today's era where the digital world and reality are intertwined, people's demand for photo-realistic dynamic scene representation and rendering is increasing. In this process, in order to ensure that texture details can be accurately presented and the realism of the scene can be fully displayed, high-quality modeling of objects with complex movements in dynamic scenes is required.

[0003] In the related technology, when performing three-dimensional object segmentation and reconstruction in a complex dynamic environment, the separation of the foreground and background in the scene is usually used as the basis. However, since the dynamic scene itself has the characteristics of flexible object movements, diverse appearances, complex environments, and easy occlusion of targets, there will be inherent ambiguity when converting two-dimensional image sequences to three-dimensional space. In addition, the complex large-scale non-rigid deformation of the human body will also cause the reconstruction results to appear unclear. Therefore, in the process of model segmentation and reconstruction, it is necessary to introduce more prior knowledge to assist the reconstruction process, which makes the entire segmentation and reconstruction process complicated. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a foreground target segmentation and reconstruction method, device, equipment and storage medium to reduce the operational complexity of foreground target segmentation and reconstruction in complex dynamic scenes and improve the segmentation and reconstruction efficiency.

[0005] To achieve the above purpose, a first aspect of an embodiment of the present application proposes a foreground object segmentation and reconstruction method, which is performed by a trained spatiotemporal Gaussian splash model, and the method includes:

[0006] At multiple acquisition moments, respectively acquiring initial acquisition data corresponding to multiple perspectives, the initial acquisition data including acquisition data corresponding to each of the acquisition moments, the acquisition data including video frames and initial Gaussian point clouds;

[0007] Inputting all the video frames into the target segmentation model for target selection, generating a foreground target segmentation mask corresponding to each video frame, and associating the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data;

[0008] The acquisition moments are used as processing moments one by one. For the processing moments, the input data corresponding to all viewing angles are input into the space-time Gaussian splash model for data processing, the input data is divided into time slices to obtain a slice Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and a rendering image corresponding to each viewing angle is obtained according to a plurality of the two-dimensional Gaussian surfaces;

[0009] A foreground target rendering sequence at the processing moment is obtained according to the rendered image corresponding to each of the viewing angles, and a foreground target reconstruction result is generated according to the foreground target rendering sequence at all the acquisition moments.

[0010] In some embodiments, inputting all the video frames into the target segmentation model for target selection and generating a foreground target segmentation mask corresponding to each video frame includes:

[0011] Obtaining prompt texts corresponding to all the video frames;

[0012] The video frame and the prompt text are input into a target segmentation model for target selection to obtain at least one foreground target, and the at least one foreground target is superimposed to obtain a foreground target segmentation mask corresponding to the video frame.

[0013] In some embodiments, dividing the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice includes:

[0014] Acquire a time slice, and obtain an initial slice point cloud based on the time slice and the initial Gaussian point cloud;

[0015] Using the foreground object segmentation mask, a mask operation is performed on each of the initial slice point clouds to obtain the corresponding slice Gaussian point cloud;

[0016] The corresponding depth value is determined according to the relationship between the center position of the slice Gaussian point cloud and the camera position of the viewing angle, and the slice Gaussian point cloud is sorted based on the order of the depth values.

[0017] In some embodiments, obtaining a two-dimensional Gaussian surface corresponding to each of the sliced ​​Gaussian point clouds includes:

[0018] Obtaining the center position, color parameter, opacity parameter, rotation matrix and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud;

[0019] Selecting a specific coordinate axis, and adjusting scale information of the initial scaling matrix based on the specific coordinate axis to obtain a target scaling matrix;

[0020] At least the center position, the color parameter, the opacity parameter, the rotation matrix and the target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

[0021] In some embodiments, obtaining a rendered image corresponding to each viewing angle according to the plurality of two-dimensional Gaussian surfaces includes:

[0022] For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces;

[0023] The rendered image corresponding to the viewing angle is obtained based on the pixel rendering color of the two-dimensional Gaussian surface.

[0024] In some embodiments, for each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces includes:

[0025] Obtaining a mixing weight of the two-dimensional Gaussian surface;

[0026] Obtaining the marginal probability density and the conditional probability density corresponding to the pixel point, and obtaining a weighted contribution value according to the product of the marginal probability density, the conditional probability density and the mixing weight;

[0027] Based on the order of the depth values, obtaining the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtaining a cumulative weight value according to the weighted contribution values;

[0028] The pixel rendering color is obtained according to the product of the weighted contribution value, the color parameter and the accumulated weight value.

[0029] In some embodiments, obtaining the mixing weight of the two-dimensional Gaussian surface includes:

[0030] Obtaining the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud;

[0031] Obtaining a camera transformation matrix and a local affine transformation matrix corresponding to the viewing angle;

[0032] Calculating a two-dimensional covariance according to the camera transformation matrix, the local affine transformation matrix and the three-dimensional covariance, and performing a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density;

[0033] The mixing weight is obtained according to the two-dimensional probability density and the corresponding opacity parameter.

[0034] In some embodiments, the training process of the spatiotemporal Gaussian splash model includes at least:

[0035] Acquire a plurality of training samples, wherein the training samples at least include a training mask, a training Gaussian point cloud, and a label image for each of the viewing angles corresponding to each training acquisition moment;

[0036] Inputting the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering processing to obtain a training rendered image, wherein the training rendered image corresponds to a two-dimensional Gaussian training surface;

[0037] During the rendering process, generating a depth consistency loss value based on the mixing weight and the training depth value of each of the two-dimensional Gaussian training surfaces;

[0038] A monocular normal regularization loss value is calculated according to normal vectors corresponding to the training rendered image and the label image respectively;

[0039] For two consecutive training acquisition moments, a time domain regularization loss value is calculated according to the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing term loss value is calculated according to the position parameters and rotation parameters respectively corresponding to the two-dimensional Gaussian training surface;

[0040] Obtaining a pixel loss value and a structure loss value according to the training rendered image and the label image, and calculating a photometric difference loss value according to the pixel loss value and the structure loss value;

[0041] Select at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness term loss value as a target loss value, and obtain a total loss value according to the target loss value and the photometric difference loss value;

[0042] The model parameters of the spatiotemporal Gaussian splash model are adjusted according to the total loss value until an iteration termination condition is reached to obtain the trained spatiotemporal Gaussian splash model.

[0043] In some embodiments, generating a depth consistency loss value based on a mixing weight and a training depth value of each of the two-dimensional Gaussian training surfaces includes:

[0044] For the two-dimensional Gaussian training surface, based on the order of training depth values, for all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all the previous two-dimensional Gaussian surfaces, obtain a first intermediate value according to a difference with the training mixing weight, accumulate the first intermediate value and multiply it by the training mixing weight of the two-dimensional Gaussian training surface to obtain a total training mixing weight, select a maximum value from the total training mixing weight to obtain a maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value;

[0045] Obtaining a training weighted contribution value and a training depth value of the two-dimensional Gaussian training surface, calculating a first product of the training weighted contribution value and the training depth value, and calculating a second product of the training weighted contribution value and the maximum training depth value;

[0046] The first products are accumulated to obtain a second intermediate value, and the second products are accumulated to obtain a third intermediate value, a difference between the second intermediate value and the third intermediate value is obtained, and the product of the maximum mixing weight and the difference is calculated to obtain the depth consistency loss value.

[0047] In some embodiments, the calculating the monocular normal regularization loss value according to the normal vectors corresponding to the training rendered image and the label image respectively includes:

[0048] Obtain the training slice Gaussian point cloud corresponding to the rendering process, and obtain the monocular estimated normal vector corresponding to the training slice Gaussian point cloud;

[0049] generating a pseudo normal vector according to the training rendered image;

[0050] The monocular normal regularization loss value is calculated according to the monocular estimated normal vector and the pseudo normal vector.

[0051] In some embodiments, for two consecutive training acquisition moments, calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface includes:

[0052] At the current training acquisition moment, obtain a preset number of adjacent Gaussian points corresponding to the central Gaussian point where the center position of the two-dimensional Gaussian training surface is located, obtain the current color parameter, current opacity parameter and current target scaling value of the central Gaussian point from the surface parameters, and obtain the total training mixture weight of the two-dimensional Gaussian training surface;

[0053] Obtaining the reference color parameter, reference opacity parameter and reference target scaling value of the adjacent Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment;

[0054] Calculate the norm of the color difference between the current color parameter and the reference color parameter, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter, and the norm of the scaling difference between the current target scaling value and the reference target scaling value;

[0055] The time domain regularization loss value is obtained by multiplying the total weight of the training mixture with the norm of the color difference, the norm of the opacity difference and the norm of the scaling difference respectively, and accumulating the three products corresponding to each of the adjacent Gaussian points.

[0056] In some embodiments, the step of calculating the motion smoothness loss value according to the position parameters and rotation parameters respectively corresponding to the two-dimensional Gaussian training surface includes:

[0057] At the current training acquisition moment, obtaining the current position parameter and current rotation parameter of the central Gaussian point from the surface parameters, and obtaining the current reference position parameter of the adjacent Gaussian point;

[0058] Acquire the reference position parameters and reference rotation parameters of the adjacent Gaussian points, and the central reference position parameters corresponding to the central Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment;

[0059] For each of the adjacent Gaussian points, a fourth intermediate value is obtained according to the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, a fifth intermediate value is obtained according to the difference between the reference position parameter and the center reference position parameter, and a sixth intermediate value is obtained according to the difference between the current reference position parameter and the current position parameter;

[0060] Convert the fourth intermediate value into a rotation intermediate matrix, subtract the sixth intermediate value from the product of the fifth intermediate value and the rotation intermediate matrix to obtain a seventh intermediate value, and multiply the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain an eighth intermediate value;

[0061] The eighth intermediate values ​​corresponding to all adjacent Gaussian points are accumulated to obtain the motion loss item corresponding to the current two-dimensional Gaussian training surface training, and all the motion loss items are accumulated to obtain the motion smoothing item loss value at the current training acquisition moment.

[0062] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a foreground object segmentation and reconstruction device, which is performed by a trained spatiotemporal Gaussian splash model, and the device includes:

[0063] Acquisition module: used to obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, respectively, wherein the initial acquisition data includes acquisition data corresponding to each of the acquisition moments, and the acquisition data includes video frames and initial Gaussian point clouds;

[0064] Mask calculation module: used to input all the video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data;

[0065] Rendering module: used to use the acquisition moments as processing moments one by one, input the input data corresponding to all viewing angles into the space-time Gaussian splash model for data processing at the processing moments, divide the input data into time slices, obtain a slice Gaussian point cloud corresponding to each time slice, obtain a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain a rendering image corresponding to each viewing angle according to multiple two-dimensional Gaussian surfaces;

[0066] Reconstruction module: used for obtaining a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each of the viewing angles, and generating a foreground target reconstruction result according to the foreground target rendering sequence at all the acquisition moments.

[0067] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0068] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a storage medium. The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0069] The foreground target segmentation and reconstruction method, device, equipment and storage medium proposed in the embodiment of the present application obtains the initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, inputs all video frames into the target segmentation model for target selection, generates the foreground target segmentation mask corresponding to each video frame, and associates the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data. For the processing moment, the input data corresponding to all perspectives are input into the spatiotemporal Gaussian splash model for data processing, the input data is divided into time slices, the sliced ​​Gaussian point cloud corresponding to each time slice is obtained, the two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained, and the rendering image corresponding to each perspective is obtained according to the multiple two-dimensional Gaussian surfaces. The foreground target rendering sequence at the processing moment is obtained according to the rendering image corresponding to each perspective, and the foreground target reconstruction result is generated according to the foreground target rendering sequence at all acquisition moments. The embodiment of the present application directly inputs the video frame into the target segmentation model to generate the foreground target segmentation mask, without the need to manually pre-set too much prior knowledge involving the specific form, position and other aspects of the target. In addition, the space-time Gaussian splash model is used to divide the input data into time slices according to the time dimension characteristics of the input data itself, without the need to know the specific change patterns of each target at different times and other prior knowledge in advance. In this way, while further reducing the dependence on external prior information, the stability of the time series can also be guaranteed. Moreover, after completing the time slice division, the space-time Gaussian splash model can track the dynamic changes of complex geometric shapes in different time slices. In the stage of generating rendered images, the shape and surface details of the object can be accurately and adaptively fitted in combination with the two-dimensional Gaussian surface, thereby improving the reconstruction accuracy. Furthermore, the space-time Gaussian splash model analyzes the motion trajectory and morphological changes of objects in time slices from different perspectives, and adjusts the inconsistencies that may occur from different perspectives with the help of the temporal coherence and spatial correlation in the data, so that the rendered images corresponding to each perspective generated in the end are more consistent in terms of key elements such as the position and shape of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of the foreground target segmentation and reconstruction method provided in an embodiment of the present application.

[0071] Figure 2 It is a schematic diagram of a video frame provided in an embodiment of the present application.

[0072] Figure 3 It is a flowchart provided by an embodiment of the present application for inputting all video frames into a target segmentation model for target selection, and generating a foreground target segmentation mask corresponding to each video frame.

[0073] Figure 4 A schematic diagram of the process of generating a foreground target mask provided in an embodiment of the present application.

[0074] Figure 5 It is a schematic diagram of input data provided in an embodiment of the present application.

[0075] Figure 6 This is a flowchart of dividing the input data into time slices and obtaining the sliced ​​Gaussian point cloud corresponding to each time slice provided in an embodiment of the present application.

[0076] Figure 7 It is a schematic diagram of a sliced ​​Gaussian point cloud provided in an embodiment of the present application.

[0077] Figure 8 It is a flowchart for obtaining the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud provided in an embodiment of the present application.

[0078] Fig. 9 This is a flowchart of obtaining the corresponding pixel rendering color according to multiple two-dimensional Gaussian surfaces for each pixel point in the two-dimensional Gaussian surface provided by an embodiment of the present application.

[0079] Fig.10 It is a flowchart of obtaining the mixing weight of a two-dimensional Gaussian surface provided in an embodiment of the present application.

[0080] Fig.11 It is a schematic diagram of a rendered image provided in an embodiment of the present application.

[0081] Fig.12 A schematic diagram of the foreground target reconstruction result provided in an embodiment of the present application.

[0082] Fig.13 It is a flow chart of the training process of the spatiotemporal Gaussian splash model provided in an embodiment of the present application.

[0083] Fig.14 It is a flowchart of generating a depth consistency loss value based on the mixing weight and training depth value of each two-dimensional Gaussian training surface provided by an embodiment of the present application.

[0084] Fig.15 This is a flowchart of calculating a monocular normal regularization loss value based on normal vectors corresponding to a training rendered image and a label image, provided in an embodiment of the present application.

[0085] Fig.16 It is a flowchart provided in an embodiment of the present application for calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments.

[0086] Fig.17 It is a flowchart of calculating the motion smoothness loss value according to the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface provided in an embodiment of the present application.

[0087] Fig.18 It is a schematic diagram of the application effect of different target loss values ​​in the embodiment of the present application.

[0088] Fig.19 It is a schematic diagram of the overall process of the foreground object segmentation and reconstruction method provided in the embodiment of the present application.

[0089] Fig. 20 This is a structural block diagram of a foreground object segmentation and reconstruction device provided by another embodiment of the present application.

[0090] Fig.21 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0092] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0094] First, some nouns involved in this application are analyzed:

[0095] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0096] In today's era where the digital world and reality are intertwined, people's demand for photo-realistic dynamic scene representation and rendering is increasing. In this process, in order to ensure that texture details can be accurately presented and the realism of the scene can be fully displayed, high-quality modeling of objects with complex movements in dynamic scenes is required.

[0097] In the related art, when performing segmentation and reconstruction of three-dimensional objects in complex dynamic environments, one method is to uniformly model the foreground and background, and the other method is to separate the foreground and background in the scene. In the first case, semantic information is integrated into the network model, and visual features that support semantic description are introduced by introducing semantic modeling or feature encoding, thereby supporting fine-grained scene understanding. However, it is necessary to further separate the foreground and background of the scene. This separation operation must rely on a large number of manually labeled data sets or three-dimensional point clouds obtained by precise scanning as input data to generate a foreground segmentation mask with a relatively high accuracy. The annotation work of such data sets will cost a high cost, and the scope of the data set is relatively limited, resulting in the reconstruction effect of the model is not very accurate. In the second case, since the dynamic scene itself has the characteristics of flexible object movements, diverse appearances, complex environments, and easy occlusion of targets, there will be inherent ambiguity when converting a two-dimensional image sequence to a three-dimensional space. Moreover, the complex large-scale non-rigid deformation of the human body will also cause the reconstruction results to be unclear. Therefore, in the process of model segmentation and reconstruction, it is necessary to introduce more prior knowledge to assist the reconstruction process, which makes the entire segmentation and reconstruction process complicated.

[0098] Based on this, the embodiment of the present application provides a foreground target segmentation and reconstruction method, device, equipment and storage medium, which directly inputs the video frame into the target segmentation model to generate a foreground target segmentation mask, without artificially presetting too much prior knowledge involving the specific shape, position and other aspects of the target. In addition, the spatiotemporal Gaussian splash model is used to divide the input data into time slices according to the time dimension characteristics of the input data itself, without knowing the prior knowledge such as the specific change mode of each target at different times in advance. In this way, while further reducing the dependence on external prior information, the stability of the time series can also be guaranteed. Moreover, after completing the time slice division, the spatiotemporal Gaussian splash model can track the dynamic changes of complex geometric shapes in different time slices. In the stage of generating rendered images, the shape and surface details of the object can be accurately and adaptively fitted in combination with the two-dimensional Gaussian surface, thereby improving the reconstruction accuracy. Furthermore, the spatiotemporal Gaussian splash model analyzes the motion trajectory and morphological changes of the object in the time slice under different perspectives, and adjusts the inconsistencies that may occur under different perspectives with the help of the temporal coherence and spatial correlation in the data, so that the rendered images corresponding to each perspective generated in the end are more consistent in terms of key elements such as the position and shape of the object.

[0099] The embodiments of the present application provide a foreground object segmentation and reconstruction method, apparatus, device and storage medium, which are specifically illustrated by the following embodiments. First, the foreground object segmentation and reconstruction method in the embodiments of the present application is described.

[0100] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0101] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0102] The foreground target segmentation and reconstruction method provided in the embodiment of the present application relates to the field of computer vision technology. The foreground target segmentation and reconstruction method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be a computer program running in a terminal or a server side. For example, the computer program can be a native program or software module in an operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a client that supports foreground target segmentation and reconstruction, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plug-in. Among them, the terminal communicates with the server via a network. The foreground target segmentation and reconstruction method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0103] In some embodiments, the terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the foreground target segmentation and reconstruction method of this embodiment to provide related services to enhance the driving experience. The server may be an independent server, or it may be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms; it may also be a service node in a blockchain system, and each service node in the blockchain system forms a peer-to-peer (Peer To Peer, P2P) network, and the P2P protocol is an application layer protocol running on the Transmission Control Protocol (Transmission Control Protocol, TCP) protocol. The terminal and the server may be connected via Bluetooth, Universal Serial Bus (Universal Serial Bus, USB) or network and other communication connection methods, which are not limited in this embodiment.

[0104] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0105] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0106] The foreground object segmentation and reconstruction method in the embodiment of the present application is described below.

[0107] Figure 1 is an optional flowchart of the foreground object segmentation and reconstruction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps 110 to 140. It can also be understood that this embodiment Figure 1 The order of step 110 to step 140 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0108] Step 110: At multiple acquisition moments, respectively obtain initial acquisition data corresponding to multiple viewing angles.

[0109] In one embodiment, for the target scene, the image acquisition device can be deployed in a 360-degree panoramic surround layout to achieve synchronous acquisition. This surround layout can cover the target scene in all directions to ensure that no angle information is missed, so that the data of the target scene can be obtained from all directions. Each image acquisition device corresponds to a specific viewing angle, which is similar to observing the same object from different positions, and the pictures seen at different positions are different. At each viewing angle, the image acquisition device is used to continuously shoot and obtain the corresponding video sequence, and the video sequence contains multiple video frames recorded at different acquisition times. For example, the image acquisition device can use a ZCAME2 camera, the number of viewing angles is set to 60, and the resolution of the video sequence is 1920×1080. If 10 acquisition moments are set, all 60 cameras can capture a video sequence containing 10 video frames.

[0110] In one embodiment, referring to Figure 2 , Figure 2 It is a schematic diagram of a video frame provided in an embodiment of the present application. Figure 2Assume that there are 8 viewing angles, each viewing angle captures a video sequence, each video sequence contains the same number of video frames, and the same acquisition time has corresponding video frames at different viewing angles. In the figure, t1, t2, ..., tn represent n acquisition times.

[0111] In one embodiment, in addition to acquiring video sequences, the image acquisition device can also acquire sparse point cloud data. In this case, all data acquired by the image acquisition device are collectively referred to as initial acquisition data. For a single perspective, its initial acquisition data is composed of the video sequence and sparse point cloud data acquired by the corresponding image acquisition device. The initial acquisition data includes the acquisition data corresponding to each acquisition time, and the acquisition data is composed of the video frame corresponding to the acquisition time and the initial Gaussian point cloud.

[0112] Next, the camera system corresponding to the image acquisition device is calibrated with the help of COLMAP to obtain the internal and external parameters of the camera. Among them, the internal parameters mainly include the focal length, principal point position and lens distortion parameters of the camera, and their role is to reflect the characteristics of the camera's internal imaging; the external parameters are related to the position and posture information of the camera in space, such as the rotation angle and translation vector of the camera, and the purpose is to determine the spatial relationship of the camera relative to the target scene being photographed. After completing the COLMAP calibration, the internal and external parameters of the camera can be used to calibrate the sparse point cloud data to obtain the initial Gaussian point cloud corresponding to each acquisition moment. Specifically: at each acquisition moment, according to the imaging characteristics and spatial position relationship provided by the internal and external parameters of the camera, the sparse point cloud data is converted and adjusted accordingly, so that it is adapted to the shooting angle and imaging characteristics of the camera, and finally the initial Gaussian point cloud corresponding to each acquisition moment is obtained.

[0113] Step 120: All video frames are input into the target segmentation model for target selection, a foreground target segmentation mask corresponding to each video frame is generated, and the foreground target segmentation mask and the corresponding initial Gaussian point cloud are associated as input data.

[0114] In one embodiment, firstly, video frames of each viewing angle are obtained, and all video frames are input into the target segmentation model for target selection, and a foreground target segmentation mask corresponding to each video frame is generated. Figure 3 , Figure 3 The flowchart of the embodiment of the present application is to input all video frames into the target segmentation model for target selection, and generate a foreground target segmentation mask corresponding to each video frame, which specifically includes the following steps:

[0115] Step 310: Obtain prompt texts corresponding to all video frames.

[0116] In one embodiment, the prompt text is used to explicitly inform the target segmentation model what kind of target features need to be segmented. Since the purpose of the embodiment of the present application is to segment dynamic targets, it is necessary to ensure the consistency of the extraction of foreground targets. Since foreground targets often have relatively uniform feature performances under different viewing angles, such as when shooting an activity scene containing a human subject, no matter from which viewing angle to observe, the key features such as the general outline and shape of the human as a foreground target are similar, therefore, for all video frames, this embodiment uses the same prompt text to perform foreground target segmentation. In other words, in the face of the video frames collected under various viewing angles, there is no need to set different prompt texts for each viewing angle or each video frame, and the extraction of targets in all video frames can be completed by relying on the same prompt text. It is understandable that the prompt text is also consistent at other processing moments.

[0117] Step 320: Input the video frame and the prompt text into the target segmentation model to select the target, obtain at least one foreground target, superimpose at least one foreground target, and obtain a foreground target segmentation mask corresponding to the video frame.

[0118] In one embodiment, the prompt text can be input in batches, and each time an object is extracted from it for segmentation, thereby obtaining the corresponding foreground object, and then all foreground objects are superimposed, and after the superposition is completed, the final total object is obtained by edge dilation operation, and then binarization is performed based on the total object to generate the foreground object segmentation mask corresponding to each video frame. In addition, the prompt text can also be input at one time to directly extract all foreground objects as the total object.

[0119] Reference Figure 4 , Figure 4 Schematic diagram of the generation process of the foreground target mask provided in the embodiment of the present application. In this example, the prompt text can be set as "person", "knife" and "beef" respectively. In this way, after three extraction operations, the corresponding foreground targets can be obtained. After superposition and expansion processing, the overall target can be obtained, and then the foreground target segmentation mask can be generated. At the same time, "person + knife + beef" can also be used as the prompt text to directly generate the overall target containing these three targets.

[0120] In one embodiment, the target segmentation model uses the GroundedSAM2 model, which can accurately process various types of video frames based on the given relevant prompt information. The model can be used to generate a foreground target segmentation mask of interest for each video frame. In the embodiment of the present application, the corresponding video sequence can be directly input into the target segmentation model according to different viewing angles to obtain the foreground target segmentation mask of the video frame. For example, if the video sequence contains multiple pedestrians and vehicles, and the prompt text is set to: "Extract pedestrians and vehicles in the picture", then for each video frame, the GroundedSAM2 model will identify the foreground targets that meet the requirements in the video frame based on the input prompt text, that is, the pedestrian and vehicle parts, and then obtain the foreground target segmentation mask corresponding to each video frame.

[0121] In one embodiment, through the above process, each video frame corresponds to a foreground object segmentation mask. Considering that the video frame also includes the initial Gaussian point cloud corresponding thereto, it is necessary to associate the foreground object segmentation mask with the corresponding initial Gaussian point cloud and use it as input data. Figure 5 , Figure 5 is a schematic diagram of input data provided by an embodiment of the present application. Figure 4 For the scene, for each acquisition moment, there is corresponding input data from view 1 to view 8, and each input data includes the corresponding foreground target segmentation mask and initial Gaussian point cloud.

[0122] Step 130: Take the acquisition moments as processing moments one by one. For the processing moments, input the input data corresponding to all viewpoints into the space-time Gaussian splash model for data processing, divide the input data into time slices, obtain the slice Gaussian point cloud corresponding to each time slice, obtain the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain the rendering image corresponding to each viewpoint based on multiple two-dimensional Gaussian surfaces.

[0123] In one embodiment, the segmentation and reconstruction process needs to be performed for each acquisition moment, so each acquisition moment is sequentially set as a processing moment. The first processing moment is taken as an example to illustrate the specific process of segmentation and reconstruction, and the execution process of the remaining processing moments is the same.

[0124] In one embodiment, at each processing moment, the input data corresponding to all viewing angles are input into the spatiotemporal Gaussian splash model for data processing to obtain a rendered image of each viewing angle at the processing moment. Figure 5 , when the reference time t1 is used as the processing time, the input data corresponding to all view angles in the reference time t1 need to be input into the spatiotemporal Gaussian splash model.

[0125] The following takes the processing time t as an example to describe the detailed processing process of the spatiotemporal Gaussian splash model.

[0126] In one embodiment, referring to Figure 6 , Figure 6 The flowchart of the embodiment of the present application for time slicing the input data to obtain the sliced ​​Gaussian point cloud corresponding to each time slice specifically includes the following steps:

[0127] Step 610: Obtain time slices, and obtain initial slice point clouds based on the time slices and the initial Gaussian point clouds.

[0128] In one embodiment, the spatiotemporal Gaussian splash model is a four-dimensional Gaussian splash model. Compared with the three-dimensional Gaussian splash model, the four-dimensional Gaussian splash model considers the influence of time. Therefore, the embodiment of the present application uses the four-dimensional Gaussian splash model to simultaneously consider time and space information. Specifically, for each three-dimensional initial Gaussian point cloud corresponding to the processing time t, time information is added therein, and the time information here is represented by the time slice μ t , μ t is a very small time frame, used to assume that all movements can be approximated as linear movements within a small time frame.

[0129] The spatiotemporal Gaussian splash model adds time information to the three-dimensional Gaussian model to transform it into a four-dimensional Gaussian model. Assuming that the three-dimensional center position of the initial Gaussian point cloud is μ 3D =(μ x ,μ y ,μ z ) T , where the three-dimensional center position represents the mean vector corresponding to the three-dimensional coordinates of each point in the initial Gaussian point cloud. At this time, adding a time slice to the three-dimensional center position can obtain the four-dimensional center position μ of the corresponding four-dimensional Gaussian function. 4D =(μ x ,μ y ,μ z ,μ t ), in constructing the four-dimensional covariance matrix Σ 4D After that, the four-dimensional Gaussian function G can be expressed as 4D (x):

[0130]

[0131] Where x represents the input Gaussian point, S 4D represents the four-dimensional scaling matrix, R 4D represents the four-dimensional rotation matrix, U represents the first decomposition matrix, V represents the second decomposition matrix, and W represents the world-to-camera camera transformation matrix, wherein the first decomposition matrix, the second decomposition matrix, the world-to-camera camera transformation matrix, and the time slice can all be adjusted during the training process.

[0132] For the processing time t, an initial slice point cloud is obtained based on the time slice and the initial Gaussian point cloud, wherein the initial slice point cloud is a three-dimensional Gaussian function.

[0133] Step 620: Use the foreground object segmentation mask to perform a mask operation on each initial slice point cloud to obtain a corresponding slice Gaussian point cloud.

[0134] In one embodiment, the foreground target segmentation mask clearly defines the area range where the foreground target is located in the video frame. The initial slice point cloud is a representation of the target scene at the corresponding acquisition time and viewing angle in the form of a point cloud, which covers multiple data points in the target scene. Therefore, according to the foreground target area marked by the foreground target segmentation mask, the corresponding point cloud data can be screened out in the initial slice point cloud as a slice Gaussian point cloud, and the range of its Gaussian points is limited according to the foreground target segmentation mask. Specifically, the three-dimensional probability density of the slice Gaussian point cloud can be obtained based on the three-dimensional probability density of the initial slice point cloud, expressed as:

[0135]

[0136] Among them, σ(t) represents the temporal opacity of the sliced ​​Gaussian point cloud corresponding to the processing time t, μ 3D (t) represents the three-dimensional center position changing with time after the linear motion term is introduced into the three-dimensional center position, Σ 3d Represents the three-dimensional covariance of the sliced ​​Gaussian point cloud. The three-dimensional covariance is used to describe the distribution shape and direction of the sliced ​​Gaussian point cloud in three-dimensional space, which can be obtained based on the initial Gaussian point cloud.

[0137] For example, in the indoor scene represented by the initial slice point cloud, there are many household items. If the foreground target segmentation mask marks the tables and chairs as the foreground target area, then with the help of mask operation, the point cloud data related to the tables and chairs can be extracted from the initial slice point cloud, while the point cloud data belonging to the background part (such as walls, floors, etc.) can be eliminated. After such mask operation, the corresponding slice Gaussian point cloud can be obtained. Compared with the initial slice point cloud, the slice Gaussian point cloud focuses more on the data related to the foreground target, and it more accurately presents the key characteristics of the foreground target in space, such as the position and shape, in the form of Gaussian distribution.

[0138] Step 630: Determine the corresponding depth value according to the relationship between the center position of the sliced ​​Gaussian point cloud and the camera position of the viewing angle, and sort the sliced ​​Gaussian point cloud based on the order of the depth values.

[0139] In one embodiment, the relationship between the center position of the sliced ​​Gaussian point cloud and the camera position corresponding to the viewing angle corresponds to a depth value, and the depth value can reflect the depth of the foreground target at the viewing angle. Specifically, the depth value is determined by the straight-line distance between the center position and the camera position that captures the viewing angle. The smaller the straight-line distance, the smaller the depth value. A small depth value means that the foreground target is relatively closer to the camera at the viewing angle, and vice versa. Next, based on the order of the depth values ​​determined in this way, the sliced ​​Gaussian point cloud is sorted, so as to sort out the front and back hierarchical relationship of each foreground target in space in an orderly manner. Figure 7 , Figure 7 is a schematic diagram of a sliced ​​Gaussian point cloud provided in an embodiment of the present application. The figure is illustrated with processing time t, where T i =t-μ t , T i+1 =t+1-μ t , T i-1 =t-1-μ t , and then sort the sliced ​​Gaussian point clouds of different perspectives according to the depth value.

[0140] Next, since the sliced ​​Gaussian point cloud is a three-dimensional Gaussian ellipsoid, each sliced ​​Gaussian point cloud can be further represented as a two-dimensional Gaussian surface by controlling the size of the z-scale direction of the rotation matrix of the three-dimensional Gaussian ellipsoid corresponding to the processing time t. Figure 8 , Figure 8 This is a flowchart of obtaining a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud provided by an embodiment of the present application, which specifically includes the following steps:

[0141] Step 810: Obtain the center position, color parameters, opacity parameters, initial rotation matrix and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud.

[0142] In one embodiment, since there are multiple slice Gaussian point clouds corresponding to the processing time t, for the kth slice Gaussian point cloud after sorting, the corresponding center position is obtained. Color parameters Opacity parameter Initial rotation matrix and the initial scaling matrix

[0143] In one embodiment, it is assumed that the distribution of illumination in the target scene can be approximated by spherical harmonics. Spherical harmonics, as a function form widely used in fields such as mathematics and physics, can effectively mathematically describe physical quantities such as illumination in space that have certain distribution laws. For example, in an indoor target scene, the intensity, direction, and distribution of light rays incident from different directions in space can be reasonably simulated and approximated by means of spherical harmonics. Therefore, the embodiment of the present application calculates color parameters based on spherical harmonics. For a two-dimensional Gaussian surface, the calculation of its color is usually associated with illumination, and the illumination information can be parameterized by the spherical harmonics set according to the actual situation. The two-dimensional Gaussian surface itself belongs to a data structure that carries scene-related information, and its color presentation is largely affected by illumination. Different illumination situations will cause the surface to show different color effects. Based on the above assumptions, the illumination information can be parameterized by spherical harmonics, that is, the complex distribution characteristics of illumination in the scene are converted into corresponding parameter values ​​in the spherical harmonics, such as spherical harmonic coefficients representing illumination or spherical harmonic coefficients representing reflectivity. Subsequently, an integral calculation is performed on the color of the pixel points in the two-dimensional Gaussian surface according to the spherical harmonic function until the color parameters are determined. The specific parameters of the spherical harmonic function can be trained during the training process.

[0144] In addition, the position, direction and shape of the two-dimensional Gaussian surface in space (determined by its center position, covariance matrix and other parameters) will affect the reception of light. For example, if the normal vector of the two-dimensional Gaussian surface (which can be obtained by deriving the parameters of the Gaussian surface) is at a different angle to the direction of illumination, the intensity of light received will also be different. Therefore, the color parameters calculated above can also be adjusted based on the geometric information of the two-dimensional Gaussian surface. For example, for diffuse reflection, the color intensity is scaled according to the cosine value of the angle between the normal vector and the direction of illumination.

[0145] Step 820: Select a specific coordinate axis, and based on the specific coordinate axis, adjust the scale information of the initial scaling matrix to obtain a target scaling matrix.

[0146] In one embodiment, the specific coordinate axis is the z axis, so based on the z axis, the z direction in the initial scaling matrix is ​​changed to zero to obtain the target scaling matrix

[0147] Step 830: At least the center position, color parameter, opacity parameter, rotation matrix and target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

[0148] In one embodiment, the parameters of the k-th two-dimensional Gaussian surface at time t are processed It is expressed as:

[0149]

[0150] According to the above process, the two-dimensional Gaussian surface corresponding to each three-dimensional sliced ​​Gaussian point cloud is obtained. Combining the flexible optimization process of the three-dimensional Gaussian and the surface alignment characteristics of the two-dimensional Gaussian surface, the quality of the reconstructed geometry is significantly improved.

[0151] In one embodiment, the two-dimensional Gaussian surface is a surface structure presented in the form of a specific mathematical function, which can describe the distribution of data points in the foreground object in the two-dimensional space, and the pixel points on the surface carry the information of the corresponding data points. Therefore, after obtaining the two-dimensional Gaussian surface, it is necessary to generate a rendering image corresponding to each viewing angle based on multiple two-dimensional Gaussian surfaces.

[0152] The specific process is: for each pixel point in the two-dimensional Gaussian surface, the corresponding pixel rendering color is determined based on multiple two-dimensional Gaussian surfaces. On this basis, the rendering image corresponding to the viewing angle is further obtained based on the pixel rendering color of the two-dimensional Gaussian surface. Since each viewing angle has its own unique observation range and angle, the rendering colors of each pixel point on the two-dimensional Gaussian surface determined by the above steps are arranged and combined according to the corresponding spatial position relationship and image construction rules to construct a complete rendering image corresponding to the viewing angle.

[0153] The detailed rendering process is described below.

[0154] In one embodiment, referring to Fig. 9 , Fig. 9 This is a flowchart of obtaining a corresponding pixel rendering color according to multiple two-dimensional Gaussian surfaces for each pixel point in the two-dimensional Gaussian surface provided by an embodiment of the present application, which specifically includes the following steps:

[0155] Step 910: Obtain the mixing weight of the two-dimensional Gaussian surface.

[0156] In one embodiment, referring to Fig.10 , Fig.10 This is a flow chart of obtaining the mixing weight of a two-dimensional Gaussian surface provided in an embodiment of the present application, which specifically includes the following steps:

[0157] Step 1010: Obtain the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud.

[0158] In one embodiment, the three-dimensional covariance is Σ 3D , the three-dimensional probability density is:

[0159]

[0160] Step 1020: Obtain the camera transformation matrix and the local affine transformation matrix corresponding to the viewing angle.

[0161] Step 1030: Calculate a two-dimensional covariance based on the camera transformation matrix, the local affine transformation matrix and the three-dimensional covariance, and perform a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density.

[0162] In one embodiment, in order to render an image, it is first necessary to use the world-to-camera camera transformation matrix W to transform the two-dimensional Gaussian surface from the world coordinate system to the camera coordinate system, and then project it to the image plane through the local affine transformation matrix J. In this process, a two-dimensional covariance is obtained. Next, a two-dimensional projection of the three-dimensional probability density is performed based on the two-dimensional covariance to obtain a two-dimensional probability density G′ 3D (x, t), this projection process can be obtained by using the transformation projection method using the covariance matrix in the related technology.

[0163] The two-dimensional covariance of the k-th two-dimensional Gaussian surface is expressed as:

[0164] Σ′ k =JWΣ 3D W T J T

[0165] Step 1040: Obtain a blending weight according to the two-dimensional probability density and the corresponding opacity parameter.

[0166] In one embodiment, assuming that the pixel point on the image plane is u, u k represents the center position of the k-th two-dimensional Gaussian surface, so the mixing weight of the k-th two-dimensional Gaussian surface is expressed as:

[0167] α k =G′(u;u k ,Σ′ k )σ k

[0168] Among them, α k represents the mixing weight, σ k represents the opacity parameter of the k-th two-dimensional Gaussian surface, that is, the opacity parameter corresponding to the processing time t It is used to measure the "visibility" of this two-dimensional Gaussian surface in the final rendering result or the weight of its influence on the rendered image, where the opacity parameter is a weighted parameter calculated based on the opacity value of the pixel. It is understandable that since the two-dimensional Gaussian surface is a surface with distribution characteristics, it presents a "hill"-like distribution form in space, composed of a large number of pixels. These pixels carry different information and together depict the overall characteristics of the surface. Therefore, the mixing weight can be in matrix form, and for each pixel on the two-dimensional Gaussian surface, there is a corresponding mixing weight value.

[0169] In addition, the coordinates of the pixel point u and the center position u of the k-th two-dimensional Gaussian surface are k and the two-dimensional covariance Σ′ k Substituting this two-dimensional probability density, the corresponding probability density value can be calculated. Therefore, the mixing weight is the result of multiplying the probability density value of the two-dimensional Gaussian surface at the pixel point with the opacity of the two-dimensional Gaussian surface. It is used to measure the weight ratio that the k-th two-dimensional Gaussian surface should have when multiple two-dimensional Gaussian surfaces participate together.

[0170] Step 920: Obtain the marginal probability density and conditional probability density corresponding to the pixel point, and obtain the weighted contribution value according to the product of the marginal probability density, the conditional probability density and the mixing weight.

[0171] In one embodiment, for a pixel u, its marginal probability density is p k (t), is the marginal probability density of time, belongs to the three-dimensional Gaussian distribution, and describes the probability distribution of the k-th two-dimensional Gaussian surface at time t. The conditional probability density is expressed as p k (u|t) is a one-dimensional Gaussian distribution, which describes the probability distribution of the k-th two-dimensional Gaussian surface at pixel u at a given time t.

[0172] In one embodiment, the weighted contribution value is expressed as:

[0173] p k (t)p k (u|t)*α k

[0174] Step 930: Based on the order of the depth values, obtain the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtain the cumulative weight value according to the weighted contribution values.

[0175] In one embodiment, for the k-th two-dimensional Gaussian surface, the previous two-dimensional Gaussian surface is the two-dimensional Gaussian surface j, and the value of j is 1 to k-1. At this time, the weighted contribution value of the two-dimensional Gaussian surface j is expressed as:

[0176] p j (t)p j (u|t)α j

[0177] The accumulated weight value is expressed as:

[0178]

[0179] The above accumulated weight values ​​are used to adjust the contribution of each two-dimensional Gaussian surface to the final color, ensuring that the sum of the contributions of all two-dimensional Gaussian surfaces is 1.

[0180] Step 940: Obtain the pixel rendering color according to the product of the reference contribution value, the color parameter and the accumulated weight value.

[0181] In one embodiment, the reference contribution value, the color parameter, and the accumulated weight value are multiplied and then accumulated to obtain the pixel rendering color, which is expressed as:

[0182]

[0183] Where c(u,t) represents the pixel rendering color corresponding to pixel u at processing time t, N represents the number of two-dimensional Gaussian surfaces, and c k Represents the color parameters of the k-th two-dimensional Gaussian surface.

[0184] In one embodiment, after obtaining the pixel rendering colors of the two-dimensional Gaussian surface according to the above process, a rendering image corresponding to each viewing angle can be generated based on these pixel rendering colors. Specifically, these pixel rendering colors are similar to tiny "color units", which together construct the subsequent rendering images of the corresponding viewing angles. For each viewing angle, there is a specific observation range and angle, and the pixel rendering colors that have been determined on the two-dimensional Gaussian surface need to be reasonably arranged and integrated according to their positions in space and the spatial relationship corresponding to the viewing angle.

[0185] For example, when the target scene is an indoor scene image containing multiple chairs and tables, different objects in the two-dimensional Gaussian surface corresponding to a certain perspective have different pixel rendering colors. For example, the pixel rendering color corresponding to the table may be a wooden brown tone, while the pixel rendering color corresponding to the chair may be another color that matches it. When generating a rendered image, the pixel rendering colors corresponding to these different objects can be orderly combined according to their spatial layout relationship at that perspective, and finally a complete rendered image corresponding to that perspective is constructed.

[0186] In one embodiment, referring to Fig.11 , Fig.11 is a schematic diagram of a rendered image provided by an embodiment of the present application. Fig.11 It can be seen that the foreground object segmentation and reconstruction method provided in the embodiment of the present application can completely extract the foreground object, and perform a suitable rendering process on it to obtain a corresponding rendered image.

[0187] Step 140: Obtain a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each viewing angle, and generate a foreground target reconstruction result according to the foreground target rendering sequences at all acquisition moments.

[0188] In one embodiment, the above process generates corresponding rendering images for each video frame of each perspective at the processing moment. It can be seen that for each acquisition moment, it contains a foreground target rendering sequence composed of rendering images of different perspectives. In this case, after obtaining the foreground target rendering sequence at each acquisition moment, the foreground target reconstruction result can be generated. In addition, the sliced ​​Gaussian point cloud obtained based on the foreground target segmentation mask can be associated with the corresponding rendering image. When displayed, the corresponding three-dimensional Gaussian points are displayed according to the color of each pixel in the rendering image, and the foreground target reconstruction result is displayed in the form of three-dimensional Gaussian points.

[0189] It is understandable that after obtaining the reconstruction result of the foreground object, the display operation can be performed according to actual needs in the display stage. For example, it can be displayed according to different acquisition times, showing the rendering images corresponding to different perspectives at a certain acquisition time; it can also be displayed according to different perspectives, showing the rendering images corresponding to a certain perspective at different acquisition times.

[0190] In a specific embodiment, referring to Fig.12 , Fig.12 This is a schematic diagram of the foreground target reconstruction result provided by the embodiment of the present application. Fig.12 In the figure, a certain perspective of the foreground target at different acquisition times can be selected according to actual needs. It can be seen that each rendered image is a foreground target rendered based on the foreground target segmentation mask.

[0191] The following describes the training process of the spatiotemporal Gaussian splash model in the embodiment of the present application.

[0192] In one embodiment, referring to Fig.13 , Fig.13 : is a flow chart of the training process of the spatiotemporal Gaussian splash model provided in an embodiment of the present application, which at least includes the following steps:

[0193] Step 1310: Obtain multiple training samples.

[0194] In one embodiment, the training samples at least include a training mask, a training Gaussian point cloud and a label image for each view corresponding to each training acquisition moment. The training mask is obtained in the same manner as the foreground object segmentation mask, the training Gaussian point cloud is obtained in the same manner as the initial Gaussian point cloud, and the label image corresponds to the rendered image.

[0195] Step 1320: Input the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering to obtain a training rendered image.

[0196] In one embodiment, the process of rendering by the spatiotemporal Gaussian splash model is the same as the above-mentioned reasoning process, which will not be repeated here. The calculation process of the training rendering image is consistent with that of the rendering image. In addition, in the rendering process, each training rendering image corresponds to a two-dimensional Gaussian training surface, and the calculation method of the two-dimensional Gaussian training surface is consistent with that of the two-dimensional Gaussian surface.

[0197] Step 1330: During the rendering process, generate a depth consistency loss value based on the mixing weights and training depth values ​​of each two-dimensional Gaussian training surface.

[0198] In one embodiment, in the process of obtaining a two-dimensional Gaussian training surface, the scale of the scaling matrix of the sliced ​​Gaussian point cloud in the Z-axis direction is directly set to 0. After completing this operation, when calculating the derivative related to the local Z-axis from the covariance matrix, the result will be zero in view of the fact that the scale has changed. In the subsequent optimization process, the photometric loss itself loses the ability to exert influence on the local Z-axis. Photometric loss generally plays an important role in measuring the differences in photometric features of images or data. When it cannot affect the local Z-axis, it is possible that during the optimization and adjustment of the data, the characteristics related to the local Z-axis cannot be reasonably corrected and improved by relying on photometric loss, thereby affecting the overall data accuracy and model training effect.

[0199] To this end, the embodiment of the present application designs a self-supervised depth consistency loss value, which requires the local Z axis to approach the normal calculated in the depth map generated by Gaussian point rendering. The normal in the depth map can reflect the directional characteristics of the object surface at different positions, and requiring the local Z axis to approach it is actually to constrain and guide the local Z axis with the help of the spatial direction information contained in the normal in the depth map.

[0200] In one embodiment, referring to Fig.14 , Fig.14 It is a flowchart of generating a depth consistency loss value based on the mixing weight and training depth value of each two-dimensional Gaussian training surface provided by an embodiment of the present application, which specifically includes the following steps:

[0201] Step 1410: For the two-dimensional Gaussian training surface, based on the order of training depth values, for all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all the previous two-dimensional Gaussian surfaces, obtain a first intermediate value according to the difference between one and the training mixing weight, accumulate the first intermediate value and multiply it by the training mixing weight of the two-dimensional Gaussian training surface to obtain the total training mixing weight, select the maximum value from the total training mixing weight to obtain the maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value.

[0202] In one embodiment, each two-dimensional Gaussian training surface is obtained from a three-dimensional Gaussian point cloud, so the training depth value corresponding to each three-dimensional Gaussian point cloud is calculated according to the above depth value calculation process, and reference is made to Figure 7 As shown in the figure, the two-dimensional Gaussian training surfaces are sorted according to the order of the training depth values. For the i-th two-dimensional Gaussian training surface, the training mixture weights of all previous two-dimensional Gaussian surfaces are expressed as: α j ′, where j ranges from 1 to i-1.

[0203] For the j-th two-dimensional Gaussian surface, the first intermediate value is obtained according to the difference between one and the training mixture weight, and the first intermediate value is expressed as:

[0204] (1-α j ′)

[0205] Then, the first intermediate value is accumulated and multiplied by the training mixture weight of the two-dimensional Gaussian training surface to obtain the total training mixture weight, the total training mixture weight ω j It is expressed as:

[0206]

[0207] Among them, m represents the m-th two-dimensional Gaussian surface corresponding to the j-th previous two-dimensional Gaussian surface, α m Represents the trained mixture weights of the m-th two-dimensional Gaussian surface.

[0208] Next, from the training total mixture weight ω j Select the maximum value from the above to get the maximum mixing weight. Assuming that the maximum value is the j′th two-dimensional Gaussian surface, the maximum mixing weight ω j′ , expressed as:

[0209] j′=argmax(ω j′ )

[0210] Then, the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight is obtained as the maximum training depth value, and the maximum training depth value is expressed as z j′ .

[0211] Step 1420: Obtain the training weighted contribution value and the training depth value of the two-dimensional Gaussian training surface, calculate a first product of the training weighted contribution value and the training depth value, and calculate a second product of the training weighted contribution value and the maximum training depth value.

[0212] In one embodiment, the first product is expressed as:

[0213] ω i z i

[0214] The second product is expressed as:

[0215] ω i z j′

[0216] Step 1430: Accumulate the first product to obtain the second intermediate value, and accumulate the second product to obtain the third intermediate value, obtain the difference between the second intermediate value and the third intermediate value, and calculate the product of the maximum mixing weight and the difference to obtain the depth consistency loss value.

[0217] In one embodiment, the second intermediate value is expressed as:

[0218] ∑ i ω i z i

[0219] The third intermediate value is expressed as:

[0220] ∑ i ω i z j′

[0221] Depth consistency loss value It is expressed as:

[0222]

[0223] Step 1340: Calculate the monocular normal regularization loss value according to the normal vectors corresponding to the training rendered image and the label image respectively.

[0224] In one embodiment, a pseudo-normal map estimated by the depth gradient of the rendered image can be used to supervise the training process. However, since there will be noise in the depth map obtained by rendering the image, especially in complex scenes, the scene contains many different types of objects, complex lighting conditions, and diverse spatial structures. These factors will make the noise in the rendered depth map more obvious. The presence of this noise will cause deviations in the pseudo-normal map estimated based on the depth gradient, and then produce artifacts in subsequent applications. The appearance of artifacts will destroy the quality of the final generated image, causing some abnormal parts of the image that are not in line with the actual situation and visually abrupt. Therefore, the embodiment of the present application uses the normal vectors corresponding to the existing training rendered image and the label image to calculate the monocular normal regularization loss value for normal constraint.

[0225] In one embodiment, referring to Fig.15 , Fig.15 It is a flowchart of calculating the monocular normal regularization loss value according to the normal vectors corresponding to the training rendered image and the label image provided by the embodiment of the present application, which specifically includes the following steps:

[0226] Step 1510: Obtain the corresponding training slice Gaussian point cloud during the rendering process, and obtain the monocular estimated normal vector corresponding to the training slice Gaussian point cloud.

[0227] In one embodiment, during the rendering process, each training rendered image has a corresponding training slice Gaussian point cloud, and at this time, a monocular estimated normal vector corresponding to the training slice Gaussian point cloud is obtained.

[0228] Step 1520: Generate a pseudo normal vector based on the training rendered image.

[0229] In one embodiment, a depth map corresponding to the training rendered image is obtained, and the depth map reflects the depth position of the scene objects corresponding to each pixel in the training rendered image in space, that is, the distance between each pixel and the observation point of the corresponding viewing angle (such as the position of the camera).

[0230] Step 1530: Calculate the monocular normal regularization loss value based on the monocular estimated normal vector and the pseudo normal vector.

[0231] In one embodiment, the monocular normal regularization loss value is the L1 loss value, expressed as:

[0232]

[0233] in, represents the monocular estimated normal vector, and N represents the pseudo normal vector.

[0234] Step 1350: For two consecutive training acquisition moments, a time domain regularization loss value is calculated based on the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing term loss value is calculated based on the corresponding position parameters and rotation parameters of the two-dimensional Gaussian training surface.

[0235] In one embodiment, in order to reduce temporal jitter while maintaining the quality of geometric surface rendering, temporal smoothing regularization is introduced to finely balance the updates of multi-view geometric priors and Gaussian properties to maintain spatiotemporal consistency. Fig.16 , Fig.16 This is a flowchart of calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments provided by an embodiment of the present application, which specifically includes the following steps:

[0236] Step 1610: At the current training acquisition moment, obtain a preset number of adjacent Gaussian points corresponding to the central Gaussian point where the center position of the two-dimensional Gaussian training surface is located, obtain the current color parameter, current opacity parameter and current target scaling value of the central Gaussian point from the surface parameters, and obtain the total training mixture weight of the two-dimensional Gaussian training surface.

[0237] In one embodiment, with respect to one of the two-dimensional Gaussian training surfaces, it presents a specific distribution pattern in space, and the central Gaussian point at the corresponding central position is usually in a key position, and the Gaussian points adjacent to it together constitute the "associated area" around it. Based on this, the embodiment of the present application determines a preset number, such as 5, 10, etc., and selects some Gaussian points that are close to the central Gaussian point from the entire two-dimensional Gaussian training surface as adjacent Gaussian points according to a certain distance metric or spatial distribution rule. These adjacent Gaussian points are helpful in analyzing the characteristics of the local area where the central Gaussian point is located and its relationship with the surrounding environment.

[0238] In one embodiment, assuming that the current training acquisition time is t, the current color parameter c of the central Gaussian point is obtained from the surface parameters of the corresponding two-dimensional Gaussian training surface. i,t , current opacity parameter σ i,t and the current target scaling value r i,t , where the target scaling value is the target scaling matrix of the i-th two-dimensional Gaussian training surface at the training acquisition time t. At the same time, the total training mixture weight ω of the two-dimensional Gaussian training surface is obtained i,t .

[0239] Step 1620: Obtain reference color parameters, reference opacity parameters, and reference target scaling values ​​of adjacent Gaussian points from the surface parameters of the corresponding viewing angle at the previous training acquisition moment.

[0240] In one embodiment, the previous training acquisition time is t-1, at which time, the reference color parameter c near the Gaussian point is obtained. i,t-1 , reference opacity parameter σ i,t-1 and the reference target scaling value r i,t-1 .

[0241] Step 1630: Calculate the norm of the color difference between the current color parameter and the reference color parameter, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter, and the norm of the scaling difference between the current target scaling value and the reference target scaling value.

[0242] In one embodiment, the norm of the color difference between the current color parameter and the reference color parameter is: k = c, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter is k = σ, the norm of the scaling difference between the current target scaling value and the reference target scaling value is k=r.

[0243] Step 1640: Three products are obtained by multiplying the total weight of the training mixture with the norm of the color difference, the norm of the opacity difference, and the norm of the scaling difference, and the three products corresponding to each adjacent Gaussian point are accumulated to obtain the time domain regularization loss value.

[0244] In one embodiment, the time domain regularization loss value It is expressed as:

[0245]

[0246] It can be understood that, since the time domain regularization loss value includes color parameters, the specific parameters of the spherical harmonic function can be adjusted through the loss value.

[0247] In one embodiment, to address unnatural artifacts caused by drastic changes in object motion, a motion smoothing term for fine-grained Gaussian motion properties is introduced. The motion smoothing term generates a local deformation that is as rigid as possible in the space-time domain to constrain consistent four-dimensional Gaussian motion. Fig.17 , Fig.17 The present invention provides a flowchart of calculating the motion smoothness loss value according to the position parameters and rotation parameters corresponding to the two-dimensional Gaussian training surface, which specifically includes the following steps:

[0248] Step 1710: At the current training acquisition moment, obtain the current position parameters and current rotation parameters of the central Gaussian point from the surface parameters, and obtain the current reference position parameters of the adjacent Gaussian points.

[0249] In one embodiment, as described above, at the training acquisition time t, the coordinates of the center position of the central Gaussian point are obtained as the current position parameter p i,t , and obtain the current rotation parameter q according to the rotation matrix of the two-dimensional Gaussian training surface i,t And, in this way, the current reference position parameter p of the neighboring Gaussian point is obtained j,t .

[0250] Step 1720: Obtain reference position parameters and reference rotation parameters of adjacent Gaussian points, and central reference position parameters corresponding to the central Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment.

[0251] In one embodiment, for the previous training acquisition time t-1, the reference position parameter p of the adjacent Gaussian point is obtained. j,t-1 and the reference rotation parameter q j,t-1 , and the central reference position parameter p corresponding to the central Gaussian point i,t-1 .

[0252] Step 1730: For each adjacent Gaussian point, obtain the fourth intermediate value according to the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, obtain the fifth intermediate value according to the difference between the reference position parameter and the center reference position parameter, and obtain the sixth intermediate value according to the difference between the current reference position parameter and the current position parameter.

[0253] In one embodiment, the fourth intermediate value, the fifth intermediate value, and the sixth intermediate value are respectively expressed as:

[0254]

[0255] p j,t-1 -p i,t-1

[0256] p j,t -p i,t

[0257] Step 1740: Convert the fourth intermediate value into a rotated intermediate matrix, subtract the sixth intermediate value from the product of the fifth intermediate value and the rotated intermediate matrix to obtain the seventh intermediate value, and multiply the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain the eighth intermediate value.

[0258] In one embodiment, the rotation intermediate matrix is ​​obtained according to the SO3 operation, which converts the quaternion into a rotation matrix, so the seventh intermediate value is expressed as:

[0259]

[0260] The eighth intermediate value is expressed as:

[0261]

[0262] Step 1750: Accumulate the eighth intermediate values ​​corresponding to all adjacent Gaussian points to obtain the motion loss item corresponding to the current two-dimensional Gaussian training surface training, and accumulate all motion loss items to obtain the motion smoothing item loss value at the current training acquisition moment.

[0263] In one embodiment, the motion loss term is expressed as:

[0264]

[0265] Motion smoothness loss value It is expressed as:

[0266]

[0267] Step 1360: Obtain a pixel loss value and a structure loss value based on the training rendered image and the label image, and calculate a photometric difference loss value based on the pixel loss value and the structure loss value.

[0268] In one embodiment, the pixel loss value is calculated based on the mean absolute error, and the structural loss value is calculated based on the D-SSIM calculation method. The D-SSIM calculation method is used to evaluate the degree of structural similarity between two images. It comprehensively considers multiple dimensions such as image brightness, contrast, and structural information to measure the similarity between the images in terms of overall structural features.

[0269] Therefore, the photometric difference loss value It is expressed as:

[0270]

[0271] Where I′ represents the training rendered image, I represents the label image, λ1 represents the preset weight, L1(I′,I) represents the pixel loss value, and L DSSIM (I′,I) represents the structural loss value.

[0272] In one embodiment, considering that only the dynamic foreground is modeled, some regularization terms may introduce background artifacts. Therefore, the embodiment of the present application also adds an additional mask loss value to minimize the training mask M in the training process to the input foreground mask label M * middle.

[0273] Specifically, the pixel point in the training mask M is recorded as r, and for the training acquisition time t, the pixel point corresponding to the pixel point r on the i-th two-dimensional Gaussian training surface is recorded as u ir , for the pixel points on the two-dimensional Gaussian training surface, obtain the corresponding mixing weight value α(u ir ), therefore, the weight value of the total training mixture weight corresponding to the pixel point is calculated, and after accumulating the values ​​corresponding to each two-dimensional Gaussian training surface, the training mask parameter value of the pixel point r of the training mask M is obtained.

[0274] Among them, the training mask parameter value M(r) is expressed as:

[0275]

[0276] Among them, u jr Represents the pixel u on the two-dimensional Gaussian training surface before the i-th two-dimensional Gaussian training surface ir The corresponding pixel point.

[0277] Therefore, the mask loss value It is expressed as:

[0278]

[0279] Among them, M * (r) represents the pixel r in the foreground mask label M *The pixel value in .

[0280] It can be understood that the target segmentation model can be jointly trained by introducing the mask loss value.

[0281] Step 1370: Select at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness term loss value as the target loss value, and obtain the total loss value according to the target loss value and the photometric difference loss value.

[0282] In one embodiment, the depth consistency loss value is given in the above process Monocular normal regularization loss value Mask loss value Temporal regularization loss value and motion smoothness loss Next, at least one of them is selected as the target loss value, and then the target loss value and the luminosity difference loss value are calculated. Get the total loss value.

[0283] Assuming all of the above are selected as target loss values, the total loss value is It is expressed as:

[0284]

[0285] Among them, α, β, δ, η, Indicates the weights corresponding to different target loss values, which can be set according to actual needs.

[0286] In one embodiment, referring to Fig.18 , Fig.18 It is a schematic diagram of the application effect of different target loss values ​​in the embodiment of the present application. Fig.18 The figure shows the depth consistency loss value selected when selecting "people and footballs" as foreground targets from the input video frame. Monocular normal regularization loss value Mask loss Schematic diagram of the segmentation obtained as a geometric consistency constraint, selecting the time domain regularization loss value and motion smoothness loss As a schematic diagram of the segmentation obtained by the spatiotemporal consistency constraint, and the segmentation obtained by the collaborative constraint by selecting all of the above as the target loss value, the training rendered image is binarized and compared with the training mask. It can be seen that whether it is the geometric consistency constraint, the spatiotemporal consistency constraint or all the collaborative constraints, good segmentation results can be obtained, and the training mask and foreground mask labels are relatively close.

[0287] Step 1380: Adjust the model parameters of the spatiotemporal Gaussian splash model according to the total loss value until the iteration termination condition is reached to obtain a trained spatiotemporal Gaussian splash model.

[0288] In one embodiment, the total loss value covers multiple aspects of loss, and these different types of losses each reflect the degree of deviation between the model output result and the expected result from the corresponding angle. Subsequently, the model parameters of the spatiotemporal Gaussian splash model are adjusted based on the total loss value. For example, the gradient information corresponding to each parameter is calculated based on the total loss value, and then the model parameters are updated in the opposite direction of the gradient, so that the parameters of the spatiotemporal Gaussian splash model are adjusted in the direction that can reduce the total loss value. This adjustment process will continue, and the total loss value will be recalculated after each adjustment to observe whether the model is improving in a better direction. Until the iteration termination condition is met, the iteration termination condition can usually be set to a total loss value less than a predetermined threshold, which means that the performance of the model has reached an acceptable accuracy range; or set to a certain number of iterations to prevent overtraining and other conditions. When the iteration termination condition is met, it means that the spatiotemporal Gaussian splash model has been fully trained and optimized, and the trained spatiotemporal Gaussian splash model is obtained at this time.

[0289] The foreground target segmentation and reconstruction method provided in the embodiment of the present application realizes accurate and high-quality segmentation and appearance reconstruction of the selected target object in four-dimensional space. It uses two-dimensional plane Gaussian to characterize the geometric shape of the target object of interest, and optimizes the Gaussian surface with the help of geometric consistency constraints such as depth consistency loss value, monocular normal regularization loss value, mask loss value, time domain regularization loss value and motion smoothing term loss value, so as to obtain an accurate geometric surface. At the same time, under the input condition of multi-viewpoint synchronous video, fine-grained object segmentation is achieved by relying on the structure, texture and geometric consistency between each view, and the continuity constraints of time and space are used to enhance geometric details and continuity. In addition, a prompt automatic video object segmentation model is introduced, which supports text prompts as input, and can carry out selection and segmentation optimization for any category of foreground target objects.

[0290] In one embodiment, referring to Fig.19 , Fig.19 It is a schematic diagram of the overall process of the foreground object segmentation and reconstruction method provided in the embodiment of the present application.

[0291] At multiple acquisition moments, multiple initial acquisition data corresponding to multiple perspectives are obtained respectively, all video frames are input into the target segmentation model for target selection, the video frames and prompt text are input into the target segmentation model for target selection, at least one foreground target is obtained, at least one foreground target is superimposed, and a foreground target segmentation mask corresponding to the video frame is obtained. The foreground target segmentation mask and the corresponding initial Gaussian point cloud are associated as input data.

[0292] For each processing moment, the input data corresponding to all viewpoints are input into the spatiotemporal Gaussian splash model for data processing. The input data is divided into time slices based on the foreground target segmentation mask to obtain the slice Gaussian point cloud corresponding to each time slice. Then, the center position, color parameters, opacity parameters, rotation matrix and target scaling matrix are used as surface parameters, where the color parameters and opacity parameters are appearance parameters, and the center position, rotation matrix and target scaling matrix are space parameters. The two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and the foreground target segmentation mask and the two-dimensional Gaussian surface are combined for rendering to obtain the rendered image corresponding to each viewpoint. Finally, the foreground target rendering sequence of the processing moment is obtained according to the rendered image corresponding to each viewpoint, and the foreground target reconstruction result is generated according to the foreground target rendering sequence of all acquisition moments. The foreground target reconstruction result is illustrated in the figure in the form of three-dimensional Gaussian points.

[0293] In addition, the figure also illustrates the training process. The training process uses at least the depth consistency loss value, the monocular normal regularization loss value, the mask loss value, and the photometric difference loss value to obtain the total loss value for model training.

[0294] The technical solution provided by the embodiment of the present application is to obtain the initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, input all video frames into the target segmentation model for target selection, generate the foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data. For the processing moment, the input data corresponding to all perspectives are input into the spatiotemporal Gaussian splash model for data processing, the input data is divided into time slices, the sliced ​​Gaussian point cloud corresponding to each time slice is obtained, the two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained, and the rendering image corresponding to each perspective is obtained according to the multiple two-dimensional Gaussian surfaces. The foreground target rendering sequence at the processing moment is obtained according to the rendering image corresponding to each perspective, and the foreground target reconstruction result is generated according to the foreground target rendering sequence at all acquisition moments. The embodiment of the present application directly inputs the video frame into the target segmentation model to generate the foreground target segmentation mask, without artificially presetting too much prior knowledge involving the specific form, position, etc. of the target. In addition, the spatiotemporal Gaussian splash model is used to divide the input data into time slices according to the time dimension characteristics of the input data itself, without knowing the prior knowledge such as the specific change mode of each target at different moments in advance. In this way, while further reducing the dependence on external prior information, the stability of the time series can also be guaranteed. Moreover, after completing the time slice division, the spatiotemporal Gaussian splash model can track the dynamic changes of complex geometric shapes in different time slices. In the stage of generating rendered images, the shape and surface details of the object can be accurately and adaptively fitted in combination with the two-dimensional Gaussian surface, thereby improving the reconstruction accuracy. Furthermore, the spatiotemporal Gaussian splash model analyzes the motion trajectory and morphological changes of objects in time slices from different perspectives, and uses the temporal coherence and spatial correlation in the data to adjust the inconsistencies that may occur from different perspectives, so that the final generated rendering images corresponding to each perspective are more consistent in key elements such as the position and shape of the object.

[0295] The present application also provides a foreground object segmentation and reconstruction device, which can implement the foreground object segmentation and reconstruction method. Fig. 20 , the device comprises:

[0296] The acquisition module 2010 is used to respectively acquire initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, wherein the initial acquisition data includes acquisition data corresponding to each acquisition moment, and the acquisition data includes video frames and initial Gaussian point clouds.

[0297] Mask calculation module 220: used to input all video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data.

[0298] Rendering module 2030: used to use the acquisition moments as processing moments one by one, and for the processing moments, input the input data corresponding to all viewpoints into the space-time Gaussian splash model for data processing, divide the input data into time slices, obtain the slice Gaussian point cloud corresponding to each time slice, obtain the two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain the rendering image corresponding to each viewpoint based on multiple two-dimensional Gaussian surfaces.

[0299] Reconstruction module 2040: used to obtain a foreground target rendering sequence at a processing moment according to the rendering image corresponding to each viewing angle, and generate a foreground target reconstruction result according to the foreground target rendering sequence at all acquisition moments.

[0300] The specific implementation of the foreground object segmentation and reconstruction device of this embodiment is basically the same as the specific implementation of the foreground object segmentation and reconstruction method described above, and will not be repeated here.

[0301] The present application also provides an electronic device, including:

[0302] at least one memory;

[0303] at least one processor;

[0304] at least one program;

[0305] The program is stored in the memory, and the processor executes the at least one program to implement the foreground object segmentation and reconstruction method of the present application. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0306] See also Fig.21 , Fig.21 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0307] The processor 2101 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0308] The memory 2102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 2102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 2102, and the processor 2101 calls and executes the foreground object segmentation and reconstruction method of the embodiment of the present application;

[0309] Input / output interface 2103, used to implement information input and output;

[0310] Communication interface 2104, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0311] A bus 2105 that transmits information between the various components of the device (e.g., the processor 2101, the memory 2102, the input / output interface 2103, and the communication interface 2104);

[0312] The processor 2101 , the memory 2102 , the input / output interface 2103 and the communication interface 2104 are connected to each other in communication within the device via the bus 2105 .

[0313] An embodiment of the present application further provides a storage medium, which is a storage medium storing a computer program, and when the computer program is executed by a processor, the foreground target segmentation and reconstruction method described above is implemented.

[0314] As a non-transitory storage medium, the memory can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0315] The foreground target segmentation and reconstruction method, device, equipment and storage medium proposed in the embodiment of the present application obtains the initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, inputs all video frames into the target segmentation model for target selection, generates the foreground target segmentation mask corresponding to each video frame, and associates the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data. For the processing moment, the input data corresponding to all perspectives are input into the spatiotemporal Gaussian splash model for data processing, the input data is divided into time slices, the sliced ​​Gaussian point cloud corresponding to each time slice is obtained, the two-dimensional Gaussian surface corresponding to each sliced ​​Gaussian point cloud is obtained, and the rendering image corresponding to each perspective is obtained according to the multiple two-dimensional Gaussian surfaces. The foreground target rendering sequence at the processing moment is obtained according to the rendering image corresponding to each perspective, and the foreground target reconstruction result is generated according to the foreground target rendering sequence at all acquisition moments. The embodiment of the present application directly inputs the video frame into the target segmentation model to generate the foreground target segmentation mask, without the need to manually pre-set too much prior knowledge involving the specific form, position and other aspects of the target. In addition, the space-time Gaussian splash model is used to divide the input data into time slices according to the time dimension characteristics of the input data itself, without the need to know the specific change patterns of each target at different times and other prior knowledge in advance. In this way, while further reducing the dependence on external prior information, the stability of the time series can also be guaranteed. Moreover, after completing the time slice division, the space-time Gaussian splash model can track the dynamic changes of complex geometric shapes in different time slices. In the stage of generating rendered images, the shape and surface details of the object can be accurately and adaptively fitted in combination with the two-dimensional Gaussian surface, thereby improving the reconstruction accuracy. Furthermore, the space-time Gaussian splash model analyzes the motion trajectory and morphological changes of objects in time slices from different perspectives, and adjusts the inconsistencies that may occur from different perspectives with the help of the temporal coherence and spatial correlation in the data, so that the rendered images corresponding to each perspective generated in the end are more consistent in terms of key elements such as the position and shape of the object.

[0316] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0317] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0318] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0319] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0320] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0321] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0322] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0323] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0324] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0325] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0326] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A foreground object segmentation and reconstruction method, characterized in that: Executed by a trained spatiotemporal Gaussian splash model, the method comprises: At multiple acquisition moments, respectively acquiring initial acquisition data corresponding to multiple perspectives, the initial acquisition data including acquisition data corresponding to each of the acquisition moments, the acquisition data including video frames and initial Gaussian point clouds; Inputting all the video frames into the target segmentation model for target selection, generating a foreground target segmentation mask corresponding to each video frame, and associating the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data; The acquisition moments are used as processing moments one by one. For the processing moments, the input data corresponding to all viewing angles are input into the space-time Gaussian splash model for data processing, the input data is divided into time slices to obtain a slice Gaussian point cloud corresponding to each time slice, a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud is obtained, and a rendering image corresponding to each viewing angle is obtained according to a plurality of the two-dimensional Gaussian surfaces; A foreground target rendering sequence at the processing moment is obtained according to the rendered image corresponding to each of the viewing angles, and a foreground target reconstruction result is generated according to the foreground target rendering sequence at all the acquisition moments.

2. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The step of inputting all the video frames into the target segmentation model to select targets and generating a foreground target segmentation mask corresponding to each video frame includes: Obtaining prompt texts corresponding to all the video frames; The video frame and the prompt text are input into a target segmentation model for target selection to obtain at least one foreground target, and the at least one foreground target is superimposed to obtain a foreground target segmentation mask corresponding to the video frame.

3. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The step of dividing the input data into time slices to obtain a slice Gaussian point cloud corresponding to each time slice includes: Acquire a time slice, and obtain an initial slice point cloud based on the time slice and the initial Gaussian point cloud; Using the foreground object segmentation mask, a mask operation is performed on each of the initial slice point clouds to obtain the corresponding slice Gaussian point cloud; The corresponding depth value is determined according to the relationship between the center position of the slice Gaussian point cloud and the camera position of the viewing angle, and the slice Gaussian point cloud is sorted based on the order of the depth values.

4. The foreground object segmentation and reconstruction method according to claim 3, characterized in that: The step of obtaining a two-dimensional Gaussian surface corresponding to each of the sliced ​​Gaussian point clouds comprises: Obtaining the center position, color parameter, opacity parameter, rotation matrix and initial scaling matrix corresponding to the sliced ​​Gaussian point cloud; Selecting a specific coordinate axis, and adjusting scale information of the initial scaling matrix based on the specific coordinate axis to obtain a target scaling matrix; At least the center position, the color parameter, the opacity parameter, the rotation matrix and the target scaling matrix are used as parameters of the two-dimensional Gaussian surface.

5. The foreground object segmentation and reconstruction method according to claim 4, characterized in that: The step of obtaining a rendered image corresponding to each viewing angle according to the plurality of two-dimensional Gaussian surfaces comprises: For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces; The rendered image corresponding to the viewing angle is obtained based on the pixel rendering color of the two-dimensional Gaussian surface.

6. The foreground object segmentation and reconstruction method according to claim 5, characterized in that: For each pixel point in the two-dimensional Gaussian surface, obtaining a corresponding pixel rendering color according to a plurality of the two-dimensional Gaussian surfaces includes: Obtaining a mixing weight of the two-dimensional Gaussian surface; Obtaining the marginal probability density and the conditional probability density corresponding to the pixel point, and obtaining a weighted contribution value according to the product of the marginal probability density, the conditional probability density and the mixing weight; Based on the order of the depth values, obtaining the weighted contribution values ​​of all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian surface, and obtaining a cumulative weight value according to the weighted contribution values; The pixel rendering color is obtained according to the product of the weighted contribution value, the color parameter and the accumulated weight value.

7. The foreground object segmentation and reconstruction method according to claim 6, characterized in that: The obtaining of the mixing weight of the two-dimensional Gaussian surface comprises: Obtaining the three-dimensional probability density and three-dimensional covariance of the corresponding slice Gaussian point cloud; Obtaining a camera transformation matrix and a local affine transformation matrix corresponding to the viewing angle; Calculating a two-dimensional covariance according to the camera transformation matrix, the local affine transformation matrix and the three-dimensional covariance, and performing a two-dimensional projection on the three-dimensional probability density based on the two-dimensional covariance to obtain a two-dimensional probability density; The mixing weight is obtained according to the two-dimensional probability density and the corresponding opacity parameter.

8. The foreground object segmentation and reconstruction method according to claim 1, characterized in that: The training process of the spatiotemporal Gaussian splash model at least includes: Acquire a plurality of training samples, wherein the training samples at least include a training mask, a training Gaussian point cloud, and a label image for each of the viewing angles corresponding to each training acquisition moment; Inputting the training mask and the training Gaussian point cloud into the spatiotemporal Gaussian splash model for rendering processing to obtain a training rendered image, wherein the training rendered image corresponds to a two-dimensional Gaussian training surface; During the rendering process, generating a depth consistency loss value based on the mixing weight and the training depth value of each of the two-dimensional Gaussian training surfaces; A monocular normal regularization loss value is calculated according to normal vectors corresponding to the training rendered image and the label image respectively; For two consecutive training acquisition moments, a time domain regularization loss value is calculated according to the surface parameters of the two-dimensional Gaussian training surface, and a motion smoothing term loss value is calculated according to the position parameters and rotation parameters respectively corresponding to the two-dimensional Gaussian training surface; Obtaining a pixel loss value and a structure loss value according to the training rendered image and the label image, and calculating a photometric difference loss value according to the pixel loss value and the structure loss value; Select at least one of the depth consistency loss value, the monocular normal regularization loss value, the temporal regularization loss value, and the motion smoothness term loss value as a target loss value, and obtain a total loss value according to the target loss value and the photometric difference loss value; The model parameters of the spatiotemporal Gaussian splash model are adjusted according to the total loss value until an iteration termination condition is reached to obtain the trained spatiotemporal Gaussian splash model.

9. The foreground object segmentation and reconstruction method according to claim 8, characterized in that: The generating a depth consistency loss value based on the mixing weight and the training depth value of each of the two-dimensional Gaussian training surfaces comprises: For the two-dimensional Gaussian training surface, based on the order of training depth values, for all the two-dimensional Gaussian surfaces before the two-dimensional Gaussian training surface, obtain the training mixing weights of all the previous two-dimensional Gaussian surfaces, obtain a first intermediate value according to a difference with the training mixing weight, accumulate the first intermediate value and multiply it by the training mixing weight of the two-dimensional Gaussian training surface to obtain a total training mixing weight, select a maximum value from the total training mixing weight to obtain a maximum mixing weight, and obtain the training depth value of the two-dimensional Gaussian training surface corresponding to the maximum mixing weight as the maximum training depth value; Obtaining a training weighted contribution value and a training depth value of the two-dimensional Gaussian training surface, calculating a first product of the training weighted contribution value and the training depth value, and calculating a second product of the training weighted contribution value and the maximum training depth value; The first products are accumulated to obtain a second intermediate value, and the second products are accumulated to obtain a third intermediate value, a difference between the second intermediate value and the third intermediate value is obtained, and the product of the maximum mixing weight and the difference is calculated to obtain the depth consistency loss value.

10. The foreground object segmentation and reconstruction method according to claim 8, characterized in that: The monocular normal regularization loss value is calculated according to the normal vectors corresponding to the training rendered image and the label image, including: Obtain the training slice Gaussian point cloud corresponding to the rendering process, and obtain the monocular estimated normal vector corresponding to the training slice Gaussian point cloud; generating a pseudo normal vector according to the training rendered image; The monocular normal regularization loss value is calculated according to the monocular estimated normal vector and the pseudo normal vector.

11. The foreground object segmentation and reconstruction method according to claim 9, characterized in that: The step of calculating the time domain regularization loss value according to the surface parameters of the two-dimensional Gaussian training surface for two consecutive training acquisition moments includes: At the current training acquisition moment, obtain a preset number of adjacent Gaussian points corresponding to the central Gaussian point where the center position of the two-dimensional Gaussian training surface is located, obtain the current color parameter, current opacity parameter and current target scaling value of the central Gaussian point from the surface parameters, and obtain the total training mixture weight of the two-dimensional Gaussian training surface; Obtaining the reference color parameter, reference opacity parameter and reference target scaling value of the adjacent Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment; Calculate the norm of the color difference between the current color parameter and the reference color parameter, the norm of the opacity difference between the current opacity parameter and the reference opacity parameter, and the norm of the scaling difference between the current target scaling value and the reference target scaling value; The time domain regularization loss value is obtained by multiplying the total weight of the training mixture with the norm of the color difference, the norm of the opacity difference and the norm of the scaling difference respectively, and accumulating the three products corresponding to each of the adjacent Gaussian points.

12. The foreground object segmentation and reconstruction method according to claim 11, characterized in that: The step of calculating the motion smoothness loss value according to the position parameters and rotation parameters respectively corresponding to the two-dimensional Gaussian training surface includes: At the current training acquisition moment, obtaining the current position parameter and current rotation parameter of the central Gaussian point from the surface parameters, and obtaining the current reference position parameter of the adjacent Gaussian point; Acquire the reference position parameters and reference rotation parameters of the adjacent Gaussian points, and the central reference position parameters corresponding to the central Gaussian point from the surface parameters of the corresponding viewing angle at the previous training acquisition moment; For each of the adjacent Gaussian points, a fourth intermediate value is obtained according to the product of the current rotation parameter and the inverse matrix of the reference rotation parameter, a fifth intermediate value is obtained according to the difference between the reference position parameter and the center reference position parameter, and a sixth intermediate value is obtained according to the difference between the current reference position parameter and the current position parameter; Convert the fourth intermediate value into a rotation intermediate matrix, subtract the sixth intermediate value from the product of the fifth intermediate value and the rotation intermediate matrix to obtain a seventh intermediate value, and multiply the norm of the seventh intermediate value by the total training mixture weight at the current training acquisition moment to obtain an eighth intermediate value; The eighth intermediate values ​​corresponding to all adjacent Gaussian points are accumulated to obtain the motion loss item corresponding to the current two-dimensional Gaussian training surface training, and all the motion loss items are accumulated to obtain the motion smoothing item loss value at the current training acquisition moment.

13. A foreground object segmentation and reconstruction device, characterized in that: Executed by a trained spatiotemporal Gaussian splash model, the device comprises: Acquisition module: used to obtain initial acquisition data corresponding to multiple perspectives at multiple acquisition moments, respectively, wherein the initial acquisition data includes acquisition data corresponding to each of the acquisition moments, and the acquisition data includes video frames and initial Gaussian point clouds; Mask calculation module: used to input all the video frames into the target segmentation model for target selection, generate a foreground target segmentation mask corresponding to each video frame, and associate the foreground target segmentation mask with the corresponding initial Gaussian point cloud as input data; Rendering module: used to use the acquisition moments as processing moments one by one, input the input data corresponding to all viewing angles into the space-time Gaussian splash model for data processing at the processing moments, divide the input data into time slices, obtain a slice Gaussian point cloud corresponding to each time slice, obtain a two-dimensional Gaussian surface corresponding to each slice Gaussian point cloud, and obtain a rendering image corresponding to each viewing angle according to multiple two-dimensional Gaussian surfaces; Reconstruction module: used for obtaining a foreground target rendering sequence at the processing moment according to the rendered image corresponding to each of the viewing angles, and generating a foreground target reconstruction result according to the foreground target rendering sequence at all the acquisition moments.

14. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the foreground object segmentation and reconstruction method according to any one of claims 1 to 12 when executing the computer program.

15. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the foreground object segmentation and reconstruction method according to any one of claims 1 to 12 is implemented.

Images