Model training method and device, scene reconstruction method and device, equipment and medium

Abstract

The invention provides a model training method and device, a scene reconstruction method and device, equipment and a medium. The method comprises the following steps: acquiring a training image of a scene to be reconstructed, wherein the training image comprises a target object and a background object; performing semantic segmentation processing on a target object and a background object in the training image to obtain a target object image and a background object image; the target object image and the background object image are input into a to-be-trained model for reconstruction, a training reconstruction result is obtained, the training reconstruction result is obtained according to three-dimensional Gaussian representation, and the three-dimensional Gaussian representation is obtained according to target object features and background object features; the target object feature and the background object feature are obtained according to the target object image and the background object image; and according to the contrast image and the training reconstruction result, the to-be-trained model is adjusted to obtain a scene reconstruction model, and the contrast image is obtained by superposing the target object image and the background object image. According to the method, the scene reconstruction efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a scene reconstruction method, a device, a equipment and a medium. Background Art

[0002] Reconstructing 3D scenes from unconstrained photo collections, such as photos of famous landmarks obtained from the Internet, is considered a cost-effective approach and has been widely studied. However, since unconstrained photos may contain transient occlusions, this poses a significant challenge to 3D scene reconstruction.

[0003] In the prior art, transient occluders in unconstrained photos are predicted by an appearance encoder and a transient mask predictor, but the appearance encoder and the transient mask predictor require a long training time, resulting in the problem of low scene reconstruction efficiency in the prior art. Summary of the invention

[0004] The present application provides a model training method, a scene reconstruction method, an apparatus, a device and a medium to solve the problem of low efficiency of scene reconstruction in the prior art.

[0005] In a first aspect, the present application provides a model training method, comprising:

[0006] Acquire a training image of a scene to be reconstructed, where the training image includes a target object and a background object;

[0007] Performing semantic segmentation processing on the target object and background object in the training image to obtain a target object image and a background object image;

[0008] Inputting the target object image and the background object image into the to-be-trained model for reconstruction to obtain a training reconstruction result, the training reconstruction result is obtained according to a three-dimensional Gaussian representation, the three-dimensional Gaussian representation is obtained according to the target object features and the background object features, the target object features and the background object features are obtained according to the target object image and the background object image;

[0009] According to the control image and the training reconstruction result, the model to be trained is adjusted to obtain a scene reconstruction model. The control image is obtained by superimposing the target object image and the background object image.

[0010] In this application, the target object image and the background object image are input into the model to be trained for reconstruction to obtain the training reconstruction result, including:

[0011] Inputting the target object image and the background object image into a feature extraction module in the model to be trained to obtain target object features and background object features;

[0012] Inputting background object features and a preset implicit cube map into a mapping module in the model to be trained to obtain a background cube map;

[0013] Input the background cube map and the target object features into the 3D Gaussian module to obtain a 3D Gaussian representation;

[0014] The 3D Gaussian representation and the background cubemap are fed into the rendering module to obtain the training reconstruction result.

[0015] In this application, the background cube map and the target object features are input into a 3D Gaussian module to obtain a 3D Gaussian representation, including:

[0016] Input the background cube map into the first encoder to obtain fine background features;

[0017] According to the characteristics of the target object, a plurality of three-dimensional Gaussian volumes representing the target object are determined;

[0018] For each three-dimensional Gaussian body, the unconstrained radiation feature is obtained according to the three-dimensional mean position in the three-dimensional Gaussian body;

[0019] Inputting the fine background features, target object features, unconstrained radiation features and initial spherical harmonic coefficients in the three-dimensional Gaussian volume into the first multi-layer perceptron to obtain updated spherical harmonic coefficients;

[0020] The initial spherical harmonic coefficients in the three-dimensional Gaussian body are updated according to the updated spherical harmonic coefficients to obtain a three-dimensional Gaussian representation.

[0021] In this application, the target object image and the background object image are input into the feature extraction module in the model to be trained to obtain the target object features and the background object features, including:

[0022] According to the second encoder, feature extraction is performed on the target object image to obtain target object features;

[0023] According to the third encoder, feature extraction is performed on the background object image to obtain background object features.

[0024] In this application, the training model is adjusted according to the reference image and the training reconstruction result to obtain a scene reconstruction model, including:

[0025] Determine a first loss function according to the control image and the training reconstruction result;

[0026] Determining a second loss function according to the target object features and the background object features;

[0027] According to the first loss function and the second loss function, the model to be trained is adjusted to obtain a scene reconstruction model.

[0028] In a second aspect, the present application provides a scene reconstruction method, comprising:

[0029] Obtaining a reconstructed image of a scene to be reconstructed, where the scene to be reconstructed is the scene to be reconstructed in the model training method provided in this application;

[0030] Performing semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image;

[0031] The target object image and the background object image are input into a scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed. The scene reconstruction model is the scene reconstruction model in the model training method provided in this application.

[0032] In this application, the method further comprises:

[0033] Obtaining target object features output by a second encoder in a scene reconstruction model;

[0034] Input the target object features and the preset background cube map into the 3D Gaussian module in the scene reconstruction model to obtain a 3D Gaussian representation;

[0035] The three-dimensional Gaussian representation and the preset background cube map are input into the rendering module in the scene reconstruction model to obtain the scene reconstruction result.

[0036] In a third aspect, the present application provides a model training device, comprising:

[0037] A first acquisition module is used to acquire a training image of a scene to be reconstructed, where the training image includes a target object and a background object;

[0038] The first segmentation module is used to perform semantic segmentation processing on the target object and the background object in the training image to obtain the target object image and the background object image;

[0039] A training module, used for inputting a target object image and a background object image into a to-be-trained model for reconstruction to obtain a training reconstruction result, wherein the training reconstruction result is obtained according to a three-dimensional Gaussian representation, and the three-dimensional Gaussian representation is obtained according to target object features and background object features, and the target object features and background object features are obtained according to the target object image and the background object image;

[0040] The adjustment module is used to adjust the training model according to the control image and the training reconstruction result to obtain the scene reconstruction model. The control image is obtained by superimposing the target object image and the background object image.

[0041] In a fourth aspect, the present application provides a scene reconstruction device, comprising:

[0042] A second acquisition module is used to acquire a reconstructed image of a scene to be reconstructed, where the scene to be reconstructed is the scene to be reconstructed in the model training method provided in this application;

[0043] The second segmentation module is used to perform semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image;

[0044] The reconstruction module is used to input the target object image and the background object image into the scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed. The scene reconstruction model is the scene reconstruction model in the model training method provided in this application.

[0045] In a fifth aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0046] Memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory to implement the method provided in the present application.

[0048] In a sixth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method provided by the present application.

[0049] The model training method, scene reconstruction method, device, equipment and medium provided in the present application obtain a training image of the scene to be reconstructed, then perform semantic segmentation processing on the target object and background object in the training image to obtain the target object image and the background object image, then input the target object image and the background object image into the model to be trained for reconstruction to obtain the training reconstruction result, and finally adjust the model to be trained according to the control image and the training reconstruction result to obtain the scene reconstruction model. The scene reconstruction model can achieve the purpose of scene reconstruction of the target object, and improve the efficiency of scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] Figure 1 A schematic diagram of a model training scenario provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a model training method provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a flow chart of a scene reconstruction method provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of a scene reconstruction device provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] In order to clearly understand the technical solution of the present application, the solution of the prior art is first introduced in detail.

[0060] Reconstructing 3D scenes from photo collections and synthesizing realistic new viewpoints has been a long-standing challenge in computer graphics and computer vision, and plays a key role in virtual reality. Neural radiance fields and their extensions significantly improve the quality of scene reconstruction and synthesis of new viewpoints by representing scenes as radiance fields.

[0061] Reconstructing 3D scenes from unconstrained photo sets (e.g., photos of famous landmarks obtained from the Internet) is considered a cost-effective way and has been widely studied. Due to changes in lighting conditions and post-processing, the same building may exhibit different radiometric properties in different images. In addition, unconstrained photos may contain transient occluders, which poses a significant challenge to 3D scene reconstruction.

[0062] In the prior art, transient occluders in unconstrained photos are predicted through appearance encoders and transient mask predictors. However, the existing appearance encoders and transient mask predictors require a long training time, which is inconsistent with the fast convergence characteristics of three-dimensional Gaussian technology, resulting in the problem of low scene reconstruction efficiency in the existing scene reconstruction.

[0063] In view of the problem of low efficiency in scene reconstruction, the inventors found in their research that predicting transient occluders is not a necessary step for scene reconstruction. By segmenting the target object and background object in the training image, the target object image and the background object image are obtained, which is equivalent to removing the transient occluders from the training image. Then, the training model is trained based on the target object image and the background object image to obtain a scene reconstruction model. The scene reconstruction model can achieve the purpose of scene reconstruction of the target object, thereby improving the efficiency of scene reconstruction. Transient occluders can refer to vehicles or pedestrians, etc.

[0064] The following introduces the application scenarios of the model training method provided in the embodiments of the present application.

[0065] Figure 1 A schematic diagram of a model training scenario provided in an embodiment of the present application, such as Figure 1 As shown, the scene includes an Internet image and a server. The server obtains an image related to the target object from the Internet image as a training image according to the target object related to the scene to be reconstructed, such as a building. The training image also includes a background object, such as the sky. Then, semantic segmentation is performed on the target object and the background object in the training image to obtain a target object image and a background object image; the target object image and the background object image are input into the model to be trained for reconstruction to obtain a training reconstruction result, which is obtained according to a three-dimensional Gaussian representation, which is obtained according to the target object feature and the background object feature, which is obtained according to the target object image and the background object image; according to the reference image and the training reconstruction result, the model to be trained is adjusted to obtain a scene reconstruction model, and the reference image is obtained by superimposing the target object image and the background object image.

[0066] Figure 2 A flow chart of a model training method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the method includes:

[0067] S201. Acquire a training image of a scene to be reconstructed, where the training image includes a target object and a background object.

[0068] In this embodiment, the target object in the scene to be reconstructed can be determined first. The target object can be any object that needs to be reconstructed, such as a building, a sculpture, a vehicle, a product, a person, a plant, or an animal. After the target object is determined, a photo containing the target object can be obtained from various image acquisition channels. In order to more accurately construct the outline of the target object, the training image can also include the background object where the target object is located, such as the sky, the ground, the water surface, the wall, and other scenes that are easy to distinguish from the target object.

[0069] S202: Perform semantic segmentation processing on the target object and background objects in the training image to obtain a target object image and a background object image.

[0070] In this embodiment, after obtaining the training image, in order to obtain the target object and the background object from the training image, the target object and the background object in the training image can be semantically segmented through the existing semantic segmentation model to obtain the target object pseudo-mask and the background object pseudo-mask, and then generate the corresponding image according to the pseudo-mask. The semantic segmentation model may include LSeg (a new training language driven semantic segmentation method and architecture), Grounded-SAM (a segmentation model) and SEEM (a large visual model) and other segmentation networks. The present application does not impose any restrictions on this, as long as the function of semantic segmentation can be achieved.

[0071] For example, for a training image containing buildings and the sky, the buildings and the sky can be input into the semantic segmentation model respectively, and then the building pseudo mask and the sky pseudo mask can be generated. It can be expressed as:

[0072]

[0073] in, is the pseudo mask of the target object, VLMSeg is the pre-trained visual large model (VLM) segmentation model, and I i is the training image, P building A text prompt for the target object. is the background object pseudo mask, P sky A text hint for the background object.

[0074] According to the above processing, the rest of the training image is regarded as a transient pseudo mask

[0075]

[0076] According to the training image and the target object pseudo mask, a target object image can be obtained, and according to the training image and the background object pseudo mask, a background object image can be obtained.

[0077] S203: Input the target object image and the background object image into the model to be trained for reconstruction to obtain a training reconstruction result.

[0078] In this embodiment, the target object image and the background object image are processed by the model to be trained to obtain a training reconstruction result, wherein the training reconstruction result can be obtained according to the three-dimensional Gaussian representation, the three-dimensional Gaussian representation can be obtained according to the target object features and the background object features, and the target object features and the background object features can be obtained according to the target object image and the background object image.

[0079] Specifically, inputting the target object image and the background object image into the to-be-trained model for reconstruction to obtain a training reconstruction result may include:

[0080] Inputting the target object image and the background object image into a feature extraction module in the model to be trained to obtain target object features and background object features;

[0081] Inputting background object features and a preset implicit cube map into a mapping module in the model to be trained to obtain a background cube map;

[0082] Input the background cube map and the target object features into the 3D Gaussian module to obtain a 3D Gaussian representation;

[0083] The 3D Gaussian representation and the background cubemap are fed into the rendering module to obtain the training reconstruction result.

[0084] In this embodiment, the model to be trained may include a feature extraction module, a mapping module, a three-dimensional Gaussian module and a rendering module.

[0085] 1. The feature extraction module is used to extract features from the target object image and the background object image to obtain target object features and background object features. Specifically, the feature extraction module may include a second encoder and a third encoder. The target object image is input into the second encoder for feature extraction to obtain target object features, and the background object image is input into the third encoder for feature extraction to obtain background object features. The second encoder and the third encoder may be independent convolutional neural networks (CNNs).

[0086] 2. The mapping module is used to tile the background object features into a preset implicit cubemap to obtain a background cubemap. The mapping module can be a coordinate-based multilayer perceptron, which is referred to as the second multilayer perceptron here for distinction. The implicit cubemap can be obtained through a learnable 4D tensor Where L×L represents the width and height of each cube face, and C represents the number of feature channels of each pixel in the implicit cube map. Then, a second multi-layer perceptron is used to generate the color of each pixel based on the features of the implicit cube map and the background object features. Each pixel of the background cube map can be represented as:

[0087]

[0088] Among them, T sky (k,u,v) represents the implicit feature corresponding to the pixel (u,v) on the kth face of the implicit cubemap. MLP γ is the second multilayer perceptron, is the background object feature, For the third encoder.

[0089] In this way, a complete background cubemap can be obtained The implicit cubemap effectively captures position-dependent details and alleviates the catastrophic forgetting problem associated with the second multilayer perceptron. However, the background object regions sampled from a single training image are sparse, which limits the number of features that can be optimized. To address this issue, the total variation (TV) loss can be introduced at the implicit cubemap level. To smooth the feature:

[0090]

[0091] 3. The 3D Gaussian module is used to fuse the features of the background cube map with the features of the target object, so that the color of the 3D Gaussian representing the target object corresponds to the color of the Beijing cube map, making the reconstruction result more accurate. For example, the same building will show different tones, different brightness, and different light and dark areas in different weather. In addition, the obtained 3D Gaussian representation can be input into the traditional 3D Gaussian rasterization process, and can also be seamlessly integrated into any downstream 3D Gaussian tasks, including scene understanding or scene editing.

[0092] Specifically, inputting the background cubemap and the target object features into a three-dimensional Gaussian module to obtain a three-dimensional Gaussian representation may include:

[0093] Input the background cube map into the first encoder to obtain fine background features;

[0094] According to the characteristics of the target object, a plurality of three-dimensional Gaussian volumes representing the target object are determined;

[0095] For each three-dimensional Gaussian body, the unconstrained radiation feature is obtained according to the three-dimensional mean position in the three-dimensional Gaussian body;

[0096] Inputting the fine background features, target object features, unconstrained radiation features and initial spherical harmonic coefficients in the three-dimensional Gaussian volume into the first multi-layer perceptron to obtain updated spherical harmonic coefficients;

[0097] The initial spherical harmonic coefficients in the three-dimensional Gaussian body are updated according to the updated spherical harmonic coefficients to obtain a three-dimensional Gaussian representation.

[0098] In this embodiment, the background cube map is input into the first encoder to obtain fine background features representing the background cube map. The first encoder can be a lightweight convolutional neural network (CNN). Then, according to the target object features, an initial three-dimensional Gaussian body is constructed, and the target object is represented by millions of three-dimensional Gaussian bodies. The learnable parameters of the three-dimensional Gaussian body may include: three-dimensional mean position μ, opacity α, rotation R, scaling factor S, initial spherical harmonic coefficients C and unconstrained radiation features F, where F is a feature vector initialized by applying the position encoding (PE) function to the three-dimensional mean position μ of each three-dimensional Gaussian. Then, the fine background features, target object features, unconstrained radiation features and initial spherical harmonic coefficients in the three-dimensional Gaussian body are input into the first multi-layer perceptron to obtain updated spherical harmonic coefficients. Finally, the initial spherical harmonic coefficients are replaced with the calculated updated spherical harmonic coefficients to obtain an updated three-dimensional Gaussian body. After obtaining each updated three-dimensional Gaussian body, a three-dimensional Gaussian representation is obtained, which can also be called an explicit three-dimensional Gaussian representation. The updated spherical harmonic coefficients C i ′ It can be expressed as:

[0099]

[0100] Among them, MLP ω is the first multilayer perceptron, For fine background features, is the target object feature, is the first encoder, For the second encoder.

[0101] 4. The rendering module is used to render the 3D Gaussian representation and the background cube map to obtain the training reconstruction result of the target object under the background cube map. By rendering the background object with the background cube map, it is avoided to expand the unnecessary 3D Gaussian volume for complex sky modeling during the training process, and the training reconstruction result It can be expressed as:

[0102]

[0103] Where x is the viewpoint position, d is the light direction, is the color of the target object after rendering, (1-O(x,d))C sky (d) is the color of the background object after rendering, and O(x, d) is the opacity of the pixel corresponding to x, d. During the training phase, we introduce random perturbations to the light direction d to enhance the anti-aliasing effect by varying its length within a unit pixel. In order to suppress the appearance of three-dimensional Gaussians in the background object area, the integrated opacity of the background object area can be constrained to be close to zero, and the loss is defined as:

[0104]

[0105] Where O is the set of all O(x,d).

[0106] S204, adjusting the model to be trained according to the control image and the training reconstruction result to obtain a scene reconstruction model, wherein the control image is obtained by superimposing the target object image and the background object image.

[0107] In this embodiment, the control image can be obtained by superimposing the target object image and the background object image, thereby removing transient objects in the training image through a greedy supervision strategy. The learning parameters in the training model can be adjusted through a loss function to obtain a scene reconstruction model.

[0108] Specifically, the loss function may include a first loss function and a second loss function. The first loss function is determined based on the reference image and the training reconstruction result; the second loss function is determined based on the target object features and the background object features. By applying the greedy mask strategy through the first loss function, the prediction of the transient mask can be avoided, the parameters that the model needs to learn are reduced, and the scene reconstruction efficiency is improved. The first loss function can be expressed as:

[0109]

[0110] Among them, λ1 and λ2 are hyperparameters, For the comparison image, is the Manhattan distance loss, is the structural similarity index (SSIM) loss.

[0111] Through the second loss function, the appearance features of background objects and target objects can be effectively aligned in the same latent space, because the inherent relationship between the appearance of background objects and target objects, and the relatively small solution space of background object features, can help guide and constrain the appearance features of target objects. Secondly, for some training images that fail to capture background objects, this alignment can estimate the background cube map only through the appearance features of buildings. The second loss function can be a mutual distillation strategy that enables the second encoder and the third encoder to learn from each other and align their features in the same latent space. The second loss can be expressed as:

[0112]

[0113] Among them, Manhat is the Manhattan distance function.

[0114] The loss function can be expressed as:

[0115]

[0116] Among them, λ3, λ4 and λ5 are hyperparameters.

[0117] The model training method provided in the embodiment of the present application obtains a training image of a scene to be reconstructed, then performs semantic segmentation processing on the target object and background object in the training image to obtain a target object image and a background object image, then inputs the target object image and the background object image into the model to be trained for reconstruction to obtain a training reconstruction result, and finally adjusts the model to be trained according to the control image and the training reconstruction result to obtain a scene reconstruction model. The scene reconstruction model can achieve the purpose of scene reconstruction of the target object, thereby improving the efficiency of scene reconstruction.

[0118] Figure 3 A schematic diagram of a scene reconstruction method provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the method includes:

[0119] S301. Obtain a reconstructed image of a scene to be reconstructed, where the scene to be reconstructed is the scene to be reconstructed in the model training method provided in this application.

[0120] In this embodiment, after obtaining the scene reconstruction model, a reconstructed image of the scene to be reconstructed corresponding to the scene reconstruction model can be obtained from the Internet image. For example, if the scene reconstruction model is a reconstruction of building A, then other images of building A can also be obtained as reconstructed images to obtain different reconstruction effects of building A.

[0121] S302 : performing semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image.

[0122] In this embodiment, after the reconstructed image is selected, a semantic segmentation model is required to perform semantic segmentation processing on the reconstructed image to obtain a target object image and a background object image in the reconstructed image.

[0123] S303. Input the target object image and the background object image into a scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed. The scene reconstruction model is the scene reconstruction model in the model training method provided in this application.

[0124] In this embodiment, the target object image and the background object image in the reconstructed image are input into the scene reconstruction model, and the target object image and the background object image in the reconstructed image are processed by the scene reconstruction model to obtain a scene reconstruction result, thereby achieving the purpose of scene reconstruction and improving the efficiency of scene reconstruction.

[0125] In some embodiments, it may also include:

[0126] Obtaining target object features output by a second encoder in a scene reconstruction model;

[0127] Input the target object features and the preset background cube map into the 3D Gaussian module in the scene reconstruction model to obtain a 3D Gaussian representation;

[0128] The three-dimensional Gaussian representation and the preset background cube map are input into the rendering module in the scene reconstruction model to obtain the scene reconstruction result.

[0129] In this embodiment, after obtaining the target object features of the target object, the scene to be reconstructed can be reconstructed according to the existing preset background cube map, and the background of the scene can be changed. For example, the scene where the target object is a building can be replaced with the reconstruction effect of different weather conditions, thereby expanding the application scenarios of the scene reconstruction model.

[0130] In some implementations, two training images may be used to generate their target object features respectively, and then new target object features may be generated by linear interpolation, and these features may be subsequently used to synthesize a completely new appearance.

[0131] A scene reconstruction method provided in this embodiment obtains a reconstructed image of the scene to be reconstructed, performs semantic segmentation processing on the target object and background object in the reconstructed image to obtain a target object image and a background object image, inputs the target object image and the background object image into a scene reconstruction model, and obtains a scene reconstruction result corresponding to the scene to be reconstructed, thereby improving the efficiency of scene reconstruction.

[0132] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, the device 40 includes:

[0133] A first acquisition module 401 is used to acquire a training image of a scene to be reconstructed, where the training image includes a target object and a background object;

[0134] The first segmentation module 402 is used to perform semantic segmentation processing on the target object and the background object in the training image to obtain a target object image and a background object image;

[0135] A training module 403 is used to input the target object image and the background object image into the model to be trained for reconstruction to obtain a training reconstruction result, the training reconstruction result is obtained according to the three-dimensional Gaussian representation, the three-dimensional Gaussian representation is obtained according to the target object features and the background object features, the target object features and the background object features are obtained according to the target object image and the background object image;

[0136] The adjustment module 404 is used to adjust the model to be trained according to the reference image and the training reconstruction result to obtain a scene reconstruction model. The reference image is obtained by superimposing the target object image and the background object image.

[0137] In some implementations, the training module 403 is further configured to:

[0138] Inputting the target object image and the background object image into a feature extraction module in the model to be trained to obtain target object features and background object features;

[0139] Inputting background object features and a preset implicit cube map into a mapping module in the model to be trained to obtain a background cube map;

[0140] Input the background cube map and the target object features into the 3D Gaussian module to obtain a 3D Gaussian representation;

[0141] The 3D Gaussian representation and the background cubemap are fed into the rendering module to obtain the training reconstruction result.

[0142] In some implementations, the training module 403 is further configured to:

[0143] Input the background cube map into the first encoder to obtain fine background features;

[0144] According to the characteristics of the target object, a plurality of three-dimensional Gaussian volumes representing the target object are determined;

[0145] For each three-dimensional Gaussian body, the unconstrained radiation feature is obtained according to the three-dimensional mean position in the three-dimensional Gaussian body;

[0146] Inputting the fine background features, target object features, unconstrained radiation features and initial spherical harmonic coefficients in the three-dimensional Gaussian volume into the first multi-layer perceptron to obtain updated spherical harmonic coefficients;

[0147] The initial spherical harmonic coefficients in the three-dimensional Gaussian body are updated according to the updated spherical harmonic coefficients to obtain a three-dimensional Gaussian representation.

[0148] In some implementations, the training module 403 is further configured to:

[0149] According to the second encoder, feature extraction is performed on the target object image to obtain target object features;

[0150] According to the third encoder, feature extraction is performed on the background object image to obtain background object features.

[0151] In some implementations, the adjustment module 404 is further configured to:

[0152] Determine a first loss function according to the control image and the training reconstruction result;

[0153] Determining a second loss function according to the target object features and the background object features;

[0154] According to the first loss function and the second loss function, the model to be trained is adjusted to obtain a scene reconstruction model.

[0155] Figure 5 A schematic diagram of the structure of a scene reconstruction device provided in an embodiment of the present application, the device 50 includes:

[0156] The second acquisition module 501 is used to acquire a reconstructed image of a scene to be reconstructed, where the scene to be reconstructed is the scene to be reconstructed in the model training method provided in this application;

[0157] The second segmentation module 502 is used to perform semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image;

[0158] The reconstruction module 503 is used to input the target object image and the background object image into the scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed. The scene reconstruction model is the scene reconstruction model in the model training method provided in this application.

[0159] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 60 includes:

[0160] The electronic device 60 may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a communication component 603 and other components. The processor 601 , the memory 602 and the communication component 603 are connected via a bus 604 .

[0161] During the specific implementation process, at least one processor 601 executes computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above model training method.

[0162] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0163] In the above Figure 6 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0164] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk storage.

[0165] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0166] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the steps of any of the above-mentioned model training methods when executed by a processor.

[0167] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0168] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores multiple instructions, and the instructions can be loaded by a processor to execute the steps in any model training method provided in the embodiment of the present application.

[0170] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0171] According to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program comprises computer instructions stored in a computer-readable storage medium.

[0172] Since the instructions stored in the storage medium can execute the steps in any model training method provided in the embodiments of the present application, the beneficial effects that can be achieved by any model training method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0173] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0174] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire a training image of a scene to be reconstructed, wherein the training image includes a target object and a background object; Performing semantic segmentation processing on the target object and the background object in the training image to obtain a target object image and a background object image; Inputting the target object image and the background object image into a to-be-trained model for reconstruction to obtain a training reconstruction result, wherein the training reconstruction result is obtained according to a three-dimensional Gaussian representation, wherein the three-dimensional Gaussian representation is obtained according to target object features and background object features, wherein the target object features and background object features are obtained according to the target object image and the background object image; The model to be trained is adjusted according to the control image and the training reconstruction result to obtain a scene reconstruction model, wherein the control image is obtained by superimposing the target object image and the background object image.

2. The method according to claim 1, characterized in that The step of inputting the target object image and the background object image into the model to be trained for reconstruction to obtain a training reconstruction result includes: Inputting the target object image and the background object image into a feature extraction module in the model to be trained to obtain target object features and background object features; Inputting the background object features and a preset implicit cube map into a mapping module in a model to be trained to obtain a background cube map; Inputting the background cubemap and the target object features into a three-dimensional Gaussian module to obtain a three-dimensional Gaussian representation; The three-dimensional Gaussian representation and the background cube map are input into a rendering module to obtain a training reconstruction result.

3. The method according to claim 2, characterized in that The step of inputting the background cubemap and the target object features into a three-dimensional Gaussian module to obtain a three-dimensional Gaussian representation includes: Inputting the background cube map into a first encoder to obtain fine background features; Determining a plurality of three-dimensional Gaussian volumes representing the target object according to the target object characteristics; For each three-dimensional Gaussian body, an unconstrained radiation feature is obtained according to the three-dimensional mean position in the three-dimensional Gaussian body; Inputting the fine background features, the target object features, the unconstrained radiation features and the initial spherical harmonic coefficients in the three-dimensional Gaussian volume into a first multilayer perceptron to obtain updated spherical harmonic coefficients; The initial spherical harmonic coefficients in the three-dimensional Gaussian volume are updated according to the updated spherical harmonic coefficients to obtain a three-dimensional Gaussian representation.

4. The method according to any one of claims 2-3, characterized in that: The step of inputting the target object image and the background object image into a feature extraction module in the model to be trained to obtain target object features and background object features includes: According to the second encoder, feature extraction is performed on the target object image to obtain target object features; According to the third encoder, feature extraction is performed on the background object image to obtain background object features.

5. The method according to any one of claims 1 to 3, characterized in that: The step of adjusting the model to be trained according to the reference image and the training reconstruction result to obtain a scene reconstruction model comprises: Determining a first loss function according to the control image and the training reconstruction result; Determining a second loss function according to the target object feature and the background object feature; According to the first loss function and the second loss function, the model to be trained is adjusted to obtain a scene reconstruction model.

6. A scene reconstruction method, characterized in that: include: Acquire a reconstructed image of a scene to be reconstructed, wherein the scene to be reconstructed is the scene to be reconstructed according to any one of claims 1 to 5; Performing semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image; The target object image and the background object image are input into a scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed, and the scene reconstruction model is the scene reconstruction model described in any one of claims 1-5.

7. The method according to claim 6, characterized in that The method further comprises: Acquire target object features output by a second encoder in the scene reconstruction model; Inputting the target object features and the preset background cube map into a three-dimensional Gaussian module in the scene reconstruction model to obtain a three-dimensional Gaussian representation; The three-dimensional Gaussian representation and the preset background cube map are input into a rendering module in the scene reconstruction model to obtain a scene reconstruction result.

8. A model training device, characterized in that: include: A first acquisition module, used to acquire a training image of a scene to be reconstructed, wherein the training image includes a target object and a background object; A first segmentation module is used to perform semantic segmentation processing on the target object and the background object in the training image to obtain a target object image and a background object image; A training module, used for inputting the target object image and the background object image into a to-be-trained model for reconstruction to obtain a training reconstruction result, wherein the training reconstruction result is obtained according to a three-dimensional Gaussian representation, wherein the three-dimensional Gaussian representation is obtained according to target object features and background object features, wherein the target object features and background object features are obtained according to the target object image and the background object image; The adjustment module is used to adjust the model to be trained according to the control image and the training reconstruction result to obtain a scene reconstruction model. The control image is obtained by superimposing the target object image and the background object image.

9. A scene reconstruction device, characterized in that: include: A second acquisition module, used to acquire a reconstructed image of a scene to be reconstructed, wherein the scene to be reconstructed is the scene to be reconstructed according to any one of claims 1 to 5; A second segmentation module is used to perform semantic segmentation processing on the target object and the background object in the reconstructed image to obtain a target object image and a background object image; A reconstruction module is used to input the target object image and the background object image into a scene reconstruction model to obtain a scene reconstruction result corresponding to the scene to be reconstructed, and the scene reconstruction model is the scene reconstruction model described in any one of claims 1-5.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images