3D gaussian panorama segmentation method based on 2D pre-training segmentation model supervision, computer device and program product

Abstract

The application relates to a 3D Gaussian panorama segmentation method based on 2D pre-training segmentation model supervision, a computer device and a program product. The 3D Gaussian panorama segmentation method comprises the following steps: inputting multiple input images of different perspectives of the same scene into a 2D pre-training segmentation model, obtaining a 2D semantic segmentation graph and a 2D instance segmentation graph, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene, and the 3D Gaussian initial expression is a scene formed by random initialization of multiple Gaussian ellipsoids; obtaining a 2D Gaussian expression obtained by projecting the 3D Gaussian initial expression under the corresponding perspective; obtaining a first generated image according to color information, a second generated image according to semantic information, and a third generated image according to instance information; training and optimizing the 3D Gaussian scene expression by using preset loss parameters, obtaining a trained 3D Gaussian segmentation model; and obtaining semantic segmentation information and instance segmentation information of the 3D Gaussian scene based on the 3D Gaussian segmentation model.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a 3D Gaussian panoramic segmentation method, computer device, and program product based on supervision of a 2D pre-trained segmentation model. Background Art

[0002] It is very necessary to establish an understanding of the semantics and other information of the entire 3D scene from 2D images. However, directly segmenting the image using only a 2D segmentation model will lack 3D information, making it difficult to establish a holistic understanding of the scene.

[0003] Existing panoptic segmentation models often suffer from the following three issues: 1) They struggle to produce consistent 3D segmentation results, meaning the segmentation results for the same object differ from viewpoint to viewpoint. 2) They are slow to train, with current methods often requiring 24 hours or more to train. 3) They are slow to render, rendering only a dozen or so segmentation maps per second. Summary of the Invention

[0004] Based on this, it is necessary to provide a 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision to address the above technical problems.

[0005] This application is based on a 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model, including:

[0006] Input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, obtain camera parameters corresponding to the input images based on the input images, and obtain a 3D Gaussian initial expression based on the input images, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids;

[0007] Based on the camera parameters, obtaining a 2D Gaussian expression obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, wherein the 2D Gaussian expression includes color information, semantic information, and instance information;

[0008] Obtain a first generated image based on the color information, obtain a second generated image based on the semantic information, and obtain a third generated image based on the instance information;

[0009] training the 3D Gaussian scene representation with preset loss parameters to obtain a trained 3D Gaussian segmentation model, the preset loss parameters including: a first preset loss of the first generated image and a corresponding input image, a second preset loss of the second generated image and a corresponding 2D semantic segmentation map, and a third preset loss of the third generated image and a mapped instance segmentation result obtained by matching mapping of the third generated image and a corresponding 2D instance segmentation map;

[0010] obtaining a 3D Gaussian panorama segmentation result based on the 3D Gaussian segmentation model, the 3D Gaussian panorama segmentation result including semantic segmentation information and instance segmentation information of a 3D Gaussian scene.

[0011] Optionally, obtaining a 3D Gaussian initial representation based on a 3D Gaussian scene according to the input image, the 3D Gaussian initial representation being a scene formed by random initialization of a plurality of Gaussian ellipsoids, and specifically including:

[0012] obtaining an initial point cloud based on a 3D Gaussian scene according to the input image, and obtaining a 3D Gaussian initial representation of the scene based on the initial point cloud.

[0013] Optionally, the mapped instance segmentation result is obtained by matching mapping of the third generated image and a corresponding 2D instance segmentation map, and specifically includes:

[0014] matching an instance ID of each pixel of the 2D instance segmentation map to the third generated image to obtain a mapped instance segmentation result for supervision.

[0015] Optionally, the 3D Gaussian initial representation of the scene is obtained based on the initial point cloud, and specifically includes: regarding a position of each initial point cloud as a center of a Gaussian ellipsoid, randomly initializing a shape of the Gaussian ellipsoid to obtain a scene composed of a plurality of Gaussian ellipsoids with different shapes, thereby obtaining the 3D Gaussian initial representation.

[0016] Optionally, based on the camera parameters, a 2D Gaussian representation obtained by projecting the 3D Gaussian initial representation at a corresponding view angle is obtained, the 2D Gaussian representation including color information, semantic information and instance information, and specifically including:

[0017] based on the camera parameters, projecting each Gaussian ellipsoid in the 3D Gaussian initial representation to a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected as a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information and instance information on the 2D plane.

[0018] Optionally, obtaining a first generated image according to the color information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, and mixing color information of each pixel point, thereby obtaining the first generated image;

[0019] Obtaining a second generated image according to the semantic information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, and mixing the semantic information of each pixel point, thereby obtaining the second generated image.

[0020] Optionally, the semantic information of each pixel is obtained using the following formula:

[0021]

[0022] Where,

[0023] κ p Represents the semantic information corresponding to pixel p;

[0024] L s Represents a neural network used to map semantic encodings into probability distributions corresponding to various semantics;

[0025] e i Represents the color information corresponding to the i-th Gaussian ball;

[0026] α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p;

[0027] α′ j Indicates the opacity of the j-th Gaussian sphere.

[0028] Optionally, obtaining a third generated image according to the instance information specifically includes:

[0029] The Gaussian ellipse is subjected to Gaussian differentiable rasterization, and instance information of each pixel is mixed to obtain a third generated image.

[0030] Optionally, the third preset loss is obtained using the following formula:

[0031]

[0032] in, is the third preset loss, used to represent the instance loss;

[0033] |I| represents the number of pixels in the image;

[0034] w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model;

[0035] h represents the hth instance;

[0036] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance;

[0037] is the probability that p pixels correspond to the p-th class instance predicted by the 2D pre-trained segmentation model;

[0038] The 2D instance segmentation map is mapped by linear matching to obtain the mapped instance segmentation result:

[0039]

[0040] in, A mapping representing a selection;

[0041] Π I Indicates the current mapping;

[0042] H I Represents all instances of an image;

[0043] |I h | represents the number of pixels in the image corresponding to instance h;

[0044] Π I (h) represents the instance number to which the h-th instance is mapped;

[0045] w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model;

[0046] h represents the hth instance;

[0047] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance.

[0048] The present application also provides a computer device, including a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0049] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0050] The present application also provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0051] The 3D Gaussian panoptic segmentation method based on 2D pre-trained segmentation model supervision in this application has at least the following effects:

[0052] The 3D Gaussian scene expression after training and optimization in this application can match the real scene corresponding to the image. Specifically, the 3D Gaussian expression is optimized through a first preset loss to complete this process. On this basis, the semantic representation of the scene is further optimized through a second preset loss; and the instance representation of the scene is further optimized through a third preset loss. The model after optimization has semantic and instance information that can fit the real scene, which can be used to obtain a panoramic expression of the entire scene, that is, to complete the entire method flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Schematic diagram of the process of a 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model in one embodiment of the present application;

[0054] Figure 2 This is a model architecture diagram of a 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model in one embodiment of the present application;

[0055] Figure 3 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] See also Figure 1 An embodiment of the present application provides a 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision, including steps S100 to S800.

[0058] In step S100 , multiple input images of the same scene from different perspectives are input into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, and camera parameters corresponding to the input images are obtained according to the input images.

[0059] Step S200 , obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0060] Step S300 : Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle. The 2D Gaussian expression includes color information, semantic information, and instance information.

[0061] Step S400: obtaining a first generated image according to color information.

[0062] Step S500: obtaining a second generated image according to semantic information.

[0063] Step S600: Obtain a third generated image according to the instance information.

[0064] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0065] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0066] See also Figure 1 and Figure 2 In some embodiments, a 3D Gaussian panoptic segmentation method based on supervision by a 2D pre-trained segmentation model is provided, which provides a detailed explanation of steps S100 to S800 described above, along with optional sub-steps. The method includes: (1) creating a 3D Gaussian representation of the scene using an input image; (2) projecting the 3D Gaussian representation into 2D pixel space; and (3) establishing an overall loss function.

[0067] (1) Using the input image to establish a 3D Gaussian representation of the scene, including steps S100 to S200:

[0068] In step S100 , multiple input images of the same scene from different perspectives are input into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, and camera parameters corresponding to the input images are obtained according to the input images.

[0069] Step S200 , obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0070] In step S100, multiple input images of the same scene from different perspectives are input into a pre-trained 2D segmentation model, and semantic segmentation information (2D semantic segmentation map) and instance segmentation information (2D instance segmentation map) of each pixel corresponding to each 2D image can be obtained. Multiple input images of the input multi-perspective picture (i.e., the actual image as shown in the figure). The 2D pre-trained segmentation model can be, for example, a "Mask2Former model". Semantic segmentation information and instance segmentation information only stay at the 2D level. It cannot integrate information from other perspectives, and there will be 3D inconsistencies.

[0071] Semantic information can be understood as categories. For example, if there is a cat in a picture, then in the semantic map, the corresponding pixel is the cat category. Instance information is like an ID card. Each object has a different instance ID. For example, if there are two cats in the picture, their instance information will be 1 and 2 respectively. Instance information is different from semantic segmentation. In semantic segmentation, the categories of the two cats are the same, but they are not the same in instance segmentation. The panoramic segmentation provided in this embodiment includes both semantic segmentation and instance segmentation, which can identify both the semantic information corresponding to each pixel and the instance information corresponding to the pixel.

[0072] Step S200 specifically includes obtaining an initial point cloud based on a 3D Gaussian scene according to the input image, regarding the position of each initial point cloud as the center of a Gaussian ellipsoid, randomly initializing the shape of the Gaussian ellipsoid, and obtaining a scene composed of multiple Gaussian ellipsoids of different shapes, that is, obtaining a 3D Gaussian initial expression.

[0073] Specifically, the input image is fed into the Colmap software to obtain an initial point cloud based on the 3D Gaussian scene and camera parameters. As you can understand, Colmap software, based on the SFM (Structure-from-Motion) algorithm, matches key information between each image to reconstruct the general information of the scene, including the initial point cloud and the corresponding perspective information of each image in this scene, namely the camera parameters.

[0074] According to steps S100 and S200, the input images are generally photos of the same scene from different perspectives. 3D information corresponding to the scenes in these photos can be obtained. This embodiment establishes a preliminary 3D Gaussian representation based on this 3D information, and subsequently obtains a rendering of this 3D Gaussian representation from any perspective. It can be understood that step S200 essentially uses ellipsoids of various shapes to fit each object in the scene. Each 3D Gaussian sphere (Gaussian ellipsoid) has color, semantic, and instance information.

[0075] Since 3D Gaussian itself is an explicit representation, compared to implicit expression methods such as NeRF (Neural Radiance Field), it does not require additional calculation of parts of the space where objects do not exist during the rendering process, thereby greatly speeding up the rendering rate. Specifically, the FPS (Frames Per Second) of panoramic segmentation using methods such as NeRF on the A6000 GPU is about 10 frames per second, while the expression based on 3D Gaussian can reach about 140 frames per second, which is more than ten times faster.

[0076] (2) Projecting the 3D Gaussian representation into the 2D pixel space, including steps S300 to S600:

[0077] Step S300 : Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle. The 2D Gaussian expression includes color information, semantic information, and instance information.

[0078] Step S300 specifically includes: based on the camera parameters, projecting each Gaussian ellipsoid in the 3D Gaussian initial expression onto a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected into a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information, and instance information on the 2D plane, that is, obtaining a 2D Gaussian expression.

[0079] It can be understood that each 3D Gaussian sphere has color information, semantic information and instance information. After projection, a single pixel or multiple pixels as a whole also have color information, semantic information and instance information accordingly. The above steps have established the initial 3D Gaussian expression of the scene, that is, the preliminary representation of the scene. With the 3D scene representation, it is only necessary to obtain the 2D information corresponding to each perspective through projection and other methods. It can be understood that the 2D Gaussian expression is the picture obtained by projection. For each picture that needs to be projected, the corresponding camera parameters are used to project the 3D Gaussian sphere onto the 2D plane of the corresponding perspective of the picture. At this time, the 3D Gaussian ellipsoid becomes a 2D Gaussian ellipse. These Gaussian ellipses have center coordinates, shape parameters, color, semantics, and instance information.

[0080] Specifically, each 3D Gaussian ellipsoid, when projected to a certain viewing angle through camera parameters, corresponds to a Gaussian ellipse. The entire 2D image is covered by a series of 2D Gaussian ellipses, and each pixel point is covered by multiple ellipses. These ellipses have color, semantics, instance and other information. Through Gaussian rasterization, the information of each ellipse is mixed on the 2D plane to obtain the color, semantics, instance and other information of the corresponding pixel.

[0081] Step S400 , obtaining a first generated image based on the color information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, and mixing the color information of each pixel to obtain the first generated image, namely the rendered image shown in the figure.

[0082] The 2D Gaussian ellipse is subjected to Gaussian differentiable rasterization, and the color information of each pixel is mixed to obtain the color of any pixel in the image, thereby obtaining the first generated image. The color calculation formula for each pixel is as follows:

[0083]

[0084] Where C(p) represents the color corresponding to pixel p, c i Represents the color information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j The opacity of the jth Gaussian sphere corresponds to this formula. The meaning of this formula is to fuse all Gaussian spheres covering each pixel by weighted opacity to obtain the color information of the pixel.

[0085] Step S500 , obtaining a second generated image based on the semantic information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, and mixing the semantic information of each pixel to obtain the second generated image, namely the rendered semantic segmentation map shown in the figure.

[0086] By rasterizing the 2D Gaussian ellipse with Gaussian differentiable rasterization and mixing the semantic information of each pixel, we can obtain the semantic segmentation result of any pixel in the image, thereby obtaining the second generated image. The semantic information calculation formula for each pixel is as follows:

[0087]

[0088] Where k p Represents the semantic information corresponding to pixel p, e i Represents the semantic information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j Corresponding to the opacity of the j-th Gaussian ball, L s represents a neural network that maps semantic encodings to probability distributions corresponding to various semantics. This formula derives the semantic information for each pixel by weighting the opacity of all Gaussian spheres covering that pixel. Given the probability distribution of all semantics for each pixel, the semantics corresponding to that pixel is selected based on the highest probability.

[0089] Step S600 , obtaining a third generated image based on the instance information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, mixing the instance information of each pixel, and obtaining the third generated image, namely the rendered instance segmentation map shown in the figure.

[0090] The 2D Gaussian ellipse is subjected to Gaussian differentiable rasterization and the instance segmentation information of each pixel is mixed to obtain the instance segmentation result corresponding to any pixel in the image, thereby obtaining the third generated image.

[0091] The instance information of each pixel is calculated as follows:

[0092]

[0093] Where π p Indicates the instance information corresponding to pixel p, o i Represents the instance information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j Corresponding to the opacity of the j-th Gaussian ball, L o represents a neural network that maps instance encodings to probability distributions corresponding to various instances. This formula essentially fuses all Gaussian spheres covering each pixel using opacity-weighted fusion to obtain the instance information for that pixel. Given the probability distribution of all instances corresponding to each pixel, we simply select the instance with the highest probability.

[0094] (3) Overall loss function, including steps S700 to S800:

[0095] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0096] The mapping instance segmentation result is obtained by matching and mapping the third generated image with the corresponding 2D instance segmentation map, including: matching and mapping the instance ID of each pixel of the 2D instance segmentation map to the third generated image to obtain a mapping instance segmentation result for supervision.

[0097] Specific mapping process of matching: The 3D instance segmentation result (third generated image) obtained by the final 3D Gaussian rendering has 3D consistency. That is, for the same object, the instance segmentation ID under different viewing angles should be consistent. The 2D instance segmentation map obtained by the 2D segmentation model is segmented in the image plane and does not have 3D consistency. This embodiment maps the instance ID of each pixel of the 2D instance segmentation map to the ID of the rendered instance segmentation map (third generated image).

[0098] Then, this embodiment proposes a criterion for measuring matching, that is, given an instance matching relationship: after mapping the instance ID of the 2D instance segmentation map through the matching relationship, a mapped 2D instance segmentation map is obtained. Then, the mapped 2D instance segmentation map is compared pixel by pixel with the rendered instance segmentation map (the third generated image). The more pixels with the same instance segmentation result, the better the matching relationship. This problem can be converted into a linear allocation problem: that is, to find an optimal matching mapping method that has the most pixels that meet the above criteria. This problem can be solved using algorithms such as KM.

[0099] In this embodiment, the contrast loss between the first generated image and the input image is used as the first preset loss. The contrast loss between the second generated image and the 2D semantic segmentation map is used as the second preset loss. The contrast loss between the third generated image and the mapped instance segmentation result is used as the third preset loss. Specifically, the preset loss parameters are obtained by adding weights to the first preset loss, the second preset loss, and the third preset loss, and then cumulatively adding them.

[0100] In this embodiment, the first, second, and third preset losses are simultaneously used to optimize the 3D Gaussian scene. This optimization process primarily enables the 3D Gaussian scene to learn information about the scene corresponding to the image, thereby expressing the semantics, instance, and other information of the entire scene.

[0101] It can be understood that the first generated image is derived from the color information of the mixed 2D Gaussian, and the first generated image is the projection of the model at that perspective. They correspond to each other, so the first preset loss can be obtained by indirectly comparing the reference image and the first generated image. The second and third preset losses are similar.

[0102] Furthermore, the optimized 3D Gaussian scene representation in this embodiment can match the real-world scene corresponding to the image. This process is completed by optimizing the 3D Gaussian representation using a first preset loss. On this basis, the semantic representation of the scene is further optimized using a second preset loss, and the instance representation of the scene is further optimized using a third preset loss. The optimized model has semantic and instance information that can match the real-world scene, which can be used to obtain a panoramic representation of the entire scene, thus completing the entire method flow.

[0103] The first preset loss, namely color loss It establishes the color and other information of the entire scene. Color loss As shown below:

[0104]

[0105] in, represents color loss; λ a Is a constant used to represent and The weight is 0.2, which represents the loss between the rendered color and the actual color; represents the L1 loss, Represents the SSIM loss of the image.

[0106] The second preset loss, namely semantic loss The semantic information of the entire scene is established. As shown below:

[0107]

[0108] in, Represents semantic loss;

[0109] |I| represents the number of pixels in the image;

[0110] w p Represents the confidence of each pixel predicted by the 2D segmentation model;

[0111] k represents the kth semantics;

[0112] κ p (k) represents the probability that the rendered p-th pixel corresponds to the k-th semantic category;

[0113] is the probability that the p pixels predicted by the 2D segmentation model correspond to the k-th semantic category.

[0114] The third preset loss, namely instance loss The instance information of the entire scene is established. As shown below:

[0115]

[0116] in, represents the instance loss;

[0117] |I| represents the number of pixels in the image;

[0118] w pdenotes the confidence of each pixel predicted by the 2D segmentation model;

[0119] h denotes the h-th instance;

[0120] π p denotes the probability that the p-th pixel corresponds to the p-th instance after rendering;

[0121] is the probability that the p-th pixel corresponds to the p-th instance predicted by the 2D segmentation model.

[0122] is a linear matching method for mapping the instance information (2D instance segmentation result) predicted by the 2D segmentation model to the instance information (mapped instance segmentation result) of the scene, and the formula is:

[0123]

[0124] wherein, denotes the selected optimal mapping;

[0125] Π I denotes the current mapping;

[0126] H I denotes all possible instances of the image;

[0127] |I h denotes the number of pixels corresponding to the h-th instance of the image;

[0128] Π I (h) denotes the instance number to which the h-th instance is mapped;

[0129] w p denotes the confidence of each pixel predicted by the 2D segmentation model;

[0130] h denotes the h-th instance;

[0131] π p denotes the probability that the p-th pixel corresponds to the p-th instance after rendering;

[0132] In step S800, a 3D Gaussian panoramic segmentation result is obtained based on a 3D Gaussian segmentation model, and the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of a 3D Gaussian scene.

[0133] Embodiments of the present application can derive the semantic and instance segmentation information of the entire scene from the multi-view pictures with the help of a 2D segmentation model, and make the panoramic segmentation model generate a segmentation result with 3D consistency with the help of a 3D Gaussian scene representation.

[0134] It should be understood that, although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a 3D Gaussian panoramic segmentation method based on supervision of a 2D pre-trained segmentation model is implemented. The display screen of the computer device can be a liquid crystal display or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse.

[0136] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0137] In step S100 , multiple input images of the same scene from different perspectives are input into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, and camera parameters corresponding to the input images are obtained according to the input images.

[0138] Step S200 , obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0139] Step S300 : Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle. The 2D Gaussian expression includes color information, semantic information, and instance information.

[0140] Step S400: obtaining a first generated image according to color information.

[0141] Step S500: obtaining a second generated image according to semantic information.

[0142] Step S600: Obtain a third generated image according to the instance information.

[0143] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0144] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0145] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0146] In step S100 , multiple input images of the same scene from different perspectives are input into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, and camera parameters corresponding to the input images are obtained according to the input images.

[0147] Step S200 , obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0148] Step S300 : Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle. The 2D Gaussian expression includes color information, semantic information, and instance information.

[0149] Step S400: obtaining a first generated image according to color information.

[0150] Step S500: obtaining a second generated image according to semantic information.

[0151] Step S600: Obtain a third generated image according to the instance information.

[0152] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0153] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0154] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:

[0155] In step S100 , multiple input images of the same scene from different perspectives are input into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, and camera parameters corresponding to the input images are obtained according to the input images.

[0156] Step S200 , obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0157] Step S300 : Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle. The 2D Gaussian expression includes color information, semantic information, and instance information.

[0158] Step S400: obtaining a first generated image according to color information.

[0159] Step S500: obtaining a second generated image according to semantic information.

[0160] Step S600: Obtain a third generated image according to the instance information.

[0161] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0162] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0163] In this embodiment, the computer program product includes a program code portion for executing the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for download with the aid of a hardware description language.

[0164] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0165] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.

[0166] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model, characterized by: include: Input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, obtain camera parameters corresponding to the input image based on the input image, and obtain a 3D Gaussian initial expression based on a 3D Gaussian scene based on the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids; based on the camera parameters, obtain a 2D Gaussian expression obtained by projecting the 3D Gaussian initial expression at the corresponding perspective, wherein the 2D Gaussian expression includes color information, semantic information, and instance information; obtain a first generated image based on the color information, obtain a second generated image based on the semantic information, and obtain a third generated image based on the instance information; Utilize preset loss parameter training to optimize the 3D Gaussian scene expression and obtain a trained 3D Gaussian segmentation model, wherein the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map, specifically including: matching and mapping the instance ID of each pixel of the 2D instance segmentation map to the third generated image to obtain a mapping instance segmentation result for supervision; Based on the 3D Gaussian segmentation model, a 3D Gaussian panoramic segmentation result is obtained, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene. The third preset loss is obtained using the following formula: ,in, is the third preset loss, used to represent the instance loss; Indicates the number of pixels in the image; Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model; It represents the hth instance; Indicates the rendered The pixel corresponds to probability of class instances; It is predicted by the 2D pre-trained segmentation model The pixel corresponds to probability of class instances; The 2D instance segmentation map is mapped by linear matching to obtain the mapped instance segmentation result: ,in, A mapping representing a selection; Indicates the current mapping; Represents all instances of an image; Indicates the number of pixels in the image corresponding to h instance; Indicates the instance number to which the hth instance is mapped; Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model; It represents the hth instance; Indicates the rendered The pixel corresponds to The probability of a class instance.

2. The 3D Gaussian panoptic segmentation method according to claim 1, wherein: Obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids, specifically comprising: An initial point cloud based on a 3D Gaussian scene is obtained according to the input image, and a 3D Gaussian initial expression of the scene is obtained based on the initial point cloud.

3. The 3D Gaussian panoptic segmentation method according to claim 2, wherein: A 3D Gaussian initial expression of the scene is obtained based on the initial point cloud, specifically including: considering the position of each initial point cloud as the center of a Gaussian ellipsoid, randomly initializing the shape of the Gaussian ellipsoid, obtaining a scene composed of multiple Gaussian ellipsoids of different shapes, and thus obtaining a 3D Gaussian initial expression.

4. The 3D Gaussian panoptic segmentation method according to claim 1, wherein: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information, specifically including: Based on the camera parameters, each Gaussian ellipsoid in the 3D Gaussian initial expression is projected onto a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected into a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information and instance information on the 2D plane.

5. The 3D Gaussian panoptic segmentation method according to claim 4, wherein: Obtaining a first generated image according to the color information, specifically comprising: performing Gaussian differentiable rasterization on the Gaussian ellipse, and mixing color information of each pixel to obtain the first generated image, wherein the first generated image is a rendered image; Obtaining a second generated image according to the semantic information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse and mixing the semantic information of each pixel point to obtain the second generated image, wherein the second generated image is a rendered semantic segmentation map.

6. The 3D Gaussian panoptic segmentation method according to claim 5, wherein: Obtaining a third generated image according to the instance information specifically includes: The Gaussian ellipse is subjected to Gaussian differentiable rasterization, and instance information of each pixel is mixed to obtain a third generated image, where the third generated image is a rendered instance segmentation map.

7. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the 3D Gaussian panoptic segmentation method based on 2D pre-trained segmentation model supervision according to any one of claims 1 to 6.

8. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision are implemented as described in any one of claims 1 to 6.

Images