3D Gaussian panoramic segmentation method based on 2D pre-training segmentation model supervision, computer equipment and program product

Abstract

The invention relates to a 3D Gaussian panoramic segmentation method based on supervision of a 2D pre-training segmentation model, computer equipment and a program product, and the 3D Gaussian panoramic segmentation method comprises the steps: inputting a plurality of input images of different visual angles of the same scene into the 2D pre-training segmentation model, and obtaining a 2D semantic segmentation image and a 2D instance segmentation image, obtaining a 3D Gaussian initial expression based on the 3D Gaussian scene, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids; obtaining a 2D Gaussian expression obtained by projecting the 3D Gaussian initial expression under a corresponding visual angle; obtaining a first generated image according to the color information, obtaining a second generated image according to the semantic information, and obtaining a third generated image according to the instance information; training and optimizing the 3D Gaussian scene expression by using a preset loss parameter to obtain a trained 3D Gaussian segmentation model; and based on the 3D Gaussian segmentation model, obtaining semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a 3D Gaussian panoramic segmentation method, computer device, and program product based on supervision of a 2D pre-trained segmentation model. Background Art

[0002] It is very necessary to establish an understanding of the semantics and other information of the entire 3D scene from a 2D image. However, directly segmenting the image using only a 2D segmentation model will lack 3D information, making it difficult to establish a holistic understanding of the scene.

[0003] Existing panoramic segmentation models often have the following three problems: 1) It is difficult to produce segmentation results with 3D consistency, that is, the segmentation results of the same object under different viewing angles are different. 2) The training speed is too slow. The current methods often take up to 24 hours or more for training. 3) The rendering speed is slow, and only more than ten segmentation maps can be rendered in one second. Summary of the invention

[0004] Based on this, it is necessary to provide a 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision to address the above technical problems.

[0005] This application is based on a 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model, including:

[0006] Input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, obtain camera parameters corresponding to the input image according to the input image, and obtain a 3D Gaussian initial expression based on a 3D Gaussian scene according to the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids;

[0007] Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information;

[0008] Obtain a first generated image according to the color information, obtain a second generated image according to the semantic information, and obtain a third generated image according to the instance information;

[0009] Optimizing the 3D Gaussian scene expression by training with preset loss parameters to obtain a trained 3D Gaussian segmentation model, wherein the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result, wherein the mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map;

[0010] Based on the 3D Gaussian segmentation model, a 3D Gaussian panoramic segmentation result is obtained, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0011] Optionally, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids, specifically includes:

[0012] An initial point cloud based on a 3D Gaussian scene is obtained according to the input image, and a 3D Gaussian initial expression of the scene is obtained based on the initial point cloud.

[0013] Optionally, the mapping instance segmentation result is obtained by matching the third generated image with the corresponding 2D instance segmentation map, specifically including:

[0014] The instance ID of each pixel in the 2D instance segmentation map is matched and mapped to the third generated image to obtain a mapped instance segmentation result for supervision.

[0015] Optionally, a 3D Gaussian initial expression of the scene is obtained based on the initial point cloud, specifically including: regarding the position of each initial point cloud as the center of a Gaussian ellipsoid, randomly initializing the shape of the Gaussian ellipsoid, and obtaining a scene composed of multiple Gaussian ellipsoids of different shapes, thereby obtaining a 3D Gaussian initial expression.

[0016] Optionally, based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, wherein the 2D Gaussian expression includes color information, semantic information, and instance information, specifically including:

[0017] Based on the camera parameters, each Gaussian ellipsoid in the 3D Gaussian initial expression is projected onto a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected into a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information and instance information on the 2D plane.

[0018] Optionally, obtaining a first generated image according to the color information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse and mixing color information of each pixel point, thereby obtaining the first generated image;

[0019] Obtaining a second generated image according to the semantic information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse and mixing the semantic information of each pixel point to obtain the second generated image.

[0020] Optionally, the semantic information of each pixel is obtained using the following formula:

[0021]

[0022] In the formula,

[0023] κ p Represents the semantic information corresponding to pixel p;

[0024] L s Represents a neural network, which is used to map semantic encoding into probability distributions corresponding to various semantics;

[0025] e i Represents the color information corresponding to the i-th Gaussian ball;

[0026] α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p;

[0027] α′ j Represents the opacity of the j-th Gaussian sphere.

[0028] Optionally, obtaining a third generated image according to the instance information specifically includes:

[0029] The Gaussian ellipse is subjected to Gaussian differentiable rasterization, and instance information of each pixel is mixed to obtain a third generated image.

[0030] Optionally, the third preset loss is obtained using the following formula:

[0031]

[0032] in, is the third preset loss, used to represent the instance loss;

[0033] |I| represents the number of pixels in the image;

[0034] w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model;

[0035] h represents the hth instance;

[0036] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance;

[0037] is the probability that p pixels correspond to the p-th class instance predicted by the 2D pre-trained segmentation model;

[0038] The 2D instance segmentation map is mapped by linear matching to obtain the mapping instance segmentation result:

[0039]

[0040] in, A mapping representing a selection;

[0041] Π I Indicates the current mapping;

[0042] H I Represents all instances of an image;

[0043] |I h | represents the number of pixels of the image corresponding to the h instance;

[0044] Π I (h) represents the instance number to which the hth instance is mapped;

[0045] w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model;

[0046] h represents the hth instance;

[0047] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance.

[0048] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0049] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0050] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision described in the present application.

[0051] The 3D Gaussian panoramic segmentation method based on the supervision of the 2D pre-trained segmentation model in this application has at least the following effects:

[0052] The 3D Gaussian scene expression after training and optimization in this application can conform to the real scene corresponding to the picture. Specifically, the 3D Gaussian expression is optimized through the first preset loss to complete the process. On this basis, the semantic representation of the scene is further optimized through the second preset loss; the instance representation of the scene is further optimized through the third preset loss. The optimized model has semantic and instance information that can fit the real scene, which can be used to obtain a panoramic expression of the entire scene, that is, to complete the entire method flow. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of a 3D Gaussian panoramic segmentation method supervised by a 2D pre-trained segmentation model in one embodiment of the present application;

[0054] Figure 2 This is a model architecture diagram of a 3D Gaussian panoramic segmentation method supervised by a 2D pre-trained segmentation model in an embodiment of the present application;

[0055] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] See also Figure 1 An embodiment of the present application provides a 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision, including steps S100 to S800.

[0058] Step S100, input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model, obtain a 2D semantic segmentation map and a 2D instance segmentation map, and obtain camera parameters corresponding to the input images according to the input images.

[0059] Step S200, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0060] Step S300: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information.

[0061] Step S400: obtaining a first generated image according to color information.

[0062] Step S500: obtaining a second generated image according to semantic information.

[0063] Step S600: obtaining a third generated image according to the instance information.

[0064] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0065] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, wherein the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0066] See also Figure 1 and Figure 2 In some embodiments, a 3D Gaussian panoptic segmentation method based on 2D pre-trained segmentation model supervision is provided, which is used to explain the above steps S100 to S800 in sections and state the sub-steps that can be optionally implemented. It includes: (i) using the input image to establish a 3D Gaussian representation of the scene; (ii) projecting the 3D Gaussian representation into a 2D pixel space; (iii) an overall loss function.

[0067] (I) Using the input image to establish a 3D Gaussian representation of the scene, including steps S100 to S200:

[0068] Step S100, input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model, obtain a 2D semantic segmentation map and a 2D instance segmentation map, and obtain camera parameters corresponding to the input images according to the input images.

[0069] Step S200, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0070] In step S100, multiple input images of the same scene from different perspectives are input into a pre-trained 2D segmentation model, and semantic segmentation information (2D semantic segmentation map) and instance segmentation information (2D instance segmentation map) of each pixel corresponding to each 2D image can be obtained. Multiple input images of the input multi-perspective picture (i.e., the actual image as shown in the figure). The 2D pre-trained segmentation model can be, for example, a "Mask2Former model". Semantic segmentation information and instance segmentation information only stay at the 2D level, and it cannot integrate information from other perspectives, and there will be 3D inconsistencies.

[0071] Semantic information can be understood as categories. For example, if there is a cat in a picture, then in the semantic map, the corresponding pixel is the category of the cat. Instance information is like an ID card. Each object has a different instance ID. For example, if there are two cats in the picture, their instance information will be 1 and 2 respectively. Instance information is different from semantic segmentation. In semantic segmentation, the categories of the two cats are the same, but they are inconsistent in instance segmentation. The panoramic segmentation provided in this embodiment includes both semantic segmentation and instance segmentation, which can identify both the semantic information corresponding to each pixel and the instance information corresponding to the pixel.

[0072] Step S200 specifically includes obtaining an initial point cloud based on a 3D Gaussian scene according to an input image, regarding the position of each initial point cloud as the center of a Gaussian ellipsoid, randomly initializing the shape of the Gaussian ellipsoid, and obtaining a scene composed of multiple Gaussian ellipsoids of different shapes, that is, obtaining a 3D Gaussian initial expression.

[0073] Specifically, the input image is input into the Colmap software to obtain the initial point cloud and camera parameters based on the 3D Gaussian scene. It can be understood that the Colmap software is based on the principle of the SFM (Structure-from-Motion) algorithm and can reconstruct the general information of the scene by matching the key information between each picture, including the initial point cloud and the perspective information corresponding to each picture in this scene, that is, the camera parameters.

[0074] According to step S100 and step S200, the input images are generally photos of the same scene at different viewing angles, and the 3D information of the scenes corresponding to these photos can be obtained. This embodiment establishes a preliminary 3D Gaussian expression based on these 3D information, and then obtains the rendering of the 3D Gaussian expression at any viewing angle. It can be understood that step S200 is essentially to fit each object in the scene with ellipsoids of different shapes, and each 3D Gaussian ball (Gaussian ellipsoid) has color, semantics and instance information.

[0075] Since 3D Gaussian itself is an explicit representation, compared to implicit expression methods such as NeRF (Neural Radiance Field), it does not require additional calculation of the parts of the space where objects do not exist during the rendering process, thus greatly speeding up the rendering rate. Specifically, the FPS (Frames Per Second) of panoramic segmentation on the A6000 GPU using methods such as NeRF is about 10 frames per second, while the expression based on 3D Gaussian can reach about 140 frames per second, which is more than ten times faster.

[0076] (ii) Projecting the 3D Gaussian representation to the 2D pixel space, including steps S300 to S600:

[0077] Step S300: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information.

[0078] Step S300 specifically includes: based on the camera parameters, projecting each Gaussian ellipsoid in the 3D Gaussian initial expression to a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected into a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information and instance information on the 2D plane, that is, obtaining a 2D Gaussian expression.

[0079] It can be understood that each 3D Gaussian sphere has color information, semantic information and instance information. After projection, a single pixel or multiple pixels as a whole also have color information, semantic information and instance information accordingly. The above steps have established the 3D Gaussian initial expression of the scene, that is, the preliminary representation of the scene. With the 3D scene representation, the 2D information corresponding to each viewing angle can be obtained only through projection and other methods. It can be understood that the 2D Gaussian expression is the picture obtained by projection. For each picture that needs to be projected, the 3D Gaussian sphere is projected onto the 2D plane of the corresponding viewing angle of the picture using its corresponding camera parameters. At this time, the 3D Gaussian ellipsoid becomes a 2D Gaussian ellipse. These Gaussian ellipses have center coordinates, shape parameters, color, semantics, and instance information.

[0080] Specifically, each 3D Gaussian ellipsoid corresponds to a Gaussian ellipse when projected to a certain viewing angle through camera parameters. The entire 2D image is covered by a series of 2D Gaussian ellipses. Each pixel point is covered by multiple ellipses. These ellipses have color, semantics, instance and other information. Through Gaussian rasterization, the information of each ellipse is mixed on the 2D plane, so as to obtain the color, semantics, instance and other information of the corresponding pixel.

[0081] Step S400, obtaining a first generated image according to the color information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, mixing the color information of each pixel point, thereby obtaining the first generated image, that is, the rendered image shown in the figure.

[0082] The 2D Gaussian ellipse is subjected to Gaussian differentiable rasterization, and the color information of each pixel is mixed to obtain the color of any pixel in the image, thereby obtaining the first generated image. The color calculation formula for each pixel is as follows:

[0083]

[0084] In the formula, C(p) represents the color corresponding to pixel p, c i Represents the color information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j The opacity of the jth Gaussian ball corresponds to this formula. The meaning of this formula is to weight the opacity of all Gaussian balls covering each pixel and obtain the color information of the pixel.

[0085] Step S500, obtaining a second generated image according to the semantic information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, mixing the semantic information of each pixel point, thereby obtaining the second generated image, that is, the rendered semantic segmentation map shown in the figure.

[0086] By rasterizing the 2D Gaussian ellipse with Gaussian differentiable rasterization and mixing the semantic information of each pixel, we can get the semantic segmentation result of any pixel in the image, thereby obtaining the second generated image. The formula for calculating the semantic information of each pixel is as follows:

[0087]

[0088] Where k p represents the semantic information corresponding to pixel p, e i Represents the semantic information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j Corresponding to the opacity of the jth Gaussian ball, L s Represents a neural network that maps semantic encodings to probability distributions corresponding to various semantics. The meaning of this formula is to fuse all Gaussian balls covering each pixel by weighted opacity to obtain the semantic information under the pixel. After obtaining the probability distribution of all semantics corresponding to each pixel, we only need to select the semantics corresponding to the pixel with the largest corresponding probability.

[0089] Step S600, obtaining a third generated image according to the instance information, specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse, mixing the instance information of each pixel point, and obtaining the third generated image, that is, the rendered instance segmentation map shown in the figure.

[0090] By performing Gaussian differentiable rasterization on the 2D Gaussian ellipse and mixing the instance segmentation information of each pixel, the instance segmentation result corresponding to any pixel in the image can be obtained, thereby obtaining the third generated image.

[0091] The instance information of each pixel is calculated as follows:

[0092]

[0093] Where π p Indicates the instance information corresponding to pixel p, o i Represents the instance information corresponding to the i-th Gaussian ball, α′ i (p) represents the opacity of the i-th Gaussian ball at pixel p, and each α′ j Corresponding to the opacity of the jth Gaussian ball, L o Represents a neural network that maps instance encodings to probability distributions corresponding to various instances. The meaning of this formula is to fuse all Gaussian balls covering each pixel by weighted opacity to obtain the instance information under the pixel. After obtaining the probability distribution of all instances corresponding to each pixel, we only need to select the instance with the largest corresponding probability as the instance corresponding to the pixel.

[0094] (III) Overall loss function, including steps S700 to S800:

[0095] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0096] The mapping instance segmentation result is obtained by matching and mapping the third generated image with the corresponding 2D instance segmentation map, including: matching and mapping the instance ID of each pixel of the 2D instance segmentation map to the third generated image to obtain a mapping instance segmentation result for supervision.

[0097] Specific mapping process of matching: The 3D instance segmentation result (third generated image) obtained by the final rendering of 3D Gaussian has 3D consistency. That is, for the same object, the instance segmentation ID under different viewing angles should be consistent. The 2D instance segmentation map obtained by the 2D segmentation model is segmented on the image plane and does not have 3D consistency. This embodiment maps the instance ID of each pixel of the 2D instance segmentation map to the ID of the rendered instance segmentation map (third generated image).

[0098] Then, this embodiment proposes a criterion for measuring matching, that is, given an instance matching relationship: after mapping the instance ID of the 2D instance segmentation map through the matching relationship, a mapped 2D instance segmentation map is obtained. Then, the mapped 2D instance segmentation map is compared pixel by pixel with the rendered instance segmentation map (the third generated image). The more pixels with the same instance segmentation result, the better the matching relationship. This problem can be transformed into a linear allocation problem: that is, to find an optimal matching mapping method, which has the most pixels that meet the above criteria. This problem can be solved using algorithms such as KM.

[0099] In this embodiment, the contrast loss between the first generated image and the input image is used as the first preset loss. The contrast loss between the second generated image and the 2D semantic segmentation map is used as the second preset loss. The contrast loss between the third generated image and the mapping instance segmentation result is used as the third preset loss. Specifically, the preset loss parameter is obtained by adding weights to the first preset loss, the second preset loss, and the third preset loss, respectively, and then accumulating and summing them.

[0100] In this embodiment, the first preset loss, the second preset loss and the third preset loss are used to optimize the 3D Gaussian scene at the same time. The optimization process is mainly to enable the 3D Gaussian scene to learn the information of the scene corresponding to the picture, so as to express the semantics, instance and other information of the entire scene.

[0101] It can be understood that the first generated image is obtained by mixing the color information of the 2D Gaussian, and the first generated image is the projection image of the model at this viewing angle, and they are corresponding, so the first preset loss can be obtained by indirectly comparing the reference image and the first generated image. The second and third preset losses are the same.

[0102] Furthermore, the optimized 3D Gaussian scene expression of this embodiment can conform to the real scene corresponding to the picture. The 3D Gaussian expression is optimized by the first preset loss to complete the process. On this basis, the semantic representation of the scene is further optimized by the second preset loss, and the instance representation of the scene is further optimized by the third preset loss. The optimized model has semantic and instance information that can fit the real scene, which can be used to obtain a panoramic expression of the entire scene, that is, to complete the entire method flow.

[0103] The first preset loss is color loss It establishes the color and other information of the entire scene. Color loss As shown below:

[0104]

[0105] in, represents color loss; a is a constant used to represent and The weight is 0.2, which represents the loss between the rendered color and the actual color; represents L1 loss, Represents the SSIM loss of the image.

[0106] The second preset loss, namely semantic loss The semantic information of the entire scene is established. As shown below:

[0107]

[0108] in, Indicates semantic loss;

[0109] |I| represents the number of pixels in the image;

[0110] w p Represents the confidence of each pixel predicted by the 2D segmentation model;

[0111] k represents the kth semantics;

[0112] κ p (k) represents the probability that the rendered p-th pixel corresponds to the k-th semantic category;

[0113] It is the probability that the p pixels predicted by the 2D segmentation model correspond to the k-th semantic category.

[0114] The third preset loss, namely instance loss The instance information of the entire scene is established. As shown below:

[0115]

[0116] in, represents the instance loss;

[0117] |I| represents the number of pixels in the image;

[0118] w pRepresents the confidence of each pixel predicted by the 2D segmentation model;

[0119] h represents the hth instance;

[0120] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance;

[0121] is the probability that p pixels correspond to the p-th class instance predicted by the 2D segmentation model.

[0122] It is a method of mapping the instance information predicted by the 2D segmentation model (2D instance segmentation map) into the instance information of the scene (mapping instance segmentation result) by linear matching. The formula is:

[0123]

[0124] in, represents the selected optimal mapping;

[0125] Π I Indicates the current mapping;

[0126] H I Represents all possible instances of an image;

[0127] |I h | represents the number of pixels of the image corresponding to the h instance;

[0128] Π I (h) represents the instance number to which the hth instance is mapped;

[0129] w p Represents the confidence of each pixel predicted by the 2D segmentation model;

[0130] h represents the hth instance;

[0131] π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance.

[0132] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, wherein the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0133] The embodiments of the present application can derive the semantic and instance segmentation information representation of the entire scene from multi-view images with the help of a 2D segmentation model, and with the help of a 3D Gaussian scene representation, enable the panoramic segmentation model to produce a segmentation result with 3D consistency.

[0134] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0135] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0136] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0137] Step S100, input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model, obtain a 2D semantic segmentation map and a 2D instance segmentation map, and obtain camera parameters corresponding to the input images according to the input images.

[0138] Step S200, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0139] Step S300: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information.

[0140] Step S400: obtaining a first generated image according to color information.

[0141] Step S500: obtaining a second generated image according to semantic information.

[0142] Step S600: obtaining a third generated image according to the instance information.

[0143] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0144] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, wherein the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0145] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0146] Step S100, input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model, obtain a 2D semantic segmentation map and a 2D instance segmentation map, and obtain camera parameters corresponding to the input images according to the input images.

[0147] Step S200, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0148] Step S300: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information.

[0149] Step S400: obtaining a first generated image according to color information.

[0150] Step S500: obtaining a second generated image according to semantic information.

[0151] Step S600: obtaining a third generated image according to the instance information.

[0152] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0153] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, wherein the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0154] In one embodiment, a computer program product is provided, comprising computer instructions, which when executed by a processor implement the following steps:

[0155] Step S100, input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model, obtain a 2D semantic segmentation map and a 2D instance segmentation map, and obtain camera parameters corresponding to the input images according to the input images.

[0156] Step S200, obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to an input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing a plurality of Gaussian ellipsoids.

[0157] Step S300: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information.

[0158] Step S400: obtaining a first generated image according to color information.

[0159] Step S500: obtaining a second generated image according to semantic information.

[0160] Step S600: obtaining a third generated image according to the instance information.

[0161] Step S700, using preset loss parameters to train and optimize the 3D Gaussian scene expression to obtain a trained 3D Gaussian segmentation model, the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result. The mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map.

[0162] Step S800: obtaining a 3D Gaussian panoramic segmentation result based on a 3D Gaussian segmentation model, wherein the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

[0163] In the present embodiment, the computer program product includes a program code portion for executing the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via a RAN, via the Internet and / or via an RBS). Alternatively or additionally, the method may be encoded in a field programmable gate array (FPGA) and / or an application specific integrated circuit (ASIC), or functionality may be provided for download by means of a hardware description language.

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0165] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. When the technical features in different embodiments are embodied in the same figure, it can be regarded that the figure also discloses the combination examples of the various embodiments involved.

[0166] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A 3D Gaussian panoptic segmentation method supervised by a 2D pre-trained segmentation model, characterized in that: include: Input multiple input images of the same scene from different perspectives into a 2D pre-trained segmentation model to obtain a 2D semantic segmentation map and a 2D instance segmentation map, obtain camera parameters corresponding to the input image according to the input image, and obtain a 3D Gaussian initial expression based on a 3D Gaussian scene according to the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids; Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, where the 2D Gaussian expression includes color information, semantic information, and instance information; Obtain a first generated image according to the color information, obtain a second generated image according to the semantic information, and obtain a third generated image according to the instance information; Optimizing the 3D Gaussian scene expression by training with preset loss parameters to obtain a trained 3D Gaussian segmentation model, wherein the preset loss parameters include: a first preset loss between the first generated image and the corresponding input image, a second preset loss between the second generated image and the corresponding 2D semantic segmentation map, and a third preset loss between the third generated image and the mapping instance segmentation result, wherein the mapping instance segmentation result is obtained by matching mapping between the third generated image and the corresponding 2D instance segmentation map; Based on the 3D Gaussian segmentation model, a 3D Gaussian panoramic segmentation result is obtained, where the 3D Gaussian panoramic segmentation result includes semantic segmentation information and instance segmentation information of the 3D Gaussian scene.

2. The 3D Gaussian panoptic segmentation method according to claim 1, characterized in that: Obtaining a 3D Gaussian initial expression based on a 3D Gaussian scene according to the input image, wherein the 3D Gaussian initial expression is a scene formed by randomly initializing multiple Gaussian ellipsoids, specifically comprising: An initial point cloud based on a 3D Gaussian scene is obtained according to the input image, and a 3D Gaussian initial expression of the scene is obtained based on the initial point cloud.

3. The 3D Gaussian panoptic segmentation method according to claim 2, characterized in that: The 3D Gaussian initial expression of the scene is obtained based on the initial point cloud, specifically including: considering the position of each initial point cloud as the center of a Gaussian ellipsoid, randomly initializing the shape of the Gaussian ellipsoid, obtaining a scene composed of multiple Gaussian ellipsoids of different shapes, and thus obtaining a 3D Gaussian initial expression.

4. The 3D Gaussian panoptic segmentation method according to claim 1, wherein: Based on the camera parameters, a 2D Gaussian expression is obtained by projecting the 3D Gaussian initial expression at a corresponding viewing angle, wherein the 2D Gaussian expression includes color information, semantic information, and instance information, specifically including: Based on the camera parameters, each Gaussian ellipsoid in the 3D Gaussian initial expression is projected onto a 2D plane corresponding to the camera parameters, so that the Gaussian ellipsoid is projected into a Gaussian ellipse, and each Gaussian ellipse forms color information, semantic information and instance information on the 2D plane.

5. The 3D Gaussian panoptic segmentation method according to claim 4, characterized in that: Obtaining a first generated image according to the color information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse and mixing color information of each pixel point to obtain the first generated image, wherein the first generated image is a rendered image; Obtaining a second generated image according to the semantic information specifically includes: performing Gaussian differentiable rasterization on the Gaussian ellipse and mixing the semantic information of each pixel point to obtain the second generated image, wherein the second generated image is a rendered semantic segmentation map.

6. The 3D Gaussian panoptic segmentation method according to claim 5, characterized in that: Obtaining a third generated image according to the instance information specifically includes: The Gaussian ellipse is subjected to Gaussian differentiable rasterization, and instance information of each pixel is mixed to obtain a third generated image, where the third generated image is a rendered instance segmentation map.

7. The 3D Gaussian panoptic segmentation method according to claim 1, characterized in that: The mapping instance segmentation result is obtained by matching the third generated image with the corresponding 2D instance segmentation map, specifically comprising: The instance ID of each pixel in the 2D instance segmentation map is matched and mapped to the third generated image to obtain a mapped instance segmentation result for supervision.

8. The 3D Gaussian panoptic segmentation method according to claim 7, characterized in that: The third preset loss is obtained using the following formula: in, is the third preset loss, used to represent the instance loss; |I| represents the number of pixels in the image; w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model; h represents the hth instance; π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance; is the probability that p pixels correspond to the p-th class instance predicted by the 2D pre-trained segmentation model; The 2D instance segmentation map is mapped by linear matching to obtain the mapping instance segmentation result: in, A mapping representing a selection; Π, represents the current mapping; H, represents all instances of the image; |I h | represents the number of pixels of the image corresponding to instance h; Π I (h) represents the instance number to which the hth instance is mapped; w p Represents the confidence of each pixel predicted by the 2D pre-trained segmentation model; h represents the hth instance; π p Indicates the probability that the rendered p-th pixel corresponds to the p-th class instance.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by the processor, the steps of the 3D Gaussian panoramic segmentation method based on 2D pre-trained segmentation model supervision as described in any one of claims 1 to 8 are implemented.

Images