Scene instance segmentation method and device, equipment, storage medium and product

Abstract

The invention discloses a scene instance segmentation method and device, equipment, a storage medium and a product, and relates to the technical field of image processing, and the scene instance segmentation method comprises the steps: obtaining image data in a target scene; processing the image data based on the preset segmentation model, and outputting a segmentation mask; and processing the segmentation mask through a three-dimensional Gaussian representation technology to generate an initial segmentation result, and complementing or removing boundary Gaussian distribution in the initial segmentation result to obtain a final segmentation result. According to the invention, the smoothness of instance segmentation is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a scene instance segmentation method, apparatus, device, storage medium and computer program product. Background Art

[0002] In recent years, as network services have grown exponentially, the supervision and management of professional network facilities has become an issue that must be considered behind network applications. Traditional private network park management mainly relies on manual labor, which puts a test on cost, efficiency, and problem location accuracy. In order to timely monitor and locate the anomalies of private network parks, park maintenance is performed to complete the subsequent scene understanding tasks.

[0003] The scene instance segmentation task is to learn the information of a scene and segment the scene according to the learned information to obtain individual instances. The traditional scene segmentation method mainly integrates artificial intelligence technology to optimize these problems. Fast instance segmentation for campus scenes is based on point cloud representation. By rendering the scene point cloud from different perspectives to obtain a two-dimensional image, and guiding the three-dimensional through two-dimensional segmentation, or directly learning the scene information of the point cloud through a large deep learning network, however, since the point cloud needs to be scanned in the scene and the scanning cost is high, it is also easy to have the problem of scanning residual point clouds, resulting in the surface of the segmented instance being not smooth enough, which greatly affects the accuracy of subsequent tasks.

[0004] Therefore, how to improve the smoothness of the generated segmentation instances is a problem that needs to be solved urgently.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present application is to provide a scene instance segmentation method, apparatus, equipment, storage medium and computer program product, aiming to solve the technical problem in the related technology that the scene is scanned by point cloud representation method and the scanning cost is high, and it is also easy to obtain residual point clouds during scanning, resulting in the surface of the segmented instance being not smooth enough, which greatly affects the accuracy of subsequent tasks.

[0007] To achieve the above objectives, the present application proposes a scene instance segmentation method, which includes:

[0008] Acquire image data of the target scene;

[0009] Based on the preset segmentation model, the image data is processed to output a segmentation mask;

[0010] The segmentation mask is processed by a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and the boundary Gaussian distribution in the initial segmentation result is supplemented or removed to obtain a final segmentation result.

[0011] In one embodiment, before the step of processing the image data based on the preset segmentation model and outputting a segmentation mask, the step further includes:

[0012] Extract historical image data under preset scenarios;

[0013] Inputting the historical image data into a pre-trained segmentation model, and performing data processing on the historical image data based on the pre-trained segmentation model to obtain a first image feature and a first segmentation mask of a plurality of instances;

[0014] Calculating a first mask loss based on the first segmentation mask and the first image feature;

[0015] The pre-trained segmentation model is iteratively optimized through the first mask loss to obtain a trained segmentation model.

[0016] In one embodiment, the step of calculating a first mask loss based on the first segmentation mask and the first image feature includes:

[0017] Based on the first segmentation mask and the first image feature, a first query mask corresponding to each instance is generated, and a feature scene corresponding to the image feature is rendered by a three-dimensional Gaussian characterization technology to obtain a rendering feature;

[0018] generating a feature mask based on the first query mask and the rendering feature;

[0019] Based on the first segmentation mask and the feature mask, a first mask loss is calculated, wherein the first mask loss includes mask loss and pixel loss, the mask loss is used to optimize the feature attributes of the scene Gaussian and the network parameters of the feature learning network, and the pixel loss is used to compare the pixel-by-pixel masks to make the features of the same instance compact.

[0020] In one embodiment, the step of processing the segmentation mask by using a three-dimensional Gaussian characterization technique to generate an initial segmentation result includes:

[0021] Extracting a second image feature of the segmentation mask, and obtaining a second query mask of the instance to be segmented based on the second image feature and the segmentation mask;

[0022] The feature attributes of the second query mask are calculated by three-dimensional Gaussian characterization technology to generate an initial segmentation result.

[0023] In one embodiment, the step of calculating the feature attributes of the second query mask by using a three-dimensional Gaussian characterization technique to generate an initial segmentation result includes:

[0024] Calculating characteristic attributes of the second query mask by using a three-dimensional Gaussian characterization technique to obtain multiple scene Gaussian distributions;

[0025] A first scene Gaussian distribution greater than a preset threshold is extracted from the scene Gaussian distribution, and the first scene Gaussian distribution is used as an initial segmentation result.

[0026] In one embodiment, the step of completing or removing the boundary Gaussian distribution in the initial segmentation result to obtain the final segmentation result includes:

[0027] Extracting instance missing points in the boundary Gaussian distribution in the initial segmentation result;

[0028] The missing points of the instance are processed by boundary completion through the neighborhood Gaussian aggregation method, and the boundary Gaussian distribution is decoupled through the boundary Gaussian decoupling method to obtain the final segmentation result.

[0029] In addition, to achieve the above-mentioned purpose, the present application also proposes a scene instance segmentation device, the scene instance segmentation device comprising:

[0030] An acquisition module is used to acquire image data in a target scene;

[0031] A first processing module, used for processing the image data based on the preset segmentation model and outputting a segmentation mask;

[0032] The second processing module is used to process the segmentation mask by using a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and to complete or remove the boundary Gaussian distribution in the initial segmentation result to obtain a final segmentation result.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a scene instance segmentation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the scene instance segmentation method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the scene instance segmentation method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the scene instance segmentation method described above are implemented.

[0036] The present application proposes a scene instance segmentation method, apparatus, device, storage medium and computer program product. The present application obtains image data in a target scene, performs instance segmentation on the image data through a preset segmentation model, obtains an instance mask after the image data is segmented, and then characterizes or calculates the segmentation mask through a three-dimensional Gaussian characterization technology to generate an initial segmentation result. At this time, the initial segmentation result cannot accurately reflect the instance distribution in the scene. Then, the Gaussian distribution on the boundary in the initial segmentation result is complemented, or the redundant part outside the boundary is removed, thereby improving the surface smoothness of the segmented instance and improving the segmentation quality, thereby avoiding affecting the accuracy of subsequent tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart of the first embodiment of the method for segmenting a scene instance of the present application is provided;

[0040] Figure 2 A schematic diagram of the segmentation reasoning process involved in the scenario instance segmentation method of this application;

[0041] Figure 3 A flowchart of the second embodiment of the method for segmenting a scene instance of the present application is provided;

[0042] Figure 4 This is a schematic diagram of the model training process involved in Example 2 of the present application;

[0043] Figure 5 A flowchart of the third embodiment of the method for scene instance segmentation of the present application is provided;

[0044] Figure 6 This is a schematic diagram of the module structure of the scene instance segmentation device according to an embodiment of the present application;

[0045] Figure 7Schematic diagram of the device structure of the hardware operating environment involved in the scene instance segmentation method in the embodiment of the present application.

[0046] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solutions of the embodiments of this application are:

[0050] Acquire image data of the target scene;

[0051] Based on the preset segmentation model, the image data is processed to output a segmentation mask;

[0052] The segmentation mask is processed by a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and the boundary Gaussian distribution in the initial segmentation result is supplemented or removed to obtain a final segmentation result.

[0053] In the existing technology, the traditional scene segmentation method mainly optimizes these problems by integrating artificial intelligence technology. The rapid instance segmentation for the park scene is based on point cloud representation, which renders the scene point cloud from different perspectives to obtain a two-dimensional image, and guides the three-dimensional through two-dimensional segmentation, or directly learns the scene information of the point cloud through a large deep learning network. However, since the point cloud needs to be scanned in the scene and the scanning cost is high, it is also easy to get the problem of residual point cloud scanned, resulting in the surface of the segmented instance is not smooth enough, which greatly affects the accuracy of subsequent tasks.

[0054] The present application proposes a scene instance segmentation method, apparatus, device, storage medium and computer program product. The present application obtains image data in a target scene, performs instance segmentation on the image data through a preset segmentation model, obtains an instance mask after the image data is segmented, and then characterizes or calculates the segmentation mask through a three-dimensional Gaussian characterization technology to generate an initial segmentation result. At this time, the initial segmentation result cannot accurately reflect the instance distribution in the scene. Then, the Gaussian distribution on the boundary in the initial segmentation result is complemented, or the redundant part outside the boundary is removed, thereby improving the surface smoothness of the segmented instance and improving the segmentation quality, thereby avoiding affecting the accuracy of subsequent tasks.

[0055] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a scene instance segmentation device, etc. The following takes the scene instance segmentation device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a scene instance segmentation method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the scene instance segmentation method of the present application.

[0057] In this embodiment, the scene instance segmentation method includes steps S10 to S30:

[0058] Step S10, acquiring image data of a target scene;

[0059] It should be noted that the target scene is an image scene that needs to be detected, and in this target scene, image data can be acquired through a camera or other devices.

[0060] It should be noted that the image data may be in jpg format, Jpeg format, etc., without any specific limitation.

[0061] Step S20, processing the image data based on the preset segmentation model and outputting a segmentation mask;

[0062] It should be noted that the preset segmentation model is a two-dimensional segmentation model SAM (Segment Anything Model). In related technologies, the real images in the training set are segmented mainly by using a general segmentation model SAM based on a neural radiation field, which usually relies on two-dimensional images, and the instance segmentation in three-dimensional space is guided by the segmentation mask. Unlike the method of projecting point clouds onto different planes, the real images in the training set completely retain the possible information in the scene, and point cloud projection may cause information errors in the two-dimensional image due to the discrete representation and occlusion of the point cloud. Therefore, the method based on SAM segmentation generally obtains higher segmentation quality.

[0063] It should be noted that the segmentation mask is a two-dimensional matrix belonging to different object categories obtained by classifying and marking each pixel in the image data through a two-dimensional segmentation model. Each object in the image data is identified through the two-dimensional segmentation model to generate the corresponding target instance.

[0064] It should be noted that the mask generation uses the general model SAM for 2D image segmentation tasks. For Gaussian representation, the segmentation task can guide the segmentation of 3D scenes by segmenting the real images in the training set. For SAM, the input is the real image I v , in the training phase, a segmentation mask of the instance contained in the camera view v is generated. In the inference phase, a single segmentation mask of the corresponding instance can be generated according to the prompt SAM. The calculation method is as follows:

[0065] M0,M1,…,M N =SAM(I v ),

[0066]

[0067] Among them I v is the real image under the viewing angle v, is the prompt in the reasoning stage, N is the number of instances in the corresponding view scene, and M is the generated mask.

[0068] Specifically, the lightweight feature learning network used for cross-view information fusion is based on ResNet and FPN. ResNet is used to perform multi-stage downsampling convolution to extract features for the entire image, and the FPN method is used to perform feature fusion operations on the feature maps of multiple stages to obtain image features. Using FPN to enhance the details of the image in the shallow network, while the deep network pays more attention to the high-level semantic information. Fusion of feature information at different levels through feature pyramid can effectively improve the network's feature extraction and representation capabilities. The feature images at each level are obtained by combining ResNet and FPN, and finally the feature fusion of each level is achieved by interpolating and fusing the image of the next layer layer by layer.

[0069] In this embodiment, the segmentation result is obtained according to the prompt of the two-dimensional segmentation model, and the feature image is learned through a lightweight network to provide view consistency for instances of cross-view images, and can provide support for subsequent work and multimodal work.

[0070] It is important to understand that there are currently the following challenges for scene segmentation:

[0071] 1. The surface obtained by segmenting the instance is not smooth enough;

[0072] 2. Current methods require large scene datasets and a lot of annotations;

[0073] 3. The computational overhead is high and the learning time of the scene is consumed.

[0074] In response to problem 1, this application proposes to use 3D Gaussian 3DGS as a scene representation to obtain a smoother and more realistic segmentation instance. As an emerging neural radiation method, 3D Gaussian has both the high-quality rendering results of NeRF and the efficient data organization of point clouds as a scene representation, and because it is obtained from real images, it can get better support with multimodal models.

[0075] In order to solve the problem of requiring a large amount of data and annotations, this proposal proposes to use SAM to generate two-dimensional segmentation results to supervise three-dimensional segmentation. As the current two-dimensional segmentation model, SAM can achieve real-time unsupervised high-quality two-dimensional segmentation. This solution generates a set of segmentation masks for each instance through high-quality two-dimensional segmentation, and optimizes the properties and network of three-dimensional Gaussian through subsequent feature learning.

[0076] To address the problem of high computational overhead, this proposal proposes using a lightweight feature learning network to learn the features of real images. Learning two-dimensional features can better guide scene segmentation. The two-dimensional image feature learning obtains the query mask through the mask generated by SAM and obtains the segmentation result through its calculation with Gaussian attributes.

[0077] Since the structure of 3D Gaussian is similar to point cloud, it can be applied to the previous 3D prior knowledge. Based on this, this proposal uses the 3D prior post-processing technology to remove outliers and complete instances of segmentation instances, and smooth the plane to obtain the final segmentation result.

[0078] Step S30, processing the segmentation mask by three-dimensional Gaussian characterization technology to generate an initial segmentation result, and completing or removing the boundary Gaussian distribution in the initial segmentation result to obtain a final segmentation result.

[0079] It should be noted that 3D Gaussian characterization technology is an advanced computer vision technology, mainly used for real-time radiation field rendering and 3D reconstruction. This technology uses 3D Gaussian distribution to represent the scene to achieve high-quality real-time rendering.

[0080] In a feasible implementation manner, the step S30 of processing the segmentation mask by using the three-dimensional Gaussian characterization technology to generate an initial segmentation result includes:

[0081] Extracting a second image feature of the segmentation mask, and obtaining a second query mask of the instance to be segmented based on the second image feature and the segmentation mask;

[0082] It should be noted that after obtaining the segmentation mask, the feature learning network in the model is used to learn or extract the two-dimensional image features in the segmentation mask, that is, the second image features. In order to learn the feature F of the real image, a lightweight feature learning network will be used in this module. The image features here are extracted during the actual image processing process.

[0083] It should be noted that, by calculating the extracted second image features and the segmentation mask, a query mask for the instance to be segmented, that is, a second query mask, is generated. Specifically, the query mask may be generated in the following manner:

[0084] The query mask Q of each instance in the scene is obtained by performing mask average pooling operation on the segmentation mask and image features. M , the specific calculation is as follows:

[0085]

[0086] in, Represents each instance mask obtained by SAM segmentation, is the indicator function, H and W are the height and width of the image and mask, that is, the image features of the corresponding pixels in the instance mask are extracted and average pooling is performed to obtain the segmentation query mask of the instance in the two-dimensional image. Query mask Q M Used for feature synthesis to query whether a pixel or Gaussian belongs to the instance to be segmented.

[0087] The feature attributes of the second query mask are calculated by three-dimensional Gaussian characterization technology to generate an initial segmentation result.

[0088] It should be noted that the method of calculating the characteristic attributes of the second query mask by using the three-dimensional Gaussian characterization technology may be to perform feature point multiplication on the second query mask and the scene Gaussian distribution to obtain an initial segmentation result mask.

[0089] In a feasible implementation manner, the step of calculating the feature attributes of the second query mask by using the three-dimensional Gaussian characterization technology to generate an initial segmentation result includes:

[0090] Calculating characteristic attributes of the second query mask by using a three-dimensional Gaussian characterization technique to obtain multiple scene Gaussian distributions;

[0091] A first scene Gaussian distribution greater than a preset threshold is extracted from the scene Gaussian distribution, and the first scene Gaussian distribution is used as an initial segmentation result.

[0092] It should be noted that the scene Gaussian distribution is a scene Gaussian that only takes feature attributes. In the inference stage, the initial segmentation result mask can be obtained by performing feature point multiplication between the second query mask and the scene Gaussian, and the initial segmentation is performed by calculating the adaptive threshold. The initial segmentation calculation is as follows:

[0093]

[0094] in is the scene Gaussian that only takes the characteristic attributes, Dot multiplication is performed to obtain the Gaussian confidence score, and the Gaussian greater than the adaptive threshold / preset threshold thresh is taken as the initial segmentation result. Thresh is adaptively controlled by taking the mean of the generated mask and the maximum value of the three-dimensional scene information. The mean and standard deviation of the generated mask reflect the feature distribution of instances in the rendered image and the real image. The three-dimensional scene information controls the minimum set of segmentation results by taking the minimum value of the Gaussian feature attributes of the top K features. The calculation of thresh is as follows, where TopK is the top k largest elements, and λ is used to control the value of k.

[0095]

[0096] Specifically, the segmentation reasoning process diagram involved in this embodiment is as follows: Figure 2 As shown in the figure, for the inference segmentation stage, a real image or image data with a fixed view angle v is first given, and SAM can generate a two-dimensional instance segmentation result of a given prompt under image v, namely, a segmentation mask v. The image features are learned through the feature learning network, and the query mask of the pre-segmented instance can be obtained by calculating the image features and the segmentation mask v.

[0097] The query mask is calculated by comparing it with the characteristic attributes of the scene Gaussian, and the Gaussian that exceeds a certain threshold is taken as the initial segmentation result. After the three-dimensional prior post-processing method proposed in this proposal, the final high-quality segmentation instance is obtained.

[0098] The present application proposes a scene instance segmentation method, apparatus, device, storage medium and computer program product. The present application obtains image data in a target scene, performs instance segmentation on the image data through a preset segmentation model, obtains an instance mask after the image data is segmented, and then characterizes or calculates the segmentation mask through a three-dimensional Gaussian characterization technology to generate an initial segmentation result. At this time, the initial segmentation result cannot accurately reflect the instance distribution in the scene. Then, the Gaussian distribution on the boundary in the initial segmentation result is complemented, or the redundant part outside the boundary is removed, thereby improving the surface smoothness of the segmented instance and improving the segmentation quality, thereby avoiding affecting the accuracy of subsequent tasks.

[0099] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 Before step S20, the scene instance segmentation method further includes steps S100 to S130:

[0100] Step S100, extracting historical image data under a preset scene;

[0101] It should be noted that the preset scene may be a scene that needs to be identified in a subsequent task. There are multiple preset scenes, and the specific number is not limited.

[0102] It should be noted that in the process of the model, the training set of the model is the training of the scene Gaussian and the adjustment of the parameters of the feature learning network. The data set used for training is the real images of each perspective of the scene, and the scene Gaussian representation is obtained by training the real images in the initial stage.

[0103] Step S110, inputting the historical image data into a pre-trained segmentation model, and performing data processing on the historical image data based on the pre-trained segmentation model to obtain a first image feature and a first segmentation mask of a plurality of instances;

[0104] It should be noted that the first image feature and the first segmentation mask of the multiple instances are used to refer to the image features and segmentation masks generated during the training process. In order to reflect the difference between the data generated during the inference process, the first image feature and the first segmentation mask are obtained in the same way as above, and are not described in detail here.

[0105] Step S120, calculating a first mask loss based on the first segmentation mask and the first image feature;

[0106] It should be noted that the first mask loss can be calculated by calculating the query mask of each instance by calculating the image features and the segmentation masks of different instances. According to the different perspectives of the real images in the real image set, the scene Gaussian is subjected to differentiable rasterization rendering calculation according to the corresponding perspective to obtain the rendering feature. The rendering feature is finally calculated with the query mask to generate a feature mask, which is used to calculate the mask loss with the segmentation mask generated by SAM to supervise the properties of the Gaussian and the parameters of the feature learning network.

[0107] Step S130: iteratively optimize the pre-trained segmentation model through the first mask loss to obtain a trained segmentation model.

[0108] It should be noted that the first mask loss is calculated through images of different perspectives to achieve cross-perspective information fusion and optimize the Gaussian network, thereby obtaining a trained segmentation model.

[0109] In a feasible implementation manner, the step S120 of calculating the first mask loss based on the first segmentation mask and the first image feature includes:

[0110] Based on the first segmentation mask and the first image feature, a first query mask corresponding to each instance is generated, and a feature scene corresponding to the image feature is rendered by a three-dimensional Gaussian characterization technology to obtain a rendering feature;

[0111] It should be noted that the first query mask of each instance in the scene is obtained by performing mask average pooling operation on the first segmentation mask and the first image feature. Then, the feature scene corresponding to the image feature is rendered using the three-dimensional Gaussian representation technology to obtain a rendered feature.

[0112] It should be noted that this embodiment uses a three-dimensional Gaussian representation to capture and represent the scene information, which can achieve better real-time rendering speed and rendering quality. A set of Gaussians with a point cloud structure is used to represent the scene. Different from the point cloud, each Gaussian has a shape and a spherical harmonic function that can express anisotropic materials. The properties of the Gaussian are determined by its position. Covariance matrix Σ 3D Represents the shape of the Gaussian ellipsoid and the spherical harmonic function to represent its color SH. The new perspective image generation of the three-dimensional Gaussian is generated by the differentiable rasterization method proposed by Gaussian sputtering. Specifically, the three-dimensional Gaussian is projected into the two-dimensional Gaussian according to the camera pose of the new perspective by the Gaussian sputtering method. The specific calculation is as follows, where S and R are the scaling factor and rotation matrix of the Gaussian. Σ 3D and Σ 2D are the covariance matrices of the three-dimensional Gaussian and the projected two-dimensional Gaussian, respectively. J is the affine transformation matrix from the three-dimensional projection to the two-dimensional one, and W is the transformation matrix from the world coordinates to the camera coordinates.

[0113] Σ 3D =RSS T R T ,

[0114] ∑ 2D =JW∑ 3D W T J T

[0115] Get the projected two-dimensional Gaussian, sort the Gaussians in a tile by depth, encode $SH$ into its RGB color $c$ according to vision, and when rendering, weight the sum of the colors from back to front according to the opacity of the Gaussian and the depth to get the final pixel color. The specific differentiable rasterization rendering is as follows, where C represents the final color of the pixel, is the number of Gaussians covering the pixel after sorting, c iand α i is the Gaussian color and opacity.

[0116]

[0117] For each pre-trained scene Gaussian adds a feature attribute It is used to learn consistent features across views in the scene, and its dimension C is consistent with the image dimension learned by the feature network. For the pre-trained Gaussian, this method freezes other properties of the Gaussian and only optimizes the feature properties.

[0118] Specifically, the rendering feature is generated by using the Gaussian sputtering method to obtain the current rendering feature according to the current camera position and viewing angle v. i is the characteristic of Gauss.

[0119]

[0120] generating a feature mask based on the first query mask and the rendering feature;

[0121] It should be noted that during the training phase, With rendering features By dot-producting the feature vector, we query the pixels in the rendered feature that belong to the instance and obtain the feature mask. The calculation is as follows:

[0122]

[0123] Where σ is the element-by-element sigmoid operation. Next, during the training phase, the mask P generated by the feature M The mask M generated by SAM SAM Calculating Losses To optimize the feature attributes of the scene Gaussian and the parameters in the feature network.

[0124] Based on the first segmentation mask and the feature mask, a first mask loss is calculated, wherein the first mask loss includes mask loss and pixel loss, the mask loss is used to optimize the feature attributes of the scene Gaussian and the network parameters of the feature learning network, and the pixel loss is used to compare the pixel-by-pixel masks to make the features of the same instance compact.

[0125] It should be noted that in order to optimize the characteristic attributes of the scene Gaussian and the parameters of the feature learning network, it is necessary to calculate the overall loss generated during the training process. The overall loss mainly includes two aspects: the mask loss supervised by the mask generated by SAM And the corresponding loss between pixels

[0126] Specifically, in order to optimize the characteristic properties of the scene Gaussian, this chapter proposes to use SAM mask to guide the learning of Gaussian features. Specifically, by generating a mask The real image mask obtained by SAM segmentation Perform binary cross entropy loss calculation, compare the distance between the two masks by calculating the loss and optimize Gaussian feature attributes and network parameters through gradient back propagation. The specific calculation is as follows:

[0127]

[0128] Where v is the real image set, v represents a single image or perspective, Represents the instance mask set under the current view v.

[0129] In order to enable the three-dimensional scene Gaussian to learn more closely the scene instance features, this chapter proposes a corresponding loss between pixels. For the mask set under the perspective of v Considering that two pixels p1 and p2 may belong to Different mask subsets in , namely and If two pixels belong to the 2D projection of the same instance, their intersection domain on the mask subset will be large, so the correlation score S between pixels is defined as corr :

[0130]

[0131] For Rendering Features A similar method is used to calculate the correlation between pixels. to calculate the cosine similarity.

[0132]

[0133] The definitions are as follows:

[0134]

[0135] The final training loss is defined as follows, where λ is the balance coefficient.

[0136]

[0137] Specifically, the model training process diagram involved in this embodiment is as follows: Figure 4As shown in FIG. 1 , in the model training stage, the training set of the model is the training of the scene Gaussian and the adjustment of the parameters of the feature learning network. The real images used for training are the real images of each perspective of the scene, and the scene Gaussian representation is obtained by training the real images in the initial stage.

[0138] In the cross-view information fusion stage, the instance segmentation mask set is obtained by SAM for the real image, and the image features are generated by the two-dimensional image feature learning network. The query mask of each instance is obtained by calculating the image features and the segmentation masks of different instances. According to the different perspectives of the real images in the real image set, the scene Gaussian is rendered by differentiable rasterization according to the corresponding perspective to obtain the rendering features. The rendering features are finally calculated with the query mask to obtain the generated mask, which is used to calculate the loss with the segmentation mask generated by SAM to supervise the properties of the Gaussian and the parameters of the feature learning network. The loss is calculated by using images from different perspectives to achieve cross-perspective information fusion and optimize the Gaussian and the network.

[0139] In this embodiment, the characteristic attributes of the scene Gaussian distribution and the parameters of the feature learning network are optimized by calculating the pixel loss and the mask loss, and the loss is calculated through images of different perspectives to achieve cross-perspective information fusion and optimize the Gaussian distribution and the feature learning network.

[0140] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those of the above-mentioned first and second embodiments can refer to the above introduction, and will not be repeated later. Figure 5 The step S30 of completing or removing the boundary Gaussian distribution in the initial segmentation result to obtain the final segmentation result includes steps S31 to S32:

[0141] Step S31, extracting instance missing points in the boundary Gaussian distribution in the initial segmentation result;

[0142] It should be noted that in the generated initial segmentation results, there will be Gaussian noise and background Gaussians in the scene, that is, interference items. In the generated Gaussian distribution, there will be Gaussian distributions that are not ignored on the boundary, and there will also be redundant Gaussian distributions outside the boundary. These redundant or missing parts are instance missing points.

[0143] Step S32, performing boundary completion processing on the missing points of the instance by using a neighborhood Gaussian aggregation method, and performing boundary Gaussian decoupling processing on the boundary Gaussian distribution by using a boundary Gaussian decoupling method, to obtain a final segmentation result.

[0144] It should be noted that in order to solve the Gaussian noise and background Gaussian in the scene, this method is based on the statistical filtering method of point cloud, and it is believed that two Gaussian with close enough spatial distance should belong to the same instance. Find the average distance of the K nearest neighbors of each Gaussian in the scene, and then calculate The mean $\mu$ and standard deviation σ of the nearest neighbor average distance of all Gaussians in the filter are obtained. Finally, this method removes the Gaussians whose average distance exceeds μ+σ, and obtains the filtered segmented Gaussians.

[0145] It should be noted that in order to solve the problem of ignoring the Gaussian belonging to the instance in the segmentation result, the method in this chapter uses the method of neighborhood Gaussian aggregation to supplement the neglected Gaussian on some boundaries. The Gaussians in the spherical neighborhood with a radius of $r$ are found to be Gaussians that do not belong to the segmentation result. These Gaussians are aggregated into the segmentation result to obtain new segmentation Gaussians. The setting of r is Each Gaussian in takes the minimum distance of the Gaussians in the result, and selects the maximum value among the minimum distances of these Gaussians as the setting of r, that is, Where D is the Euclidean distance.

[0146] It should be noted that in order to make the instance Gaussian obtained by segmentation have a smoother boundary, this chapter proposes a boundary Gaussian decoupling method, which can effectively decouple the Gaussian at the instance boundary and remove the part outside the boundary. Each Gaussian can be projected onto the corresponding pose plane to obtain a two-dimensional Gaussian. If the endpoint on the longest axis of the Gaussian exceeds the boundary of the current segmentation mask, that is, the ellipsoid shape of the Gaussian exceeds the boundary, the Gaussian that meets the conditions is marked as a boundary Gaussian. As shown in the figure, let the boundary points of the longest axis of the two-dimensional Gaussian be A and B, and the boundary point be O. The corresponding three-dimensional Gaussian can be adjusted by the ratio λ. Its mean (position) and scaling factor are as follows: where s and p are the position of the original three-dimensional Gaussian and the scaling factor of the longest axis, s' and p' are the position and the scaling factor of the longest axis of the new three-dimensional Gaussian after decoupling, and e is the unit vector of the three-dimensional Gaussian position $p$ pointing to the endpoint of the long axis.

[0147]

[0148] s′=λs,

[0149]

[0150] In this embodiment, the initial segmentation result is optimized by using a neighborhood Gaussian aggregation method and a boundary Gaussian decoupling method, so that the instance Gaussian obtained by segmentation has a smoother boundary.

[0151] The Gaussian representation scene segmentation method with cross-view information fusion proposed in this application solves the problem that previous methods require a large amount of data sets and annotations and have high computational overhead. At the same time, due to the information learning and fusion of two-dimensional images, it is more supportive of more multimodal models. The scene segmentation method proposed in this proposal is more efficient and accurate, and has better support for subsequent tasks. It can be used as an automated high-performance scene understanding tool in the process of building a private network park informatization.

[0152] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the scene instance segmentation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0153] This application also provides a scene instance segmentation device, please refer to Figure 6 , the scene instance segmentation device comprises:

[0154] An acquisition module 10 is used to acquire image data of a target scene;

[0155] A first processing module 20, configured to process the image data based on the preset segmentation model and output a segmentation mask;

[0156] The second processing module 30 is used to process the segmentation mask by using a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and to complete or remove the boundary Gaussian distribution in the initial segmentation result to obtain a final segmentation result.

[0157] Optionally, the device further comprises:

[0158] An extraction module, used to extract historical image data under a preset scenario;

[0159] An input module, used for inputting the historical image data into a pre-trained segmentation model, and performing data processing on the historical image data based on the pre-trained segmentation model to obtain a first image feature and a first segmentation mask of a plurality of instances;

[0160] A calculation module, configured to calculate a first mask loss based on the first segmentation mask and the first image feature;

[0161] An optimization module is used to iteratively optimize the pre-trained segmentation model through the first mask loss to obtain a trained segmentation model.

[0162] Optionally, the calculation module includes:

[0163] A rendering unit, configured to generate a first query mask corresponding to each instance based on the first segmentation mask and the first image feature, and render a feature scene corresponding to the image feature by using a three-dimensional Gaussian characterization technique to obtain a rendering feature;

[0164] A generating unit, configured to generate a feature mask based on the first query mask and the rendering feature;

[0165] A first computing unit is used to calculate a first mask loss based on the first segmentation mask and the feature mask, wherein the first mask loss includes a mask loss and a pixel loss, the mask loss is used to optimize the feature attributes of the scene Gaussian and the network parameters of the feature learning network, and the pixel loss is used to compare the pixel-by-pixel masks to compact the features of the same instance.

[0166] Optionally, the second processing module includes:

[0167] a first extraction unit, configured to extract a second image feature of the segmentation mask, and obtain a second query mask of the instance to be segmented based on the second image feature and the segmentation mask;

[0168] The second calculation unit is used to calculate the feature attributes of the second query mask by using a three-dimensional Gaussian characterization technology to generate an initial segmentation result.

[0169] Optionally, the second computing unit includes:

[0170] A calculation subunit, configured to calculate characteristic attributes of the second query mask by using a three-dimensional Gaussian characterization technique to obtain multiple scene Gaussian distributions;

[0171] The extraction subunit is used to extract the first scene Gaussian distribution that is greater than a preset threshold from the scene Gaussian distribution, and use the first scene Gaussian distribution as an initial segmentation result.

[0172] Optionally, the second processing module further includes:

[0173] A second extraction unit, used to extract instance missing points in the boundary Gaussian distribution in the initial segmentation result;

[0174] The processing unit is used to perform boundary completion processing on the missing points of the instance by using a neighborhood Gaussian aggregation method, and to decouple the boundary Gaussian distribution by using a boundary Gaussian decoupling method to obtain a final segmentation result.

[0175] The scene instance segmentation device provided by the present application adopts the scene instance segmentation method in the above embodiment, which can solve the technical problem of scene instance segmentation. Compared with the prior art, the beneficial effects of the scene instance segmentation device provided by the present application are the same as the beneficial effects of the scene instance segmentation method provided by the above embodiment, and other technical features in the scene instance segmentation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0176] The present application provides a scene instance segmentation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the scene instance segmentation method in the above-mentioned embodiment one.

[0177] Reference below Figure 7 , which shows a schematic diagram of the structure of a scene instance segmentation device suitable for implementing the embodiment of the present application. The scene instance segmentation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The scene instance segmentation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0178] like Figure 7As shown, the scene instance segmentation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the scene instance segmentation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the scene instance segmentation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a scene instance segmentation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0179] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0180] The scene instance segmentation device provided by the present application adopts the scene instance segmentation method in the above embodiment to solve the technical problem of scene instance segmentation. Compared with the prior art, the beneficial effects of the scene instance segmentation device provided by the present application are the same as the beneficial effects of the scene instance segmentation method provided by the above embodiment, and other technical features in the scene instance segmentation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0181] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0182] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0183] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the scene instance segmentation method in the above-mentioned embodiment.

[0184] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0185] The computer-readable storage medium may be included in the scene instance segmentation device; or may exist independently without being assembled into the scene instance segmentation device.

[0186] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the scene instance segmentation device, the scene instance segmentation device:

[0187] Acquire image data of the target scene;

[0188] Based on the preset segmentation model, the image data is processed to output a segmentation mask;

[0189] The segmentation mask is processed by a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and the boundary Gaussian distribution in the initial segmentation result is supplemented or removed to obtain a final segmentation result.

[0190] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0193] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned scene instance segmentation method, and can solve the technical problem of scene instance segmentation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the scene instance segmentation method provided in the above-mentioned embodiment, and will not be repeated here.

[0194] The present application also provides a computer program product, including a computer program, which implements the steps of the scene instance segmentation method as described above when executed by a processor.

[0195] The computer program product provided in this application can solve the technical problem of scene instance segmentation. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the scene instance segmentation method provided in the above embodiment, which will not be repeated here.

[0196] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A scene instance segmentation method, characterized in that: The method comprises: Acquire image data of the target scene; Based on the preset segmentation model, the image data is processed to output a segmentation mask; The segmentation mask is processed by a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and the boundary Gaussian distribution in the initial segmentation result is supplemented or removed to obtain a final segmentation result.

2. The method according to claim 1, characterized in that Before the step of processing the image data based on the preset segmentation model and outputting a segmentation mask, the method further includes: Extract historical image data under preset scenarios; Inputting the historical image data into a pre-trained segmentation model, and performing data processing on the historical image data based on the pre-trained segmentation model to obtain a first image feature and a first segmentation mask of a plurality of instances; Calculating a first mask loss based on the first segmentation mask and the first image feature; The pre-trained segmentation model is iteratively optimized through the first mask loss to obtain a trained segmentation model.

3. The method according to claim 2, characterized in that The step of calculating a first mask loss based on the first segmentation mask and the first image feature comprises: Based on the first segmentation mask and the first image feature, a first query mask corresponding to each instance is generated, and a feature scene corresponding to the image feature is rendered by a three-dimensional Gaussian characterization technology to obtain a rendering feature; generating a feature mask based on the first query mask and the rendering feature; Based on the first segmentation mask and the feature mask, a first mask loss is calculated, wherein the first mask loss includes mask loss and pixel loss, the mask loss is used to optimize the feature attributes of the scene Gaussian and the network parameters of the feature learning network, and the pixel loss is used to compare the pixel-by-pixel masks to make the features of the same instance compact.

4. The method according to any one of claims 1 to 3, characterized in that The step of processing the segmentation mask by using the three-dimensional Gaussian characterization technology to generate an initial segmentation result includes: Extracting a second image feature of the segmentation mask, and obtaining a second query mask of the instance to be segmented based on the second image feature and the segmentation mask; The feature attributes of the second query mask are calculated by three-dimensional Gaussian characterization technology to generate an initial segmentation result.

5. The method according to claim 4, characterized in that The step of calculating the feature attributes of the second query mask by using the three-dimensional Gaussian characterization technology to generate an initial segmentation result includes: Calculating feature attributes of the second query mask using a three-dimensional Gaussian characterization technique to obtain multiple scene Gaussian distributions; A first scene Gaussian distribution greater than a preset threshold is extracted from the scene Gaussian distribution, and the first scene Gaussian distribution is used as an initial segmentation result.

6. The method according to claim 1, characterized in that The step of completing or removing the boundary Gaussian distribution in the initial segmentation result to obtain the final segmentation result includes: Extracting instance missing points in the boundary Gaussian distribution in the initial segmentation result; The missing points of the instance are processed by boundary completion through the neighborhood Gaussian aggregation method, and the boundary Gaussian distribution is decoupled through the boundary Gaussian decoupling method to obtain the final segmentation result.

7. A scene instance segmentation device, characterized in that: The device comprises: An acquisition module is used to acquire image data in a target scene; A first processing module, used for processing the image data based on the preset segmentation model and outputting a segmentation mask; The second processing module is used to process the segmentation mask by using a three-dimensional Gaussian characterization technology to generate an initial segmentation result, and to complete or remove the boundary Gaussian distribution in the initial segmentation result to obtain a final segmentation result.

8. A scene instance segmentation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the scene instance segmentation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the scene instance segmentation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the scene instance segmentation method according to any one of claims 1 to 6 are implemented.

Images