Image segmentation method in multi-view scene
By combining the multi-view encoder module and the patch decoder module, the problem of unsupervised learning of object representation in complex 3D scenes is solved, and accurate object and background segmentation is achieved.
Patent Information
- Application Number
- CN202410750735.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-12
- Publication Date
- 2025-12-12
AI Technical Summary
Existing methods struggle to learn object-centered representations in unsupervised manner in complex 3D scenes, especially those requiring camera position annotations, which perform poorly in real-world scenarios.
The algorithm employs a multi-view encoder module, a patch decoder module, and an image decoder module to identify and extract corresponding features from images at different viewpoints in an unsupervised manner. It uses viewpoint representation extraction units, background representation extraction units, object representation extraction units, and connection units. The connection unit connects the viewpoint representation and the object representation to generate viewpoint-object representation and viewpoint-background representation. The patch decoder module decodes these representations to generate masks for the object and background. Finally, the image decoder module reconstructs the representation to generate the segmentation result.
It achieves accurate object and background recognition and segmentation in complex 3D scenes, and can extract features from images from multiple perspectives and generate accurate segmentation results.
Smart Images
Figure CN121120681A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of machine learning, and particularly relates to an image segmentation method in a multi-view scene. BACKGROUND
[0002] We observe the world with our eyes and form visual representations of everything in the world through learning, so we can visually recognize various objects. In order to enable artificial intelligence systems to also observe and learn objects in the environment like people, in recent years, object-centric representation learning methods have been proposed, which can well segment and recognize objects and backgrounds in complex visual scene images. Unlike traditional image representation learning, which converts an image into a representation, i.e., a vector, the goal of object-centric representation learning is to convert an image usually containing multiple objects and backgrounds into representations, i.e., vectors, of multiple objects and backgrounds.
[0003] According to the category of input visual data, existing methods can be divided into three categories: single image scene-based, video scene-based, and multi-view scene-based. The single image scene-based method aims to learn the representation of all objects including the background from a single image. The video scene-based method uses the temporal information between consecutive frames to extract the representation of objects and backgrounds in the video scene. The multi-view scene-based method uses multi-view images or camera pose annotations to learn the three-dimensional representation of objects and backgrounds from three-dimensional scenes. The existing OCLOC method and SIMONe method can learn object-centric representations unsupervisedly from three-dimensional scenes, but only satisfactory results have been achieved on simple synthetic scenes. The existing SRT method and OSRT method can handle complex synthetic scenes and even real scenes, but they need camera position annotations for training and testing.
[0004] Therefore, it is still challenging to learn object-centric representations unsupervisedly from complex three-dimensional scenes. SUMMARY
[0005] The present application is proposed to solve the above problems, and aims to provide an image segmentation method in a multi-view scene.
[0006] The application provides a multi-view scene image segmentation method, which is used for obtaining segmentation results corresponding to multiple different view images of the same scene according to a trained multi-view image segmentation model. The multi-view image segmentation model comprises a multi-view encoder module, a Patch decoder module and an image decoder module. The multi-view encoder module comprises a Patch feature extraction unit, which is used for extracting corresponding Patch features of each view image; a view representation extraction unit, which comprises a view encoder and is used for extracting corresponding view representations according to each Patch feature; a background representation extraction unit, which comprises a background encoder and is used for extracting corresponding background representations according to each Patch feature; an object representation extraction unit, which is used for obtaining multiple object-centered representations as object representations according to all Patch features and corresponding view representations and background representations; and a connection unit, which is used for connecting corresponding view representations and each object representation respectively to obtain corresponding view-object representations, and connecting corresponding view representations and corresponding background representations to obtain corresponding view-background representations. The Patch decoder module comprises an object decoding unit, which is used for decoding each view-object representation to obtain corresponding object Patch representations and object masks; a background decoding unit, which is used for decoding each view-background representation to obtain corresponding background Patch representations and background masks; and a reconstruction feature unit, which is used for obtaining corresponding reconstruction Patch features according to corresponding object Patch representations, object masks, background Patch representations and background masks for each view image. The image decoder module comprises a format conversion unit, which is used for converting each reconstruction Patch feature into a preset input form; and an image reconstruction unit, which is used for obtaining corresponding segmentation results according to corresponding reconstruction Patch features for each view image.
[0007] In the multi-view scene image segmentation method provided by the application, the multi-view image segmentation model can be trained according to a training set comprising multiple existing view images of different scenes to obtain the trained multi-view image segmentation model. The process comprises a scene segmentation stage and a scene reconstruction stage. The scene segmentation stage is the first stage and is used for training the multi-view encoder module and the Patch decoder module. The scene reconstruction stage is the second stage and is used for training the image decoder module.
[0008] In the multi-view scene image segmentation method provided by the application, in the scene segmentation stage, the reconstruction loss L seg The parameters of the multi-view encoder module and the Patch decoder module are optimized, and the calculation expression of the reconstruction loss L seg is as follows: In the formula, V is the total number of existing view images of the same scene. is the Patch feature corresponding to the vth existing view image, is the reconstructed Patch feature corresponding to the vth existing view image, is the background reconstructed Patch feature corresponding to the vth existing view image, and a is a hyperparameter.
[0009] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: in the training, the image reconstruction unit is also used to reconstruct the reconstructed Patch feature corresponding to each view image to obtain the corresponding reconstructed image, and in the scene reconstruction stage, the reconstruction loss L rec The parameters of the image decoder module are optimized, and the calculation expression of the reconstruction loss L rec is as follows: In the formula, V is the total number of existing view images of the same scene, x v is the vth existing view image, is the reconstructed image corresponding to the vth existing view image.
[0010] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: the Patch feature extraction unit is a pre-trained visual basic model DINO.
[0011] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: the view encoder is composed of a convolutional neural network and two layers of fully connected networks, used to extract view representations and capture the same view-dependent attributes of all objects and backgrounds in each view image, the background encoder includes two layers of fully connected networks, and the object representation extraction unit includes a slot attention encoder.
[0012] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: the object decoding unit is an object decoder, which includes multiple layers of fully connected networks, and the background decoding unit is a background decoder, which includes multiple layers of fully connected networks.
[0013] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: the reconstruction feature unit takes the normalized background mask and object mask as weights for each view image, and performs weighted summation on the object Patch representation and the background Patch representation to obtain the corresponding reconstructed Patch feature.
[0014] In the image segmentation method in the multi-view scene provided by the application, it can also have the following characteristics: the image reconstruction unit is a VQ-VAE decoder, the format conversion unit is an MLP network, and the preset input form is a form consistent with the input of the VQ-VAE decoder.
[0015] Effects of the Invention
[0016] According to the image segmentation method in the multi-view scene of the present application, on the one hand, the Patch feature reconstruction is realized by the Patch decoder module, i.e., the reconstructed Patch feature is generated, so that the complex scene can be processed; on the other hand, the two-stage unsupervised training of the scene segmentation stage and the scene reconstruction stage makes the trained multi-view image segmentation model be able to accurately identify and segment the object and the background. Therefore, the image segmentation method in the multi-view scene of the present application can accurately obtain the segmentation results corresponding to the multi-view images from the complex three-dimensional scene. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 is a block diagram of the multi-view image segmentation model in the embodiment of the present application;
[0018] Figure 2 is a flowchart of the image segmentation using the multi-view image segmentation model in the embodiment of the present application;
[0019] Figure 3 is a schematic diagram of the different view images of the same scene and the corresponding segmentation results in the embodiment of the present application. DETAILED DESCRIPTION
[0020] In order to make the technical means, creative features, purposes and effects achieved by the present application easy to understand, the following embodiments will be specifically described in combination with the drawings.
[0021] The present embodiment provides an image segmentation method in a multi-view scene, which is used to obtain the segmentation results corresponding to the multiple different view images of the same scene according to the trained multi-view image segmentation model. In the present embodiment, the multiple different view images of the same scene are input into the trained multi-view image segmentation model together, so as to obtain the segmentation results corresponding to each view image. In the present embodiment, one scene contains one background and at least one object.
[0022] Figure 1 is a block diagram of the multi-view image segmentation model in the embodiment of the present application.
[0023] As shown in Figure 1 , the multi-view image segmentation model 100 includes a multi-view encoder module 10, a Patch decoder module 20 and an image decoder module 30.
[0024] In this embodiment, the multi-view encoder module 10 is used to identify objects in images captured from different perspectives in an unsupervised manner. The patch decoder module 20 is used to reconstruct image features, which helps the model to process complex three-dimensional scene images. The image decoder module 30 is used to reconstruct the entire scene and the corresponding single object and background images in the scene.
[0025] The multi-view encoder module 10 includes a patch feature extraction unit 101, a perspective representation extraction unit 102, a background representation extraction unit 103, an object representation extraction unit 104, and a connection unit 105.
[0026] The patch feature extraction unit 101 is used to extract corresponding patch features for each perspective image. The patch feature extraction unit 101 is a pre-trained visual base model DINO.
[0027] The perspective representation extraction unit 102 includes a perspective encoder, which is used to extract corresponding perspective representations according to each patch feature.
[0028] The perspective encoder is composed of a convolutional neural network and two fully connected networks, which are used to extract perspective representations to capture the same perspective-dependent properties of all objects and backgrounds in each perspective image.
[0029] The background representation extraction unit 103 includes a background encoder, which is used to extract corresponding background representations according to each patch feature. The background encoder includes two fully connected networks.
[0030] The object representation extraction unit 104 is used to obtain multiple object-centered representations as object representations according to all patch features and their corresponding perspective representations and background representations. The object representation extraction unit 104 includes a slot attention encoder.
[0031] The connection unit 105 is used to connect the corresponding perspective representations and each object representation for each perspective image to obtain the corresponding perspective-object representation, and to connect the corresponding perspective representation and the corresponding background representation to obtain the corresponding perspective-background representation.
[0032] In this embodiment, the representations to be connected are expanded to the same dimension for connection.
[0033] The patch decoder module 20 includes an object decoding unit 201, a background decoding unit 202, and a reconstructed feature unit 203.
[0034] The object decoding unit 201 is configured to decode each view-object representation to obtain a corresponding object patch representation and an object mask. The object decoding unit 201 is an object decoder and includes a multi-layer fully connected network.
[0035] The background decoding unit 202 is configured to decode each view-background representation to obtain a corresponding background patch representation and a background mask. The background decoding unit 202 is a background decoder and includes a multi-layer fully connected network.
[0036] In this embodiment, the object patch representation and the background patch representation are reconstructed by the two multi-layer fully connected networks, i.e., the object decoder and the background decoder, so that the complex scene is processed.
[0037] The reconstruction feature unit 203 is configured to obtain a corresponding reconstructed patch feature for each view image according to the corresponding object patch representation, object mask, background patch representation, and background mask. In this embodiment, the segmentation result of the corresponding view image is obtained according to the object mask and the background mask.
[0038] The reconstruction feature unit 203 is configured to obtain a corresponding reconstructed patch feature for each view image according to the corresponding object patch representation, object mask, background patch representation, and background mask. In this embodiment, the segmentation result of the corresponding view image is obtained according to the object mask and the background mask.
[0039] The image decoder module 30 includes a format conversion unit 301 and an image reconstruction unit 302.
[0040] The format conversion unit 301 is configured to convert each reconstructed patch feature into a preset input form. The format conversion unit is an MLP network.
[0041] The image reconstruction unit 302 is configured to obtain a corresponding segmentation result for each view image according to the corresponding reconstructed patch feature. The image reconstruction unit is a VQ-VAE decoder, and the preset input form is a form that conforms to the input of the VQ-VAE decoder.
[0042] The following describes a process of image segmentation by using the multi-view image segmentation model 100 with reference to the accompanying drawings.
[0043] Figure 2 FIG. 1 is a flowchart of image segmentation by using the multi-view image segmentation model in an embodiment of the present application.
[0044] As shown in FIG. 1, the image segmentation by using the multi-view image segmentation model 100 includes the following steps. Figure 2
[0045] Step S1, the Patch feature extraction unit 101 is used to extract corresponding Patch features for each view image.
[0046] Step S2, the view representation extraction unit 102 is used to extract corresponding view representations according to each Patch feature.
[0047] Step S3, the background representation extraction unit 103 is used to extract corresponding background representations according to each Patch feature.
[0048] Step S4, the object representation extraction unit 104 is used to obtain multiple object-centered representations as object representations according to all Patch features and corresponding view representations and background representations.
[0049] Step S5, the connection unit 105 is used to connect corresponding view representations and each object representation respectively to obtain corresponding view-object representations, and connect corresponding view representations and corresponding background representations to obtain corresponding view-background representations for each view image.
[0050] Step S6, the object decoding unit 201 is used to decode each view-object representation to obtain corresponding object Patch representation and object mask.
[0051] Step S7, the background decoding unit 202 is used to decode each view-background representation to obtain corresponding background Patch representation and background mask.
[0052] Step S8, the reconstructed feature unit 203 is used to obtain corresponding reconstructed Patch features according to corresponding object Patch representation, object mask, background Patch representation and background mask for each view image.
[0053] Step S9, the format conversion unit 301 is used to convert each reconstructed Patch feature into a preset input form.
[0054] Step S10, the image reconstruction unit 302 is used to reconstruct corresponding reconstructed Patch features of each view image to obtain corresponding segmentation results.
[0055] Figure 3 FIG. 1 is a schematic diagram of different view images and corresponding segmentation results of the same scene in an embodiment of the present application.
[0056] As shown in FIG. 1, the leftmost view image is the view image of the leftmost scene, and the rightmost view image is the view image of the rightmost scene. Figure 3As shown, in this embodiment, multiple view images, i.e., Image, of the same scene are selected from the existing CLEVR dataset, SHOP dataset and GSO dataset respectively, and these view images are processed by the multi-view image segmentation model 100 to obtain the corresponding segmentation results, i.e., Segment. As can be seen, the multi-view image segmentation model 100 of this embodiment can accurately identify the background and each object in the same scene and accurately segment them.
[0057] In this embodiment, an object-centered unsupervised representation learning method is used to train the multi-view image segmentation model 100. This method can effectively learn object-centered representations unsupervisedly from complex three-dimensional scenes. The specific method is as follows: a training set containing multiple existing view images of different scenes is used to train the multi-view image segmentation model to obtain a trained multi-view image segmentation model. This process includes a scene segmentation stage and a scene reconstruction stage.
[0058] The scene segmentation stage is the first stage, which is used to train the multi-view encoder module 10 and the Patch decoder module 20. In the scene segmentation stage, a reconstruction loss L seg The parameters of the multi-view encoder module 10 and the Patch decoder module 20 are optimized, and the reconstruction loss L seg The calculation expression of L is as follows:
[0059]
[0060] In the formula, V is the total number of existing view images of the same scene, is the Patch feature corresponding to the vth existing view image, is the reconstructed Patch feature corresponding to the vth existing view image, is the reconstructed Patch feature corresponding to the vth existing view image is the background reconstructed Patch feature in the reconstructed Patch feature of the vth existing view image, and a is a hyperparameter.
[0061] In the training, the image reconstruction unit 302 is further configured to reconstruct the reconstructed Patch feature corresponding to each view image to obtain a corresponding reconstructed image.
[0062] In this embodiment, a channel is added to the VQ-VAE decoder to output a mask of each object or background, i.e., an object mask or a background mask, so that it can not only output the appearance of each object or background, i.e., a reconstructed image, but also output the shape of each object or background, i.e., a segmentation result.
[0063] The scene reconstruction stage is the second stage, which is used to train the image decoder module 30. In the scene reconstruction stage, a reconstruction loss L recOptimizing parameters of the image decoder module 30, reconstruction loss L rec The calculation expression is:
[0064]
[0065] In the formula, V is the total number of existing view images of the same scene, x v is the vth existing view image, is the reconstruction image corresponding to the vth existing view image.
[0066] Effects of the embodiments
[0067] According to the image segmentation method in the multi-view scene according to the embodiments, on the one hand, the Patch feature reconstruction is realized by the Patch decoder module, that is, the reconstruction Patch feature is generated, so that the complex scene can be processed; on the other hand, through the two-stage unsupervised training of the scene segmentation stage and the scene reconstruction stage, the trained multi-view image segmentation model can accurately identify and segment the objects and the background. In summary, the method can accurately obtain the segmentation results corresponding to multiple view images from the complex three-dimensional scene.
[0068] Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.
Claims
1. An image segmentation method for multi-view scenes, used to obtain segmentation results corresponding to multiple images of the same scene from different viewpoints based on a trained multi-view image segmentation model, characterized in that, The multi-view image segmentation model includes: Multi-view encoder module, Patch decoder module, and image decoder module. The multi-view encoder module includes: The Patch feature extraction unit is used to extract corresponding Patch features from each of the aforementioned viewpoint images; The viewpoint representation extraction unit includes a viewpoint encoder, which is used to extract the corresponding viewpoint representation based on the features of each patch. The background representation extraction unit includes a background encoder, which is used to extract the corresponding background representation based on the features of each patch. The object representation extraction unit is used to obtain multiple object-centered representations as object representations based on all the Patch features and their corresponding viewpoint representations and background representations; The connection unit is used to connect the corresponding viewpoint representation and the corresponding object representation for each of the viewpoint images to obtain the corresponding viewpoint-object representation, and to connect the corresponding viewpoint representation and the corresponding background representation to obtain the corresponding viewpoint-background representation. The Patch decoder module includes: The object decoding unit is used to decode each of the viewpoint-object representations to obtain the corresponding object patch representation and object mask; The background decoding unit is used to decode each of the viewpoint-background representations to obtain the corresponding background patch representation and background mask; The reconstructed feature unit is used to obtain corresponding reconstructed patch features for each of the aforementioned viewpoint images based on the corresponding object patch representation, object mask, background patch representation, and background mask. The image decoder module includes: The format conversion unit is used to convert each of the reconstructed patch features into a preset input format; The image reconstruction unit is used to obtain the corresponding segmentation result for each of the aforementioned viewpoint images based on the corresponding reconstruction patch features.
2. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The multi-view image segmentation model is trained using a training set containing multiple existing viewpoint images of different scenes to obtain the trained multi-view image segmentation model. This process includes a scene segmentation stage and a scene reconstruction stage. The scene segmentation stage is the first stage, used to train the multi-view encoder module and the Patch decoder module. The scene reconstruction stage is the second stage, used to train the image decoder module.
3. The image segmentation method for multi-view scenes according to claim 2, characterized in that: in, In the scene segmentation stage, a reconstruction loss L is used. seg Optimize the parameters of the multi-view encoder module and the patch decoder module. The reconstruction loss L seg The calculation expression is: In the formula, V represents the total number of existing viewpoint images of the same scene. The Patch feature corresponding to the v-th existing viewpoint image. The reconstructed patch features corresponding to the v-th existing viewpoint image, Reconstruct the Patch feature for the background corresponding to the v-th existing viewpoint image, where α is a hyperparameter.
4. The image segmentation method for multi-view scenes according to claim 2, characterized in that: in, In the training process, the image reconstruction unit is also used to reconstruct the reconstruction patch features corresponding to each of the viewpoint images to obtain the corresponding reconstructed images. In the scene reconstruction phase, a reconstruction loss L is used. rec Optimize the parameters of the image decoder module. The reconstruction loss L rec The calculation expression is: In the formula, V represents the total number of existing viewpoint images of the same scene, and x v For the v-th existing viewpoint image, This is the reconstructed image corresponding to the v-th existing viewpoint image.
5. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The Patch feature extraction unit is a pre-trained visual base model DINO.
6. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The view encoder consists of a convolutional neural network and two fully connected layers, used to extract view representations and capture view-dependent attributes common to all objects and the background in each view image. The background encoder comprises two fully connected networks. The object representation extraction unit includes a slotted attention encoder.
7. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The object decoding unit is an object decoder, which includes a multi-layer fully connected network. The background decoding unit is a background decoder, which contains a multi-layer fully connected network.
8. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The reconstructed feature unit uses the normalized background mask and object mask as weights to perform a weighted summation of the object patch representation and the background patch representation for each viewpoint image, thereby obtaining the corresponding reconstructed patch feature.
9. The image segmentation method for multi-view scenes according to claim 1, characterized in that: in, The image reconstruction unit is a VQ-VAE decoder. The format conversion unit is an MLP network. The preset input format is the input format that conforms to the VQ-VAE decoder.