Method, apparatus, device and product for image instance segmentation

Abstract

This disclosure relates to a method, apparatus, device, and product for image instance segmentation. The method includes determining, based on an input image, the region of interest (ROI) features of a target object and a target region image of the target object. The method further includes extracting shape features of the target object based on the target region image. The method also includes determining fusion features of the target object based on the ROI features and the shape features. Furthermore, the method includes performing instance segmentation on the input image based on the fusion features to generate a mask instance of the target object in the input image. In this way, the fusion features of the target object take into account the shape characteristics of the target object. When the embodiments of this disclosure utilize the fusion features of the target object for image instance segmentation, they can focus on the overall region of the target object, especially the region occluded by other target objects, thereby more accurately identifying the boundary contour of the object region.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to methods, apparatus, devices, and products for image instance segmentation. Background Technology

[0002] When using computer vision technology for scene understanding, many objects (including people) are occluded. In scenarios where objects are partially occluded, the complete shape of the occluded object is perceived based on prior knowledge, and the boundary information of the occluded region is described to perform instance segmentation of the image. Summary of the Invention

[0003] Embodiments of this disclosure provide a method, apparatus, device, and product for image instance segmentation.

[0004] In a first aspect of this disclosure, an image instance segmentation method is provided. The method includes determining, based on an input image, a region of interest (ROI) feature of a target object in the input image and a target region image of the target object. The method further includes extracting shape features of the target object based on the target region image. The method also includes determining fusion features of the target object based on the ROI feature and the shape features. Furthermore, the method includes performing instance segmentation on the input image based on the fusion features to generate a mask instance of the target object in the input image.

[0005] In a second aspect of this disclosure, a method for training an image segmentation model is provided, the image segmentation model including a main model and an auxiliary model. The method includes acquiring training samples, which include multiple sample images and multiple mask sample instances corresponding to sample objects in the sample images. The method further includes extracting shape features of the sample objects in the sample images by the auxiliary model. The method also includes performing instance segmentation on the sample images by the main model based on fused features to generate mask prediction instances of the sample objects in the sample images, the fused features being determined based on region of interest features and shape features of the sample objects in the sample images. The method further includes generating mask projection instances of the sample objects by the auxiliary model based on the shape features of the sample objects in the sample images and the camera pose. Furthermore, the method includes jointly training the main model and the auxiliary model based on the mask prediction instances and mask projection instances, using the mask sample instances corresponding to the sample images as supervision signals.

[0006] In a third aspect of this disclosure, an image instance segmentation apparatus is provided. The apparatus includes a generation module configured to generate, based on an input image, region-of-interest (ROI) features of a target object in the input image and a target region image of the target object. The apparatus also includes an extraction module configured to extract shape features of the target object based on the target region image. Furthermore, the apparatus includes a fusion module configured to determine fusion features of the target object based on the RIO features and the shape features. Additionally, the apparatus includes a segmentation module configured to perform instance segmentation on the input image based on the fusion features to generate mask instances of the target object in the input image.

[0007] In a fourth aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor. The electronic device also includes a memory coupled to the at least one processor and having instructions stored thereon that, when executed by the at least one processor, cause the electronic device to perform the method provided according to the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that is executed by a processor to implement the method provided in the first aspect.

[0009] In a sixth aspect of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, which are executed by a processor to implement the method according to a first aspect of this disclosure.

[0010] It should be understood that the description in the Summary of the Invention section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram of an example environment in which some embodiments of this disclosure may be implemented is shown;

[0013] Figure 2A A flowchart illustrating an image instance segmentation method according to some embodiments of this disclosure is shown;

[0014] Figure 2B A flowchart illustrating a method for training an image segmentation model according to some embodiments of this disclosure is shown;

[0015] Figure 3A schematic diagram of the overall training architecture of an image segmentation model according to some embodiments of the present disclosure is shown;

[0016] Figure 4 A schematic diagram illustrating the training process of an image segmentation model according to some embodiments of the present disclosure is shown;

[0017] Figure 5 A schematic diagram illustrating the reasoning process for image instance segmentation in some embodiments of this disclosure is shown;

[0018] Figure 6 A block diagram of an image instance segmentation apparatus according to some embodiments of the present disclosure is shown;

[0019] Figure 7 A block diagram of a device that can implement some embodiments of the present disclosure is shown. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] Traditional instance segmentation tasks focus on the visible parts of an object. Unlike traditional instance segmentation, nonmodal instance segmentation requires inferring the shape of an occluded object to obtain a complete object outline. The object can be any object in an image, such as a person or animal. This ability to infer cues of invisible objects from visible object shapes is called nonmodal prediction. Related deep learning methods can perform nonmodal instance segmentation tasks. While these methods can segment and predict the entire object region, including both visible and invisible parts, they suffer from low-quality results when dealing with occluded areas.

[0023] To address this, embodiments of this disclosure propose an image instance segmentation scheme capable of nonmodal prediction. In embodiments of this disclosure, the scheme determines the ROI features (also referred to herein as region of interest features) and shape features of an object based on an input image. Then, based on the object's ROI features and shape features, the scheme determines the object's fusion features and performs image instance segmentation based on these fusion features to generate a mask instance of the object.

[0024] In this way, the scheme combines the shape information of an object with its ROI features, enabling more accurate identification of the boundary contours of the object region based on its shape characteristics. Thus, when performing image instance segmentation, this scheme can focus on the entire object region, especially areas occluded by other objects, thereby addressing the problem of insufficient prediction accuracy in existing instance segmentation tasks.

[0025] Figure 1 A schematic diagram of an example environment 100 in which some embodiments of this disclosure may be implemented is shown. For example... Figure 1 As shown, environment 100 includes camera 102 for acquiring images of the environment. In some examples, camera 102 is mounted on a vehicle to acquire images of the environment surrounding the vehicle, which may be images of the front, rear, or sides of the vehicle. In some examples, camera 102 is mounted on a door to acquire images of the environment outside the door. Furthermore, camera 102 can also be mounted in other application scenarios to acquire environmental images of that scenario for image instance segmentation tasks.

[0026] like Figure 1As shown, the environment 100 also includes a controller 106, used to receive and process the environmental image 104 captured by the camera 102. The controller 106 is communicatively connected to the camera 102, communicating via wired or wireless transmission. The controller 106 contains an image segmentation model, including a main model 106-1 and an auxiliary model 106-2. The main model 106-1 in the controller 106 can determine the ROI features of objects based on the environmental image 104. For example, if the bicycle in the environmental image 104 is object A, then the main model 106-1 can determine the ROI features of the bicycle, i.e., the ROI features of object A. Similarly, if the car in the environmental image 104 is object B, then the main model 106-1 can determine the ROI features of the car, i.e., the ROI features of object B. The auxiliary model 106-2 in the controller 106 can determine the shape features 108 of objects based on the environmental image 104. For example, if the bicycle in environment image 104 is object A, then auxiliary model 106-2 can determine the shape features of the bicycle, i.e., the shape features of object A. Similarly, if the car in environment image 104 is object B, then auxiliary model 106-2 can determine the shape features of the car, i.e., the shape features of object B. The main model 106-1 of the controller 106 performs feature fusion based on the object's ROI features and shape features 108, and performs image instance segmentation on environment image 104 based on the fused features to generate a mask instance 110 for the object. For example, the main model 106-1 generates a mask instance for object A based on the ROI features and shape features of object A. The main model 106-1 also generates a mask instance for object B based on the ROI features and shape features of object B.

[0027] In this way, when the main model performs instance segmentation on the input image, it considers the shape characteristics of the objects determined by the auxiliary model. It can utilize the fused features formed by ROI features and shape features to accurately predict the masked instances of objects, especially identifying the boundary contours of occluded areas of objects with high quality. Thus, by using the shape characteristics (3D shape features) of objects in the input image determined by the model to predict occlusion regions between objects, the accuracy of object masked instance prediction is improved. Furthermore, based on the object masked instances, a target detection image containing all objects can be generated, improving the accuracy of target detection in any detection scenario.

[0028] Figure 2A A flowchart of an image instance segmentation method 200A according to some embodiments of the present disclosure is shown. Method 200A may, for example, be derived by... Figure 1 The controller 106 in the environment 100 shown executes. For example... Figure 2AAs shown in box 202, method 200A can determine the region of interest features of the target object in the input image and the target region image of the target object based on the input image. For example, in... Figure 1 In the environment 100 shown, the environment image 104 is the input image, and the controller 106 can determine the features of interest and the target area image of the environment image 104 acquired by the camera 102.

[0029] In box 202, determining the region of interest (ROI) features of a target object in the input image based on the input image includes extracting image features from the input image and determining ROI features based on the image features. In some embodiments, a feature extraction network, such as a convolutional neural network (CNN), can be used to extract image features from the input image. In some embodiments, a target bounding box containing the region of the target is generated using a region proposal network (RPN) based on the image features. Then, ROI features are extracted from the target bounding box using a region of interest (ROI) network. Optionally, the ROI network can employ ROI pooling or ROI alignment networks.

[0030] In box 202, determining the target region image of a target object in the input image based on the input image includes extracting image features from the input image, determining the target bounding box of the target object based on the image features, extracting region of interest (ROI) features from the target bounding box, and generating the target region image of the target object based on the ROI features. In some embodiments, a feature extraction network, such as a CNN, can be used to extract image features from the input image. In some embodiments, a target bounding box containing the region of the target is generated using an RPN based on the image features. Then, the ROI network is used to extract ROI features from the target bounding box. In some embodiments, the feature map is cropped based on the ROI features to obtain the target region image of the target object. For example, when there are multiple target objects in the input image, the feature map is cropped based on the target bounding box of each target object to obtain the target region image of each target object. That is, one target object has one target region image.

[0031] In some embodiments, when determining the region of interest features of the target object in the input image and the target region image of the target object based on the input image, the feature extraction network and the RPN network can be shared.

[0032] In box 204, method 200A can extract shape features of a target object based on a target region image. In some embodiments, a pre-trained encoder is used to extract the shape features of the target object from the target region image. During the training phase, the encoder learns the 3D shape features of the object using a 3D perspective of the object in the image. During the inference phase, the encoder can extract the shape features of the object.

[0033] In block 206, method 200A can determine the fusion features of the target object based on region of interest (ROI) features and shape features. In some embodiments, the fusion features are determined by concatenating the ROI features and shape features along the depth channel. For example, concatenating the ROI feature tensor and the shape feature tensor along the depth channel yields fusion features containing 3D shape information of the target object.

[0034] In box 208, method 200A can perform instance segmentation on the input image based on fusion features to generate mask instances of target objects in the input image. In some embodiments, based on fusion features, mask instances of target objects containing visible parts, invisible parts, and occlusion parts of the input image are generated. Visible parts generally refer to objects or regions in the image that are not occluded or covered. Invisible parts refer to parts of objects that theoretically exist but do not appear in the current view or image. These parts may not appear in the image due to factors such as camera viewpoint, image cropping boundaries, or the position of the object itself. Occlusion parts refer to parts that are blocked by other objects, i.e., these parts cannot be directly seen because there are other objects in front of them. Optionally, a mask prediction network can be used to output predicted mask instances based on fusion features. For example, the mask prediction network can use AIS Former to output mask instances of visible parts, invisible parts, and occlusion parts.

[0035] In this way, since shape features embody the shape information of an object, combining shape features with ROI features for image instance segmentation allows the image segmentation model to focus on the entire object region, including visible, invisible, and occluded parts. This enables accurate prediction of the object's boundary contours, especially the boundary contours of parts of the object occluded by other objects.

[0036] Figure 2BA flowchart of a method 200B for training an image segmentation model, according to some embodiments of the present disclosure, is shown. The image segmentation model includes a main model and an auxiliary model. The main model is capable of generating a mask instance of an object based on the object's fusion features. The auxiliary model assists the main model in training, helping the shape encoder in the main model learn the 3D shape features of the object, thereby enabling the main model to obtain fusion features containing the object's 3D shape information. The method 200B includes, for example, methods that can be... Figure 1 The controller 106 in the environment 100 shown executes. For example... Figure 2B As shown in box 212, method 200B can obtain training samples, which include multiple sample images and multiple mask sample instances corresponding to sample objects in the sample images.

[0037] In box 214, method 200B can extract shape features of sample objects in a sample image based on an auxiliary model. In some implementations, the auxiliary model includes a shape encoder for extracting shape features of the sample objects.

[0038] In box 216, method 200B can generate mask prediction instances of sample objects in the sample image by performing instance segmentation on the sample image based on fused features by the main model. The mask prediction instances of the sample objects are determined based on fused feature segmentation obtained by fusing the region of interest (ROI) features and shape features of the sample objects. In some implementations, the main model includes a backbone network, a shape encoder, a fusion module, and a mask prediction network. The fusion module fuses the shape features of the sample objects extracted by the shape encoder with the ROI features output by the backbone network to obtain fused features. The mask prediction network generates mask prediction instances based on the fused features.

[0039] In box 218, method 200B can generate a mask projection instance of a sample object by an auxiliary model based on the shape features of the sample object in the sample image and the camera pose. In some implementations, the auxiliary model includes a shape decoder and a view estimator. The view estimator obtains the angle at which the sample object is captured by the virtual camera, and the shape decoder determines a shape feature volume based on the shape features output by the shape encoder. The auxiliary model projects the shape feature volume based on the camera pose, converting the 3D shape into a 2D shape, and generating a mask projection instance.

[0040] In box 220, method 200B can use the mask sample instances corresponding to the sample images as supervision signals to jointly train the main model and the auxiliary model based on the mask prediction instances and mask projection instances. In some implementations, the loss between the mask prediction instances and the mask sample instances, as well as the loss between the mask projection instances and the mask sample instances, are calculated. Based on the losses, the parameters of the shape encoder, shape decoder, and fusion module in the image segmentation model are adjusted until training converges. During this process, there is no need to adjust the view estimator parameters. After training is complete, the auxiliary model (including the view estimator and shape decoder) is removed, and inference is performed using the main model.

[0041] In this way, an auxiliary model is used to assist the main model in training, enabling the main model's shape encoder to learn the shape features of objects. Subsequently, the main model can accurately segment image instances based on fused features containing 3D shape information, especially occluded instances.

[0042] Figure 3 A schematic diagram of the overall training architecture 300 of an image segmentation model according to some embodiments of the present disclosure is shown. Figure 2A The image instance segmentation method 200A shown is implemented by an image segmentation model. For example... Figure 3 As shown, the overall training structure 300 includes an image segmentation model 306. The image segmentation model 306 includes a main model 306-1 and an auxiliary model 306-2. The main model 306-1 includes a fusion module for determining the fusion features of the target object. The auxiliary model 306-2 includes a shape encoder for extracting the shape features of the target object. Figure 3 As shown, the auxiliary model 306-2 extracts the shape features of the sample object based on the sample image 304-1 in the training samples and sends them to the fusion module in the main model 306-1. The main model 306-1 extracts the ROI features of the sample object based on the sample image 304-1. The fusion module of the main model 306-1 connects the ROI features and shape features of the sample object to obtain fused features. Based on the fused features, the main model 306-1 obtains the mask prediction instance 308 of the sample object. Based on the shape features, the auxiliary model 306-2 obtains the mask projection instance 310 of the sample object. Then, using the mask sample instance 304-2 in the training sample set as the supervision signal, the main model 306-1 and the auxiliary model 306-2 are jointly trained.

[0043] In some embodiments, a first loss 312 is calculated based on mask prediction instance 308 and mask sample instance 304-2 of the sample objects, and a second loss 314 is calculated based on mask projection instance 310 and mask sample instance 304-2 of the sample objects. The first loss 312 includes the loss for all target objects. The second loss 314 includes the loss for all target objects. The first loss 312 and the second loss 314 are superimposed to obtain the total loss 316 of the image segmentation model 306. This total loss 316 can be a large value, and then the total loss 316 can be backpropagated to the image segmentation model 306 to guide the optimization of the parameters of the main model 306-1 and the parameters of the shape encoder in the auxiliary model 306-2.

[0044] By jointly training the two branches, the shape encoder of the auxiliary model can learn the 3D shape characteristics of the object, and the learned 3D features can be well integrated with the ROI features. This allows the mask prediction instances of the sample objects output by the main model to be obtained with consideration of the 3D shape information of the object, thereby improving the accuracy of the main model in instance segmentation, especially in predicting mask instances of objects with occlusion relationships with high quality.

[0045] Figure 4 A schematic diagram illustrating the training process 400 of an image segmentation model according to some embodiments of this disclosure is shown. For example... Figure 4 As shown, the image segmentation model 436 is trained based on training samples. These training samples include multiple sample images 404-1 and multiple mask sample instances 404-2 corresponding to sample objects in the sample images 404-1. The image segmentation model 436 includes a main model and an auxiliary model. The main model includes a backbone network 406, a fusion module 414, and a mask prediction network 422. The auxiliary model includes a shape encoder 412, a shape decoder 420, and a view estimator 430. During the training process 400, based on the sample images 404-1, the main model generates mask prediction instances 426 of the sample objects in the sample images 404-1. Based on the shape features 416 of the sample objects in the sample images 404-1 and the camera pose 432, the auxiliary model generates mask projection instances 428 of the sample objects. Then, using the mask sample instances 404-2 corresponding to the sample images 404-1 as supervision signals, the main model and the auxiliary model are jointly trained.

[0046] like Figure 4As shown, the process of the main model generating the mask prediction instance 426 of the sample object in sample image 404-1 requires sequential processing through the backbone network 406, the fusion module 414, and the mask prediction network 422. In some embodiments, the backbone network 406 in the main model determines the sample region image 408 of the sample object in the sample image based on sample image 404-1. Then, based on the sample region image 408, the shape encoder 412 in the auxiliary model extracts the shape features 416 of the sample object. The backbone network 406 also determines the region of interest features 410 of the sample object in the sample image based on sample image 404-1. The fusion module 414 in the main model determines the fusion features 418 of the sample object based on the region of interest features 410 and the shape features 416. The mask prediction network 422 in the main model generates the mask prediction instance 426 of the sample object in sample image 404-1 based on the fusion features 418 of the sample object. The mask prediction instance contains multiple mask prediction instances for the target object. Each mask prediction instance masks the target object with binary 0s and the non-target object parts with binary 1s.

[0047] like Figure 4As shown, the process of generating the mask projection instance 428 of the sample object in the sample image 404-1 by the auxiliary model requires the shape encoder 412, the shape decoder 420, and the view estimator 430. In some embodiments, the backbone network 406 in the main model determines the sample region image 408 of the sample object in the sample image 404-1 based on the sample image 404-1. For example, when there are multiple sample objects in the sample image, multiple sample region images 408 can be obtained, each containing one sample object. The sample region image 408 can be obtained by using the feature extraction network, RPN network, and ROI network in the backbone network to obtain the region of interest features, and then cropping. Next, the sample region image 408 is input to the shape encoder 412 and the view estimator 430 respectively. The shape encoder 412 extracts the shape features 416 of the sample object and sends them to the shape decoder 420. The shape decoder 410 extracts the shape feature body 424 of the sample object based on the shape features 416 of the sample object. Optionally, using a ray casting method, shape feature 416 is projected onto shape feature volume 424 with the same width, height, and channel size. This shape feature volume 424 represents the reconstruction of the entire object without any occlusion, where each element in shape feature volume 424 represents the probability of a voxel mesh occupying a volume. The shape feature volume can be a cube or a cuboid. View estimator 430 extracts the camera pose 432 of the sample object captured by the virtual camera based on the sample region image 408. In some examples, the camera pose includes the angle and distance of the object captured by the virtual camera. The angle can include azimuth and pitch angles. In other examples, the virtual camera has a rotation matrix relative to the visual coordinate system of the object, and the rotation angle in the rotation matrix is ​​used to characterize the camera pose. Finally, the auxiliary model generates a mask projection instance 428 of the sample object based on the camera pose 432 and the shape feature volume 424. In some embodiments, based on the camera pose 432 of the sample object captured by the virtual camera in the sample image and the shape feature 424 of the sample object, a differential rendering module can be used to generate a mask projection instance 428 of the sample object in the sample image through differential rendering. In other embodiments, the camera pose 432 of the sample object captured by the virtual camera and the shape feature 424 of the sample object are used to perform vector dot product calculation using a dot product configuration between networks to obtain the mask projection instance 428 of the sample object. That is, based on the camera pose 432, the shape feature 424 is projected to convert the 3D shape into a 2D shape, and finally a 2D mask projection instance is obtained.

[0048] In training process 400, the image segmentation model 436 is trained using the mask sample instance 404-2 corresponding to the sample image as the supervision signal. Mask sample instance 404-2 is a mask instance with labeled occlusions. As shown by the dashed line in the figure, the mask prediction instance 426 of the sample object output by the main model, the mask projection instance 428 of the sample object output by the auxiliary model, and mask sample instance 404-2 are all input into the generator loss 434, which calculates the total loss value. This total loss value is equal to the sum of the first loss value and the second loss value. Specifically, the first loss value is calculated based on mask sample instance 404-2 and mask prediction instance 426 using the first loss function. The second loss value is calculated based on mask sample instance 404-2 and mask projection instance 428 using the second loss function. Then, the parameters of the fusion module in the main model and the shape encoder and shape decoder in the auxiliary model are adjusted using the total loss value. The first and second loss functions can be cross-entropy loss functions.

[0049] Using the above method, a set of camera rays passing through the shape feature volume and camera pose is formed, and points are sampled along the rays. The occupancy state of each point can be obtained by trilinear interpolation of voxels in the volume. Then, differential rendering can be used to accumulate the ray points onto the image plane to obtain mask projection instances. Subsequently, the mask sample instances are used as 2D supervision to train and optimize the shape encoder, enabling it to learn shape features representing the 3D shape information of the object during the training phase. Finally, the shape encoder fuses the output shape features with the ROI features, and the fused features are input into the mask head in the mask prediction model, with each mask head yielding a mask instance. In this way, the trained image segmentation model can predict the mask instances of the target object with high quality.

[0050] Figure 5 A schematic diagram illustrating the inference process 500 for image instance segmentation according to some embodiments of this disclosure is shown. Based on... Figure 4 After training is completed as shown in the training process 400, the shape decoder 420 and view estimator 430 in the auxiliary model are removed, and the shape encoders in the main model and auxiliary model are put into the inference process for image instance segmentation.

[0051] like Figure 5 As shown, input image 502 is input to backbone network 504, resulting in ROI features 508 and target region image 506. Target region image 506 is input to shape encoder 510, outputting shape features 512 of the target object. Fusion module 514 fuses ROI features 508 and shape features 512 to obtain fused features 516. Mask prediction network 518 predicts and outputs a mask instance 520 of the target object based on the fused features 516.

[0052] Furthermore, when applied to object detection, the image segmentation model can also generate an object detection image based on the mask instance 520 of the predicted object. The object detection image displays the visible, invisible, and occluded portions of the object in the input image.

[0053] In this way, the inference phase eliminates the need for shape decoders and view estimators from the training phase to perform shape reconstruction. Instead, it utilizes a shape encoder to obtain the shape information of the target object in the input image. This not only simplifies the model structure but also eliminates the need for 3D object reconstruction, improving inference and prediction efficiency. Furthermore, the fusion module of the main model integrates shape features with ROI features, enabling the main model to predict non-modal mask instances more accurately based on the complete shape information of the object.

[0054] Figure 6 A block diagram of an image instance segmentation apparatus 600 according to some embodiments of the present disclosure is shown. For example... Figure 6 As shown, the apparatus 600 includes a generation module 602, configured to generate region-of-interest (ROI) features of a target object in the input image and a target region image of the target object based on the input image. The apparatus 600 also includes an extraction module 604, configured to extract shape features of the target object based on the target region image. The apparatus also includes a fusion module 606, configured to determine fusion features of the target object based on the RIO features and shape features. Furthermore, the apparatus 600 includes a segmentation module 608, configured to perform instance segmentation on the input image based on the fusion features to generate mask instances of the target object in the input image.

[0055] In some embodiments, the generation module 602 includes a backbone network. This backbone network includes a feature extraction network, an RPN network, and a ROI network. The input image is processed by the feature extraction network to obtain image features. The image features are then processed by the RPN network to generate bounding boxes, which contain the region of the target. Subsequently, the ROI network is used to extract region of interest features from the bounding boxes. Furthermore, based on the region of interest features, cropping is performed to obtain an image of the target region of the target object.

[0056] In some embodiments, the extraction module 604 includes a shape encoder configured to extract shape features of a target object based on a target region image of the target object.

[0057] In some embodiments, the fusion module 606 is used to connect the region of interest features and shape features of the target object along the depth channel to form a fused feature.

[0058] In some embodiments, the segmentation module 608 includes a mask prediction network configured to predict mask instances of a target object based on fusion features.

[0059] In some embodiments, the apparatus 600 further includes a training module configured to train the generation module 602, the extraction module 604, the fusion module 606, and the segmentation module 608.

[0060] In some embodiments, the training module includes a shape decoder, a view estimator, and a generator loss. During the training phase, an auxiliary model for the image segmentation model is constructed using the shape encoder, shape decoder, and view estimator, and is jointly trained with the main model, which consists of the generation module 602, the fusion module 606, and the segmentation module 608. The auxiliary model learns the 3D shape information of the object. The main model, based on the fused features obtained by fusing the object's shape information, can accurately predict the mask instance of the object.

[0061] The training process can be referred to Figure 4 The training process 400 shown uses mask sample instances labeled with occlusions as 2D supervision to train the main model and auxiliary model. During training, the parameters of the main model, shape encoder, and shape decoder are adjusted. After training, inference is performed based on the generation module 602, extraction module 604, fusion module 606, and segmentation module 608.

[0062] Figure 7 A schematic block diagram of an example device 700 that can be used to implement embodiments of the present disclosure is shown. Device 700 may correspond to the controller in the foregoing method embodiments. Figure 7 As shown, device 700 includes a processor 701, which can perform various appropriate actions and processes based on computer program instructions loaded into random access memory (RAM) 703 according to computer program instructions stored in read-only memory (ROM) 702. RAM 703 may also store various programs and data required for the operation of device 700. The processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0063] The various processes and procedures described above, such as method 200A, method 200B, and training process 400, can be executed by processor 701. For example, in some embodiments, method 200A, method 200B, and training process 400 can be implemented as software programs tangibly contained in a machine-readable medium. In some embodiments, part or all of the software program can be loaded and / or installed on device 700 via ROM 702. When the software program is loaded into RAM 703 and executed by processor 701, one or more actions of method 200A, method 200B, and training process 400 described above can be performed.

[0064] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload programmable logic devices (CPLDs), and so on.

[0065] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0066] This disclosure can be a method, apparatus, and / or program product. The program product may include a machine-readable storage medium on which machine-readable program instructions for performing various aspects of this disclosure are loaded. The machine-readable program instructions described herein can be downloaded from the machine-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the machine-readable program instructions from the network and forwards them to the machine-readable storage medium in the respective computing / processing device.

[0067] Machine program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. Machine-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the machine-readable program instructions to implement various aspects of this disclosure.

[0068] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. Furthermore, although operations are depicted in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0069] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for image instance segmentation (200A), comprising: Based on the input image, determine (202) the region of interest features of the target object in the input image and the target region image of the target object; Based on the target region image, extract (204) the shape features of the target object; Based on the region of interest features and the shape features, determine (206) the fusion features of the target object; as well as Based on the fusion features, the input image is segmented to generate (208) a mask instance of the target object in the input image.

2. The method (200A) according to claim 1, wherein determining (202) the region of interest features of the target object in the input image and the target region image of the target object based on the input image comprises: Based on the input image, extract the image features of the input image; Based on the image features, the features of the region of interest and the bounding box of the target object are determined; as well as Based on the target bounding box, the target region image of the target object is generated.

3. The method (200A) according to claim 1, wherein determining (206) the fusion features of the target object based on the region of interest features and the shape features includes: The fused features are determined by connecting the region of interest features and the shape features along the depth channel.

4. The method (200A) according to claim 1, wherein the input image contains at least a partially invisible target object and / or at least a partially occluded target object, and based on the fusion features, the input image is segmented to generate (208) a mask instance of the target object in the input image, comprising: Based on the fusion features, non-modal instance segmentation is performed on the input image to generate a mask instance containing the visible portion, as well as the invisible portion and / or occluded portion of the target object in the input image.

5. The method (200A) according to claim 1, wherein the method is implemented by an image segmentation model, the image segmentation model comprising a main model and an auxiliary model, the main model comprising a fusion module for determining the fusion features of the target object, and the auxiliary model comprising a shape encoder for extracting the shape features of the target object.

6. A method (200B) for training an image segmentation model, said image segmentation model comprising a main model and an auxiliary model, said method comprising: (212) Obtain training samples, the training samples including multiple sample images and multiple mask sample instances corresponding to sample objects in the sample images; The auxiliary model extracts (214) the shape features of the sample objects in the sample image; The main model performs instance segmentation on the sample image based on the fusion features to generate (216) a mask prediction instance of the sample object in the sample image. The fusion features are determined based on the region of interest features and shape features of the sample object in the sample image. The auxiliary model generates (218) a mask projection instance of the sample object based on the shape features of the sample object in the sample image and the camera pose; as well as Using the mask sample instance corresponding to the sample image as the supervision signal, the main model and the auxiliary model are jointly trained based on the mask prediction instance and the mask projection instance (220).

7. The method (200B) according to claim 6, wherein the main model performs instance segmentation on the sample image based on fusion features to generate (216) a mask prediction instance of the sample object in the sample image, comprising: Based on the sample image, determine the region of interest features of the sample object in the sample image and the sample region image of the sample object; Based on the sample region image, the shape encoder in the auxiliary model extracts the shape features of the sample object; Based on the region of interest features and the shape features of the sample object, the fusion module in the main model determines the fusion features of the sample object; as well as Based on the fusion features of the sample objects, the mask prediction network of the main model generates mask prediction instances of the sample objects in the sample images.

8. The method of claim 6 (200B), further comprising: Based on the sample region image of the sample object, the shape encoder in the auxiliary model extracts the shape features of the sample object, and the view estimator in the auxiliary model extracts the camera pose of the sample object.

9. The method (200B) according to claim 6 or 8, wherein generating (218) a mask projection instance of the sample object by the auxiliary model based on the shape features of the sample object in the sample image and the camera pose comprises: Based on the shape features of the sample object, the shape feature volume of the sample object is extracted by the shape decoder in the auxiliary model; as well as Based on the camera pose and the shape feature, the auxiliary model generates a mask projection instance of the sample object.

10. The method (200B) according to claim 9, wherein generating a mask projection instance of the sample object by the auxiliary model based on the camera pose and the shape feature volume comprises: Based on the camera pose and shape features of the sample object in the sample image, a mask projection instance of the sample object in the sample image is generated by differential rendering.

11. The method (200B) according to claim 9, wherein using the mask sample instance corresponding to the sample image as a supervision signal, and jointly training the main model and the auxiliary model based on the mask prediction instance and the mask projection instance (220) comprises: Based on the mask sample instance and the mask prediction instance, a first loss value is calculated using a first loss function; Based on the mask sample instance and the mask projection instance, a second loss value is calculated using a second loss function; as well as Based on the first loss value and the second loss value, the parameters of the fusion module in the main model and the shape encoder and shape decoder in the auxiliary model are adjusted.

12. The method (200B) according to claim 11, wherein the fusion module of the main model and the shape encoder in the auxiliary model are applied to the image instance segmentation inference process after the image segmentation model has been trained.

13. An image instance segmentation apparatus (600), comprising: The generation module (602) is configured to generate, based on the input image, the region of interest features of the target object in the input image and the target region image of the target object; The extraction module (604) is configured to extract the shape features of the target object based on the target region image; The fusion module (606) is configured to determine the fusion features of the target object based on the region of interest features and the shape features; as well as The segmentation module (608) is configured to perform instance segmentation on the input image based on the fusion features to generate a mask instance of the target object in the input image.

14. An electronic device (700), comprising: At least one processor (701); as well as A memory (702) coupled to the at least one processor (701) and having instructions stored thereon, which, when executed by the at least one processor (701), cause the electronic device (700) to perform the method according to any one of claims 1-12.

15. A computer program product comprising a computer program that is executed by a processor to implement the method according to any one of claims 1 to 12.

Images