A method for completing non-visible regions of an image based on non-modal instance segmentation

Abstract

This invention provides a method for image non-visible region completion based on non-modal instance segmentation, belonging to the field of computer vision technology. This method establishes a non-modal instance segmentation model based on weakly supervised learning using the Swing Transformer. A non-visible region completion network is trained on an improved Swing Transformer Unet segmentation network, thus eliminating the need for manual annotation of occluded region masks. Simultaneously, occlusion boundary estimation is introduced, and the prediction weights for occluded and visible regions are redesigned. The concept of adversarial generative learning is introduced, feeding the predicted non-visible region masks into the discriminator, adding an adversarial generative discriminator loss function, and improving the mask shape output by the discriminator. This method employs weakly supervised learning, significantly reducing the cost of sample occlusion annotation, while also addressing the problem of scarce non-modal datasets. The completion network effectively allows image data with ground-value non-modal masks to learn supervised pixel completion information during completion learning, thereby generating pixel completion capabilities.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for completing non-visible regions of an image based on non-modal instance segmentation. Background Technology

[0002] In scene understanding using computer vision technology, many objects are occluded, and sometimes objects even occlude each other. In scenarios where objects are partially occluded, humans can perceive the complete shape of the occluded object based on prior knowledge, describe the boundary information of the occluded area, and possess the ability to collect prior information from the shapes of visible objects and infer invisible cues—this ability is called amodal prediction. With the popularization of deep learning technology, visual technologies such as image classification, object detection, and image segmentation in various fields have developed rapidly, even surpassing the human visual system in the ability to predict and recognize visible objects. However, these technologies are mainly used for predicting visible areas in images and lack the ability to predict invisible areas. Completing the shape of invisible parts of an image helps to assemble disordered, fragmented, and incomplete objects into a coherent and complete scene.

[0003] The ability to predict the shape of non-visible regions in an image can solve many problems in practical applications. For example, in agriculture, when developing machine vision systems for harvesting fruits, the prediction of non-visible regions enables robots to grasp and manipulate objects hidden behind complex and cluttered scenes. The most common method for restoring the complete shape of occluded objects is non-modal instance segmentation, the core idea of ​​which is to segment a joint mask of the visible and non-visible regions of the object.

[0004] Non-modal instance segmentation techniques are mainly divided into two types:

[0005] One approach is a two-stage instance segmentation method combining object detection. For example, Mask R-CNN is a detector improved from Faster R-CNN, segmenting instances within their bounding boxes (ROIs) to generate instance masks. Another approach is a two-stage non-modal instance segmentation method based on CNNs. This method updates the bounding boxes of occluded regions predicted by the Faster R-CNN object detector based on the probability of each pixel in the segmentation heatmap being an ROI object, and then iteratively recalculates the heatmap to design the bounding boxes. Mask R-CNN and PANet incorporate features to determine if occlusion exists within an ROI, combining more features to adapt to the task of segmenting occluded masks. BCNet decouples overlapping objects within the same ROI into non-intersecting two-layer images, combining a graph convolutional network to infer the mask of the non-visible region based on the occlusion relationship between the two layers. Unlike CNN models, context-aware composite convolutional neural networks integrate the advantages of symbolic models and convolutional neural networks, predicting interpretable bounding boxes for instances that have lost contextual information due to moderate to severe occlusion, and then using a segmentation model to output the instance.

[0006] Another approach is to directly predict the mask of the occluded region of an instance without relying on bounding boxes predicted by object detection. Cubic Bézier curves, a mathematical curve derived from computer graphics, can be used to complete the shape of the occluded region. The ORCNN algorithm, built on Mask R-CNN, employs two segmentation branches to simultaneously predict both visible and invisible masks, calculating the difference between them to output the predicted mask of the occluded region. SLN provides a semantically aware distance map for all instances in the image and generates bounding box proposals for invisible objects in the ROI-Align layer to obtain instance-level and global-level feature relationships.

[0007] The methods described above are all based on supervised learning and require a large number of pixel-level instance mask labels for model training. However, label annotation for occluded regions is very costly. Therefore, researchers have attempted to reduce the reliance on manual labels. For example, PCNet uses a self-supervised approach to train the segmentation network to obtain the ability to complete the mask of occluded regions. It only needs to provide the mask of the visible region as input, and sets an occluder on the visible mask to recover the previous visible mask. During the completion process, it optimizes the segmentation results based on the occlusion order of instances in the image. ASBUNet uses occlusion boundaries instead of the occluder mask in PCNet, abandoning the intervention of occlusion order. Variational autoencoders use probability estimation to predict occluded region masks, generating diverse non-modal masks and making proposal decisions, thus eliminating the need for ground truth mask labels. CSDNet generates pseudo non-modal masks from real images. By dividing the image into many instance layers, modeling the occlusion of layers, and estimating the occlusion order of layers, it detects that fully visible objects detected on each layer of the image are actually occluders on the next layer. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a method for completing the non-visible region of an image based on non-modal instance segmentation. This method uses pixel prediction, non-modal prediction, and image instance segmentation techniques, based on weakly supervised deep learning, non-modal prediction, etc., to achieve morphological completion of the non-visible region of an image.

[0009] To solve the above-mentioned technical problems, the technical solution adopted by this invention is: an image non-visible region completion method based on non-modal instance segmentation. This method establishes a non-modal instance segmentation model based on weakly supervised learning using Swing Transformer, and trains a non-visible region completion network on an improved Swing Transformer Unet segmentation network, thus eliminating the need for manual annotation of the occluded region mask. Simultaneously, it introduces occlusion boundary estimation from ASBUNet, redesigns the prediction weights for occluded and visible regions, and incorporates the idea of ​​adversarial generative learning. The predicted non-visible region mask is fed into the discriminator, and an adversarial generative discriminator loss function is added to improve the mask shape output by the discriminator. Specifically, the method includes the following steps:

[0010] Step 1: Obtain the RGB image dataset and perform image preprocessing;

[0011] Acquire RGB image data; based on the acquired images, use cosine similarity calculation to remove images with a repetition greater than a set threshold, calculate the variance of the Laplacian transform of the images to remove images with a blur greater than a set threshold, and then filter out images with cluttered backgrounds and poor shooting angles for deletion.

[0012] Step 2: Label the images in the image dataset to construct the training set and validation set;

[0013] The Labelme tool is used to annotate image semantic labels. Combined with scene structure, both visible and invisible parts of the image are semantically labeled. For unoccluded objects, a length mask annotation method is used. For occluded objects, a hierarchical annotation method is used. The first layer annotates the visible part, the second layer annotates the invisible part, and the third layer annotates the joint area of ​​the visible and invisible parts to obtain nonmodal ground truth labels.

[0014] The training and validation sets were constructed at a 9:1 ratio. After horizontal flipping, translation, and random cropping, the training set was augmented to 2430 images.

[0015] Step 3: Construct a non-modal instance segmentation model based on weakly supervised learning using the Swing Transformer; the non-modal instance segmentation model includes a 4×4 convolutional kernel, a linear layer, an encoder, a decoder, and a discriminator;

[0016] The model uses the occlusion boundaries and visible portion masks of the image as input. First, a 4×4 convolution kernel is used to convolve the image, resulting in a tensor with 96 channels. Then, a linear layer is applied to the model for dimensionality transformation. After the linear layer, the image becomes a series of segmented features, which are further input into the encoder. Both the encoder and decoder employ the SwinTransformer Block based on the improved SwinTransformer Unet symmetric architecture. The improved SwinTransformer Unet serves as the backbone of the segmentation network for the entire non-modal instance segmentation model. The encoder collects shallow features through four sampling layers. These shallow features are then fused with the deep features collected by the decoder through jump connections. The decoder incorporates joint weights for predicting visible and invisible region masks to adjust the model's loss function. The discriminator introduces adversarial generative learning to judge and improve the mask morphology of the invisible region masks predicted by the decoder.

[0017] The non-modal instance segmentation model uses a boundary estimation method to input the contact boundary between the occluder and the occluded object into the model in the form of a mask. The model will output an uncertain boundary map of the other side of the occluded part. The contents in the two boundary parts will be filled with pixels to obtain a reasonable shape.

[0018] The nonmodal instance segmentation model employs a partial completion algorithm to implement a weakly supervised learning process for nonmodal mask completion. Based on the idea of ​​allowing the model to autonomously learn and recover invisible masks, during the training phase, the images are processed to randomly sample occluded and occluded instances, and only their visible masks are used for model training. Specifically, two images with an occlusion relationship are randomly selected, and the image containing instance A is denoted as I. A Another image containing instance B is denoted as I. B And the visible masks of the two instances marked in the image are denoted as M respectively. A M B ;

[0019] The first case is that instance A is occluded by instance B, and the mask image of the occlusion is denoted as M. A / M B Then combine the image of instance A (Image) A Masked M B The occluded image is denoted as I. A / M B Together, they are fed into an hourglass-structured segmentation network for segmentation, and the recovered target is the visible mask of instance A. Meanwhile, to prevent the model from over-filling pixels of other instances, a second case is where instance B is occluded by instance A; the mask image of the occlusion is denoted as M. B / M A Joint input I A / M B / M A The restored target is also the visible mask of instance A, allowing the completion network in the non-modal instance segmentation model to learn whether to complete the target pixels, thus achieving the purpose of the regularization learning process; subsequently, a discriminator D is introduced. m This improves the quality of the restored mask.

[0020] The ordering of pairwise occlusion relationships is represented by pairwise instances formed by two visible masks connected together; the pairwise occlusion order matrix is ​​used to represent object relationships, and a directed graph is drawn to visualize the object relationships.

[0021] The occlusion order directed graph is represented as G = (Y, T), where Y is the set of all objects in the image, with a size of N, and T is an N×N matrix. i,j The occlusion relationship between two adjacent pairs of objects i and j is expressed as follows:

[0022]

[0023] In the above formula, and M represents the completed nonmodal mask for objects i and j.i With M j Represents the visibility mask of objects i and j. and These represent the pixel value increments of the non-modal masks after two visible mask completions. If the increments of objects i and j after the mask completion network are equal and both are 0, it means that they do not occlude each other and are on the same layer. In the matrix, T... i,j If the value of T is 0, and the increment of the former is less than that of the latter, it means that object i occludes object j. i,j If the value of T is 1, and the increment of the former is not less than the increment of the latter, it means that object j occludes object i. i,j The value is -1; based on the occlusion order generated in pairs, the occlusion order between objects in the whole scene is gradually inferred, providing clear prior information support and interpretability for non-modal mask completion;

[0024] Step 4: Train the non-modal instance segmentation model through three stages to determine the optimal model parameters;

[0025] Step 5: Test the completion capability and occlusion order prediction capability of the non-modal instance segmentation model;

[0026] After training, the non-modal instance segmentation model is tested using a test set to evaluate its non-modal completion and order prediction capabilities. The non-modal object segmentation model is evaluated using pairwise occlusion order accuracy and the intersection-over-union (IoU) ratio between the completed mask and the ground truth. The completed non-modal mask is visualized to determine its shape and regularity. Subsequently, all non-modal completion results are used as pseudo-non-modal annotation labels and fed into Mask R-CNN for training to obtain corresponding object segmentation results. The object segmentation results are visualized, and the accuracy and precision of the mask segmented by the non-modal object segmentation model are evaluated using evaluation metrics from the COCO dataset.

[0027] The beneficial effects of adopting the above technical solution are as follows: The image non-visible region completion method based on non-modal instance segmentation provided by this invention allows the constructed non-modal object segmentation model to be masked and trained under the premise that only visible regions exist in the image. The model utilizes prior information such as occlusion order and object shape to extract features from the visible parts of the object, ultimately predicting information about the invisible regions of the object in the image, and finally achieving shape completion of the occluded object. This technology can significantly reduce the difficulty and cost of annotating training samples, and achieve morphological information recovery of invisible regions of objects in images through weakly supervised learning. This helps improve the positioning accuracy of the vision system of harvesting robots and the depth recovery accuracy of harvesting scenes.

[0028] Employing weakly supervised learning significantly reduces the cost of occlusion annotation for samples in practical applications, while also addressing the scarcity of amodal datasets. The completion network effectively enables image data with ground truth amodal masks to learn supervised pixel completion information during completion learning, thereby generating pixel completion capabilities. Based on accurate occlusion order, the generated pseudo-annotation labels can replace ground truth labels in model training.

[0029] The model's segmentation network uses Transformer UNet, with a Swin-tiny backbone network to improve convergence speed. By introducing occlusion boundaries between adjacent objects, a relatively regular tomato-shaped uncertainty map is used as a shape prior. The pixels of the target object are determined by defining the boundaries of the mask shape. The shape prior significantly improves the model's performance when predicting the complete shape of the object. The prior enables the model to achieve high accuracy in non-modal mask completion, making it suitable for handling complex visual tasks such as occlusion prediction. Attached Figure Description

[0030] Figure 1 The mask labeling effect of the visible and non-visible areas of tomatoes provided in the embodiments of the present invention is shown in the following: (a) is the original image with three tomatoes, (b) is the semantic label of the visible part, (c) is the mask of the occluded part with manual annotation, and (d) is the complete non-modal truth label.

[0031] Figure 2 This is a network diagram of a non-modal instance segmentation model provided in an embodiment of the present invention;

[0032] Figure 3 This is a diagram of the improved Swing Transformer Unet structure provided in an embodiment of the present invention;

[0033] Figure 4 This is a structural diagram of the Swing Transformer Block provided in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram illustrating the process of completing invisible areas of a tomato image according to an embodiment of the present invention.

[0035] Figure 6 The occlusion order prediction diagram of the tomato instance provided in the embodiment of the present invention is shown in (a) as an instance diagram of the occlusion scene and (b) as an occlusion order diagram.

[0036] Figure 7 This is a comparison chart of non-modal masking results for different models provided in embodiments of the present invention;

[0037] Figure 8These are instance segmentation result diagrams corresponding to different models provided in the embodiments of the present invention;

[0038] Figure 9 This is a diagram showing the non-modal instance segmentation result provided in an embodiment of the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0040] In this embodiment, an image non-visible region completion method based on non-modal instance segmentation is proposed. A non-modal instance segmentation model based on weakly supervised learning using Swing Transformer is established. A non-visible region completion network is trained on an improved Swing TransformerUnet segmentation network, thus eliminating the need for manual annotation of the occluded region mask. Simultaneously, occlusion boundary estimation from ASBUNet is introduced, and the prediction weights for occluded and visible regions are redesigned. The concept of adversarial generative learning is introduced, feeding the predicted non-visible region mask into the discriminator, adding an adversarial generative discriminator loss function, and improving the mask shape output by the discriminator. Specifically, the method includes the following steps:

[0041] Step 1: Obtain the RGB image dataset and perform image preprocessing;

[0042] Use the Azure Kinect depth camera to acquire RGB image data with a resolution of 1280×720 or higher; based on the acquired images, use cosine similarity calculation to remove images with a repetition greater than a set threshold, calculate the variance of the Laplacian transform of the images to remove images with a blur greater than a set threshold, and then filter out images with cluttered backgrounds and poor shooting angles for deletion.

[0043] Step 2: Label the images in the image dataset to construct the training set and validation set;

[0044] The Labelme tool is used to annotate image semantic labels. Combined with scene structure, both visible and invisible parts of the image are semantically labeled. For unoccluded objects, a length mask annotation method is used. For occluded objects, a hierarchical annotation method is used. The first layer annotates the visible part, the second layer annotates the invisible part, and the third layer annotates the joint area of ​​the visible and invisible parts to obtain nonmodal ground truth labels.

[0045] This embodiment uses the Labelme tool to create mask labels in the image. By designing and annotating each part with different semantic labels, they are merged into a complex scene, such as... Figure 1As shown in Figure (a), the original image contains three tomatoes in prominent positions. The tomatoes on the left and right are fully visible, while the tomatoes in the darker middle area are largely obscured, indicating high ripeness. The remaining tomatoes in the background are discarded due to their small pixel size and incorrect angle, and are treated as background pixels. The tomatoes on either side only require ordinary pixel-level labeling and do not need layer-level labeling. Figure (b) shows the ground truth semantic labels for the visible parts of the obscured tomatoes in the image, which can be used as ground truth for visible regions in instance segmentation tasks. Figure (c) shows the ground truth regions (masks of the obscured areas) of the invisible parts of the tomatoes in the manually labeled image. To avoid the influence of subjective randomness on the ground truth regions, each image is labeled by 10 people. Images with obvious labeling errors are removed, and the label regions of the remaining images are binarized with the background. For pixels at the same location in different binary images, if the pixel belongs to the foreground (obscured area) in more than half of the images, the pixel is set as a pixel of the obscured area; otherwise, the pixel is a background pixel. Figure (d) shows the non-modal mask after combining the visible and non-visible areas of the tomato in the image.

[0046] Non-modal instance segmentation requires labeling both visible and invisible regions separately, and the labeling of invisible regions requires multiple people, resulting in high manual costs for ground truth labels. To reduce costs, this invention proposes a non-modal instance segmentation model based on weakly supervised learning, which uses a mask of the visible region as a label during training.

[0047] In this embodiment, for each image's occluded region, we selected 20 annotators to annotate the invisible areas of randomly assigned images. The average of their annotated regions was then calculated; if more than 50% of the annotators selected a pixel region as foreground, that region was considered foreground. This embodiment carefully distinguishes between obvious errors and subjective ambiguities, using only reasonable amodal annotations as the basis for evaluation in the test set. This approach helps ensure the quality and reliability of the annotations used for evaluation.

[0048] The training and validation sets were constructed at a 9:1 ratio. After horizontal flipping, translation, and random cropping, the training set was augmented to 2430 images.

[0049] Step 3: Construct a non-modal instance segmentation model based on weakly supervised learning using the Swing Transformer; the non-modal instance segmentation model is as follows: Figure 2 As shown, it includes a 4×4 convolutional kernel, a linear layer, an encoder, a decoder, and a discriminator;

[0050] The nonmodal instance segmentation model, based on the improved Swin Transformer UNet, employs a partial completion algorithm and trains a completion network, thereby reducing the reliance on a large number of manual modal annotations for training. The model uses the occlusion boundaries and visible portion masks of the image as input. First, a 4×4 convolution kernel is used to convolve the image, resulting in a tensor with 96 channels. Then, a linear layer is applied to the model for dimensionality transformation. After the linear layer, the image becomes a series of segmented features, which are further input into the encoder. Both the encoder and decoder employ a Swin Transformer Block based on an improved Swin Transformer Unet symmetric architecture. The improved Swin Transformer Unet serves as the backbone of the segmentation network for the entire non-modal instance segmentation model. The encoder collects shallow features through four sampling layers. These shallow features are then fused with deep features collected by the decoder through jump connections. The decoder incorporates joint weights for predicting visible and invisible region masks to adjust the model's loss function. An adversarial generative learning approach is introduced in the discriminator to judge and improve the mask morphology of the invisible region masks predicted by the decoder.

[0051] Swin Transformer is a novel paradigm based on deep learning architecture and a visual backbone plus learning tasks. This architecture addresses the computational complexity issues of traditional CNN networks when handling large images by introducing a hierarchical mechanism and the sliding window concept. Traditional Transformer models require calculating global attention weights when performing multi-head self-attention mechanisms. This computation is extremely expensive, especially when the input sequence is very long. Swin Transformer employs a block-based mechanism, dividing the input sequence into multiple blocks, with each block considering only local information from its neighboring blocks, significantly reducing computational complexity. We noted that after deep downsampling, segmentation is required. SwinTransformer UNet proposes a U-shaped symmetric Swin Transformer Block decoder to upsample the extracted deep features. We then add a 1×1 convolutional layer after the improved extended layer with skip connections to resize and flatten the output. The network first segments the input image into a series of non-overlapping patches, each 4×4 in size. Since each pixel has three RGB channels, each patch has a dimension of 4×4×3, and is finally transformed to dimension C through a linear embedding layer. Then, through four stacked modules, the PatchMerging layer concatenates adjacent 2×2 patches to obtain a 4C-dimensional feature map, which is then reduced to 2C by a linear layer. After downsampling through four modules, a feature map F is obtained. The decoder is designed for upsampling. Each Patch Expendging layer upsamples the feature map to restore it to twice the resolution of the input. The final Expending Up layer restores it to four times the resolution. The network is then connected to a 4×4 convolutional layer to output the class. Each encoder and decoder is connected to a jump connection to fuse shallow and deep features. The structure is as follows: Figure 3 As shown.

[0052] In the Swing Transformer, multiple Transformer layers are not completely independent but connected through multi-layer cross-attention. Each layer receives input from the previous layer and other branches, weights and fuses them using the multi-layer attention mechanism, and outputs it to the next layer. Each layer also introduces local attention for encoding image patches. It focuses only on local locations within the input sequence, thus reducing computation. The encoder and decoder of the Transformer are similar in that they are primarily composed of multiple Swing Transformer Blocks. Each Swing Transformer Block contains a relative position encoding layer, a multi-head self-attention layer, an MLP layer, a fully connected feedforward network layer, and a residual connection. These blocks can be computed in parallel, improving computational efficiency and significantly enhancing model performance. Swing Transformer Blocks have two structures, such as... Figure 4 As shown, the only difference between the two structures is that one uses a W-MSA structure while the other uses a SW-MSA structure; these two structures are used in pairs. For each window consisting of patches, W-MSA computes its Query, Key, and Value vectors, still following the Self-Attention algorithm, which is represented as:

[0053] Attention(Q,K,V)=SoftMax(dQKT+B)V

[0054] Next, the Query, Key, and Value vectors of each window are multiplied by matrices to obtain the Attention matrix for each window. These Attention matrices are then concatenated to obtain the final Attention matrix. To address information exchange between windows, the SW-MSA moving window operation shifts each window a certain distance to the left or right, creating some overlap between adjacent windows. This allows the model to consider information from neighboring windows when calculating self-attention.

[0055] The non-modal instance segmentation model uses a boundary estimation method to input the contact boundary between the occluder and the occluded object into the model in the form of a mask. The model will output an uncertain boundary map of the other side of the occluded part. The contents in the two boundary parts will be filled with pixels to obtain a reasonable shape.

[0056] Most previous supervised methods added a non-modal prediction branch to complete the task, while another branch focused only on visible mask features. However, completing the non-modal mask actually requires both the occluded and visible parts to have similar features. The method used in this invention employs a partially completed training approach, based on the idea of ​​allowing the model to autonomously learn and recover the invisible mask. During the training phase, the data images are processed, and occluded and partially occluded instances are randomly sampled. The non-modal instance segmentation model uses a partially completed algorithm to implement the weakly supervised learning process for non-modal mask completion; for example... Figure 5 As shown, based on the idea of ​​allowing the model to autonomously learn and recover the invisible mask, during the training phase, the image is processed to randomly sample occlusion instances and occluded instances, and only their visible masks are used for model training. Specifically, two images with an occlusion relationship are randomly selected, and the image containing instance A is denoted as I. A Another image containing instance B is denoted as I. B And the visible masks of the two instances marked in the image are denoted as M respectively. A M B ;

[0057] The first case is that instance A is occluded by instance B, and the mask image of the occlusion is denoted as M. A / M B Then combine the image of instance A (Image) A Masked M B The occluded image is denoted as I. A / M B Together, they are fed into an hourglass-structured segmentation network for segmentation, and the recovered target is the visible mask of instance A. Meanwhile, to prevent the model from over-filling pixels of other instances, a second case is where instance B is occluded by instance A; the mask image of the occlusion is denoted as M. B / M A Joint input I A / M B / M A The restored target is also the visible mask of instance A, allowing the completion network in the non-modal instance segmentation model to learn whether to complete the target pixels, thus achieving the purpose of the regularization learning process; subsequently, a discriminator D is introduced. m This improves the quality of the restored mask. The process can be described as follows:

[0058] M A M B =N(A, B)

[0059] M preA =D m (PCM θ (M A / MB ,I A / M B ))

[0060] M preA =D m (PCM θ (M B / M A ,I A / M B / M A ))

[0061] Among them, M preA Let θ be the visible mask of instance A obtained in both cases, N be the segmentation model or manual annotation, PCM be the partial completion module, and θ be the parameters of the partial completion module.

[0062] Uncertainty estimation in instance segmentation quantifies the model's uncertainty regarding instance boundaries in an image. This is achieved by introducing a probabilistic model to handle blurred and uncertain boundary locations. In the partial completion task above, when the model recovers occluded instances, the invisible parts will be filled with different pixels due to boundary uncertainty. For example... Figure 5 As shown, the model feeds the original image, the occluded mask of the occluded object, and the mask of the occluded object into the hourglass network module, and then provides a mask branch to obtain the occlusion boundary. When the uncertainty is estimated to be large, the model will capture a reasonable variable shape size. The visible portion of the mask takes a size of 224×224×2 as input, and through the SwinTransformer Unet, it yields an H×W×2 feature map, where H represents the height and W represents the width. The feature map has two dimensions: one channel for non-modal segmentation prediction and one channel for estimating boundary uncertainty. By calculating the confidence interval, the range of the prediction result is obtained. The uncertainty map output by the model produces a relatively low segmentation loss in regions of high uncertainty, while the generated shape prior helps the model better understand spatial distribution information.

[0063] Humans can intuitively perceive the order of overlapping objects in natural scenes. Analyzing the depth ordering of instances in an image helps recover non-modal masks; however, sequential ordering is not ideal for understanding the order of cyclically occluded objects. While ordering pairwise occlusion relationships, using pairs of adjacent instances connected by two visible masks to represent occlusion between objects, fails to address the issue of two objects occluding each other. Considering only ROI (Region of Interest) occlusion without reasoning about occlusion order makes it unsuitable for complex scenes with many overlapping objects. When the model is trained using a partial completion algorithm, it gains the ability to determine whether to complete the pixels of occluded objects and also solves the problem of predicting the occlusion order between objects in an image. This invention borrows the occlusion boundary concept proposed by ASBUNet and uses the image completion network to fill in different boundary segments to generate different instance pixel values. This combination compensates for the errors caused by the image completion network's inability to follow consistent object geometry and missing boundaries. We use a paired occlusion order directed graph to represent instance relationships. The ordering of pairwise occlusion relationships is represented by pairing instances connected by two visible masks to form adjacent instance pairs; pairwise occlusion order matrices are used to represent object relationships, and directed graphs are drawn to visualize these relationships, such as... Figure 6 As shown:

[0064] The occlusion order directed graph is represented as G = (Y, T), where Y is the set of all objects in the image, with a size of N, and T is an N×N matrix. i,j The occlusion relationship between two adjacent pairs of objects i and j is expressed as follows:

[0065]

[0066] In the above formula, and M represents the completed nonmodal mask for objects i and j. i With M j This represents the visible (modal) mask of objects i and j. and These represent the pixel value increments of the non-modal masks after two visible mask completions. If the increments of objects i and j after the mask completion network are equal and both are 0, it means that they do not occlude each other and are on the same layer. In the matrix, T... i,j If the value of T is 0, and the increment of the former is less than that of the latter, it means that object i occludes object j. i,j If the value of T is 1, and the increment of the former is not less than the increment of the latter, it means that object j occludes object i. i,jThe value is -1; based on the occlusion order generated in pairs, the occlusion order between objects in the whole scene is gradually inferred, providing clear prior information support and interpretability for non-modal mask completion.

[0067] Step 4: Train the non-modal instance segmentation model through three stages to determine the optimal model parameters;

[0068] In this embodiment, both model training and testing were performed on the experimental platform built in this study, ensuring the consistency of the comparison conditions. The main hardware configuration of this platform is as follows: CPU: AMD R5 3600X, 3.8GHz; RAM: 64GB; GPU: NVIDIA GeForce RTX 3090, 24GB VRAM. The main software environment includes: Ubuntu 20.04 operating system, PyTorch 1.10 deep learning framework, CUDA 11.7 general-purpose parallel computing architecture, cuDNN 8.3.4 GPU acceleration library for deep neural networks, Python 3.7 programming language, and OpenCV 4.5.1 computer vision library.

[0069] The model is trained in three stages. The first stage trains on low-resolution images, which can be accelerated by using a larger batch size and fewer iterations, i.e., as a pre-training process. The second stage trains on high-resolution images with a smaller learning rate and a longer training time. The Swin Transformer has been pre-trained on the public dataset ImageNet-1K. At the beginning of training, the pre-trained weights are loaded to backpropagate the model's training weights. The model can better adapt to downstream tasks on small datasets by using this transfer learning method. The third stage trains the model's predicted non-modal mask results on Mask R-CNN.

[0070] In this embodiment, the input size of the visible mask is 224x224. Due to limitations in physical memory and learning adjustment, the number of iterations is 50,000, and the initial learning rate is 5e. -4 The batch size is 32, and an SGD (Stochastic Gradient Descent) optimizer with a momentum of 0.9 and a weight decay of 0.0001 is used to optimize the model's backpropagation. A learning rate multiplier of 0.1 is used to adjust the learning rate for different layer parameters. Predictions are made on the validation set every 2000 iterations, and the loss is evaluated.

[0071] Step 5: Test the completion capability and occlusion order prediction capability of the non-modal instance segmentation model;

[0072] After training, the non-modal instance segmentation model is tested using a test set to evaluate its non-modal completion and order prediction capabilities. The model is evaluated using pairwise occlusion order accuracy and the intersection-over-union (IoU) ratio between the completed mask and the ground truth. The completed non-modal mask is visualized to determine its shape and regularity. All non-modal completion results are then used as pseudo-non-modal annotation labels and fed into Mask R-CNN for training to obtain corresponding object segmentation results. The object segmentation results are visualized, and the accuracy and precision of the mask segmented by the non-modal object segmentation model are evaluated using evaluation metrics from the COCO dataset. This demonstrates that our proposed weakly supervised learning approach is feasible and comparable to supervised learning methods.

[0073] In this embodiment, the non-modal instance segmentation model (AC-BET) of the present invention is trained on the three backbone networks of the encoders swin-s, swin-b, and swin-T, respectively, and the training performance is included in the evaluation. The thresholds for sequential inference and non-modal masks are set to 0.2 and 0.2, respectively, when evaluating the dataset on PCNet and ASBUNet. UNet and a UNet with partial convolutions are used as the segmentation network for training. Our proposed network model trains an encoder-decoder segmentation network with a Swin transformer structure. Similar to the two models above, we use the visible portion mask as the model input. The output results are shown in Table 1.

[0074] Table 1. Segmentation results of different models and the same segmentation network.

[0075] method Segmentation network All crosses and comparisons Pixel accuracy Invisible parts are all intersected and compared PCNet Unet 90.62 91.36 - ASBUNet partial convolution Unet 92.47 94.19 38.99 AC-BET Swin-Small-Unet 92.76 94.06 35.91 AC-BET Swin-base-Unet 93.56 94.89 43.37 AC-BET Swin-Tiny-Unet 94.13 97.83 57.79

[0076] Among the three different feature extraction networks, Swin-Tiny-Unet demonstrated the best performance. Compared to the other two networks, it improved mIOU accuracy by approximately 1.4% and 0.6%, respectively, while inv-mIOU accuracy improved by approximately 21.8% and 14.4%. Furthermore, Swin-Tiny has significantly fewer parameters than the other two, resulting in faster computation and inference speeds. Compared to PCNet and ASBUNet, its average intersection-over-union ratio (AUI) was also 3.5% and 1.7% higher, respectively, demonstrating better segmentation performance and providing more convincing segmentation results. This embodiment... Figure 7 The image shows a comparison of the segmentation model of the present invention on a non-modal mask.

[0077] The nonmodal instance segmentation model of the present invention provides a solution for nonmodal mask completion. When faced with similar shapes and colors and moderate or severe occlusion, it can complete a more reasonable shape of the occluded area than other methods.

[0078] Non-modal instance segmentation requires detecting instances from a given image and predicting the instance's mask, which contains the complete shape of the object under occlusion perception. This embodiment applies the network's predictive capabilities to tomato images to generate pseudo-non-modal labels for tomatoes, eliminating the need for manual annotation of tomato instances in complex scenes. This embodiment uses these non-modal pseudo-labels for Mask-RCNN training and evaluates them on a tomato test set, producing relatively reasonable shape estimation and instance segmentation results. In this embodiment, the classes on MRCNN are set to two categories: tomatoes as foreground and the rest as background. Table 2 shows the experimental results;

[0079] Table 2. Segmentation results of pseudo-label training for different models.

[0080]

[0081]

[0082] On the mask AP in the table, the non-modal instance segmentation model of this invention achieves an improvement of nearly 10% and 7.3% compared to PCNet and ASBU, respectively. The AP50 and AP75 metrics are also close to the results of ground truth training, indicating that the multi-layered attention Transformer is helpful in acquiring image feature information, and the discriminator's judgment of the output mask also improves the subsequent instance segmentation accuracy. Because the Tomato dataset contains many close-up images, small targets are not included in training by default, resulting in mediocre performance on APm. Figure 8 The results of nonmodal instance segmentation trained with different pseudo-labels are shown.

[0083] Finally, this embodiment used the AC-BET model of the present invention to test more results on the Tomato test set, such as... Figure 9 As shown, the model predicts the complete shape of the invisible region with high quality. The jagged edges of the completed region are due to the occluding and occluded tomatoes being too similar in shape, color, and other features, and the visible part differing too much from the occluder, indicating severe occlusion. Analysis reveals that the invisible part contains too little content. During training on a large amount of data, the model tends to complete pixels, but it also learns the ability to segment the occluder and occluded object at confusing boundaries. Leaves and branches are a type of irregular instance; when combined with tomatoes of the same type that have high similarity to the target, they act as occluders, resulting in severe occlusion of the target. This leads to extremely irregular masks for both the invisible and visible parts, making prediction more difficult.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for completing non-visible regions of an image based on non-modal instance segmentation, characterized in that: A nonmodal instance segmentation model based on weakly supervised learning using SwinTransformer is established. A non-visible region completion network is trained on the improved SwinTransformer Unet segmentation network, thus eliminating the need for manual annotation of the occluded region mask. At the same time, the occlusion boundary estimation from ASBUNet is introduced, the prediction weights in the occluded and visible regions are redesigned, and the idea of ​​adversarial generative learning is introduced. The predicted non-visible region mask is fed into the discriminator, and an adversarial generative discriminator loss function is added to improve the mask shape output by the discriminator. Includes the following steps: Step 1: Obtain the RGB image dataset and perform image preprocessing; Step 2: Label the images in the image dataset to construct the training set and validation set; Step 3: Construct a non-modal instance segmentation model based on weakly supervised learning using the Swing Transformer; the non-modal instance segmentation model includes a 4×4 convolutional kernel, a linear layer, an encoder, a decoder, and a discriminator; The nonmodal instance segmentation model uses the occlusion boundaries and visible portion masks of the image as input. First, a 4×4 convolution kernel is used to convolve the image, resulting in a tensor with 96 channels. Then, a linear layer is applied to the model for dimensionality transformation. After the linear layer, the image becomes a series of segmented features, which are further input into the encoder. Both the encoder and decoder employ a Swin Transformer Block based on an improved Swin Transformer Unet symmetric architecture. The improved Swin Transformer Unet serves as the backbone of the entire nonmodal instance segmentation model. The encoder collects shallow features through four sampling layers. These shallow features are then fused with deep features collected by the decoder through jump connections. The decoder incorporates joint weights for predicting visible and invisible region masks to adjust the model's loss function. An adversarial generative learning approach is introduced in the discriminator to judge and improve the mask morphology of the invisible region masks predicted by the decoder. Step 4: Train the non-modal instance segmentation model through three stages to determine the optimal model parameters; Step 5: Test the non-modal instance segmentation model's completion capability and occlusion order prediction capability.

2. The image non-visible region completion method based on non-modal instance segmentation according to claim 1, characterized in that: The specific method for step 1 is as follows: Acquire RGB image data; based on the acquired images, use cosine similarity calculation to remove images with a repetition greater than a set threshold, calculate the variance of the Laplacian transform of the images to remove images with a blur greater than a set threshold, and then filter and delete images with cluttered backgrounds and poor shooting angles.

3. The image non-visible region completion method based on non-modal instance segmentation according to claim 1, characterized in that: The specific method for step 2 is as follows: The Labelme tool is used to annotate image semantic labels. Combined with scene structure, both visible and invisible parts of the image are semantically labeled. For unoccluded objects, a length mask annotation method is used. For occluded objects, a hierarchical annotation method is used. The first layer annotates the visible part, the second layer annotates the invisible part, and the third layer annotates the joint area of ​​the visible and invisible parts to obtain nonmodal ground truth labels. The training and validation sets were constructed at a 9:1 ratio. The training set was then augmented by horizontal flipping, translation, and random cropping.

4. The image non-visible region completion method based on non-modal instance segmentation according to claim 1, characterized in that: The non-modal instance segmentation model uses a boundary estimation method to input the contact boundary between the occluder and the occluded object into the model in the form of a mask. The model will output an uncertain boundary map of the other side of the occluded part. The content in the two boundary parts will be filled with pixels to obtain a reasonable shape.

5. The image non-visible region completion method based on non-modal instance segmentation according to claim 4, characterized in that: The non-modal instance segmentation model employs a partial completion algorithm to implement a weakly supervised learning process for non-modal mask completion; Based on the idea of ​​allowing the model to autonomously learn and recover invisible masks, during the training phase, the images are processed to randomly sample occlusion instances and occluded instances, and only their visible masks are used for model training. Specifically, two images with an occlusion relationship are randomly selected, and the image containing instance A is denoted as... Another image containing instance B is denoted as And the visible masks of the two instances marked in the image are denoted as follows: , ; The first case is that instance A is occluded by instance B. The mask image used to cover the occlusion is denoted as... / Then combine the image of instance A Masked Occluded image Together, they are fed into an hourglass-structured segmentation network for segmentation, and the recovered target is the visible mask of instance A. Simultaneously, to prevent the model from over-filling pixels of other instances, a second case is where instance B is occluded by instance A; the occluded mask image is denoted as... / Joint input The restored target is also the visible mask of instance A, so that the completion network in the non-modal instance segmentation model can learn to judge whether the target pixels should be completed, thus achieving the purpose of regularization learning process; A discriminator was then introduced. This improves the quality of the restored mask.

6. The image non-visible region completion method based on non-modal instance segmentation according to claim 5, characterized in that: In the non-modal instance segmentation model, the ordering of pairwise occlusion relationships is represented by pairing instances connected by two visible masks to form adjacent instance pairs. Pairwise occlusion order matrices are used to represent object relationships, and a directed graph is drawn to visualize these relationships. The specific method is as follows: The occlusion order directed graph is represented as follows: Let Y be the set of all objects in the image, with a size of N, and T be an N×N matrix. The occlusion relationship between two adjacent pairs of objects i and j is expressed as follows: ； In the above formula, and The non-modal mask representing the completed representation of objects i and j. and Represents the visibility mask of objects i and j. and These represent the pixel value increments of the non-modal mask after the two visible masks are completed; If the increments generated by the mask completion network for objects i and j are equal and both are 0, it means that they do not obstruct each other and are in the same layer. If the value of is 0, and the increment of the former is less than that of the latter, it means that object i occludes object j. If the value of is 1, and the increment of the former is not less than the increment of the latter, it means that object j occludes object i. The value is 1. Based on the occlusion order generated in pairs, the occlusion order between objects in the entire scene is gradually inferred, providing clear prior information support and interpretability for nonmodal mask completion.

7. The image non-visible region completion method based on non-modal instance segmentation according to claim 6, characterized in that: The specific method for step 5 is as follows: After training, the non-modal instance segmentation model is tested using a test set to evaluate its non-modal completion and order prediction capabilities. The non-modal object segmentation model is evaluated using pairwise occlusion order accuracy and the intersection-over-union (IoU) ratio between the completed mask and the ground truth. The completed non-modal mask is visualized to determine its shape and regularity. Subsequently, all non-modal completion results are used as pseudo-non-modal annotation labels and fed into Mask R-CNN for training to obtain corresponding object segmentation results. The object segmentation results are visualized, and the accuracy and precision of the mask segmented by the non-modal object segmentation model are evaluated using evaluation metrics from the COCO dataset.

Images