Interactive modal-free instance segmentation method based on click-aware shape priors

Abstract

The application provides an interactive mode-free instance segmentation method based on click-aware shape prior, which comprises the following steps: performing data enhancement processing on an image dataset to obtain a training set instance; constructing a shape prior codebook by using the training set instance; constructing a triple loss by using a prediction mask, a user click in a current round, an encoder and the shape prior codebook, training the encoder by using the triple loss to obtain a trained encoder; obtaining a feature map of an input image by using an image block in the training set instance; obtaining a shape prior feature map by using the prediction mask, the user click in the current round, the trained encoder, a real mask and the shape prior codebook; aligning the feature map of the input image and the shape prior feature map to obtain an aligned feature map; and obtaining a segmentation result by using the aligned feature map. The deformable attention mechanism is used to adaptively shift the sampling points, so as to make up for the mismatching problem of the retrieved aligned shape prior feature.

Description

Technical Field

[0001] This invention relates to the field of instance segmentation technology, and in particular to an interactive, modal-free instance segmentation method based on click-aware shape prior. Background Technology

[0002] Instance segmentation, a fundamental task focused on segmenting masks for each object instance, has shown broad application prospects in fields such as autonomous driving, robotics, and augmented reality. Building upon this foundation, due to the complex occlusion relationships between objects in the real world, modal-free instance segmentation has emerged. It aims to predict the complete mask of an object, covering both visible and occluded regions. This extended task has significant practical value in areas such as virtual object placement, safe navigation in autonomous driving, and 3D scene reconstruction and understanding.

[0003] However, modal nonmodal instance segmentation faces significant technical challenges, primarily due to the lack of direct visual information supporting the occlusion regions of object instances, making prediction difficult. To address this challenge, existing methods generally focus on constructing various auxiliary mechanisms to facilitate the prediction of occluded regions. However, prior knowledge is not always effectively applied to unseen instances during the testing phase.

[0004] Therefore, how to accurately retrieve and align shape priors for unseen instances remains an open problem that urgently needs to be solved. Summary of the Invention

[0005] In view of the above, the main objective of this invention is to propose an interactive, modal-free instance segmentation method based on click-aware shape priors to solve the aforementioned technical problems.

[0006] This invention proposes an interactive, modal-free instance segmentation method based on click-aware shape priors, the method comprising the following steps:

[0007] Step 1: Obtain the image dataset and perform data augmentation on the image dataset to obtain training set instances;

[0008] Step 2: Construct a shape prior codebook using training set instances;

[0009] Step 3: Use the prediction mask from the previous round, the user clicks in the current round, and the encoder to obtain the query vector. Construct a triplet loss using the query vector and the shape prior codebook. Use the triplet loss to train the encoder to obtain a trained encoder.

[0010] Step 4: Input the image patches from the training set instances into the backbone network and perform feature extraction to obtain the feature map of the input image;

[0011] Step 5: Use the prediction mask from the previous round, the user clicks in this round, and the trained encoder to obtain a new query vector. Input the real mask from the training set instance into the trained encoder to obtain a new key vector. Use the new query vector and the new key vector to search in the shape prior codebook to obtain the shape prior. Flatten the shape prior and then upsample it to obtain the shape prior feature map.

[0012] Step 6: Align the feature map and shape prior feature map of the input image using a deformable attention mechanism to obtain the aligned feature map;

[0013] Step 7: Concatenate and compress the aligned feature map and the concatenated feature map to obtain the IMCA feature map; input the IMCA feature map into the segmentation head to obtain the prediction mask for this round, and use the prediction mask for this round as the segmentation result.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] 1. This invention learns the shape prior for click perception through triplet loss. By reducing the distance between the query vector and the key vectors in the positive and negative sample sets, the shape prior has a higher intersection-over-union (IoU) ratio and a true mask, so as to retrieve a more matching shape prior.

[0016] 2. This invention aligns the retrieved shape priors with deformable attention and adaptively shifts the sampling points using a deformable attention mechanism to compensate for the mismatch problem of the retrieved aligned shape prior features. Attached Figure Description

[0017] Figure 1 This is a flowchart of the interactive, modal-free instance segmentation method based on click-aware shape prior proposed in this invention;

[0018] Figure 2 This is a schematic diagram of the formal training process of the interactive modal-free instance segmentation method based on click-aware shape prior proposed in this invention.

[0019] Figure 3 This is a curve comparing the impact of different click counts on the average intersection-union ratio (IUU) of the occluded area for the present invention and three representative models on the KINS dataset.

[0020] Figure 4 This is a schematic diagram of the pre-training process of the interactive modal-free instance segmentation method based on click-aware shape prior proposed in this invention. Detailed Implementation

[0021] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0022] These and other aspects of the embodiments of the present invention will become clear from the following description and accompanying drawings. In these descriptions and drawings, some specific embodiments of the present invention are specifically disclosed to illustrate some ways of implementing the principles of the embodiments of the present invention; however, it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0023] Please see Figure 1 This invention proposes an interactive, modal-free instance segmentation method based on click-aware shape priors, which includes the following steps:

[0024] Step 1: Obtain the image dataset and perform data augmentation on the image dataset to obtain training set instances.

[0025] Furthermore, the specific details of data augmentation in this step are as follows:

[0026] Images in the image dataset smaller than 448x448 are automatically padded with edges to ensure they are large enough. Then, a 448x448 region is randomly cropped from the image, ensuring the target instance is within the region, thus obtaining the processed image.

[0027] The processed image and the real mask are subjected to data augmentation operations including random scaling, random flipping, random 90-degree rotation, and fine affine transformation. In addition, the processed image is subjected to data augmentation operations including random brightness and contrast, and random RGB color shift.

[0028] Specifically, in this step, data augmentation operations include two categories: geometric and spatial transformations, and color and photometric transformations.

[0029] On one hand, geometric and spatial transformations are applied simultaneously to the input image and its corresponding mask to ensure the consistency of their spatial positions after transformation. Specifically, the geometric and spatial transformations include: uniform random scaling, which adjusts the size of the image and mask randomly between 0.75 and 1.25 times their original size; random horizontal flipping with a 50% probability; random 90-degree rotations (0, 90, 180, or 270 degrees); and a fine affine transformation, with a 75% probability, applying a set of random translations within ±3% of the original size and random rotations within ±3 degrees to the image and mask.

[0030] On the other hand, color and photometric transformations only change the pixel values ​​of the image without affecting the mask, and each transformation is performed independently with a 75% probability. Specifically, color and photometric transformations include: random brightness and contrast adjustments within ±25% of the image brightness and within -15% to +40% of the contrast; and random RGB color offsets applied to the pixel values ​​of the R, G, and B color channels of the image, each with a random offset within the range of -10 to +10.

[0031] Step 2: Construct a shape prior codebook using training set instances;

[0032] In step 2, a shape prior codebook is constructed using training set instances. The specific process is as follows:

[0033] Obtain the ground mask of all training set instances for each category to obtain the ground mask set;

[0034] For each category in the set of true masks, construct a collection of... The codebook has one slot, so as to obtain the initial codebook structure;

[0035] In each slot of the initialization codebook structure, the real mask of the current training set instance is stored to obtain the codebook filled with key-value pairs;

[0036] When the number of categories in the codebook for filling key-value pairs is less than When the number of categories exceeds a certain threshold, spatial augmentation is performed on the existing samples to obtain a shape prior codebook; when the number of categories exceeds a certain threshold... At that time, K-Means clustering is performed on the real mask embeddings of the training set instances to obtain the shape prior codebook.

[0037] Specifically, in this step, to ensure that the codebook for each category reaches... Given the preset capacity, this invention employs an adaptive filling strategy.

[0038] For categories with fewer than 1024 training set instances, the samples are augmented through a spatial data augmentation process. The spatial augmentation may include transformations such as random horizontal flipping, random rotation at small angles within ±15 degrees, and random scaling between 0.8 and 1.2 times. The existing real masks are processed cyclically to generate new masks until all 1024 slots are filled.

[0039] For categories with more than 1024 instances in the training set, the K-Means clustering method is used to select the most representative shapes. The process first uses a mask encoder to convert each real modal mask into a high-dimensional feature vector (i.e., mask embedding). Then, using all mask embeddings as data points, the K-Means clustering algorithm is executed with the target cluster number set to 1024. Finally, the 1024 cluster center vectors obtained after the algorithm converges are used as the shape priors for that category and stored in the 1024 slots of the codebook.

[0040] Step 3: Use the prediction mask from the previous round, the user clicks in the current round, and the encoder to obtain the query vector. Construct a triplet loss using the query vector and the shape prior codebook. Use the triplet loss to train the encoder to obtain a trained encoder.

[0041] In step 3, the query vector is obtained using the prediction mask from the previous round, the user clicks in the current round, and the encoder. A triplet loss is constructed using the query vector and the shape prior codebook. The encoder is then trained using the triplet loss to obtain a trained encoder. The specific process is as follows:

[0042] The previous round prediction mask and the current round user clicks are concatenated to obtain a concatenated tensor. The concatenated tensor is input into the encoder to obtain a concatenated feature map. The concatenated feature map is flattened to obtain a query vector.

[0043] The key vector is obtained by searching the shape prior codebook using the query vector;

[0044] Calculate the intersection-union ratio (IoU) of the ground truth mask of the current training set instance with the ground truth masks of all slots;

[0045] The slots are sorted using the intersection-union ratio and divided into a positive sample set and a negative sample set;

[0046] The triplet loss is constructed by query vector, key vector, positive sample set, and negative sample set;

[0047] The encoder is trained using triplet loss to obtain a trained encoder;

[0048] In the process of concatenating the previous round's prediction mask and the current round's user clicks to obtain a concatenated tensor, inputting the concatenated tensor into the encoder to obtain a concatenated feature map, and flattening the concatenated feature map to obtain the query vector, the following relationship exists:

[0049] ;

[0050] in, Represents the query vector. This indicates that the process is handled by querying the encoder. This represents the prediction mask from the previous round. This indicates the number of clicks made by users in this round. Indicates a splicing operation;

[0051] In the process of constructing the triplet loss using the query vector, key vector, positive sample set, and negative sample set, the following relationship exists:

[0052] ;

[0053] in, Represents the triplet loss function. This indicates that the process is performed using the ReLU activation function. The index representing the slot number. Indicates the first The key vector in each slot This represents the margin hyperparameter. Represents the set of positive samples. Represents the set of negative samples. This indicates the calculation of Euclidean distance.

[0054] Specifically, in this step, the triplet loss learning mechanism aims to optimize the parameters of the encoder (including the query encoder and the key encoder) to construct a feature embedding space that is more semantically meaningful.

[0055] For each target instance during training, its own real mask is used as a benchmark. The shape similarity between them is quantified by calculating the intersection-union ratio (IoU) between the mask and the real masks stored in all 1024 slots of the codebook.

[0056] Based on the IoU value, positive and negative sample sets are constructed for supervised learning: the top N slot indices with the highest IoU value are defined as the positive sample set S+, representing the "correct" matches that are most similar to the shape of the current instance; at the same time, the top N indices of slots with IoU values ​​less than 0.8 (excluding items already in S+) are defined as the negative sample set S-, representing those matches that may be somewhat misleading to the target shape but should be judged as "incorrect".

[0057] Subsequently, this invention employs the aforementioned triplet loss function to apply a relative distance constraint. The core idea of ​​this loss function is not simply to bring all positive samples closer and push all negative samples further away, but rather to focus on the most challenging samples: that is, to find the positive sample furthest from the query vector in the feature space. and distance query vector Recent negative samples .

[0058] The goal of the loss function is to drive the model so that the distance between the nearest negative sample and the query vector is at least larger than the distance between the furthest positive sample and the query vector by a preset margin hyperparameter. The aforementioned The ReLU activation function ensures that the loss value is zero once the distance constraint is met, thus stabilizing the training process.

[0059] By backpropagating this loss, the query encoder and key encoder are jointly optimized, ultimately enabling the Euclidean distance in the feature space to directly reflect the IoU similarity between shape masks, thus allowing for more accurate retrieval of highly relevant shape priors during the inference phase.

[0060] Step 4: Input the image patches from the training set instances into the backbone network and perform feature extraction to obtain the feature map of the input image.

[0061] Specifically, in this step, the backbone network uses a Vision Transformer (ViT-B) model to extract deep semantic features of image patches. This process is as follows:

[0062] First, the input 448x448 image block is divided into a series of 16x16 pixel image patches in a non-overlapping manner, resulting in 28x28=784 image patches. A learnable positional code is then embedded and superimposed on each image patch.

[0063] Subsequently, each two-dimensional image patch is flattened into a one-dimensional vector and mapped to a fixed-length feature vector through a linear projection layer, i.e., image patch embedding.

[0064] To preserve the original spatial location information, a learnable location code is embedded and superimposed on each image patch, thereby transforming the two-dimensional image data into a one-dimensional vector sequence containing 784 vectors with temporal and spatial information.

[0065] Then, the one-dimensional vector sequence is input into the core component of the ViT-B model, namely a Transformer Encoder composed of multiple stacked layers.

[0066] At each layer of the encoder, each vector in the sequence is computed with its association weight with all other vectors in the sequence through a multi-head self-attention mechanism, thereby capturing global contextual information and long-distance dependencies of the image.

[0067] Finally, the vector sequence output by the transformer encoder is reshaped and restored to a two-dimensional feature map with spatial dimensions. Its spatial size H×W corresponds to the way the image patch is divided, i.e., 28x28, while the number of channels C corresponds to the feature embedding dimension of the transformer model.

[0068] Step 5: Use the prediction mask from the previous round, the user clicks in this round, and the trained encoder to obtain a new query vector. Input the real mask from the training set instance into the trained encoder to obtain a new key vector. Use the new query vector and the new key vector to search in the shape prior codebook to obtain the shape prior. Flatten the shape prior and then upsample it to obtain the shape prior feature map.

[0069] Furthermore, in this step, the process of obtaining the new query vector is similar to step 3. The prediction mask from the previous round is concatenated with the user clicks in the current round to obtain a concatenated tensor. The concatenated tensor is then input into the trained encoder to obtain a new concatenated feature map. The new concatenated feature map is then flattened to obtain the new query vector.

[0070] Furthermore, in this step, the trained encoder includes a trained query encoder and a trained key encoder. The predicted mask from the previous round and the user clicks from the current round are concatenated and then input into the trained query encoder for encoding to obtain an encoded feature map. The encoded feature map is then flattened to obtain a new query vector. The real mask is input into the trained key encoder to obtain a new vector.

[0071] Specifically, in this step, the user's interaction information (user clicks in this round) will be processed. ) and the previous round's predicted segmentation result (previous round's predicted mask) The system transforms the feature representation into one that can be efficiently retrieved within a shape-prior codebook, and executes an asymmetric encoding process. The current user click is automatically generated within a simulated interactive protocol that can run for up to 24 rounds. In the t-th round of interaction, the system determines missegmented regions by comparing the predicted mask from the previous round with the true mask.

[0072] Furthermore, the total area of ​​all missed detection regions (i.e., false negatives) and the total area of ​​all false positive regions (i.e., false positives) are calculated. If the total area of ​​missed detection regions is greater than the total area of ​​false positive regions, a positive click is generated; otherwise, a negative click is generated. This click is precisely placed at the geometric center of the largest connected missegmented region.

[0073] Ultimately, the user's click information in this round is encoded into a click mask for two channels: if it is a positive click, it is marked at the corresponding position in the first channel; if it is a negative click, it is marked at the corresponding position in the second channel.

[0074] Subsequently, the mask predicted by the previous model is concatenated with the click masks of these two channels to form a multi-channel 448x448 tensor, which is then fed into the trained query encoder.

[0075] The trained query encoder compresses the input tensor into a new 288-dimensional query vector through six layers of convolution operations including downsampling and ReLU activation.

[0076] On the other hand, the trained key encoder uses the exact same network architecture as the trained query encoder to independently encode the real modalityless mask into a new key vector, so as to ensure that the new query vector and the new key vector are in the same comparable feature space.

[0077] Step 6: Align the feature map and shape prior feature map of the input image using a deformable attention mechanism to obtain the aligned feature map;

[0078] In step 6, the feature map and shape prior feature map of the input image are aligned using a deformable attention mechanism to obtain an aligned feature map. The specific process is as follows:

[0079] The concatenated feature map and the feature map of the input image are concatenated to obtain the intermediate feature map;

[0080] The intermediate feature map is concatenated with the shape prior feature map and processed by a deformable attention function to obtain the sampling offset and attention weights.

[0081] By utilizing shape prior feature maps, sampling offsets, attention weights, and reference points for query features, a deformable attention mechanism is used to process the data to obtain aligned feature maps.

[0082] In the process of concatenating the intermediate feature map with the shape prior feature map and processing it through a deformable attention function to obtain the sampling offset and attention weights, the following relationship exists:

[0083] ;

[0084] in, Indicates the sampling offset. Indicates attention weights. This indicates that the process is handled using a deformable attention function. This represents the intermediate feature map. Represents the shape prior feature map;

[0085] In the process of obtaining the aligned feature map by utilizing the shape prior feature map, sampling offset, attention weights, and reference points of the query feature through a deformable attention mechanism, the following relationship exists:

[0086] ;

[0087] in, This represents the aligned feature map. Indicates the number of attention heads. This indicates the number of sampling points for each attention head. and All indicate the first The learnable weight matrix in each attention head. Indicates the first The first attention Attention weights for each sampling point Indicates the first The first attention Sampling offset of each sampling point The reference point that represents the query feature.

[0088] Specifically, in this step, to address the inevitable mismatch between the general shape prior retrieved from the shape prior codebook and a specific instance in the current image, a shape prior alignment mechanism based on deformable attention is employed.

[0089] The core idea of ​​this mechanism is that it does not directly use the retrieved shape prior feature map. Instead, it dynamically and non-rigidly distorts and reassembles the features based on the query context composed of current image features and user interaction information, thereby generating an aligned feature map that is precisely aligned with the target instance. .

[0090] Specifically, the process first fuses information from multiple sources through a splicing operation:

[0091] The feature map containing the previous mask and user click information is concatenated with the feature map containing the current image's visual content to generate a comprehensive intermediate feature map of the query context. ;

[0092] Subsequently, this feature map is compared with the shape prior feature map retrieved from the shape prior codebook. The components are then concatenated again and fed into an offset prediction network (i.e., a deformable attention function). In this context, the network learns and predicts two sets of key parameters by comparing query context and shape prior: one set is the sampling offset. One set provides the direction and distance for spatial movement in subsequent sampling operations; the other set is the attention weights. It assigns different importance to the features of each sampling point;

[0093] Finally, in the shape prior feature map Alignment and reorganization of the execution features.

[0094] Unlike traditional attention mechanisms that sample at fixed grid points, deformable attention mechanisms utilize the offset predicted in the previous step. For a series of preset reference points on the feature map Through formula The new sampling point location is dynamically calculated.

[0095] These new sampling point locations are therefore adaptively adjusted to the regions most relevant to the query context features. Then, at these new locations... Perform feature sampling and utilize the predicted attention weights. The sampled features are then weighted and summed. This process is performed in... The process is executed in parallel across several different attention heads, each learning to focus on different alignment patterns, ultimately resulting in a learnable weight matrix. and The aggregation process is performed to generate the finally aligned feature map. .

[0096] In this way, a general shape prior can be adaptively deformed to accurately match the specific pose, shape, and occlusion of the current target instance.

[0097] Step 7: Concatenate and compress the aligned feature map and the concatenated feature map to obtain the IMCA feature map; input the IMCA feature map into the segmentation head to obtain the prediction mask for this round, and use the prediction mask for this round as the segmentation result.

[0098] Please see Figure 4 It should be noted that the pre-training of the present invention includes two steps: constructing a shape prior codebook using training set instances and training the query encoder and key encoder using triplet loss, which correspond to steps 2 and 3 above, respectively.

[0099] Please see Figure 2 The formal training of this invention includes four steps: input image feature extraction, shape prior retrieval, shape prior alignment, and complete mask prediction, which correspond to steps 4, 5, 6, and 7 above, respectively.

[0100] It should be noted that the method of this invention is based on the prediction mask of the previous round, while for the initial round, an all-zero mask is initialized as the "prediction mask" for the initial round.

[0101] It should be noted that, in the formal training process of this invention, the decoder implementation includes, but is not limited to, inverse flattening and upsampling. Specifically, this invention involves inverse flattening the shape prior obtained from the shape prior codebook and then upsampling it to obtain a shape prior feature map, which serves to recover the spatial structure.

[0102] Furthermore, during the formal training phase, the overall objective for each input image can be obtained by averaging the segmentation loss over T interactive iterations. The expression for the segmentation loss is:

[0103] ;

[0104] in, This represents the segmentation loss (normalized focal loss). Indicates the total loss; This represents the total number of interaction iterations, with a maximum of 24. Indicates the number of rounds of interactive iteration; Indicates the first The mask obtained from round prediction; Indicates the true mask;

[0105] The learning mechanism of segmentation loss is to measure the difference between the predicted mask XX in round t and the real complete mask XX, and guide the model to update more closely to the real segmentation in this round of interaction.

[0106] Furthermore, preferred embodiments of the present invention were developed and verified in a high-performance computing environment. Specifically, the present invention was tested on three authoritative public datasets: COCOA cls, KINS, and D2SA, and implemented using the Python 3.8 programming language and the PyTorch 1.13.1 deep learning platform.

[0107] In terms of hardware configuration, it uses an Intel Xeon Gold 6326 CPU @ 2.90GHz as the central processing unit and is equipped with four NVIDIA RTX A5000 graphics processors for accelerated computing. The operating environment of this embodiment is Ubuntu 20.04.6 LTS 64-bit operating system.

[0108] During model training, all configurations were kept consistent to ensure fairness and reproducibility of results. The ViT-B model was used as the backbone network, and the size of all input images was uniformly adjusted to 448x448 pixels. The training process employed a normalized focal loss function for optimization, using the Adam optimizer with the following hyperparameters: Set to 0.9, Set it to 0.999. The initial learning rate of the model is set to... The algorithm is dynamically adjusted following a multi-step decay strategy. To ensure consistency of experimental results, all experiments use 42 as the global random seed.

[0109] To comprehensively and specifically evaluate the performance of this invention on interactive modal-free instance segmentation tasks, this invention employs two types of core evaluation metrics.

[0110] Firstly, to focus on examining the model's efficiency in interactively correcting occlusion areas, this invention introduces the occlusion area click count metric (Number of Clicks for Occlusion). ↓), the statistical model of this indicator reaches the preset average intersection-union ratio (... The number of interaction clicks required to achieve the target. In this embodiment, three target accuracy thresholds are set: 80%, 85%, and 90%, respectively corresponding to... , and Furthermore, to focus on challenging scenarios, this evaluation was conducted specifically on test samples with an occlusion rate exceeding 10%.

[0111] Table 1 shows the comparison results between the model of this invention and existing interactive segmentation models. It can be seen that all models require a large number of clicks to achieve high IoU. For example, the baseline of the MFP model is... At that time it was 1.93, while The performance then increases to 4.76. Combining MFP with C2F-Seg significantly outperforms interactive baselines; for example, on KINS, NoC is reduced by 0.11 compared to MFP, demonstrating the advantage of incorporating shape priors. Furthermore, the model of this invention achieves substantial improvements on all baselines, for example, on KINS, it is reduced by 0.34 compared to MFP+C2F. This fully demonstrates the effectiveness of simultaneously utilizing user clicks and shape priors.

[0112] Table 1: Comparison results between the model of this invention and existing interactive segmentation models

[0113]

[0114] To measure the accuracy of the model's segmentation results, this invention uses the mean Intersection over Union (IoU) ratio. As another core metric, this metric is further subdivided into: the average intersection-union ratio (AUC) of the complete regions used to evaluate the accuracy of complete instance segmentation. ↑), and the average intersection-union ratio (IUU) of the occluded region, which is specifically used to evaluate the segmentation accuracy of the occluded part. (↑).

[0115] Table 2 shows the comparison results between the model of this invention and existing modal-free segmentation models. Since the mIoU metric does not involve clicks, the number of clicks was fixed at 1, 3, and 5 during evaluation. The results show that the MFP+C2F model already outperforms traditional modal-free segmentation methods at the baseline, and the model of this invention further surpasses this baseline at different click counts. On the KINS dataset, it achieves a +0.49% improvement over MFP+C2F with only 1 click. and +1.21% It increased by +2.15% on each of the 5 clicks. and +4.37% This demonstrates that the model of the present invention can achieve improvements with a small number of clicks, and its advantages are more significant in scenarios with multiple clicks.

[0116] Table 2 Comparison results of the model of this invention and existing modeless segmentation models

[0117]

[0118] Please see Figure 3 , Figure 3 The figure shows the average intersection-union ratio (IUU) of the occluded region for the model of this invention and the MFP+C2F model, the MFP model, and the C2F model on the KINS dataset for different numbers of clicks (1 to 24). A comparative analysis chart showing the impact of indicator results.

[0119] Figure 3 The curves, from top to bottom, represent: the model of this invention, the MFP+C2F model, the MFP model, and the C2F model. (From...) Figure 3 It can be seen that when the number of clicks exceeds approximately 15, the performance of different algorithm models tends to converge.

[0120] Overall, the model of this invention achieves a higher average crossover / union ratio with the same number of clicks. For example... Figure 3 As shown, the AUC (area under the curve) score is calculated to evaluate overall performance, and the model of this invention achieves the optimal AUC score.

[0121] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0123] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. An interactive, modal-free instance segmentation method based on click-aware shape prior, characterized in that, The method includes the following steps: Step 1: Obtain the image dataset and perform data augmentation on the image dataset to obtain training set instances; Step 2: Construct a shape prior codebook using training set instances; Step 3: Use the prediction mask from the previous round, the user clicks in the current round, and the encoder to obtain the query vector. Construct a triplet loss using the query vector and the shape prior codebook. Use the triplet loss to train the encoder to obtain a trained encoder. Step 4: Input the image patches from the training set instances into the backbone network and perform feature extraction to obtain the feature map of the input image; Step 5: Use the prediction mask from the previous round, the user clicks in this round, and the trained encoder to obtain a new query vector. Input the real mask from the training set instance into the trained encoder to obtain a new key vector. Use the new query vector and the new key vector to search in the shape prior codebook to obtain the shape prior. Flatten the shape prior and then upsample it to obtain the shape prior feature map. Step 6: Align the feature map and shape prior feature map of the input image using a deformable attention mechanism to obtain the aligned feature map. The specific process is as follows: The concatenated feature map and the feature map of the input image are concatenated to obtain the intermediate feature map; The intermediate feature map is concatenated with the shape prior feature map and processed by a deformable attention function to obtain the sampling offset and attention weights, which have the following relationship: ； in, Indicates the sampling offset. Indicates attention weights. This indicates that the process is handled using a deformable attention function. This represents the intermediate feature map. Represents the shape prior feature map; Using shape prior feature maps, sampling offsets, attention weights, and reference points for query features, a deformable attention mechanism is employed to obtain aligned feature maps, corresponding to the following relationship: ； in, This represents the aligned feature map. Indicates the number of attention heads. This indicates the number of sampling points for each attention head. and All indicate the first The learnable weight matrix in each attention head. Indicates the first The first attention Attention weights for each sampling point Indicates the first The first attention Sampling offset of each sampling point Reference points indicating query features; Step 7: Concatenate and compress the aligned feature map and the concatenated feature map to obtain the IMCA feature map; input the IMCA feature map into the segmentation head to obtain the prediction mask for this round, and use the prediction mask for this round as the segmentation result.

2. The interactive modal-free instance segmentation method based on click-aware shape prior as described in claim 1, characterized in that, In step 2, a shape prior codebook is constructed using training set instances, which specifically includes the following sub-steps: Obtain the ground mask of all training set instances for each category to obtain the ground mask set; For each category in the set of true masks, construct a collection of... The codebook has one slot, so as to obtain the initial codebook structure; In each slot of the initialization codebook structure, the real mask of the current training set instance is stored to obtain the codebook filled with key-value pairs; When the number of categories in the codebook for filling key-value pairs is less than When the number of categories exceeds a certain threshold, spatial augmentation is performed on the existing samples to obtain a shape prior codebook; when the number of categories exceeds a certain threshold... At that time, K-Means clustering is performed on the real mask embeddings of the training set instances to obtain the shape prior codebook.

3. The interactive modal-free instance segmentation method based on click-aware shape prior as described in claim 2, characterized in that, In step 3, the query vector is obtained using the prediction mask from the previous round, the user clicks in the current round, and the encoder. A triplet loss is constructed using the query vector and the shape prior codebook. The encoder is then trained using the triplet loss to obtain a trained encoder. The specific process is as follows: The previous round prediction mask and the current round user clicks are concatenated to obtain a concatenated tensor. The concatenated tensor is input into the encoder to obtain a concatenated feature map. The concatenated feature map is flattened to obtain a query vector. The key vector is obtained by searching the shape prior codebook using the query vector; Calculate the intersection-union ratio (IoU) of the ground truth mask of the current training set instance with the ground truth masks of all slots; The slots are sorted using the intersection-union ratio and divided into a positive sample set and a negative sample set; The triplet loss is constructed by query vector, key vector, positive sample set, and negative sample set; The encoder is trained using triplet loss to obtain a trained encoder.

4. The interactive modal-free instance segmentation method based on click-aware shape prior as described in claim 3, characterized in that, In the process of concatenating the previous round's prediction mask and the current round's user clicks to obtain a concatenated tensor, inputting the concatenated tensor into the encoder to obtain a concatenated feature map, and flattening the concatenated feature map to obtain the query vector, the following relationship exists: ； in, Represents the query vector. This indicates that the process is handled by querying the encoder. This represents the prediction mask from the previous round. This indicates the number of clicks made by users in this round. This indicates a splicing operation.

5. The interactive modal-free instance segmentation method based on click-aware shape prior according to claim 4, characterized in that, In the process of constructing the triplet loss using the query vector, key vector, positive sample set, and negative sample set, the following relationship exists: ； in, Represents the triplet loss function. This indicates that the process is performed using the ReLU activation function. The index representing the slot number. Indicates the first The key vector in each slot This represents the margin hyperparameter. Represents the set of positive samples. Represents the set of negative samples. This indicates the calculation of Euclidean distance.

Images