Interactive image segmentation method and system based on lightweight adapter fine-tuning

Abstract

The application discloses an interactive image segmentation method and system based on fine tuning of a lightweight adapter, and the method comprises the following steps: acquiring an input image and a click image, and performing image segmentation on the input image to obtain a plurality of non-overlapping image blocks; inputting the click image and the image blocks into a Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image; inputting the input image into a self-adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image; acquiring a historical prediction image and a mask feature, performing down-sampling on the historical prediction image to obtain a historical prediction image thumbnail; and inputting the historical prediction image thumbnail, the mask feature, the multi-scale feature map and the high-quality feature map into a ViT backbone network adapter for image decoding to obtain a prediction image. The application combines a self-adaptive domain feature extraction network with a backbone network to extract image features, and better image segmentation effect and efficiency are obtained.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an interactive image segmentation method and system based on lightweight adapter fine-tuning. Background Technology

[0002] Interactive segmentation aims to segment target objects in images using limited user-interactive information. Due to the addition of extra user-annotated information, interactive segmentation can often achieve higher accuracy than traditional segmentation methods in complex scenes. Existing image segmentation techniques mainly focus on improving the performance of algorithms on specific datasets, thus ignoring the data diversity problem common in real-world crowdsourced annotation tasks. Diverse data can lead to cross-domain performance losses for algorithms, thereby reducing the efficiency of annotation algorithms.

[0003] Therefore, existing technologies still need improvement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an interactive image segmentation method and system based on lightweight adapter fine-tuning, which addresses the above-mentioned deficiencies of the prior art and aims to solve the problem of poor image segmentation performance in the prior art.

[0005] The technical solution adopted by this invention to solve the problem is as follows:

[0006] In a first aspect, embodiments of the present invention provide an interactive image segmentation method based on lightweight adapter fine-tuning, wherein the method includes:

[0007] The input image and the clicked image are acquired, and the input image is segmented to obtain several non-overlapping image blocks;

[0008] The clicked image and the image block are input into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image.

[0009] The input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image;

[0010] Obtain historical prediction images and mask features, downsample the historical prediction images, and obtain thumbnails of the historical prediction images;

[0011] The historical predicted image thumbnail, the mask feature, the multi-scale feature map, and the high-quality feature map are input into the ViT backbone network adapter for image decoding to obtain the predicted image.

[0012] In one implementation, the Plain ViT backbone network includes a first patch embedding module, a second patch embedding module, N Transformer encoder modules, and a feature pyramid module. The step of inputting the clicked image and the image patch into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image includes:

[0013] The clicked image is input into the first Patch embedding module for feature mapping to obtain the clicked image feature sequence;

[0014] The image patch is input into the second Patch embedding module for feature mapping to obtain the image patch feature sequence;

[0015] The click image feature sequence and the image block feature sequence are sequentially input into the N Transformer encoder modules for image encoding to obtain the global feature image of the input image;

[0016] The global feature image is input into the feature pyramid module for image feature fusion to obtain a multi-scale feature map.

[0017] In one implementation, the adaptive domain feature extractor includes a spatial prior module, an injection module, a precision feature extraction module, and a feature compression module. The step of inputting the input image into the adaptive domain feature extractor for feature extraction to obtain a high-quality feature map of the input image includes:

[0018] The input image is standardized;

[0019] The standardized input image is input into the spatial prior module for image detail extraction to obtain the output image of the spatial prior module.

[0020] The spatial prior module output image and the global feature image are input into the injection module for image feature fusion to obtain a feature fusion map.

[0021] The feature fusion map is input into the precision feature extraction module to extract image precision features, thereby obtaining a high-precision feature map;

[0022] The high-precision feature map is input into the feature compression module for image feature compression to obtain a high-quality feature map.

[0023] In one implementation, the ViT backbone network adapter includes a dense input embedding module, a multi-scale feature fusion module, and a bidirectional attention module. The step of inputting the historical predicted image thumbnail, the mask features, the multi-scale feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image includes:

[0024] The dense input embedding module is used to input the mask features into the bidirectional attention module;

[0025] The multi-scale feature map is input into the multi-scale feature fusion module for feature fusion, and the fused image is input into the bidirectional attention module;

[0026] The historical predicted image thumbnail and the high-quality feature map are input into the bidirectional attention module;

[0027] The bidirectional attention module is used to decode the historical prediction image thumbnail, the mask feature, the multi-scale feature map, and the high-quality feature map to obtain the prediction image.

[0028] In one implementation, after inputting the historical predicted image thumbnail, the mask feature, the multi-scale fused feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image, the method further includes:

[0029] The predicted image and the multi-scale feature map are logarithmically summed to obtain the corrected output image.

[0030] In one implementation, the spatial prior module includes a backbone layer, a convolutional filtering layer, and a fully connected layer;

[0031] The backbone layer comprises three deformable convolutional layers and one max pooling layer;

[0032] The backbone layer is used to capture the spatial features of the input image, and the convolutional filtering layer is used to double the number of feature channels and reduce the size of the input image.

[0033] In one implementation, the dense input embedding module includes four two-dimensional convolutional layers, a LayerNorm layer, and an activation function.

[0034] Secondly, embodiments of the present invention also provide an interactive image segmentation system based on lightweight adapter fine-tuning, wherein the system includes:

[0035] The first image acquisition module is used to acquire the input image and the clicked image, and to perform image segmentation on the input image to obtain several non-overlapping image blocks;

[0036] The feature fusion module is used to input the clicked image and the image block into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image;

[0037] The feature extraction module is used to input the input image into an adaptive domain feature extractor to extract image features and obtain a high-quality feature map of the input image.

[0038] The second image acquisition module is used to acquire historical prediction images and mask features, and to downsample the historical prediction images to obtain thumbnails of the historical prediction images.

[0039] The image decoding module is used to input the historical predicted image thumbnail, the mask feature, the multi-scale fusion feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image.

[0040] In one implementation, the feature fusion module includes:

[0041] The click image feature sequence acquisition unit is used to input the click image into the first Patch embedding module for feature mapping to obtain the click image feature sequence.

[0042] The image patch feature sequence acquisition unit is used to input the image patch into the second Patch embedding module for feature mapping to obtain the image patch feature sequence;

[0043] A global feature image acquisition unit is used to sequentially input the click image feature sequence and the image block feature sequence into the N Transformer encoder modules for image encoding to obtain the global feature image of the input image;

[0044] The multi-scale feature map acquisition unit is used to input the global feature image into the feature pyramid module for image feature fusion to obtain a multi-scale feature map.

[0045] In one implementation, the feature extraction module includes:

[0046] A standardization unit is used to standardize the input image;

[0047] An image detail extraction unit is used to input the standardized input image into the spatial prior module to extract image details and obtain the output image of the spatial prior module.

[0048] An image feature fusion unit is used to input the image output by the spatial prior module and the global feature image into the injection module to perform image feature fusion and obtain a feature fusion map.

[0049] An image precision feature extraction unit is used to input the feature fusion map into the precision feature extraction module to extract image precision features and obtain a high-precision feature map.

[0050] The image feature compression unit is used to input the high-precision feature map into the feature compression module for image feature compression to obtain a high-quality feature map.

[0051] In one implementation, the image decoding module includes:

[0052] The first input unit is used to input the mask features into the bidirectional attention module using the dense input embedding module;

[0053] The second input unit is used to input the multi-scale feature map into the multi-scale feature fusion module for feature fusion, and input the fused image into the bidirectional attention module;

[0054] The third input unit is used to input the historical prediction image thumbnail and the high-quality feature map into the bidirectional attention module;

[0055] The decoding unit is used to perform image decoding on the historical prediction image thumbnail, the mask feature, the multi-scale feature map and the high-quality feature map using the bidirectional attention module to obtain the prediction image.

[0056] In one implementation, the system further includes:

[0057] The image correction module is used to perform logarithmic summation on the predicted image and the multi-scale feature map to obtain a corrected output image.

[0058] In one implementation, the spatial prior module includes a backbone layer, a convolutional filtering layer, and a fully connected layer; the backbone layer includes three deformation convolutional layers and a max pooling layer; the backbone layer is used to capture the spatial features of the input image, and the convolutional filtering layer is used to double the number of feature channels and reduce the size of the input image.

[0059] In one implementation, the dense input embedding module includes four two-dimensional convolutional layers, a LayerNorm layer, and an activation function.

[0060] Thirdly, the present invention also provides a terminal device, including a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the one or more programs including an interactive image segmentation method based on lightweight adapter fine-tuning as described in any of the preceding claims.

[0061] Fourthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is able to perform the interactive image segmentation method based on lightweight adapter fine-tuning as described in any of the preceding embodiments.

[0062] The beneficial effects of this invention are as follows: Compared with the prior art, this invention provides an interactive image segmentation method based on lightweight adapter fine-tuning. First, an input image and a clicked image are acquired, and the input image is segmented to obtain several non-overlapping image blocks. Then, the clicked image and the image blocks are input into a Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image. Further, the input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image. Next, historical prediction images and mask features are acquired, and the historical prediction images are downsampled to obtain thumbnails of the historical prediction images. Finally, the thumbnails of the historical prediction images, the mask features, the multi-scale feature map, and the high-quality feature map are input into a ViT backbone network adapter for image decoding to obtain the predicted image. This application employs an adaptive domain feature extraction network and combines it with a ViT backbone network to better extract image features, achieving better cross-domain performance, and simultaneously obtaining better image segmentation results and efficiency. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a flowchart illustrating the interactive image segmentation method based on lightweight adapter fine-tuning provided in an embodiment of the present invention.

[0065] Figure 2 This is an overall schematic diagram of the interactive image segmentation method based on lightweight adapter fine-tuning provided in the embodiments of the present invention.

[0066] Figure 3 This is a schematic diagram of the adaptive domain feature extractor of the interactive image segmentation method based on lightweight adapter fine-tuning provided in an embodiment of the present invention.

[0067] Figure 4 This is a schematic diagram of the ViT backbone network adapter for the interactive image segmentation method based on lightweight adapter fine-tuning provided in an embodiment of the present invention.

[0068] Figure 5 This is a schematic diagram illustrating the effectiveness of the interactive image segmentation method based on lightweight adapter fine-tuning provided in this embodiment of the invention in target segmentation in a fine image domain.

[0069] Figure 6 This is a block diagram illustrating the principle of an interactive image segmentation system based on lightweight adapter fine-tuning provided in an embodiment of the present invention.

[0070] Figure 7 This is a block diagram illustrating the internal structure of the terminal device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0072] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0073] Existing technologies suffer from poor performance and low efficiency in image segmentation.

[0074] To address the problems in existing technologies, this embodiment provides an interactive image segmentation method based on lightweight adapter fine-tuning. This method enables cross-domain operations on image data and improves the performance and efficiency of the segmentation algorithm. Specifically, the method first acquires an input image and a clicked image, and then segments the input image to obtain several non-overlapping image blocks. Next, the clicked image and the image blocks are input into a Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image. Further, the input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map. Then, historical prediction images and mask features are acquired, and the historical prediction images are downsampled to obtain thumbnails. Finally, the historical prediction image thumbnails, the mask features, the multi-scale feature map, and the high-quality feature map are input into a ViT backbone network adapter for image decoding to obtain the predicted image. This application employs an adaptive domain feature extraction network and combines it with a ViT backbone network to better extract image features, achieving better cross-domain performance, and simultaneously obtaining better segmentation results and efficiency.

[0075] For example, the terminal device acquires the user's input image and clicked image. The system performs preliminary image segmentation on the input image to obtain several non-overlapping image blocks. Further, the obtained image blocks and clicked image are input into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map. Then, the input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image. The system also acquires the previous historical prediction image and mask features, and performs downsampling on the historical prediction image to obtain a thumbnail. Finally, the historical prediction image thumbnail, mask features, multi-scale feature map, and high-quality feature map obtained in the previous steps are input together into the ViT network adapter for image decoding to obtain the predicted image. Furthermore, the predicted image and multi-scale feature map can be logarithmically summed to obtain a corrected output image. By employing an adaptive domain feature extraction network and combining it with the ViT backbone network, image features are extracted more effectively, achieving better cross-domain performance, and simultaneously obtaining better segmentation results and efficiency.

[0076] Exemplary methods

[0077] This embodiment provides an interactive image segmentation method based on lightweight adapter fine-tuning, which can be applied to terminal devices. Specifically, as follows... Figure 1 As shown, the method includes:

[0078] Step S100: Obtain the input image and the clicked image, and perform image segmentation on the input image to obtain several non-overlapping image blocks.

[0079] In this embodiment, the network input includes the input image x∈R B×3×H×W Click on image x c ∈R B×2×H×W , wherein the clicked image x c It contains two channels, representing the positions of positive and negative clicks respectively; B represents the number of training samples; H represents the height of the input image; W represents the width of the input image; x p This represents the location distribution of the target object, i.e., the initial segmentation output of the segmentation network. It indicates the probability value of whether there is a target at each location, x. m The final segmentation mask is obtained by taking a threshold value for this variable, resulting in a mask where each value is either True or False.

[0080] In specific implementation, the input image and the clicked image are first obtained, and then the input image is segmented into a series of non-overlapping image blocks of size 16×16, so that they can be input into the Patch embedding module for feature mapping.

[0081] Step S200: Input the clicked image and the image block into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image.

[0082] In this embodiment, the Plain ViT backbone network includes a first Patch embedding module, a second Patch embedding module, N Transformer encoder modules, and a feature pyramid module.

[0083] In specific implementation, such as Figure 2 As shown, the clicked image is input into the first Patch embedding module for feature mapping to obtain a clicked image feature sequence. Image patches from the segmented input image are then input into the second Patch embedding module for feature mapping to obtain an image patch feature sequence. Through linear transformation between the two Patch embedding modules, the features of each image patch can be mapped into a feature vector of dimension C, forming a feature sequence of length L. Then, the clicked image feature sequence and the image patch feature sequence are sequentially input into N stacked Transformer encoder modules for image encoding to obtain the global feature image of the input image. Each Transformer encoder module contains a multi-head self-attention layer. The output of the feature vector sequence after passing through the nth layer is... The output after the last layer is: The output contains the strongest global image features. Finally, the global feature image is input into the feature pyramid module for image feature fusion to obtain a multi-scale feature map. The feature pyramid module consists of a set of convolutions and deconvolutions. By setting different convolution strides, a feature map with a size greater than the original image can be obtained. The feature maps are denoted as follows: Multi-scale processing enables algorithms to better handle objects of different sizes in input images. By learning target representations at different scales, more robust multi-scale target segmentation performance can be achieved.

[0084] Step S300: Input the input image into an adaptive domain feature extractor to extract image features and obtain a high-quality feature map of the input image.

[0085] In this embodiment, the adaptive domain feature extractor includes a spatial prior module, an injection module, a precision feature extraction module, and a feature compression module. The spatial prior module includes a backbone layer, a convolutional filtering layer, and a fully connected layer. This module captures local pixel relationships and spatial context of the image through deformable convolution. The backbone layer includes three deformable convolutional layers and a max-pooling layer, used to capture the spatial features of the input image. Batch normalization layers and the ReLU generalization function are used between the convolutional layers to improve generalization ability. The convolutional filtering layer is used to double the number of feature channels and reduce the size of the input image. The adaptive domain feature extractor extracts image features from the input image to obtain a high-quality feature map.

[0086] In practical implementation, to obtain more accurate segmentation results, we need to acquire rich global semantic context information and local edge detail information. Therefore, an adaptive domain feature extractor is introduced to obtain more spatial prior information and edge detail information, such as... Figure 3 The diagram shows the specific structure of the adaptive domain feature extractor. First, the input image is standardized to obtain the standardized image x. in Then, the standardized input image is input into the spatial prior module for image detail extraction, resulting in the output image of the spatial prior module. The output image of the spatial prior module and the global feature image are then input into the injection module for image feature fusion, resulting in a feature fusion map. The feature fusion map is then input into the precision feature extraction module for image precision feature extraction, resulting in a high-precision feature map. Finally, the high-precision feature map is input into the feature compression module for image feature compression, resulting in a high-quality feature map. In the spatial prior module, the input image is first input into the backbone layer, and after three convolutional layers and a max-pooling layer in the backbone layer, the output is... The backbone layer reduces the spatial scale of the image and maps channels to hidden dimensions. Then, the output of the backbone layer is input into the convolutional filter layer to further double the number of feature channels, reduce the image size, and the output of the convolutional filter layer is... The convolutional filtering layer consists of cascaded deformable convolutions with a stride of 2 and a kernel size of 3×3. Finally, the output of the convolutional filtering layer is input into the fully connected layer to project the feature map onto the feature dimension C of ViT, resulting in the output image of the spatial prior module. The fully connected layer consists of deformable convolutions with 1×1 kernels. After the input image is processed by the spatial prior module, an injection module is used to fuse the output image of the spatial prior module and the global feature map output by the Transformer encoder module. Specifically, this involves fusing the spatial prior features based on cross-attention. Features in the early layers of the PlainViT backbone network We then performed a fusion. Research on ViT shows that later blocks in the backbone network have longer attention distances, while earlier blocks are more localized and contain less semantically meaningful details. Based on this characteristic, we extracted the outputs of early attention blocks in the ViT backbone network as features. Specifically, we selected the output of the 3rd block out of a total of 12 blocks, denoted as... Will As a query, the spatial prior features As keys and values, spatial features are introduced into early ViT features using cross-attention, as shown in Equation (1).

[0087]

[0088] Where A(·) represents the cross-attention layer, the input at position one is the query, and the input at position two is the key and value, and norm(·) represents the LayerNorm layer. γ is a learnable parameter used to correlate the output of the attention layer in the residual connection. It has an initial value of 0. Since the output of the attention layer is non-zero, the parameter γ will be optimized to be non-zero in the first gradient descent iteration. In this way, the fusion efficiency of the adaptive domain module and the ViT backbone network can be gradually balanced in a learning manner. After introducing spatial features into the early ViT features, a precision feature extraction module is applied to extract high-precision features. The precision feature extraction module consists of a feedforward network (FFN) and a cross-attention layer. The process of high-precision feature extraction is shown in equations (2) and (3) below:

[0089]

[0090]

[0091] The output of the feedforward network As a query in the cross-attention layer, the output of the previous layer The cross-attention layer serves as both keys and values; finally, a feature compression module is designed to compress the output of the cross-attention layer through two deconvolutional layers. By increasing the dimension of the feature map, the size of the feature map can be obtained, resulting in a high-quality final feature map.

[0092] Step S400: Obtain historical prediction images and mask features, downsample the historical prediction images, and obtain thumbnails of the historical prediction images.

[0093] In this embodiment, the historical prediction image is the previous prediction image. t represents the t-th click. After acquiring the historical prediction image, downsampling is performed to obtain a thumbnail of the historical prediction image.

[0094] Step S500: Input the thumbnail of the historical predicted image, the mask feature, the multi-scale feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image.

[0095] In this embodiment, the ViT backbone network adapter includes a dense input embedding module, a multi-scale feature fusion module, and a bidirectional attention module. The dense input embedding module includes four two-dimensional convolutional layers, a LayerNorm layer, and an activation function. This module can input the mask features into the ViT backbone network adapter to enhance the stability of the model's prediction results and prevent significant changes in the predicted mask during two consecutive clicks. The dense input embedding module can also reduce the size of the mask features to the original size. The dimension mapped to the feature is C. The dense input embedding module is connected to the image features obtained by the Plain ViT backbone network and input into the bidirectional attention module for decoding.

[0096] In practical implementation, we introduce an efficient token learning method into the ViT backbone network adapter to improve its ability to learn high-quality domain information and predict masks, such as... Figure 4As shown, the token is first decoded through two decoder layers. In each decoder layer, the token undergoes self-attention update, followed by feature update through a bidirectional token-to-image and image-to-to token cross-attention layer. After passing through the decoder layer, the output token is associated with global image context features. Finally, a three-layer MLP is added, and the updated output token can predict dynamic MLP weights, generating dynamic convolutional kernels. Furthermore, we introduce a multi-scale feature fusion module in the ViT backbone network adapter. This module fuses multi-scale features obtained from the Plain ViT backbone network based on a simple feature pyramid network and inputs them into the ViT backbone network adapter for image decoding. According to ViT training, this simple feature pyramid structure can effectively extract visually specific inductive biases from the backbone network. A multi-scale feature fusion layer is designed in the ViT backbone network adapter. For features of different scales, deconvolutional layers of different lengths are used for compression, compressing the feature dimensions to a specific level. Unify the feature size to the input image size. Finally, the compressed multi-scale features are concatenated along their respective dimensions and then passed through a fully connected layer to output a dimension of [missing value]. Its characteristics.

[0097] In one implementation, the dense input embedding module inputs the mask features into the bidirectional attention module; the multi-scale feature map is input into the multi-scale feature fusion module for feature fusion, and the fused image is input into the bidirectional attention module; the historical prediction image thumbnail and the high-quality feature map are input into the bidirectional attention module; the bidirectional attention module performs image decoding on the historical prediction image thumbnail, the mask features, the multi-scale feature map, and the high-quality feature map to obtain the prediction image. During model training, we fix the model parameters of the pre-trained ViT model, while only making the parameters of the proposed domain adaptive adapter learnable. Therefore, the learnable parameters only include the convolutional layer and cross-attention layer in the adaptive domain feature extractor, the output token in the adapter, its associated three-layer MLP, the downsampling convolutional layer of the dense input embedding module, and the two decoder layers.

[0098] In one implementation, after inputting the thumbnail of the historical predicted image, the mask features, the multi-scale fused feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image, the method further includes: performing a logarithmic summation on the predicted image and the multi-scale feature map to obtain a corrected output image. During inference, the prediction from the adapter is used as the high-quality prediction result. The output result of the ViT after passing through the ordinary semantic segmentation head is logarithmically summed with the adapter's high-quality mask prediction to correct the output prediction mask. The corrected result is then upsampled to obtain the final output.

[0099] To verify the effectiveness of this invention, we evaluated it on multiple benchmarks, and the experimental results are shown in Table 1. As can be seen from Table 1, this invention exhibits the best performance among the compared algorithms. By adopting the scheme of this invention, the efficiency of interactive segmentation is effectively improved. Overall, the interactive segmentation algorithm proposed in this invention achieves state-of-the-art performance.

[0100]

[0101] Table 1. Performance Comparison Results of the Invention and Prior Art

[0102] In this invention, the main technical features include improving the performance of the algorithm on cross-domain datasets, including performance on fine-grained data segmentation tasks. In this respect, the present invention demonstrates superior performance compared to benchmark algorithms, such as... Figure 5 As shown. By Figure 5 As can be seen, the algorithm proposed in this invention can effectively improve the performance of algorithms on fine-grained object segmentation tasks, demonstrating the effectiveness of the proposed scheme. This invention utilizes a domain-adaptive network to learn domain features in cross-domain data. These domain features include image texture details and color style characteristics. The proposed domain-adaptive model can effectively learn representations of this specific domain style and achieve efficient segmentation and annotation on cross-domain data. The domain-adaptive model is deployed in the decoder head and also in the image feature extraction network.

[0103] Exemplary System

[0104] like Figure 6As shown in the illustration, this embodiment of the invention provides an interactive image segmentation system based on lightweight adapter fine-tuning. The system includes: a first image acquisition module 10, a feature fusion module 20, a feature extraction module 30, a second image acquisition module 40, and an image decoding module 50. Specifically, the first image acquisition module 10 is used to acquire an input image and a clicked image, and to segment the input image to obtain several non-overlapping image blocks. The feature fusion module 20 is used to input the clicked image and the image blocks into a Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image. The feature extraction module 30 is used to input the input image into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image. The second image acquisition module 40 is used to acquire historical prediction images and mask features, and to downsample the historical prediction images to obtain thumbnails of the historical prediction images. The image decoding module 50 is used to input the historical prediction image thumbnails, the mask features, the multi-scale fused feature map, and the high-quality feature map into a ViT backbone network adapter for image decoding to obtain a predicted image.

[0105] In one implementation, the feature fusion module includes:

[0106] The click image feature sequence acquisition unit is used to input the click image into the first Patch embedding module for feature mapping to obtain the click image feature sequence.

[0107] The image patch feature sequence acquisition unit is used to input the image patch into the second Patch embedding module for feature mapping to obtain the image patch feature sequence;

[0108] A global feature image acquisition unit is used to sequentially input the click image feature sequence and the image block feature sequence into the N Transformer encoder modules for image encoding to obtain the global feature image of the input image;

[0109] The multi-scale feature map acquisition unit is used to input the global feature image into the feature pyramid module for image feature fusion to obtain a multi-scale feature map.

[0110] In one implementation, the feature extraction module includes:

[0111] A standardization unit is used to standardize the input image;

[0112] An image detail extraction unit is used to input the standardized input image into the spatial prior module to extract image details and obtain the output image of the spatial prior module.

[0113] An image feature fusion unit is used to input the image output by the spatial prior module and the global feature image into the injection module to perform image feature fusion and obtain a feature fusion map.

[0114] An image precision feature extraction unit is used to input the feature fusion map into the precision feature extraction module to extract image precision features and obtain a high-precision feature map.

[0115] The image feature compression unit is used to input the high-precision feature map into the feature compression module for image feature compression to obtain a high-quality feature map.

[0116] In one implementation, the image decoding module includes:

[0117] The first input unit is used to input the mask features into the bidirectional attention module using the dense input embedding module;

[0118] The second input unit is used to input the multi-scale feature map into the multi-scale feature fusion module for feature fusion, and input the fused image into the bidirectional attention module;

[0119] The third input unit is used to input the historical prediction image thumbnail and the high-quality feature map into the bidirectional attention module;

[0120] The decoding unit is used to perform image decoding on the historical prediction image thumbnail, the mask feature, the multi-scale feature map and the high-quality feature map using the bidirectional attention module to obtain the prediction image.

[0121] In one implementation, the system further includes:

[0122] The image correction module is used to perform logarithmic summation on the predicted image and the multi-scale feature map to obtain a corrected output image.

[0123] In one implementation, the spatial prior module includes a backbone layer, a convolutional filtering layer, and a fully connected layer; the backbone layer includes three deformation convolutional layers and a max pooling layer; the backbone layer is used to capture the spatial features of the input image, and the convolutional filtering layer is used to double the number of feature channels and reduce the size of the input image.

[0124] In one implementation, the dense input embedding module includes four two-dimensional convolutional layers, a LayerNorm layer, and an activation function.

[0125] Based on the above embodiments, the present invention also provides a terminal device, the principle block diagram of which is shown in Figure 7. The terminal device may include one or more processors 100 ( Figure 7(Only one is shown in the image), memory 101, and computer program 102 stored in memory 101 and executable on one or more processors 100, such as a program for an interactive image segmentation method based on lightweight adapter fine-tuning. When one or more processors 100 execute computer program 102, they can implement the various steps in the embodiment of the interactive image segmentation method based on lightweight adapter fine-tuning. Alternatively, when one or more processors 100 execute computer program 102, they can implement the functions of various modules / units in the embodiment of the interactive image segmentation system based on lightweight adapter fine-tuning, which is not limited here.

[0126] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0127] In one embodiment, memory 101 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 101 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 101 may include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the terminal device. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0128] Those skilled in the art will understand that Figure 7 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal device to which the present invention is applied. The specific terminal device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0129] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the methods described above. Any references to memory, storage, operational databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operating data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0130] In summary, this invention discloses an interactive image segmentation method based on lightweight adapter fine-tuning. First, an input image and a clicked image are acquired, and the input image is segmented to obtain several non-overlapping image blocks. Then, the clicked image and the image blocks are input into a Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image. Further, the input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image. Next, historical prediction images and mask features are acquired, and the historical prediction images are downsampled to obtain thumbnails of the historical prediction images. Finally, the thumbnails of the historical prediction images, the mask features, the multi-scale feature map, and the high-quality feature map are input into a ViT backbone network adapter for image decoding to obtain the predicted image. This application employs an adaptive domain feature extraction network and combines it with a ViT backbone network to better extract image features, achieving better cross-domain performance, and simultaneously obtaining better segmentation results and efficiency.

[0131] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An interactive image segmentation method based on lightweight adapter fine-tuning, characterized in that, The method includes: The input image and the clicked image are acquired, and the input image is segmented to obtain several non-overlapping image blocks; The clicked image and the image block are input into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image. The input image is input into an adaptive domain feature extractor for image feature extraction to obtain a high-quality feature map of the input image; Obtain historical prediction images and mask features, downsample the historical prediction images, and obtain thumbnails of the historical prediction images; The historical predicted image thumbnail, the mask feature, the multi-scale feature map, and the high-quality feature map are input into the ViT backbone network adapter for image decoding to obtain the predicted image; The adaptive domain feature extractor includes a spatial prior module, an injection module, a precision feature extraction module, and a feature compression module. The step of inputting the input image into the adaptive domain feature extractor for feature extraction to obtain a high-quality feature map of the input image includes: The input image is standardized; The standardized input image is input into the spatial prior module for image detail extraction, and the output image of the spatial prior module is obtained. The spatial prior module output image and the global feature image are input into the injection module for image feature fusion to obtain a feature fusion map; The feature fusion map is input into the precision feature extraction module to extract image precision features, thereby obtaining a high-precision feature map; The high-precision feature map is input into the feature compression module for image feature compression to obtain a high-quality feature map. The ViT backbone network adapter includes a dense input embedding module, a multi-scale feature fusion module, and a bidirectional attention module. The step of inputting the historical predicted image thumbnail, the mask features, the multi-scale feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image includes: The dense input embedding module is used to input the mask features into the bidirectional attention module; The multi-scale feature map is input into the multi-scale feature fusion module for feature fusion, and the fused image is input into the bidirectional attention module; The historical predicted image thumbnail and the high-quality feature map are input into the bidirectional attention module; The bidirectional attention module is used to decode the historical prediction image thumbnail, the mask feature, the multi-scale feature map, and the high-quality feature map to obtain the prediction image.

2. The interactive image segmentation method based on lightweight adapter fine-tuning according to claim 1, characterized in that, The Plain ViT backbone network includes a first patch embedding module, a second patch embedding module, N Transformer encoder modules, and a feature pyramid module. The step of inputting the clicked image and the image patch into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image includes: The clicked image is input into the first Patch embedding module for feature mapping to obtain the clicked image feature sequence; The image patch is input into the second Patch embedding module for feature mapping to obtain the image patch feature sequence; The click image feature sequence and the image block feature sequence are sequentially input into the N Transformer encoder modules for image encoding to obtain the global feature image of the input image; The global feature image is input into the feature pyramid module for image feature fusion to obtain a multi-scale feature map.

3. The interactive image segmentation method based on lightweight adapter fine-tuning according to claim 1, characterized in that, After inputting the historical predicted image thumbnail, the mask feature, the multi-scale fused feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image, the process further includes: The predicted image and the multi-scale feature map are logarithmically summed to obtain the corrected output image.

4. The interactive image segmentation method based on lightweight adapter fine-tuning according to claim 1, characterized in that, The spatial prior module includes a backbone layer, a convolutional filter layer, and a fully connected layer. The backbone layer comprises three deformable convolutional layers and one max pooling layer; The backbone layer is used to capture the spatial features of the input image, and the convolutional filtering layer is used to double the number of feature channels and reduce the size of the input image.

5. The interactive image segmentation method based on lightweight adapter fine-tuning according to claim 1, characterized in that, The dense input embedding module includes four two-dimensional convolutional layers, a LayerNorm layer, and an activation function.

6. An interactive image segmentation system based on lightweight adapter fine-tuning, characterized in that, The system is used to implement the steps of the interactive image segmentation method based on lightweight adapter fine-tuning as described in any one of claims 1-5, the system comprising: The first image acquisition module is used to acquire the input image and the clicked image, and to perform image segmentation on the input image to obtain several non-overlapping image blocks; The feature fusion module is used to input the clicked image and the image block into the Plain ViT backbone network for image feature fusion to obtain a multi-scale feature map of the input image; The feature extraction module is used to input the input image into an adaptive domain feature extractor to extract image features and obtain a high-quality feature map of the input image. The second image acquisition module is used to acquire historical prediction images and mask features, and to downsample the historical prediction images to obtain thumbnails of the historical prediction images. The image decoding module is used to input the historical predicted image thumbnail, the mask feature, the multi-scale fusion feature map, and the high-quality feature map into the ViT backbone network adapter for image decoding to obtain the predicted image.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an interactive image segmentation program based on lightweight adapter fine-tuning stored in the memory and executable on the processor. When the processor executes the interactive image segmentation program based on lightweight adapter fine-tuning, it implements the steps of the interactive image segmentation method based on lightweight adapter fine-tuning as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an interactive image segmentation program based on lightweight adapter fine-tuning, which, when executed by a processor, implements the steps of the interactive image segmentation method based on lightweight adapter fine-tuning as described in any one of claims 1-5.

Images