An interactive image segmentation method, system, computer device and storage medium

Abstract

The application provides an interactive image segmentation method, system, computer device and storage medium, and belongs to the technical field of computer vision and deep learning, and the method comprises the following steps: a data set comprising an image, a mask and corresponding granularity values is constructed; a LoRA algorithm is used to efficiently fine-tune a VIT model to obtain a LoRA-ViT model; through a multi-modal feature fusion mechanism, visual contour features, structured features generated by user clicks and image blocks converted by granularity values are deeply fused; finally, an adaptive inner loop refinement strategy is adopted to iteratively optimize the segmentation result without additional user interaction until convergence. The granularity control mechanism is innovatively introduced, the ambiguity of user input is effectively reduced, the number of interactions is significantly reduced, and the segmentation accuracy is improved through an improved balanced normalized focal loss function, and the application is especially suitable for precise image segmentation tasks in complex scenes.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision and deep learning, and specifically relates to an interactive image segmentation method, system, computer device and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, interactive image segmentation, as a core supporting technology for human-computer collaborative tasks, has been widely applied in fields such as image editing, smart device interface interaction, and medical image analysis. Its core requirement is to quickly and accurately segment the target region based on a small amount of user input, achieving an efficient closed loop of "instruction-execution".

[0003] However, current mainstream interactive segmentation methods suffer from several key problems. For example, user input is highly ambiguous. Traditional methods rely solely on click input, and when the target contains multiple components, the model cannot distinguish whether the user intends to segment the whole or a part through a single or few clicks, leading to model errors. Furthermore, to optimize segmentation accuracy, existing methods require multiple iterative clicks from the user, and the model needs to be re-inferred after each click, making it difficult to meet real-time response requirements. Additionally, interactive segmentation tasks commonly suffer from imbalances in the number of positive and negative pixels and imbalances in pixel classification difficulty. Traditional focus loss methods can only alleviate the latter, failing to address both imbalances simultaneously, resulting in blurred segmentation boundaries and missed foreground detections.

[0004] The aforementioned problems limit the practicality of existing methods, and there is an urgent need for a solution that can coordinate multiple interactive information inputs, reduce the number of interactions, and optimize loss calculation. Summary of the Invention

[0005] To address the problems of high input ambiguity, numerous interactions, and insufficient segmentation accuracy in existing interactive segmentation methods, this invention provides an interactive image segmentation method, system, computer device, and storage medium.

[0006] To achieve the above objectives, the present invention provides an interactive image segmentation method, comprising:

[0007] Acquire the original target image group, and divide the target image group into multiple different component masks and patch image blocks; calculate the granularity value of each component mask, and convert the granularity value into a learnable image block embedding.

[0008] Linear encoding is performed on the patch image blocks, and visual contour features of each encoded patch image block are extracted; interactive click information of users dividing image segmentation regions on the target image is statistically analyzed, and the node association strength of all interactive click information is evaluated; based on the node association strength, the K strongest neighbor nodes are selected for each node; the information of the K neighbor nodes is fused to generate click intent features; the click intent features and visual contour features are fused to generate structured click features; the dimensions of the image block embedding and the visual contour features are aligned and added together to form visual-granular fusion features; the structured click features and visual-granular fusion features are fused into multimodal fusion features.

[0009] The multimodal fusion features are initially segmented into mask segmentation results; the current mask segmentation results are updated iteratively based on the multimodal fusion features; if the number of pixels that differ between the current mask segmentation results and the previous mask segmentation results is less than a set threshold, the loop stops, and the target image region segmentation results are obtained.

[0010] Preferably, the original target image is processed into the target image region segmentation result using the LoRA-ViT model; the LoRA-ViT model specifically comprises: using the traditional Vision Transformer as a framework, using the LoRA algorithm to introduce two low-rank matrices to calculate the weights of the traditional Vision Transformer model, adding a linear transformation function after the input layer of the traditional Vision Transformer model; replacing the fully connected attention mechanism of the traditional Vision Transformer model with a Top-k sparse attention mechanism to obtain the LoRA-ViT model, and updating the weights of the LoRA-ViT model by updating the low-rank matrix.

[0011] Preferably, the method of using a Top-k sparse attention mechanism to replace the fully connected attention mechanism of the traditional Vision Transformer model specifically includes: evaluating the node association strength of all interactive click information using a similarity calculation function; using a Top-k selection function to select the K strongest neighbor nodes for each node with the node association strength; and fusing the information of the K neighbor nodes through an average aggregation function to generate click intent features.

[0012] Preferably, the step of iteratively updating the current mask segmentation result based on multimodal fusion features specifically includes: keeping the multimodal fusion features unchanged; if the number of pixels differing between the current mask segmentation result and the previous mask segmentation result is greater than a set threshold, performing mask segmentation iteratively using the multimodal fusion features and the previous segmentation mask until the number of pixels differing between the current mask segmentation result and the previous mask segmentation result is less than the set threshold, then stopping the loop to obtain the final region image segmentation result.

[0013] Preferably, the training process of the LoRA-ViT model further includes optimizing parameters using the Balanced Normalized Focus Loss Function (LBNFL), specifically including: obtaining the true mask value of the original training target image; the loss function LBNFL, by introducing a balance coefficient and a normalization term, balances the difference between the target image region segmentation result and the true mask value of the original training target image, updates the parameters of the low-rank matrix of the LoRA algorithm through backpropagation, optimizes the segmentation accuracy of the LoRA-VIT model, and obtains the trained LoRA-VIT model.

[0014] Preferably, the image blocks are embedded and aligned with the dimensions of the visual contour features through a broadcast operation, and then added together to form a visual-granular fusion feature.

[0015] Preferably, the method further includes preprocessing the target image, specifically including: performing region segmentation and click partitioning on different components of the original target image to generate a preliminary component mask; the preliminary component mask is subjected to morphological hole removal and filtering to obtain a final component mask; calculating the granularity value of the final component mask to form a dataset of image, mask and granularity value; the granularity value is determined by the ratio of the pixel area of ​​the final component mask to the overall mask pixel area of ​​the original target image.

[0016] The present invention also provides an interactive image segmentation system, comprising:

[0017] The data acquisition module is used to acquire the original target image group, divide the target image group into multiple different component masks and patch image blocks; calculate the granularity value of each component mask, and convert the granularity value into a learnable image block embedding.

[0018] The feature processing module is used to linearly encode the patch image blocks, extract the visual contour features of each encoded patch image block; statistically analyze the interactive click information of the user dividing the image segmentation region on the target image, and evaluate the node association strength of all interactive click information; based on the node association strength, select the K strongest neighbor nodes for each node; fuse the information of the K neighbor nodes to generate click intent features; fuse the click intent features and visual contour features to generate structured click features; align the dimensions of the image block embedding with the visual contour features and add them together to form a visual-granular fusion feature; and fuse the structured click features and the visual-granular fusion feature into a multimodal fusion feature.

[0019] The segmentation module is used to initially segment the multimodal fusion features into mask segmentation results; it iteratively updates the current mask segmentation results based on the multimodal fusion features; if the number of pixels differing between the current mask segmentation result and the previous mask segmentation result is less than a set threshold, the loop stops, and the target image region segmentation result is obtained.

[0020] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any one of the interactive image segmentation methods.

[0021] The present invention also provides a computer-readable storage medium storing a computer program that, when loaded by a processor, can execute any of the steps in the interactive image segmentation method.

[0022] The interactive image segmentation method provided by this invention has the following beneficial effects:

[0023] This invention calculates component mask granularity values, allowing users to control the segmentation range by setting granularity requirements. This satisfies both fast coarse segmentation and fine component segmentation. It evaluates the node association strength of all interactive click information, selecting the K strongest neighbor nodes for each node and fusing the information from these K neighbor nodes to generate click intent features. This accurately captures user click intent, reduces click ambiguity, and pre-constrains the segmentation range with granular information, minimizing invalid clicks. It uses multimodal fusion features to generate preliminary mask segmentation results and iteratively optimizes these results to obtain the final refined region image segmentation. This reduces the number of user interactions while improving segmentation accuracy. Without sacrificing visual feature extraction capabilities, it significantly reduces training memory usage and time, achieving a balance between lightweight training and high-performance segmentation. Attached Figure Description

[0024] To more clearly illustrate the embodiments and design schemes of the present invention, the accompanying drawings required for this embodiment will be briefly described below. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an interactive image segmentation method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the interactive segmentation method based on granular click control collaborative optimization according to an embodiment of the present invention;

[0027] Figure 3This is a structural diagram of the attention module for fusing click intent feature maps according to an embodiment of the present invention;

[0028] Figure 4 This is a structural diagram of the adaptive inner loop refinement in an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand and implement the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and should not be construed as limiting the scope of protection of the present invention.

[0030] Some studies have attempted to introduce various interactive information to assist segmentation, but have not achieved deep fusion of multiple features and lack "controllable segmentation granularity," making it impossible for users to flexibly switch the segmentation range according to their needs.

[0031] The interactive segmentation method of this invention first utilizes the LoRA algorithm to modify the weights of the traditional Vision Transformer model by introducing a low-rank matrix; then, it replaces the fully connected attention mechanism of the traditional Vision Transformer model with a Top-k sparse attention mechanism, constructing an improved LoRA-VIT model. The LoRA-VIT model is a segmentation model that mainly includes an input module, a graph attention module, an interactive segmentation module, and an output module. Figure 2 As shown, the Vision Transformer is fine-tuned using LoRA technology to efficiently extract visual features. User click information is encoded into structured click intent features via a graph attention module, and the cues representing segmentation granularity are embedded and aligned with the visual features for fusion. Subsequently, the model deeply fuses visual-granular features and click features through a dual cross-attention mechanism to generate preliminary segmentation results. Finally, the system enters the core adaptive inner loop refinement stage, where the current segmentation result and fixed multimodal features are input into the model again for iterative optimization until the output mask converges to a stable state. This results in a final segmentation result with both high accuracy and controllable granularity, achieved without additional user intervention.

[0032] This invention provides an interactive image segmentation method, specifically as follows: Figure 1 As shown, it includes:

[0033] S1. Acquire the original target image group, and divide the target image group into multiple different component masks and patch image blocks; calculate the granularity value of each component mask, and convert the granularity value into a learnable image block embedding.

[0034] Granularity-mask pair dataset generation: Based on pre-trained models and public datasets, a granularity-mask pair dataset is generated to fine-tune the ability to control granularity.

[0035] By using a pre-trained model to perform click loops on different components of the target in the training set, a preliminary component mask is obtained by performing one positive click on a single component and 3-5 negative clicks at a certain distance from the positive click. The generated component masks are then post-processed, including morphological hole removal and link filtering. The ratio of the component mask pixel area to the overall target mask pixel area is calculated as the "granularity value" of that component, forming a "granularity-mask pair" dataset.

[0036] S2. Linearly encode the patch image blocks and extract the visual contour features of each encoded patch image block; statistically analyze the interactive click information of the user dividing the image segmentation region on the target image, and evaluate the node association strength of all interactive click information; based on the node association strength, select the K strongest neighbor nodes for each node; fuse the information of the K neighbor nodes to generate click intent features; fuse the click intent features and visual contour features to generate structured click features; align the dimensions of the image block embedding with the visual contour features and add them together to form a visual-granular fusion feature; fuse the structured click features and the visual-granular fusion feature into a multimodal fusion feature.

[0037] Specifically, the LoRA-ViT model is used to segment the original target image into patch image blocks and encode them, and the visual contour features of the encoded patch image blocks are extracted; the interactive click information of the user in the segmented regions of the target image is counted, and the similarity of all interactive click information is calculated through the graph attention module to obtain the click intent feature; the click intent feature and the visual contour feature are fused to generate the structured click feature.

[0038] Encoding of Multiple Interactive Inputs and Generation of Structured Click Features Based on Graph Attention. This invention introduces additional inputs to enable interactive models to control the granularity of segmentation results. It improves the accuracy of interactive segmentation tasks by enhancing the graph attention module used to generate structured click features, employing an adaptive inner loop refinement strategy, and balancing the normalized focus loss. Finally, it performs interactive segmentation of target-level and component-level objects using a Vision Transformer-based model.

[0039] To enable the model to control the granularity of its output, a series of granularity-mask pairs are needed for training. First, a pre-trained interactive segmentation model is selected, and clicks are performed repeatedly on its training set. Only one positive click is made on each component of the segmentation target, followed by multiple negative clicks. Negative clicks are kept at a certain distance from the single positive click to prevent collisions. This series of component masks is then processed through morphological hole removal and filtering to obtain the component masks used for training. The granularity values ​​for different components are then determined. The calculation method is as follows:

[0040] (1)

[0041] Where j represents a segmentation target, and i represents a component on the segmentation target. This represents the granularity value of the i-th component within the j-th objective. This is the area calculation function. The component mask of the i-th component in the j-th target. The overall mask of the j-th target.

[0042] According to formula (1), this granularity value is clearly in the range of 0 to 1. Therefore, these granularity values ​​are discretized into ten inputs with a step size of 0.1, and the discrete granularity is mapped to a high-dimensional embedding. The granularity embedding is aligned with the dimension of each image patch embedding, and then directly added to the patch embedding. Through a broadcast operation, the granularity information is propagated to each patch of each image, forming a visual-granularity fusion feature. Using the dual cross-attention mechanism of the interactive segmentation module, the structured click features and the visual-granularity fusion features are fused into a multimodal fusion feature.

[0043] LoRA fine-tuning is integrated with various interactive features. For this learnable patch embedding, LoRA fine-tuning allows the model to control the granularity of its output. Specifically, LoRA fine-tuning uses two small-rank matrices; the product of these two matrices has the same dimension as the full parameter matrix. Therefore, by training these two small-rank matrices and adding their product to the model's parameters, the full parameter matrix can be updated.

[0044] (2)

[0045] Where W is the weight matrix before model fine-tuning, and B and A represent the two low-rank matrices used for LoRA fine-tuning. The input matrix of the model, The output of the model.

[0046] As shown in Equation (2), since LoRA fine-tuning requires a small matrix rank, it fine-tunes only a few parameters instead of retraining the entire model, greatly reducing the number of parameters that need to be trained and preserving the knowledge learned during the pre-training phase, thus reducing the risk of forgetting. Furthermore, LoRA fine-tuning can accelerate the model training process. By only fine-tuning a matrix with a small rank, instead of updating the parameters of the entire model, the computational cost and storage requirements during training are effectively reduced. This saves a significant amount of computational resources during training, making it particularly suitable for scenarios requiring rapid iteration or processing large-scale datasets.

[0047] An adaptive inner loop refinement strategy is employed to optimize the interactive segmentation results. Specifically, the improved interactive segmentation refinement strategy uses an adaptive inner loop refinement algorithm. The model is trained on commonly used target-level datasets for interactive segmentation applications and tested on commonly used target-level and component-level datasets. The loss function is calculated based on the segmentation results and the ground truth values, and the model parameters are continuously optimized using gradient descent.

[0048] like Figure 3 As shown, the graph attention module adds a linear transformation function after the input layer of the traditional Vision Transformer model; it replaces the fully connected attention mechanism of the traditional Vision Transformer model with a Top-k sparse attention mechanism, optimizing the graph attention from a globally fully connected mechanism to a locally most relevant connection. User clicks are considered nodes in the graph attention, and these nodes are dynamically selected in the graph based on user intent. First, the similarity of click features is calculated using a similarity calculation function, a Top-k selection function, and an average aggregation function to determine which graph nodes are most relevant to the user's click intent, thus generating click intent features. Then, the features of these relevant nodes are fused to generate click intent features, forming structured interaction features. This dynamic adjustment of the graph structure enables the model to predict target regions more accurately when facing complex user interactions, improving the stability and robustness of the segmentation task. Finally, these structured interaction features are fused with the visual-granular fusion features of the Vision Transformer through a dual cross-attention fusion mechanism to form multimodal fusion features, thereby enhancing the control of user clicks in the segmentation results.

[0049] S3. Initially segment the multimodal fusion features into mask segmentation results; cyclically update the current mask segmentation results based on the multimodal fusion features; if the number of pixels differing between the current mask segmentation result and the previous mask segmentation result is less than a set threshold, stop the loop and obtain the target image region segmentation result.

[0050] The multimodal fusion features are input into the segmentation head module to generate a preliminary mask segmentation result. The current mask segmentation result is updated iteratively based on the multimodal fusion features. The loop stops when the number of pixels differing between the current and previous mask segmentation results is less than a set threshold, yielding the final region image segmentation result. The inner loop is a segmentation optimization loop without user interaction, while the outer loop is an optimization iteration process based on user interaction. When a user completes an interaction (e.g., one positive example click and 3-5 negative example clicks), the model first generates a preliminary mask segmentation result. If the result has not reached pixel-level stability, meaning there is still room for optimization in the segmentation details, the inner loop is automatically triggered. Figure 4 As shown, the segmentation model represents the main framework of the LoRA-ViT model, not any individual segmentation module within the model. The segmentation model is responsible for executing the entire segmentation process. The LoRA-ViT model employs an adaptive inner-loop refinement strategy. Within the user's iterative click loop, an additional nested loop for refinement is introduced. Each iteration of the click loop incorporates several inner loops. In the outer loop, without the user providing new click sequences, the inner loop updates the previous round's segmentation results and reruns the model while keeping other inputs unchanged, continuously refining the segmentation results under limited click interaction information. When the number of pixels changed in the segmentation result of a certain inner loop is less than a certain threshold, the inner loop refinement adaptively stops, and new user clicks are added to start a new click iteration loop.

[0051] The model is continuously optimized while addressing the issues of "imbalance in the number of positive and negative pixels" and "imbalance in the difficulty of pixel classification" by balancing the normalized focus loss.

[0052] (3)

[0053] Among them, in formulas (3)-(4), For the focal loss value, Here is the predicted value, and γ is the focus coefficient. Represents the true label, .

[0054] (4)

[0055] (5)

[0056] The improved loss function specifically adopts balanced normalized focus loss. The test set is input into an interactive segmentation model based on granular click control co-optimization to obtain the prediction results. The segmentation results are obtained by comparing the prediction results with the true labels.

[0057] To address the impact of imbalances in the number of positive and negative pixels and the difficulty of pixel classification on model training, the focus loss is optimized into a balanced normalized focus loss. The focus loss function, as shown in Equation (3), relies on an exponential product to partially solve the problem of imbalanced pixel classification difficulty, but further research is still lacking. To better address these two issues, a Taylor expansion is performed on the focus loss, and the gradient of its Taylor expansion is calculated, as shown in Equations (4) and (5). Analysis of the gradient expression shows that the first term of the Taylor expansion can be considered the gradient center; therefore, a balancing coefficient can be added to the first term to address the imbalance in the difficulty of positive and negative pixels. In addition, a normalization term is introduced to dynamically adjust the gradient weights and reduce rapid gradient decay. The final balanced normalized focus loss is shown in Equations (6)-(8). Here is the predicted value, and γ is the focus coefficient. Represents the actual label.

[0058] (6)

[0059] in, This is a sample weight adjustment factor used to correct the loss of samples of different categories and difficulties, thereby alleviating the problems of class imbalance and loss dominated by easily distinguishable samples. This is the balance coefficient, which takes a value between (0,1) and is used to adjust the weight balance between positive and negative class samples.

[0060] (7)

[0061] in, The normalization coefficients for positive class samples (foreground pixels) are... represents the normalization coefficient for the negative class samples (background pixels), and n is the total number of samples. The normalization coefficient eliminates the imbalance in the magnitude of loss caused by differences in the number of samples of different categories.

[0062] (8)

[0063] in, To balance the normalized focus loss function, weight adjustment was incorporated. and normalization The core logic of focus loss is used to guide model parameter updates.

[0064] This invention proposes an interactive segmentation method and system based on granular click control and collaborative optimization. By fusing adjustable granularity values, structured click features, and visual contour features, it solves the problems of high input ambiguity and numerous interactions in traditional interactive segmentation. Simultaneously, it innovatively designs a balanced normalized focus loss and an adaptive inner loop refinement strategy, improving segmentation accuracy while reducing computational costs. This method is highly adaptable to real-time human-computer collaboration scenarios such as image editing, significantly improving the interactive segmentation model's ability to understand user intent and the efficiency of segmentation task execution. The invention's embodiments define the minimum number of clicks required for the user to achieve an IoU of 85% or 90% for the first time, and the IoU value between the model's output segmentation result and the ground truth annotation after only one click by the user.

[0065] Based on a trained segmentation model, component masks are generated iteratively by performing "1 positive click + 3-5 negative clicks" on different components of the target. Morphological hole removal and post-filtering processing are used to optimize mask quality. Granularity values ​​are calculated as "component mask pixel area / overall target mask pixel area," forming a granular-mask pair dataset that requires no additional manual annotation, providing a training foundation for the model's granular control capabilities. The granular values ​​are then upscaled into trainable Patch Embeddings, aligned with visual contour features, and broadcast to each visual contour feature, forming visual-granular fusion features. Structured click features generated using a graph attention module are then fused with visual contour features to create multimodal fusion features, improving the completeness of feature representation.

[0066] A LoRA fine-tuning strategy is adopted, introducing two low-rank matrices into the ViT backbone model. Training only the parameters of these low-rank matrices enables granular control learning, reducing the parameter size by over 90%. An adaptive inner-loop refinement process is designed. Without additional user clicks, the segmentation mask is continuously refined by re-inputting the previous round's segmentation results into the model. The inner loop adaptively stops when the segmentation result differs sufficiently from the previous round, and a new round of clicks is accepted. A balance coefficient is introduced based on the traditional focus loss. To address the imbalance in the number of positive and negative pixels, a normalization term was added. This addresses the imbalance in pixel classification difficulty and improves the model's ability to recognize pixels that are difficult to classify.

[0067] Based on the same inventive concept, the present invention also provides an interactive image segmentation system, comprising:

[0068] The data acquisition module is used to acquire the original target image group, divide the target image group into multiple different component masks and patch image blocks; calculate the granularity value of each component mask, and convert the granularity value into a learnable image block embedding.

[0069] The feature processing module is used to linearly encode the patch image blocks, extract the visual contour features of each encoded patch image block; statistically analyze the interactive click information of the user dividing the image segmentation region on the target image, and evaluate the node association strength of all interactive click information; based on the node association strength, select the K strongest neighbor nodes for each node; fuse the information of the K neighbor nodes to generate click intent features; fuse the click intent features and visual contour features to generate structured click features; align the dimensions of the image block embedding with the visual contour features and add them together to form a visual-granular fusion feature; and fuse the structured click features and the visual-granular fusion feature into a multimodal fusion feature.

[0070] The segmentation module is used to initially segment the multimodal fusion features into mask segmentation results; it iteratively updates the current mask segmentation results based on the multimodal fusion features; if the number of pixels differing between the current mask segmentation result and the previous mask segmentation result is less than a set threshold, the loop stops, and the target image region segmentation result is obtained.

[0071] This invention also provides a computer device, which, at the hardware level, includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the interactive image segmentation method provided above.

[0072] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the interactive image segmentation method provided above.

[0073] Specific limitations regarding the computational system for interactive image segmentation methods can be found in the limitations of interactive image segmentation methods described above, and will not be repeated here. Each module in the aforementioned interactive image segmentation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independent of the processor in a computer device, or stored in software in the memory of a computer device, so that the processor can call and execute the corresponding operations of each module.

[0074] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. Furthermore, the above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An interactive image segmentation method, characterized in that, include: Acquire the original target image group, and then divide the target image group into multiple different component masks and patch image blocks; Calculate the granularity value of the mask for each component, and convert the granularity value into a learnable image patch embedding; Linear encoding is performed on the Patch image blocks, and the visual contour features of each encoded Patch image block are extracted. Collect user interaction click information on the target image segmentation region and evaluate the node association strength of all interaction click information. Based on the node association strength, select the K strongest neighbor nodes for each node; fuse the information of the K neighbor nodes to generate click intent features; By fusing click intent features and visual contour features, structured click features are generated; The image patches are embedded and aligned with the dimensions of the visual contour features, and then added together to form a visual-granular fusion feature; The structured click features and visual-granular fusion features are fused into a multimodal fusion feature; The multimodal fusion features are initially segmented into mask segmentation results, and the current mask segmentation results are updated iteratively based on the multimodal fusion features. If the number of pixels that have changed between the current mask segmentation result and the previous mask segmentation result is less than a set threshold, the loop stops, and the target image region segmentation result is obtained.

2. The interactive image segmentation method of claim 1, wherein, The LoRA-ViT model is used to process the original target image into the target image region segmentation result. Specifically, the LoRA-ViT model is as follows: using the traditional Vision Transformer as a framework, the LoRA algorithm is used to introduce two low-rank matrices to calculate the weights of the traditional Vision Transformer model. A linear transformation function is added after the input layer of the traditional Vision Transformer model. The Top-k sparse attention mechanism is used to replace the fully connected attention mechanism of the traditional Vision Transformer model to obtain the LoRA-ViT model. The weights of the LoRA-ViT model are updated by updating the low-rank matrix.

3. The interactive image segmentation method of claim 2, wherein, The method of using a Top-k sparse attention mechanism to replace the fully connected attention mechanism of the traditional Vision Transformer model specifically includes: evaluating the node association strength of all interactive click information using a similarity calculation function; using a Top-k selection function to select the K strongest neighbor nodes for each node with the node association strength; and fusing the information of the K neighbor nodes through an average aggregation function to generate click intent features.

4. The method of claim 1, wherein, The step of iteratively updating the current mask segmentation result based on multimodal fusion features specifically includes: keeping the multimodal fusion features unchanged; if the number of pixels that have changed between the current mask segmentation result and the previous mask segmentation result is greater than a set threshold, performing mask segmentation iteratively using the multimodal fusion features and the previous segmentation mask until the number of pixels that have changed between the current mask segmentation result and the previous mask segmentation result is less than the set threshold, then stopping the loop and obtaining the final region image segmentation result.

5. The method of claim 2, wherein, It also includes, The training process of the LoRA-ViT model uses the Balanced Normalized Focus Loss Function (LBNFL) to optimize parameters. Specifically, it includes: obtaining the true mask value of the original training target image; the loss function LBNFL, by introducing a balance coefficient and a normalization term, balances the difference between the target image region segmentation result and the true mask value of the original training target image, updates the parameters of the low-rank matrix of the LoRA algorithm through backpropagation, optimizes the segmentation accuracy of the LoRA-ViT model, and obtains the trained LoRA-ViT model.

6. The method of claim 1, wherein, The image blocks are embedded and aligned with the dimensions of the visual contour features through a broadcast operation, and then added together to form a visual-granular fusion feature.

7. The method of claim 1, wherein, It also includes preprocessing of the target image, specifically including: performing region segmentation and click partitioning on different components of the original target image to generate a preliminary component mask; the preliminary component mask is subjected to morphological hole removal and filtering to obtain a final component mask; calculating the granularity value of the final component mask to form a dataset of image, mask and granularity value; the granularity value is determined by the ratio of the pixel area of ​​the final component mask to the overall mask pixel area of ​​the original target image.

8. An interactive image segmentation system, characterized in that, include: The data acquisition module is used to acquire the original target image group and divide the target image group into multiple different component masks and patch image blocks; Calculate the granularity value of the mask for each component, and convert the granularity value into a learnable image patch embedding; The feature processing module is used to perform linear encoding on the Patch image blocks and extract the visual contour features of each encoded Patch image block. Collect user interaction click information on the target image segmentation region and evaluate the node association strength of all interaction click information. Based on the node association strength, select the K strongest neighbor nodes for each node; fuse the information of the K neighbor nodes to generate click intent features; By fusing click intent features and visual contour features, structured click features are generated; The image patch embeddings are aligned with the dimensions of the visual contour features and then added together to form a visual-granular fusion feature; the structured click feature and the visual-granular fusion feature are fused into a multimodal fusion feature; The segmentation module is used to initially segment the multimodal fusion features into masked segmentation results; The current mask segmentation result is updated cyclically based on multimodal fusion features; If the number of pixels that have changed between the current mask segmentation result and the previous mask segmentation result is less than a set threshold, the loop stops, and the target image region segmentation result is obtained.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is loaded by the processor, it is able to perform the steps of the method according to any one of claims 1 to 7.

Images