Image segmentation method and device, computer device and storage medium

Abstract

The application relates to an image segmentation method and device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring an image to be segmented, and extracting multi-scale features of the image to be segmented; performing re-calibration processing on the multi-scale features to generate enhanced multi-scale features; performing feature splicing on the enhanced multi-scale features to generate unified semantic features, and generating pseudo prompts based on the unified semantic features to obtain initial pseudo prompts; if a user prompt is received, merging the user prompt and the initial pseudo prompts to generate a merged prompt; performing initial mask prediction based on the merged prompt and the enhanced multi-scale features to generate an initial segmentation mask; and performing iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and generating a target segmentation mask when the iterative optimization is completed. The application effectively improves the segmentation accuracy and interactive flexibility in a complex scene.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image segmentation method, apparatus, computer device and storage medium. Background Technology

[0002] Image segmentation, a core task in computer vision, aims to divide pixels in an image into multiple regions with semantic or visual consistency to achieve accurate extraction of targets of interest. This technology has wide applications in medical image analysis, remote sensing image interpretation, autonomous driving environmental perception, and human-computer interaction. In recent years, the rapid development of deep learning technology has driven significant progress in image segmentation, with convolutional neural network-based segmentation methods becoming the mainstream solution. Among them, the fully convolutional network (FCN) achieves end-to-end pixel-level prediction by replacing fully connected layers in classification networks with convolutional layers, achieving breakthrough progress in semantic segmentation tasks. The U-Net architecture, with its symmetrical encoder-decoder structure and skip connection mechanism, demonstrates superior performance in medical image processing, effectively mitigating the loss of feature information during upsampling.

[0003] With increasing task complexity and higher demands for long-range dependency modeling capabilities, the Transformer architecture has been introduced into the field of image segmentation, pioneering a new research direction. Representative works such as Vision Transformer and Swin Transformer have significantly improved the modeling ability and computational efficiency of the model through global self-attention mechanisms and local sliding window strategies. Against this backdrop, cue-guided image segmentation methods have emerged. These methods utilize cue information in the form of points, boxes, or text as input to guide the model in segmenting the target region, thereby reducing the dependence on large amounts of pixel-level labeled data and enhancing the model's flexibility and scalability.

[0004] However, existing technologies still face numerous challenges in practical applications. Particularly when processing complex images, traditional methods often struggle to accurately capture the correlations between multi-scale features, limiting segmentation accuracy. Furthermore, in interactive segmentation scenarios, effectively integrating user prompts with automatically generated pseudo-prompts, and progressively improving segmentation quality through iterative optimization mechanisms, are critical issues that urgently need to be addressed. In addition, existing methods still have room for improvement in areas such as feature recalibration, unified semantic feature generation, and multi-scale feature fusion. These factors all constrain the effectiveness of image segmentation technology in complex scenarios. Summary of the Invention

[0005] The purpose of this application is to provide an image segmentation method, apparatus, computer device, and storage medium to improve segmentation accuracy and interactive flexibility in complex scenes.

[0006] To address the aforementioned technical problems, embodiments of this application provide an image segmentation method, including:

[0007] Obtain the image to be segmented and extract its multi-scale features;

[0008] The multi-scale features are recalibrated to generate enhanced multi-scale features;

[0009] The enhanced multi-scale features are concatenated to generate unified semantic features, and pseudo-prompts are generated based on the unified semantic features to obtain initial pseudo-prompts;

[0010] If a user prompt is received, the user prompt is merged with the initial pseudo-prompt to generate a merged prompt;

[0011] Based on the merged prompt and the enhanced multi-scale features, an initial mask prediction is performed to generate an initial segmentation mask;

[0012] Based on the initial segmentation mask and the enhanced multi-scale features, iterative optimization is performed, and when the iterative optimization is completed, the target segmentation mask is generated.

[0013] To address the aforementioned technical problems, embodiments of this application provide an image segmentation apparatus, comprising:

[0014] The feature extraction module is used to acquire the image to be segmented and extract multi-scale features of the image to be segmented;

[0015] The recalibration module is used to recalibrate the multi-scale features to generate enhanced multi-scale features.

[0016] The pseudo-hint generation module is used to perform feature concatenation on the enhanced multi-scale features to generate unified semantic features, and generate pseudo-hints based on the unified semantic features to obtain initial pseudo-hints;

[0017] The prompt merging module is used to merge the user prompt with the initial pseudo prompt if a user prompt is received, and generate a merged prompt.

[0018] The initial segmentation module is used to perform initial mask prediction based on the merged prompts and the enhanced multi-scale features to generate an initial segmentation mask.

[0019] The target segmentation mask generation module is used to perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and generate the target segmentation mask when the iterative optimization is completed.

[0020] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is to provide a computer device, including one or more processors; and a memory for storing one or more programs, such that the one or more processors implement the image segmentation method described in any one of the above-mentioned methods.

[0021] To solve the above-mentioned technical problems, one technical solution adopted by the present invention is: a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the image segmentation method described in any one of the above-mentioned methods.

[0022] This invention provides an image segmentation method, apparatus, computer device, and storage medium. The method includes: acquiring an image to be segmented and extracting multi-scale features from the image; recalibrating the multi-scale features to generate enhanced multi-scale features; concatenating the enhanced multi-scale features to generate unified semantic features, and generating initial pseudo-hints based on the unified semantic features; if a user hint is received, merging the user hint with the initial pseudo-hint to generate a merged hint; performing initial mask prediction based on the merged hint and the enhanced multi-scale features to generate an initial segmentation mask; and iteratively optimizing based on the initial segmentation mask and the enhanced multi-scale features, generating a target segmentation mask when the iterative optimization is complete. This invention, through a multi-scale feature recalibration, pseudo-hint generation, and user hint fusion mechanism, combined with an iterative optimization strategy, effectively solves the problems of insufficient segmentation accuracy and poor interactive flexibility in complex scenes found in existing methods, effectively improving segmentation accuracy and interactive flexibility in complex scenes. Attached Figure Description

[0023] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the implementation of the image segmentation method provided in this application embodiment;

[0025] Figure 2 This is a flowchart illustrating the implementation of the first sub-process in the image segmentation method provided in this application embodiment;

[0026] Figure 3 This is a flowchart illustrating the implementation of the second sub-process in the image segmentation method provided in this application embodiment;

[0027] Figure 4This is a flowchart illustrating the implementation of the third sub-process in the image segmentation method provided in this application embodiment;

[0028] Figure 5 This is a flowchart illustrating the implementation of an image segmentation method according to another embodiment of this application;

[0029] Figure 6 This is a flowchart illustrating the implementation of the fourth sub-process in the image segmentation method provided in this application embodiment;

[0030] Figure 7 This is a schematic diagram of the image segmentation apparatus provided in an embodiment of this application;

[0031] Figure 8 This is a schematic diagram of the computer device provided in the embodiments of this application. Detailed Implementation

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0035] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0036] It should be noted that the image segmentation method provided in this application embodiment is generally executed by a server, and correspondingly, the image segmentation device is generally configured in the server.

[0037] The image segmentation method provided in this application can be applied to the segmentation of bird images in traditional Chinese paintings. Existing image segmentation methods often fail to achieve ideal segmentation results when faced with the unique painting style, blurred boundaries, and complex backgrounds of traditional Chinese paintings. These methods exhibit problems such as inability to clearly segment the subject, inaccurate prompt responses, and rough mask edges, limiting their practical application value in traditional cultural image processing. This application aims to address the shortcomings of existing models in the task of segmenting bird images in traditional Chinese paintings by proposing a technical method that better adapts to the characteristics of traditional Chinese painting images, effectively improves segmentation accuracy and prompt utilization efficiency, and meets the practical needs for high-quality segmentation tools in the context of the digitization of traditional culture.

[0038] In existing technologies, image segmentation techniques face significant challenges when processing images with complex styles. General segmentation models are typically trained on natural image data. When applied to artistic images such as traditional paintings, issues such as incomplete target segmentation, biased prompting responses, and jagged mask boundaries often arise due to differences in painting styles, blurred edges, and complex backgrounds. For example, in the segmentation of bird elements in traditional Chinese paintings, the gradient edges and freehand brushstrokes created by ink wash techniques make it difficult for existing models to accurately identify the target contours. Manual prompts require repeated corrections, and the segmentation results often fail to meet the needs of cultural relic preservation.

[0039] To address the aforementioned issues, existing methods suffer from shortcomings in feature representation and optimization mechanisms. Analysis reveals that the complex textures of traditional paintings require effective fusion of multi-level features, while current single-scale feature extraction methods struggle to capture the morphological features of artistic elements. Furthermore, over-reliance on manual prompts increases operational complexity and lacks an automatic guidance mechanism. Regarding the issue of rough edges, it was found that single predictions are insufficient to eliminate interference from stylized brushstrokes, necessitating an iterative optimization mechanism. Based on these findings, a technical approach of multi-scale feature enhancement, automatic prompt generation, and iterative optimization has gradually emerged. Therefore, this application proposes a technical solution including: acquiring the image to be segmented and extracting multi-scale features; recalibrating the multi-scale features to generate enhanced features; concatenating the enhanced features to generate unified semantic features and produce initial pseudo-prompts; merging user prompts and initial pseudo-prompts; performing initial mask prediction based on the merged prompts; and generating the target segmentation mask through iterative optimization.

[0040] Specifically, this application first acquires visual features of different granularities through a multi-level feature extraction network, and then adaptively enhances these features using a channel attention mechanism, effectively improving the representation ability of painting elements. The enhanced multi-scale features are then fused into a unified semantic representation, and initial guidance information is automatically generated by analyzing the global context. When a user correction prompt is received, manual input and automatic prompts are organically combined to form a more accurate guidance signal. After initial segmentation based on the fused prompts and enhanced features, a multi-round optimization mechanism is established. Through continuous feature re-extraction and mask correction, the edge blurring problem caused by ink wash rendering in traditional paintings is gradually eliminated, ultimately obtaining a segmentation result that conforms to artistic characteristics. This application effectively improves the accuracy and robustness of artistic image segmentation. Multi-scale feature recalibration enhances the model's ability to capture painting style features, the pseudo-prompt generation mechanism reduces the need for manual intervention, and the iterative optimization process significantly improves the smoothness of the segmentation boundaries. When processing complex artistic images such as birds in traditional Chinese paintings, it can accurately identify the outline of the target subject, reduce repeated correction operations, and obtain segmentation results that conform to artistic expression characteristics.

[0041] Please see Figure 1 , Figure 1 A specific implementation of the image segmentation method is shown.

[0042] It should be noted that if substantially the same result is obtained, the method of this invention does not necessarily require further clarification. Figure 1 Limited to the sequence shown, this method includes the following steps:

[0043] S1: Obtain the image to be segmented and extract the multi-scale features of the image to be segmented.

[0044] Specifically, multi-scale features of the image to be segmented are extracted through a backbone encoder. The encoder gradually reduces the image resolution and increases the channel dimension during the feature extraction process by stacking convolutional and pooling layers.

[0045] Among them, multi-scale features refer to capturing image details and overall structure through feature maps at different resolution levels. Convolutional neural network encoders can be used to extract feature sets containing high, medium and low resolution features.

[0046] Further, step S1 includes: acquiring the image to be segmented, extracting features from the image to be segmented using an encoder, and generating the multi-scale features, wherein the multi-scale features include a first high-resolution feature, a second medium-resolution feature, a third medium-resolution feature, and a fourth low-resolution feature.

[0047] Specifically, the first high-resolution feature is output from the first stage of the encoder, with 64 channels, maintaining a high spatial resolution to store local details. The second and third medium-resolution features are output from the second and third stages of the encoder, respectively, with the number of channels expandable to 256 or 512, capturing medium-scale structures such as the bird's trunk and wings by increasing the receptive field. The fourth low-resolution feature is output from the end of the encoder, with the number of channels increased to 1024, establishing a global association between the bird and the background in a highly abstract feature space. The four feature layers, through a complementary relationship of decreasing resolution and increasing semantics, form a multi-layered expression covering microscopic details to macroscopic semantics, providing a feature foundation for subsequent processing that adapts to the complex forms of traditional Chinese painting.

[0048] The encoder, implemented using a convolutional neural network, is an image feature extraction module that can employ ResNet, VGG, or Transformer architectures. It extracts features of varying abstraction levels from the input image. The first high-resolution feature is the feature map output from the shallow layer of the encoder, with a resolution that can be 1 / 4 of the input image size. It is used to preserve the texture of bird feathers and edge details. The second and third medium-resolution features are the feature maps output from the intermediate layers of the encoder, with a resolution that can be 1 / 8 or 1 / 16 of the input image size. They are used to capture medium-grained morphological structures. The fourth low-resolution feature is the feature map output from the deep layers of the encoder, with a resolution that can be 1 / 32 of the input image size. It is used to characterize the overall posture of the bird and its relationship to the background.

[0049] S2: Recalibrate the multi-scale features to generate enhanced multi-scale features.

[0050] Among them, recalibration processing refers to dynamically adjusting the feature channel weights through a channel attention mechanism. Specifically, it can be implemented by combining spatial pyramid pooling with a gating mechanism to enhance the feature response that is sensitive to painting style.

[0051] Please see Figure 2 , Figure 2 A specific implementation of step S2 is shown below:

[0052] S21: Extract the context information of each scale feature in the multi-scale features.

[0053] S22: Compress the context information to generate bottleneck features.

[0054] S23: Upsample the bottleneck feature to obtain the upsampled feature, and fuse the upsampled feature with the corresponding scale feature in the multi-scale feature to obtain the decoded feature.

[0055] S24: Based on the decoding features and the multi-scale features, perform channel recalibration to generate the enhanced multi-scale features.

[0056] Specifically, the semantic associations of features at each scale are first obtained through a context information extraction module to acquire context information. This context information is then compressed using a convolutional layer, normalization, and activation function to generate bottleneck features. Subsequently, an upsampling operation restores the spatial resolution of the bottleneck features to the original scale, and they are then fused element-wise with the original features of the corresponding level to form decoded features that simultaneously contain deep semantics and shallow details. Further, a channel attention mechanism is used to dynamically adjust the weights of the decoded features. Activation coefficients are generated by calculating the dependencies between channels, and these coefficients are multiplied channel-wise with the original multi-scale features. Finally, the recalibrated features of all levels are projected onto a unified channel dimension to form an enhanced multi-scale feature representation. This application effectively solves the problem of decreased segmentation accuracy caused by insufficient multi-scale feature integration. By constructing a context-aware feature enhancement mechanism, the model's ability to express complex semantic features is improved. This method can adaptively fuse semantic information and detail features at different levels, enhancing the recognition ability of weakly textured regions while preserving the clarity of target boundaries, thereby achieving more accurate image segmentation in complex scenes.

[0057] Upsampling refers to the process of restoring a high-resolution image from a low-resolution feature map. Common methods include nearest neighbor interpolation and deconvolution. Contextual information refers to the semantic association information captured in different receptive fields, which can be implemented using dilated convolution or global average pooling to enhance the semantic consistency of features at different scales. Channel recalibration refers to dynamically adjusting the weight distribution of feature channels, which can be implemented using a channel attention mechanism combined with the sigmoid function to strengthen important feature channels and suppress noise interference.

[0058] Please see Figure 3 , Figure 3 A specific implementation of step S24 is shown below:

[0059] S241: The channel attention mechanism is used to perform average pooling calculation on the decoded features to generate average pooling results, and a gating vector is generated based on the average pooling results by a multilayer perceptron.

[0060] S242: The gated vector is activated using the Sigmoid activation function to obtain the activated features.

[0061] S243: Multiply the activated features with the multi-scale features channel by channel to generate recalibrated features.

[0062] S244: Project all the recalibrated features onto the same number of channels to obtain the enhanced multi-scale features.

[0063] Specifically, in the channel recalibration process, a global average pooling operation is first performed on the decoded features to compress the spatial information of each channel into a single statistic, capturing the global contextual information of the channel dimension. Then, a multilayer perceptron is used to perform a nonlinear transformation on the pooling result to generate a gating vector representing the importance of each channel. This gating vector, after activation by the sigmoid function, forms weight coefficients in the 0-1 interval, used to indicate the retention strength of each channel. The activated weight coefficients are multiplied channel-by-channel with the original multi-scale features to suppress redundant channels and enhance key channels. Finally, a 1×1 convolution is used to unify the number of channels in all recalibrated features, ensuring dimensionality consistency of multi-scale features in subsequent processing. This process, by dynamically adjusting channel weights, effectively solves the problem of insufficient feature response caused by fixed weights in traditional multi-scale fusion. This application effectively solves the problems of channel information redundancy and insufficient key feature response in multi-scale feature fusion. In the segmentation scene of birds in traditional Chinese paintings, this technology can accurately enhance the texture feature channels of the bird's main edge while weakening the interference channels in the background area, making the generated segmentation mask edges smoother and more coherent, significantly improving the segmentation accuracy of complex artistic style images.

[0064] The channel attention mechanism refers to the mechanism of dynamically adjusting the feature channel weights by modeling the dependencies between channels. Specifically, it can be implemented using a combination of global average pooling and a multilayer perceptron structure to capture global contextual information of different channels. Average pooling is the operation of compressing decoded features in the spatial dimension, specifically implemented using a global average pooling layer to extract statistical features in the channel dimension. A multilayer perceptron is a neural network module composed of fully connected layers and nonlinear activation functions. Specifically, it can be implemented using a structure with a ReLU activation function added between two fully connected layers to generate channel weight vectors. The Sigmoid activation function is a nonlinear function that maps the input to the 0-1 interval. Specifically, it can be implemented using standard Sigmoid operations to generate channel selection masks. Channel-wise multiplication is the operation of element-wise multiplication of the activated features with the original features in the channel dimension. Specifically, it can be implemented using matrix dot multiplication to achieve dynamic selection of feature channels. Channel projection is the operation of unifying features with different numbers of channels to the same dimension. Specifically, it can be implemented using a 1×1 convolution kernel to eliminate dimensional differences between multi-scale features.

[0065] S3: Perform feature concatenation on the enhanced multi-scale features to generate unified semantic features, and generate pseudo-prompts based on the unified semantic features to obtain initial pseudo-prompts.

[0066] Specifically, pseudo-cue generation refers to automatically generating initial position information for guided segmentation. This can be achieved through global feature analysis combined with position weight maps, reducing reliance on manual input.

[0067] Please see Figure 4 , Figure 4 A specific implementation of step S3 is shown below:

[0068] S31: Perform feature concatenation on the enhanced multi-scale features to generate the unified semantic features.

[0069] S32: Generate encoded features based on the unified semantic features using the Transformer encoder.

[0070] S33: Perform global average pooling on the encoded features to obtain the global average pooling result, and generate the sparse cue based on the global average pooling result through the first layer multilayer perceptron.

[0071] S34: A dense location weight map is generated by performing a nonlinear transformation based on the encoded features using a second-layer multilayer perceptron, and the dense location weight map is upsampled to generate the dense prompt.

[0072] Specifically, multi-scale features are concatenated through channels to form unified semantic features, which are then input into a Transformer encoder for spatial relationship modeling, outputting encoded features containing global semantic associations. Global average pooling is applied to the encoded features to eliminate spatial dimensional differences. The compressed feature vectors are mapped to a sparse cue coordinate set through a first-layer multilayer perceptron, a process that focuses on the location information of key semantic regions. Simultaneously, the encoded features are input into a second-layer multilayer perceptron for nonlinear transformation, generating a low-resolution dense position weight map, which is then upsampled to the original image size via bilinear interpolation. This process reconstructs the detailed distribution features of the target region. Sparse and dense cuees provide segmentation guidance information from two dimensions: discrete keypoints and continuous regions, forming a complementary cue mechanism. This application solves the problem of inaccurate pseudo-cue generation leading to blurred segmentation edges in traditional Chinese painting scenes using general models. Sparse cuees accurately locate the main bird region, while dense cuees refine details such as feather texture. The synergistic effect of both results in a clear boundary contour for the initial segmentation mask, laying an accurate foundation for subsequent iterative optimization.

[0073] Sparse cues refer to discrete location indications generated through global statistical features. Specifically, this can be achieved by extracting spatially independent features using global average pooling and then mapping them through a fully connected layer. This method captures the core semantic features of salient regions in an image. Dense cues refer to a continuous weight distribution map covering all pixels in the image. This can be achieved by upsampling the encoded features after nonlinear transformation using a multilayer perceptron to restore resolution. This method preserves the spatial distribution information of target edge details. Unified semantic features refer to the fusion of multi-scale features. This can be achieved by aligning and connecting feature maps of different resolutions using channel concatenation. This method integrates semantic information from different receptive fields. The Transformer encoder refers to a sequence modeling module based on a self-attention mechanism. This can be achieved by stacking multiple layers of a standard Transformer structure. This method can establish semantic relationships across regions.

[0074] S4: If a user prompt is received, the user prompt and the initial pseudo-prompt are merged to generate a merged prompt.

[0075] Specifically, if a user prompt is received, the user prompt is merged with the initial pseudo-prompt to obtain a merged prompt, thus forming a complete prompt vector.

[0076] Please see Figure 5 , Figure 5 A specific implementation method following step S3 is shown below:

[0077] S3A: If the user prompt is not received, perform an initial mask prediction based on the initial pseudo-prompt and the enhanced multi-scale features to generate a new initial segmentation mask.

[0078] S3B: Based on the new initial segmentation mask and the enhanced multi-scale features, perform iterative optimization. When the iterative optimization is completed, generate a new target segmentation mask.

[0079] Specifically, when the system detects that the user has not provided interactive prompts, it first uses a pre-generated initial pseudo-prompt as a segmentation guidance signal, inputting it along with multi-scale features into the mask prediction module to generate an initial segmentation result. Then, it enters an iterative optimization phase. In each iteration, morphological opening and closing operations are performed based on the mask output from the previous round to extract the edge-corrected pseudo-prompt region. This pseudo-prompt is then fused with multi-scale features to generate an updated segmentation mask. This process, through multiple rounds of feature reuse and mask correction, gradually eliminates local errors in the initial segmentation while maintaining the semantic consistency of the multi-scale features. Finally, after reaching a preset number of iterations, the optimized target segmentation mask is output.

[0080] The initial pseudo-hint refers to a hint signal automatically generated by the system. Specifically, it can be obtained by using a multilayer perceptron to perform nonlinear transformation on the encoded features to generate a dense positional weight map, and then upsampling it to provide initial segmentation guidance when there is no user input. Iterative optimization refers to the process of gradually improving the segmentation accuracy through multiple iterations. Specifically, it can be achieved by using morphological operations to extract the current pseudo-hint and combining it with multi-scale features to generate an iterative segmentation mask. The termination condition is controlled by a preset number of iterations to ensure that the segmentation result converges to a stable state.

[0081] Furthermore, when the amount of data is small, manual methods and traditional image processing techniques based on threshold segmentation, region growing, or edge detection can be used to distinguish birds from the background using pixel grayscale or texture features.

[0082] S5: Based on the merged prompt and the enhanced multi-scale features, perform initial mask prediction to generate an initial segmentation mask.

[0083] Specifically, the initial segmentation mask is generated by performing initial mask prediction based on the merged cue and enhanced multi-scale features through the SAM2 (Segment Anything Model v2) decoder.

[0084] S6: Perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features. When the iterative optimization is completed, generate the target segmentation mask.

[0085] Specifically, iterative optimization refers to gradually refining the boundary through multiple rounds of mask correction, which can be achieved using morphological operations and feature feedback mechanisms, and is used to eliminate edge discontinuities caused by artistic brushstrokes.

[0086] Please see Figure 6 , Figure 6 A specific implementation of step S6 is shown below:

[0087] S61: Iterative optimization is performed based on the initial segmentation mask and the enhanced multi-scale features, and morphological operations are performed based on the previous output mask in each iteration to extract the current pseudo-hint.

[0088] S62: Generate the current iteration segmentation mask based on the current pseudo-hint and the enhanced multi-scale features, and perform the next iteration based on the current iteration segmentation mask until the number of iterations reaches the preset number, and then generate the target segmentation mask.

[0089] Specifically, after the initial segmentation mask is generated, the system enters a multi-round optimization phase. Each iteration first performs morphological opening and closing operations on the previously output mask to eliminate small holes and smooth boundaries, extracting a pseudo-cue region with spatial continuity. This pseudo-cue serves as prior location information and is fused across layers with enhanced multi-scale features. An attention mechanism is used to weight the contribution of features of different resolutions to the current region. The fused features are processed by the decoder to generate an updated segmentation mask, whose edge accuracy is gradually improved through iterations. When the system detects that the iteration counter has reached a preset threshold, the optimization process terminates and the final target segmentation mask is output. This application effectively solves the problem of boundary blurring caused by insufficient utilization of contextual information during the iterative optimization of the segmentation mask. The pseudo-cue generated by morphological operations can accurately identify effective target regions, avoiding the accumulation and amplification of noise interference in multiple iterations. The dynamic fusion mechanism of multi-scale features ensures the synergistic effect of features of different resolutions in the boundary correction process, significantly improving the segmentation accuracy of complex texture regions. The preset number of iterations ensures the convergence of the optimization process while avoiding over-smoothing of details due to over-optimization.

[0090] Morphological operations refer to the process of enhancing the spatial structural features of the mask, specifically through dilation, erosion, or combinations thereof, to eliminate isolated noise regions in the mask and enhance the continuity of the target boundary. False hints are intermediate guiding information extracted from the iterative process through morphological operations, specifically generated through boundary contour extraction or connected component analysis, used to correct the segmentation results in subsequent iterations. Multi-scale features refer to image semantic information containing different resolution levels, specifically implemented by extracting feature maps of different depths through a convolutional neural network encoder, used to provide complementary contextual information at different iteration stages. The preset number of iterations is a conditional parameter controlling the termination of the optimization process, specifically set to an integer range of 3 to 5, used to balance computational efficiency and segmentation accuracy. Furthermore, iterative optimization can be stopped when the mask change is less than a threshold (convergence), and the mask generated in the last iteration can be used as the target segmentation mask.

[0091] In this embodiment, an image to be segmented is acquired, and multi-scale features of the image are extracted. The multi-scale features are recalibrated to generate enhanced multi-scale features. The enhanced multi-scale features are then concatenated to generate unified semantic features, and pseudo-hints are generated based on these unified semantic features to obtain initial pseudo-hints. If a user hint is received, the user hint is merged with the initial pseudo-hint to generate a merged hint. An initial mask prediction is performed based on the merged hint and the enhanced multi-scale features to generate an initial segmentation mask. Iterative optimization is performed based on the initial segmentation mask and the enhanced multi-scale features. When the iterative optimization is complete, a target segmentation mask is generated. This embodiment of the invention effectively solves the problems of insufficient segmentation accuracy and poor interaction flexibility in complex scenes by using a multi-scale feature recalibration, pseudo-hint generation, and user hint fusion mechanism, combined with an iterative optimization strategy. This effectively improves the segmentation accuracy and interaction flexibility in complex scenes.

[0092] Please refer to Figure 7 As a response to the above Figure 1 To implement the method shown, this application provides an embodiment of an image segmentation apparatus, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0093] like Figure 7 As shown, the image segmentation device in this embodiment includes: a feature extraction module 71, a recalibration module 72, a pseudo-hint generation module 73, a hint merging module 74, an initial segmentation module 75, and a target segmentation mask generation module 76, wherein:

[0094] The feature extraction module 71 is used to acquire the image to be segmented and extract multi-scale features of the image to be segmented;

[0095] The recalibration module 72 is used to recalibrate the multi-scale features to generate enhanced multi-scale features.

[0096] The pseudo-prompt generation module 73 is used to perform feature concatenation on the enhanced multi-scale features to generate unified semantic features, and generate pseudo-prompts based on the unified semantic features to obtain initial pseudo-prompts;

[0097] The prompt merging module 74 is used to merge the user prompt with the initial pseudo prompt if a user prompt is received, to generate a merged prompt;

[0098] The initial segmentation module 75 is used to perform initial mask prediction based on the merged prompt and the enhanced multi-scale features to generate an initial segmentation mask.

[0099] The target segmentation mask generation module 76 is used to perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and generate a target segmentation mask when the iterative optimization is completed.

[0100] Furthermore, the recalibration module 72 includes:

[0101] A context information extraction unit is used to extract the context information of each scale feature in the multi-scale features;

[0102] A feature compression unit is used to compress the context information to generate bottleneck features;

[0103] The decoding feature generation unit is used to upsample the bottleneck feature to obtain the upsampled feature, and fuse the upsampled feature with the corresponding scale feature in the multi-scale feature to obtain the decoding feature;

[0104] The channel recalibration unit is used to perform channel recalibration based on the decoding features and the multi-scale features to generate the enhanced multi-scale features.

[0105] Furthermore, the channel recalibration unit includes:

[0106] The gated vector generation unit is used to perform average pooling calculation on the decoded features using a channel attention mechanism, generate an average pooling result, and generate a gated vector based on the average pooling result using a multilayer perceptron.

[0107] An activation unit is used to activate the gated vector using the Sigmoid activation function to obtain activated features.

[0108] The recalibrated feature generation unit is used to multiply the activated features with the multi-scale features channel by channel to generate recalibrated features;

[0109] The feature projection unit is used to project all the recalibrated features to the same number of channels to obtain the enhanced multi-scale features.

[0110] Furthermore, the pseudo-hint generation module 73 includes:

[0111] The feature concatenation unit is used to concatenate the enhanced multi-scale features to generate the unified semantic features.

[0112] The encoding feature generation unit is used to generate encoding features based on the unified semantic features through the Transformer encoder;

[0113] The sparse cue generation unit is used to perform global average pooling on the encoded features to obtain the global average pooling result, and generate the sparse cue based on the global average pooling result through a first-layer multilayer perceptron.

[0114] The dense cue generation unit is used to generate a dense position weight map by performing a nonlinear transformation based on the encoded features through a second-layer multilayer perceptron, and to perform upsampling processing on the dense position weight map to generate the dense cue.

[0115] Furthermore, the pseudo-hint generation module 73 also includes:

[0116] The new mask generation module is used to perform an initial mask prediction based on the initial pseudo-hint and the enhanced multi-scale features if the user prompt is not received, and generate a new initial segmentation mask.

[0117] The new segmentation mask generation module is used to perform iterative optimization based on the new initial segmentation mask and the enhanced multi-scale features. When the iterative optimization is completed, a new target segmentation mask is generated.

[0118] Furthermore, the target segmentation mask generation module 76 includes:

[0119] The gear position pseudo-hint unit is used to perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and to perform morphological operations based on the previous output mask in each iteration to extract the current pseudo-hint.

[0120] The optimization completion unit is used to generate the current iteration segmentation mask based on the current pseudo-hint and the enhanced multi-scale features, and to perform the next iteration based on the current iteration segmentation mask until the number of iterations reaches a preset number, and then generate the target segmentation mask.

[0121] Furthermore, the feature extraction module 71 includes:

[0122] A multi-scale feature extraction unit is used to acquire the image to be segmented, extract features from the image to be segmented by an encoder, and generate the multi-scale features, wherein the multi-scale features include a first high-resolution feature, a second medium-resolution feature, a third medium-resolution feature, and a fourth low-resolution feature.

[0123] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.

[0124] Computer device 8 includes a memory 81, a processor 82, and a network interface 83 that are interconnected via a system bus. It should be noted that... Figure 8 Only a computer device 8 with three components—memory 81, processor 82, and network interface 83—is shown. It should be understood that implementing all shown components is not required; more or fewer components may be implemented alternatively. Those skilled in the art will understand that this computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), and embedded devices.

[0125] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0126] The memory 81 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as the hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 8. Of course, the memory 81 may include both internal storage units and external storage devices of the computer device 8. In this embodiment, the memory 81 is typically used to store the operating system and various application software installed on the computer device 8, such as program code for image segmentation methods. In addition, the memory 81 may also be used to temporarily store various types of data that have been output or will be output.

[0127] In some embodiments, processor 82 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 82 is typically used to control the overall operation of the computer device 8. In this embodiment, processor 82 is used to run program code stored in memory 81 or process data, for example, to run the program code of the image segmentation method described above to implement various embodiments of the image segmentation method.

[0128] The network interface 83 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 8 and other electronic devices.

[0129] This application also provides another embodiment, namely, a computer-readable storage medium storing a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the image segmentation method described above.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0131] Obviously, the embodiments described above are merely some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the scope of this application. This application can be implemented in many different forms; rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of protection of this application.

Claims

1. An image segmentation method characterized by, The method comprises the following steps: acquire an image to be segmented, and extract multi-scale features of the image to be segmented; perform re-labeling processing on the multi-scale features to generate enhanced multi-scale features; perform feature stitching on the enhanced multi-scale features to generate unified semantic features, and generate initial pseudo prompts based on the unified semantic features; if a user prompt is received, merge the user prompt with the initial pseudo prompts to generate merged prompts; perform initial mask prediction based on the merged prompts and the enhanced multi-scale features to generate an initial segmentation mask; perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and generate a target segmentation mask when the iterative optimization is completed; the initial pseudo prompts include sparse prompts and dense prompts; the step of performing feature stitching on the enhanced multi-scale features to generate unified semantic features, and generating initial pseudo prompts based on the unified semantic features comprises: perform feature stitching on the enhanced multi-scale features to generate the unified semantic features; generate encoded features based on the unified semantic features through a Transformer encoder; perform global average pooling processing on the encoded features to obtain a global average pooling result, and generate the sparse prompts based on the global average pooling result through a first multi-layer perception; perform nonlinear transformation on the encoded features based on the encoded features through a second multi-layer perception to generate a dense position weight map, and perform upsampling processing on the dense position weight map to generate the dense prompts.

2. The image segmentation method of claim 1, wherein, the step of performing re-labeling processing on the multi-scale features to generate enhanced multi-scale features comprises: extract context information of each scale feature in the multi-scale features; perform feature compression on the context information to generate bottleneck features; perform upsampling on the bottleneck features to obtain upsampled features, and fuse the upsampled features with corresponding scale features in the multi-scale features to obtain decoded features; perform channel re-labeling on the decoded features and the multi-scale features to generate the enhanced multi-scale features.

3. The image segmentation method of claim 2, wherein, the step of performing channel re-labeling on the decoded features and the multi-scale features to generate the enhanced multi-scale features comprises: perform average pooling calculation on the decoded features through a channel attention mechanism to generate an average pooling result, and generate a gating vector based on the average pooling result through a multi-layer perception; perform activation processing on the gating vector through a Sigmoid activation function to obtain activated features; multiply the activated features with the multi-scale features channel by channel to generate re-labeled features; project all the re-labeled features to the same channel number to obtain the enhanced multi-scale features.

4. The image segmentation method of claim 1, wherein, after the step of performing feature stitching on the enhanced multi-scale features to generate unified semantic features, and generating initial pseudo prompts based on the unified semantic features, the method further comprises: if the user prompt is not received, perform initial mask prediction based on the initial pseudo prompts and the enhanced multi-scale features to generate a new initial segmentation mask; performing iterative optimization based on the new initial segmentation mask and the enhanced multi-scale features, and generating a new target segmentation mask when the iterative optimization is completed.

5. The image segmentation method of claim 1, wherein, The iterative optimization based on the initial segmentation mask and the enhanced multi-scale features includes: performing iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and performing morphological operations based on the output mask of the last iteration in each iteration to extract the current pseudo prompt; generating a current iteration segmentation mask according to the current pseudo prompt and the enhanced multi-scale features, and performing the next iteration based on the current iteration segmentation mask until the number of iterations reaches a preset number, and generating the target segmentation mask.

6. The image segmentation method according to any one of claims 1 to 5, characterized in that, The method includes: obtaining the to-be-segmented image, and extracting multi-scale features of the to-be-segmented image.

7. An image segmentation apparatus characterized by comprising: obtaining the to-be-segmented image, and extracting multi-scale features of the to-be-segmented image. The method includes: a feature extraction module configured to obtain a to-be-segmented image and extract multi-scale features of the to-be-segmented image; a re-calibration module configured to perform re-calibration processing on the multi-scale features to generate enhanced multi-scale features; a pseudo prompt generation module configured to perform feature concatenation on the enhanced multi-scale features to generate unified semantic features, and generate pseudo prompts based on the unified semantic features to obtain initial pseudo prompts; a prompt merging module configured to merge the user prompts with the initial pseudo prompts to generate merged prompts if the user prompts are received; an initial segmentation module configured to perform initial mask prediction based on the merged prompts and the enhanced multi-scale features to generate an initial segmentation mask; a target segmentation mask generation module configured to perform iterative optimization based on the initial segmentation mask and the enhanced multi-scale features, and generate a target segmentation mask when the iterative optimization is completed. The initial pseudo prompts include sparse prompts and dense prompts. The pseudo prompt generation module includes: a feature concatenation unit configured to perform feature concatenation on the enhanced multi-scale features to generate the unified semantic features; an encoded feature generation unit configured to generate encoded features based on the unified semantic features by using a Transformer encoder; a sparse prompt generation unit configured to perform global average pooling processing on the encoded features to obtain a global average pooling result, and generate the sparse prompts based on the global average pooling result by using a first multilayer perceptron; 8. A computer device, comprising: a dense prompt generation unit configured to perform nonlinear transformation on the encoded features by using a second multilayer perceptron to generate a dense position weight map, and perform up-sampling processing on the dense position weight map to generate the dense prompts. The device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the image segmentation method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the image segmentation method in any one of claims 1 to 6.

Images