High-precision Image Segmentation Method and Device Based on Feature Interaction and Fusion of Early and Late Features

Abstract

The present application provides a high-precision image segmentation method and apparatus based on feature interaction and early and late feature fusion, which relates to the field of computer vision technology and aims to capture global features and retain detailed information to achieve high-precision image segmentation. The method includes: obtaining a hint feature corresponding to hint information, and obtaining an image feature corresponding to an image to be segmented, where the hint information is used to represent a segmented area; fusing an interaction feature of the hint feature with the image feature to obtain a fused image feature; fusing the fused image feature with early and late features of the image feature, where the early feature has spatial detailed information and the late feature has global semantic information, to obtain an early and late fusion feature; fusing the interaction feature of the fused image feature with the early and late fusion feature to obtain a final image feature; and performing image segmentation based on the final image feature to obtain a segmentation result.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a high-precision image segmentation method and device based on feature interaction and front-end and back-end feature fusion. Background Art

[0002] High-precision image segmentation has important applications in computer vision, and is widely used in scenarios such as medical image processing, remote sensing image analysis, and target recognition. The key to achieving high-precision segmentation lies in fully leveraging the global correlation and local detail of features.

[0003] However, existing methods often struggle to strike a balance between capturing long-range dependencies and preserving local details, resulting in segmentation results with missing details or blurred boundaries in complex scenes. Therefore, how to capture global features while preserving detailed information to achieve high-precision image segmentation is a pressing technical challenge. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a high-precision image segmentation method and device based on feature interaction and front-end and back-end feature fusion, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a high-precision image segmentation method based on feature interaction and front-end and back-end feature fusion is disclosed, the method comprising:

[0006] Obtaining a prompt feature corresponding to the prompt information, and obtaining an image feature corresponding to the image to be segmented, wherein the prompt information is used to represent the segmented area;

[0007] fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features;

[0008] Fusing the fused image features with the early features and late features of the image features to obtain early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information;

[0009] Fusing the interactive features of the fused image features with the front-end and rear-end fusion features to obtain final image features, wherein the interactive features of the fused image features have fine-grained information between features;

[0010] Image segmentation is performed based on the final image features to obtain a segmentation result.

[0011] Optionally, the prompt feature includes a first type of feature and a second type of feature, where the first type of feature is used for image segmentation and the second type of feature is used to provide mask information; the method further includes:

[0012] Performing attention interaction on the first type of feature and the second type of feature to obtain a first interaction feature;

[0013] Fusing the first interaction feature with the first type of feature to obtain a first type of feature with mask information;

[0014] Using the first type of feature with mask information and the second type of feature as the interaction feature of the prompt feature.

[0015] Optionally, fusing a third type of feature and a fourth type of feature of the fused image feature, where the third type of feature is used for image segmentation and the fourth type of feature is used to provide mask information; the method further includes:

[0016] Performing attention interaction on the third type of feature and the fourth type of feature to obtain a second interaction feature;

[0017] Fusing the second interaction feature with the third type of feature to obtain a third type of feature with mask information;

[0018] Using the third type of feature with mask information and the fourth type of feature as the interaction feature of the fused image feature.

[0019] Optionally, fusing the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late fusion feature, including:

[0020] Fusing the fused image feature with the early-stage feature to obtain an updated early-stage feature;

[0021] Fusing the updated early-stage feature with the late-stage feature to obtain an early-late fusion feature.

[0022] Optionally, fusing the updated early-stage feature with the late-stage feature to obtain an early-late fusion feature, including:

[0023] Performing upsampling on the updated early-stage feature to obtain an updated early-stage feature with the same size as the late-stage feature;

[0024] Fusing the updated early-stage feature with the same size as the late-stage feature with the late-stage feature to obtain an early-late fusion feature.

[0025] Optionally, fusing the interaction feature of the prompt feature with the image feature to obtain a fused image feature, including:

[0026] Input the interaction features of the prompt features and the image features into multiple first attention interaction layers for processing to obtain first attention interaction features;

[0027] Input the first attention interaction features into a second attention interaction layer to obtain fused image features, where the second attention interaction layer is used for the interaction between image features and prompt features.

[0028] Optionally, fuse the interaction features of the fused image features with the front and back fused features to obtain final image features, including:

[0029] Process the interaction features of the fused image features into the same dimension as the front and back fused features to obtain dimension-processed features;

[0030] After performing convolutional downsampling on the dimension-processed features, fuse them with the front and back fused features to obtain final image features.

[0031] In the second aspect of the embodiments of the present application, a high-precision image segmentation device based on feature interaction and front and back feature fusion is disclosed. The device includes:

[0032] A first acquisition module, configured to acquire prompt features corresponding to prompt information and image features corresponding to an image to be segmented, where the prompt information is used to represent the segmented region;

[0033] A first fusion module, configured to fuse the interaction features of the prompt features with the image features to obtain fused image features, where the interaction features of the prompt features have explicit association information between features;

[0034] A second fusion module, configured to fuse the fused image features with the front and back features of the image features to obtain front and back fused features, where the front features have spatial detail information and the back features have global semantic information;

[0035] A third fusion module, configured to fuse the interaction features of the fused image features with the front and back fused features to obtain final image features, where the interaction features of the fused image features have fine-grained information between features;

[0036] A result segmentation module, configured to perform image segmentation based on the final image features to obtain a segmentation result.

[0037] In the third aspect of the embodiments of the present application, an electronic device is disclosed, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the high-precision image segmentation method based on feature interaction and front and back feature fusion described in the first aspect of the embodiments of the present application are implemented.

[0038] In the fourth aspect of the embodiments of the present application, a computer-readable storage medium is disclosed, on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and early-late feature fusion described in the first aspect of the embodiments of the present application are implemented.

[0039] In the fifth aspect of the embodiments of the present application, a computer program product is disclosed, including a computer program. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and early-late feature fusion described in the first aspect of the embodiments of the present application are implemented.

[0040] The embodiments of the present application have the following advantages:

[0041] In the embodiments of the present application, a hint feature corresponding to hint information is obtained, and an image feature corresponding to the image to be segmented is obtained, where the hint information is used to represent the segmented region; then, the interaction feature of the hint feature is fused with the image feature to obtain a fused image feature; the fused image feature is fused with the early and late features of the image feature to obtain an early-late fused feature, where the early feature has spatial detail information and the late feature has global semantic information; finally, the interaction feature of the fused image feature is fused with the early-late fused feature to obtain a final image feature; and image segmentation is performed based on the final image feature to obtain a segmentation result.

[0042] Since the interaction feature of the hint feature has explicit association information between features, fusing the interaction feature of the hint feature with the image feature enhances the ability of interactive modeling of the input features, and the interaction feature of the fused image feature has fine-grained information between features. Fusing the interaction feature of the fused image feature with the early-late fused feature enhances the ability of interactive modeling of the output features. Therefore, the balance problem between global modeling and local detail capture is solved, thereby improving the semantic consistency and boundary clarity of the segmentation result. At the same time, fusing the fused image feature with the early and late features of the image feature makes full use of the high-resolution spatial detail information in the early feature and the global semantic information in the late feature, effectively compensating for the loss of detail information caused by multiple downsamplings, and improving the segmentation performance of the model for small targets and complex scenes. Description of the Drawings

[0043] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 It is a flowchart of the steps of a high-precision image segmentation method based on feature interaction and fusion of early and late features provided by an embodiment of the present application;

[0045] Figure 2 It is a flowchart of the steps of another high-precision image segmentation method based on feature interaction and fusion of early and late features provided by an embodiment of the present application.

[0046] Figure 3 It is a schematic structural diagram of an image segmentation model provided by an embodiment of the present application;

[0047] Figure 4 It is a flowchart of obtaining the interaction features of prompt features provided by an embodiment of the present application;

[0048] Figure 5 It is a flowchart of obtaining the interaction features of fused image features provided by an embodiment of the present application;

[0049] Figure 6 It is a flowchart of obtaining the early and late fusion features provided by an embodiment of the present application;

[0050] Figure 7 It is a flowchart of obtaining the early and late fusion features in another way provided by an embodiment of the present application;

[0051] Figure 8 It is a schematic structural diagram of a high-precision image segmentation device based on feature interaction and fusion of early and late features provided by an embodiment of the present application;

[0052] Figure 9 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific embodiments

[0053] To make the above objects, features, and advantages of the present application more clearly understood, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present application belong to the scope of protection of the present application.

[0054] The existing high-precision image segmentation methods include the following five. The first high-precision image segmentation method uses an adapter and visual cues to achieve the purpose of adapting to different segmentation tasks without changing the main structure of the SAM model (Segment Anything Model, a model for high-precision image segmentation). The second high-precision image segmentation method uses a prompt-driven adapter to optimize the sparse and dense cues of SAM, aiming to improve the segmentation mask quality of SAM, so as to enhance the adaptability of the model to specific tasks. The third high-precision image segmentation method adopts a two-level optimization strategy to fine-tune the SAM model. By optimizing the weight parameters and prompt embeddings of the model on two different subsets of the training dataset respectively, overfitting is prevented, thereby improving the performance of the model in semantic segmentation tasks. The fourth high-precision image segmentation method improves the mask prediction quality by introducing a learnable high-quality output Token in the mask decoder of SAM, enabling the model to generate higher-quality segmentation masks, especially when dealing with objects with complex structures. The fifth high-precision image segmentation method combines a complementary convolutional neural network and a standard SAM network for medical image segmentation. By only fine-tuning the additional CNN network and the SAM decoder part, the training time is reduced and the performance of the model in medical image segmentation tasks is improved. These methods jointly promote the application and performance improvement of the SAM model in different fields.

[0055] Although the above methods can achieve high-precision image segmentation, they still face many challenges in capturing global features and retaining detailed information. On the one hand, they perform poorly in global feature modeling and are difficult to comprehensively capture long-range global dependencies. Especially when dealing with complex scenes or large-scale targets, the problem of missing context information is likely to occur. On the other hand, due to multiple downsampling operations during the segmentation process, the high-resolution spatial detail information is severely weakened, and the result often shows blurred boundaries or inaccurate segmentation of small targets. In addition, the decoder part of the existing methods lacks an explicit feature interaction mechanism, and the flow of generated feature information is restricted, making the model unable to fully exploit the potential feature expression ability, thus limiting the overall accuracy and detail characterization ability of the segmentation result. Therefore, these methods have become the main bottleneck for the further development of existing segmentation methods under complex scenarios and high-precision requirements.

[0056] To overcome the limitations of related technologies, the embodiments of this application provide a high-precision image segmentation method based on feature interaction and early-late feature fusion. By enhancing the model's ability to interactively model input and output features, the segmentation accuracy and generalization ability are significantly improved; moreover, the high-resolution spatial detail information in the early-stage features and the global semantic information in the late-stage features are fully utilized, effectively compensating for the loss of detail information caused by multiple downsamplings and improving the model's segmentation performance for small targets and complex scenes.

[0057] The following will combine the accompanying drawings to provide a detailed description of the high-precision image segmentation method based on feature interaction and early-late feature fusion according to the embodiments of the present application.

[0058] Referring to Figure 1 as shown Figure 1 is a flowchart of the steps of a high-precision image segmentation method based on feature interaction and early-late feature fusion provided by the embodiments of the present application. As Figure 1 shown, the high-precision image segmentation method based on feature interaction and early-late feature fusion may include steps S110 to S150:

[0059] Step S110: Obtain the prompt feature corresponding to the prompt information, and obtain the image feature corresponding to the image to be segmented, where the prompt information is used to characterize the segmented region.

[0060] In the embodiments of the present application, the prompt information may be prompt information in different forms. For example, the prompt information may be a box or a point. Obtaining the prompt feature corresponding to the prompt information may be obtained through a Prompt Encoder, that is, the prompt information is converted into a prompt feature through the prompt encoder, and the prompt feature is used to characterize the region to be segmented in the image.

[0061] Obtaining the image feature corresponding to the image to be segmented may be obtained through an Image Encoder, that is, the image to be segmented is converted into an image feature through the image encoder. Moreover, when obtaining the image feature corresponding to the image to be segmented through the image encoder, multi-layer features from shallow to deep can be obtained. Among them, the shallow features have a higher resolution and can retain more spatial details, while the deep features are often good at capturing global semantic information.

[0062] Step S120: Fuse the interaction feature of the prompt feature with the image feature to obtain a fused image feature, where the interaction feature of the prompt feature has explicit association information between features.

[0063] In the embodiments of the present application, the interaction feature of the prompt feature refers to the interaction feature obtained by performing cross-attention interaction on the prompt feature. In order to improve the model's ability to interactively model the input features, when performing mask decoding, the interaction feature of the prompt feature is used to fuse with the image feature to obtain a fused image feature. Among them, the fused image feature refers to the image feature containing mask information, that is, the mask probability of each image pixel can be generated through the fused image feature subsequently.

[0064] In some embodiments, fusing the interaction feature of the prompt feature with the image feature to obtain a fused image feature includes steps S120-1 to S120-2:

[0065] Step S120-1: Input the interaction feature of the prompt feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature.

[0066] Step S120-2: Input the first attention interaction feature into a second attention interaction layer to obtain a fused image feature, where the second attention interaction layer is used for the interaction between the image feature and the prompt feature.

[0067] In the embodiments of the present application, the first attention interaction layer refers to a Transformer layer, and the second attention interaction layer refers to a Token to image attn layer. Fusing the interaction feature of the prompt feature with the image feature can be implemented through a Mask Decoder. The plurality of first attention interaction layers and the second attention interaction layer are both included in the Mask Decoder. When performing mask decoding, first, the interaction feature of the input prompt feature and the image feature are processed by multiple first attention interaction layers to obtain a first attention interaction feature, and then the first attention interaction feature is processed by the second attention interaction layer.

[0068] Step S130: Fuse the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late fused feature, where the early-stage feature has spatial detail information and the late-stage feature has global semantic information.

[0069] In the embodiments of the present application, the image feature includes multiple layers of features from shallow to deep. Considering that in high-precision segmentation tasks, deep features are often good at capturing global semantic information but are prone to losing local details, especially in the expression of the boundaries of segmentation targets or small targets. On the contrary, due to the higher resolution, shallow features can retain more spatial details but lack the support of global context information.

[0070] To make up for this defect, the present application introduces the early-stage feature (shallow feature) and the late-stage feature (deep feature) of the image feature for early-late feature fusion. That is, after obtaining the fused image feature in step S120, instead of directly performing image segmentation, the fused image feature is fused with the early-stage feature and the late-stage feature of the image feature to obtain an early-late fused feature. Among them, the early-late fused feature is a feature that contains mask information and has both global semantic information and spatial detail information. In this way, through this method, not only the modeling ability of the Mask Decoder for global semantic information is enhanced, but also the retention effect of boundary details is significantly improved, thereby achieving a more accurate and robust segmentation result.

[0071] Step S140: Fuse the interaction feature of the fused image feature with the front-and-back fused feature to obtain the final image feature, where the interaction feature of the fused image feature has fine-grained information between features.

[0072] In the embodiment of the present application, the interaction feature of the fused image feature refers to the interaction feature obtained by performing cross attention on the fused image feature. To further enhance the global modeling ability and detail expression ability of the feature, after obtaining the front-and-back fused feature, the interaction feature with the fused image feature is also used to fuse with the front-and-back fused feature to obtain the final image feature. Since the interaction feature of the fused image feature has fine-grained information between features, the interaction modeling ability of the output feature is enhanced, and the balance problem between global modeling and local detail capture is further solved.

[0073] Step S150: Perform image segmentation based on the final image feature to obtain a segmentation result.

[0074] In the embodiment of the present application, after obtaining the final image feature, the mask probability of each image pixel can be generated based on the final image feature, and then the segmentation result can be obtained according to the mask probability of each image pixel. For example, if the mask probability threshold is 0.6, the pixel region with a mask probability greater than 0.6 is determined as the mask region (or segmentation region).

[0075] Adopting the technical solution of the embodiment of the present application, since the interaction feature of the prompt feature has explicit association information between features, fusing the interaction feature of the prompt feature with the image feature enhances the interaction modeling ability of the input feature, and the interaction feature of the fused image feature has fine-grained information between features. Fusing the interaction feature of the fused image feature with the front-and-back fused feature enhances the interaction modeling ability of the output feature. Therefore, the balance problem between global modeling and local detail capture is solved, thereby improving the semantic consistency and boundary clarity of the segmentation result. At the same time, fusing the fused image feature with the early and late features of the image feature makes full use of the high-resolution spatial detail information in the early feature and the global semantic information in the late feature, effectively compensating for the loss of detail information caused by multiple downsamplings, and improving the segmentation performance of the model for small targets and complex scenes.

[0076] Combined with the above embodiments, the embodiments of the present application provide a high-precision image segmentation method based on feature interaction and front-back period feature fusion. In this method, the prompt features include a first type of feature and a second type of feature. The first type of feature is used for image segmentation, and the second type of feature is used to provide mask information. Before performing step S120, the interaction features of the prompt features are obtained through the following steps, specifically including:

[0077] Perform attention interaction on the first type of feature and the second type of feature to obtain a first interaction feature; fuse the first interaction feature with the first type of feature to obtain a first type of feature with mask information; use the first type of feature with mask information and the second type of feature as the interaction features of the prompt features.

[0078] In the embodiments of the present application, cross-attention interaction (i.e., crossattention) is performed on the two types of features in the prompt features to establish an explicit association between the features, so as to better achieve high-precision image segmentation.

[0079] Exemplarily, performing attention interaction on the first type of feature and the second type of feature can be expressed as:

[0080] ,

[0081] ,

[0082] Among them, represents the first interaction feature; represents the first type of feature, that is, the newly introduced high-quality segmentation token; represents the second type of feature, that is, the mask token of the SAM model; represents the feature dimension; 、 、 all represent weight parameters, represents the query linear mapping layer, represents the key linear mapping layer, represents the query (Value) linear mapping layer.

[0083] Fusing the first interaction feature with the first type of feature is equivalent to introducing the information of the mask token into the first type of feature to obtain a first type of feature with mask information. Exemplarily, fusing the first interaction feature with the first type of feature can be expressed as:

[0084] ,

[0085] Among them, represents the first type of feature with mask information.

[0086] Finally, the first type of features and the second type of features containing mask information are used as the interaction features of the prompt features. Since the first type of features for image segmentation in the interaction features of the prompt features contains mask information, the interaction features based on the prompt features can better achieve image segmentation.

[0087] In a specific implementation, obtaining the interaction features of the prompt features can be achieved based on the first feature interaction module. Specifically, after obtaining the prompt features corresponding to the prompt information, the two types of features of the prompt features (i.e., the first type of features and the second type of features) are input into the first feature interaction module for cross attention to obtain the first type of features containing mask information, so as to use the first type of features containing mask information and the second type of features as the interaction features of the prompt features.

[0088] Combined with the above implementation manners, the embodiments of the present application provide a high-precision image segmentation method based on feature interaction and front-back period feature fusion. In this method, the fused image features include a third type of features and a fourth type of features. The third type of features is used for image segmentation, and the fourth type of features is used to provide mask information. Before performing step S140, the interaction features of the fused image features are also obtained through the following steps, specifically including:

[0089] Performing attention interaction on the third type of features and the fourth type of features to obtain a second interaction feature; fusing the second interaction feature with the third type of features to obtain the third type of features containing mask information; using the third type of features containing mask information and the fourth type of features as the interaction features of the fused image features.

[0090] In the embodiments of the present application, by performing cross attention interaction (i.e., cross attention) on two types of features in the fused image features to optimize the output fine-grained interaction, high-precision image segmentation can be better achieved.

[0091] Exemplarily, performing attention interaction on the third type of features and the fourth type of features can be expressed as:

[0092] ,

[0093] ,

[0094] where represents the second interaction feature; represents the third type of features, that is, the output high-quality segmentation token; represents the fourth type of features, that is, the mask token of the SAM model; represents the feature dimension; , , both represent weight parameters, represents the query linear mapping layer, represents the key linear mapping layer, represents the value linear mapping layer.

[0095] Fusing the second interaction feature with the third type of feature is equivalent to introducing the information of the masked token into the third type of feature, obtaining the third type of feature with masked information. Exemplarily, fusing the second interaction feature with the third type of feature can be expressed as:

[0096] ,

[0097] wherein, represents the third type of feature with masked information.

[0098] Finally, the third type of feature with masked information and the fourth type of feature are used as the interaction feature of the fused image feature. Since in the interaction feature of the fused image feature, the third type of feature for image segmentation contains masked information, image segmentation can be better realized based on the interaction feature of the fused image feature.

[0099] In specific implementation, obtaining the interaction feature of the fused image feature can be realized based on the second feature interaction module. Specifically, after obtaining the fused image feature, the two types of features (i.e., the third type of feature and the fourth type of feature) of the fused image feature are input into the second feature interaction module for cross attention to obtain the third type of feature with masked information, so that the third type of feature with masked information and the fourth type of feature are used as the interaction feature of the fused image feature.

[0100] Combined with the above embodiments, the embodiments of the present application provide a high-precision image segmentation method based on feature interaction and early and late feature fusion. In this method, in the above step S130, "fusing the fused image feature with the early feature and the late feature of the image feature to obtain the early and late fused feature" specifically includes steps S130-1 to S130-2:

[0101] Step S130-1: Fusing the fused image feature with the early feature to obtain the updated early feature.

[0102] Step S130-2: Fusing the updated early feature with the late feature to obtain the early and late fused feature.

[0103] In the embodiments of the present application, the early feature is denoted as early_feature ( ), and the later feature is denoted as later_feature ( ), after obtaining the fused image feature, fuse the early feature and the later feature with the fused image feature.

[0104] First, execute step S130-1 to update the early feature to obtain an updated early feature. Exemplarily, fusing the fused image feature with the early feature can be expressed as:

[0105] ,

[0106] where, represents the updated early feature, fused image feature, represents a convolution operation.

[0107] Next, execute step S130-2 to fuse the updated early feature with the later feature to obtain a fused early and later feature. Specifically, perform upsampling on the updated early feature to obtain an updated early feature with the same size as the later feature; fuse the updated early feature with the same size as the later feature with the later feature to obtain a fused early and later feature.

[0108] Among them, the updated early feature can be upsampled to the same size as the later feature through transposed convolution. Performing upsampling on the updated early feature can be expressed as:

[0109] ,

[0110] where, represents the updated early feature with the same size as the later feature, represents transposed convolution.

[0111] Finally, fuse (add) the updated early feature with the same size as the later feature with the later feature to obtain a fused early and later feature, where the fused early and later feature is expressed as:

[0112] ,

[0113] In the embodiment of the present application, since the early feature has spatial detail information and the later feature has global semantic information, by fusing the fused image feature with the early and later features of the image feature, the obtained fused early and later feature is a feature that contains mask information and has both global semantic information and spatial detail information. In this way, the ability of the mask decoder to model global semantic information is enhanced, and the retention effect of boundary details is significantly improved, realizing a more accurate and robust segmentation result.

[0114] In specific implementation, the fusion of the fused image feature with the early and late features of the image feature can be achieved through the early-late fusion module. Specifically, after obtaining the fused image feature, the fused image feature, the early feature, and the late feature are input into the early-late fusion module for feature fusion, so as to obtain the early-late fusion feature.

[0115] Combined with the above embodiments, the embodiments of the present application provide a high-precision image segmentation method based on feature interaction and early-late feature fusion. In this method, in the above step S140, "fusing the interaction feature of the fused image feature with the early-late fusion feature to obtain the final image feature" specifically includes steps S140-1 to step S140-2:

[0116] Step S140-1: Process the interaction feature of the fused image feature into the same dimension as the early-late fusion feature to obtain the dimension-processed feature.

[0117] Step S140-2: After performing convolutional downsampling on the dimension-processed feature, fuse it with the early-late fusion feature to obtain the final image feature.

[0118] In the embodiments of the present application, the interaction feature of the fused image feature is processed into the same dimension as the early-late fusion feature through transposed convolution. The dimension-processed feature can be expressed as:

[0119] ,

[0120] where represents the interaction feature of the fused image feature, represents the dimension-processed feature, represents transposed convolution.

[0121] Exemplarily, the final image feature can be expressed as:

[0122] ,

[0123] where represents the convolution operation (convolutional downsampling).

[0124] Through the above process, based on the interaction feature of the fused image feature and the early-late fusion feature, the final image feature is obtained. Since the interaction feature of the fused image feature is obtained through attention interaction between two types of features it contains and has fine-grained information between features, and the early-late fusion feature is a feature that contains mask information and has global semantic information and spatial detail information, the final image feature contains rich detail information, global information, and mask information. Therefore, high-quality image segmentation can be achieved based on the final image feature.

[0125] The following uses a specific embodiment to illustrate the high-precision image segmentation method based on feature interaction and early-late feature fusion in the application embodiment.

[0126] Referring to Figure 2 as shown, Figure 2 is the flowchart of steps of another high-precision image segmentation method based on feature interaction and early-late feature fusion provided by the application embodiment. Specifically, it includes the following steps S210 to step S270:

[0127] Step S210: Obtain the hint feature corresponding to the hint information, and obtain the image feature corresponding to the image to be segmented, where the hint information is used to represent the segmented region.

[0128] Step S220: The hint feature includes a first type of feature and a second type of feature. Perform attention interaction between the first type of feature and the second type of feature to obtain a first interaction feature; fuse the first interaction feature with the first type of feature to obtain a first type of feature with mask information; use the first type of feature with mask information and the second type of feature as the interaction feature of the hint feature.

[0129] Step S230: Fuse the interaction feature of the hint feature with the image feature to obtain a fused image feature, and the interaction feature of the hint feature has explicit association information between features.

[0130] Step S240: The fused image feature includes a third type of feature and a fourth type of feature. Perform attention interaction between the third type of feature and the fourth type of feature to obtain a second interaction feature; fuse the second interaction feature with the third type of feature to obtain a third type of feature with mask information; use the third type of feature with mask information and the fourth type of feature as the interaction feature of the fused image feature.

[0131] Step S250: Fuse the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late fused feature, where the early-stage feature has spatial detail information and the late-stage feature has global semantic information.

[0132] Step S260: Fuse the interaction feature of the fused image feature with the early-late fused feature to obtain the final image feature, and the interaction feature of the fused image feature has fine-grained information between features.

[0133] Step S270: Perform image segmentation based on the final image feature to obtain a segmentation result.

[0134] In the embodiments of the present application, by performing cross-attention interaction on two types of features in the prompt features to establish explicit associations between the features, and by performing cross-attention interaction on two types of features in the fused image features, the fine-grained interaction of the output is optimized, thereby enhancing the interaction modeling ability of the input features and the output features, thus solving the balance problem between global modeling and local detail capture, and improving the semantic consistency and boundary clarity of the segmentation result.

[0135] Moreover, by fusing the fused image features with the early-stage features and late-stage features of the image features, the high-resolution spatial detail information in the early-stage features and the global semantic information in the late-stage features are fully utilized, effectively compensating for the loss of detail information caused by multiple downsamplings, and improving the segmentation performance of the model for small targets and complex scenes.

[0136] In this way, the method not only fully utilizes the detail advantages of the early-stage features and the global information of the late-stage features, but also improves the overall segmentation effect through feature interaction, providing a feasible implementation method for high-precision segmentation in complex scenes.

[0137] Exemplarily, the high-precision image segmentation method based on feature interaction and early-late stage feature fusion can be implemented based on an image segmentation model, such as Figure 3 shown, Figure 3 is a schematic structural diagram of an image segmentation model provided by an embodiment of the present application. Among them, the image segmentation model includes an image encoder, a prompt encoder, and a mask decoder. When performing image segmentation, the prompt information and the image to be segmented are input into the image segmentation model. The prompt information is converted into prompt features by the prompt encoder, and the image to be segmented is converted into image features by the image encoder. Then, the prompt features and the image features are input into the mask decoder for decoding to obtain the segmentation result.

[0138] Among them, the mask decoder includes: a first feature interaction module, multiple first attention interaction layers (Transformer), a second attention interaction layer (Token to image attn), a second feature interaction module, an early-late stage feature fusion module, and a transposed convolutional layer (Trans Conv). The specific decoding process is as follows: First, use the first feature interaction module to obtain the interaction features of the prompt features. As Figure 4 shown, the prompt features include a first type of feature and a second type of feature. The two types of features of the prompt features are input into the first feature interaction module for cross attention to obtain the first type of feature containing mask information. Thus, the first type of feature containing mask information and the second type of feature are used as the interaction features of the prompt features.

[0139] Secondly, the interaction features of the prompt features and the image features are input into multiple first attention interaction layers for processing to obtain first attention interaction features, and the first attention interaction features are input into a second attention interaction layer to obtain fused image features.

[0140] Next, the second feature interaction module is used to obtain the interaction features of the fused image features. As Figure 5 shown, the fused image features include a third type of feature and a fourth type of feature. Furthermore, the two types of features of the fused image features are input into the second feature interaction module for cross attention to obtain the third type of feature with masked information. Thus, the third type of feature with masked information and the fourth type of feature are used as the interaction features of the fused image features.

[0141] After that, as Figure 6 shown, the fused image features, the early-stage features, and the late-stage features of the image features are input into the early-late stage feature fusion module for feature fusion to obtain early-late stage fusion features. Finally, the interaction features of the fused image features are fused with the early-late stage fusion features to obtain the final image features, and image segmentation is performed based on the final image features to obtain the segmentation result.

[0142] Specifically, as Figure 7 shown, the process of fusing the early-late stage fusion features and obtaining the final image features is as follows: The fused image features are fused with the early-stage features to obtain updated early-stage features, and the updated early-stage features are upsampled to obtain updated early-stage features of the same size as the late-stage features; The updated early-stage features of the same size as the late-stage features are fused with the late-stage features to obtain early-late stage fusion features. After that, the interaction features of the fused image features are processed by transposed convolution to the same dimension as the early-late stage fusion features to obtain dimension-processed features; And after the dimension-processed features are convolved and downsampled, they are fused with the early-late stage fusion features to obtain the final image features.

[0143] In this way, through the innovative masked decoder structure and feature fusion method, the performance of the high-precision image segmentation task is significantly improved. By introducing the feature interaction module (Token-to-Token Attention mechanism), the model's ability to interactively model the input and output features is effectively enhanced, solving the balance problem between global modeling and local detail capture, thereby improving the semantic consistency and boundary clarity of the segmentation result. At the same time, the early-late stage feature fusion decoder combines the early-stage features of the image encoder and the deep semantic features of the late stage, making full use of the high-resolution spatial information in the early-stage features and the global semantic information in the late stage features, effectively compensating for the loss of detail information caused by multiple downsamplings in the traditional decoder, and improving the model's segmentation performance for small targets and complex scenes.

[0144] The embodiment of the present application further provides a high-precision image segmentation device based on feature interaction and early-late feature fusion. Refer to Figure 8 as shown in Figure 8 FIG. Figure 8 is a schematic structural diagram of a high-precision image segmentation device based on feature interaction and early-late feature fusion provided by an embodiment of the present application. The device includes:

[0145] A first acquisition module 810, configured to acquire a hint feature corresponding to hint information, and acquire an image feature corresponding to an image to be segmented, where the hint information is used to represent a segmented region;

[0146] A first fusion module 820, configured to fuse the interaction feature of the hint feature with the image feature to obtain a fused image feature, where the interaction feature of the hint feature has explicit association information between features;

[0147] A second fusion module 830, configured to fuse the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late fusion feature, where the early-stage feature has spatial detail information and the late-stage feature has global semantic information;

[0148] A third fusion module 840, configured to fuse the interaction feature of the fused image feature with the early-late fusion feature to obtain a final image feature, where the interaction feature of the fused image feature has fine-grained information between features;

[0149] A result segmentation module 850, configured to perform image segmentation based on the final image feature to obtain a segmentation result.

[0150] In an optional embodiment, the hint feature includes a first type of feature and a second type of feature. The first type of feature is used for image segmentation, and the second type of feature is used to provide mask information;

[0151] The device further includes:

[0152] A first interaction module, configured to perform attention interaction on the first type of feature and the second type of feature to obtain a first interaction feature; fuse the first interaction feature with the first type of feature to obtain a first type of feature with mask information; use the first type of feature with mask information and the second type of feature as the interaction feature of the hint feature.

[0153] In an optional embodiment, the fused image feature includes a third type of feature and a fourth type of feature. The third type of feature is used for image segmentation, and the fourth type of feature is used to provide mask information;

[0154] The device further includes:

[0155] The first interaction module is used to perform attention interaction between the third category features and the fourth category features to obtain a second interaction feature; fuse the second interaction feature with the third category features to obtain a third category feature containing mask information; and use the third category features containing mask information and the fourth category features as interaction features of the fused image features.

[0156] In an optional embodiment, the second fusion module includes:

[0157] A first fusion submodule is used to fuse the fused image features with the previous features to obtain updated previous features;

[0158] The second fusion submodule is used to fuse the updated early features with the late features to obtain early and late fusion features.

[0159] In an optional embodiment, the second fusion submodule is further used to upsample the early update features to obtain early update features of the same size as the late update features; and fuse the early update features of the same size as the late update features with the late features to obtain front-end and late fusion features.

[0160] In an optional embodiment, the first fusion module includes:

[0161] A first input module is configured to input the interaction feature of the prompt feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature;

[0162] The second input module is used to input the first attention interaction feature into the second attention interaction layer to obtain a fused image feature. The second attention interaction layer is used for the interaction between the image feature and the prompt feature.

[0163] In an optional embodiment, the third fusion module includes:

[0164] A dimension processing module, configured to process the interactive features of the fused image features into the same dimension as the front-end and rear-end fusion features, thereby obtaining dimension-processed features;

[0165] The third fusion submodule is used to perform convolution downsampling on the dimension processing features and then fuse them with the front and rear fusion features to obtain the final image features.

[0166] The present application also provides an electronic device, Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 9As shown in the figure, the electronic device 900 includes: a memory 910 and a processor 920. The memory 910 and the processor 920 are communicatively connected via a bus. A computer program is stored in the memory 910, and this computer program can run on the processor 920, thereby implementing the steps of the high-precision image segmentation method based on feature interaction and front-and-back period feature fusion described in the embodiments of the present application.

[0167] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and front-and-back period feature fusion described in the embodiments of the present application are implemented.

[0168] The embodiments of the present application also provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and front-and-back period feature fusion described in the embodiments of the present application are implemented.

[0169] Each embodiment in this specification is described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other.

[0170] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0171] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the specified functions in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one process or multiple processes and / or blocks. Figure 1 One process or multiple processes and / or blocks Figure 1 Steps for implementing the functions specified in one block or multiple blocks.

[0173] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0174] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the said element.

[0175] The above has introduced in detail a high-precision image segmentation method and device based on feature interaction and pre- and post-stage feature fusion provided by the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A high-precision image segmentation method based on feature interaction and fusion of early and late features, characterized in that, The method includes: Obtaining a hint feature corresponding to hint information, and obtaining an image feature corresponding to an image to be segmented, where the hint information is used to represent a segmented region, the hint feature includes a first type of feature and a second type of feature, the first type of feature is used for image segmentation, and the second type of feature is used to provide mask information; Performing attention interaction between the first type of feature and the second type of feature to obtain a first interaction feature, fusing the first interaction feature with the first type of feature to obtain a first type of feature with mask information, and using the first type of feature with mask information and the second type of feature as the interaction feature of the hint feature; Fusing the interaction feature of the hint feature with the image feature to obtain a fused image feature, where the interaction feature of the hint feature has explicit association information between features, and the fused image feature includes a third type of feature and a fourth type of feature, the third type of feature is used for image segmentation, and the fourth type of feature is used to provide mask information; Performing attention interaction between the third type of feature and the fourth type of feature to obtain a second interaction feature, fusing the second interaction feature with the third type of feature to obtain a third type of feature with mask information, and using the third type of feature with mask information and the fourth type of feature as the interaction feature of the fused image feature; Fusing the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late-stage fused feature, where the early-stage feature has spatial detail information and the late-stage feature has global semantic information; Fusing the interaction feature of the fused image feature with the early-late-stage fused feature to obtain a final image feature, where the interaction feature of the fused image feature has fine-grained information between features; Performing image segmentation based on the final image feature to obtain a segmentation result.

2. The method according to claim 1, wherein Fusing the fused image feature with the early-stage feature and the late-stage feature of the image feature to obtain an early-late-stage fused feature, including: Fusing the fused image feature with the early-stage feature to obtain an updated early-stage feature; Fusing the updated early-stage feature with the late-stage feature to obtain an early-late-stage fused feature.

3. The method according to claim 2, wherein Fusing the updated early-stage feature with the late-stage feature to obtain an early-late-stage fused feature, including: Performing upsampling processing on the updated early-stage feature to obtain an updated early-stage feature with the same size as the late-stage feature; Fusing the updated early-stage feature with the same size as the late-stage feature with the late-stage feature to obtain an early-late-stage fused feature.

4. The method according to any one of claims 1-3, characterized in that, Fusing the interaction feature of the hint feature with the image feature to obtain a fused image feature, including: Inputting the interaction feature of the hint feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature; Inputting the first attention interaction feature into a second attention interaction layer to obtain a fused image feature, where the second attention interaction layer is used for interaction between the image feature and the hint feature.

5. The method according to any one of claims 1-3, characterized in that Fusing the interaction feature of the fused image feature with the early-late-stage fused feature to obtain a final image feature, including: Process the interaction features integrating the image features to the same dimension as the pre-and post-stage fusion features to obtain dimension-processed features; After performing convolutional downsampling on the dimension-processed features, fuse them with the pre-and post-stage fusion features to obtain the final image features.

6. A high-precision image segmentation device based on feature interaction and fusion of front and back period features, characterized in that, The device includes: A first acquisition module, configured to acquire hint features corresponding to hint information and image features corresponding to an image to be segmented, where the hint information is used to represent a segmented region, the hint features include first-type features and second-type features, the first-type features are used for image segmentation, and the second-type features are used to provide mask information; A first interaction module, configured to perform attention interaction between the first-type features and the second-type features to obtain first interaction features, fuse the first interaction features with the first-type features to obtain first-type features with mask information, and use the first-type features with mask information and the second-type features as interaction features of the hint features; A first fusion module, configured to fuse the interaction features of the hint features with the image features to obtain fused image features, the interaction features of the hint features have explicit association information between features, the fused image features include third-type features and fourth-type features, the third-type features are used for image segmentation, and the fourth-type features are used to provide mask information; A second interaction module, configured to perform attention interaction between the third-type features and the fourth-type features to obtain second interaction features, fuse the second interaction features with the third-type features to obtain third-type features with mask information, and use the third-type features with mask information and the fourth-type features as interaction features of the fused image features; A second fusion module, configured to fuse the fused image features with the early-stage features and late-stage features of the image features to obtain pre-and post-stage fusion features, the early-stage features have spatial detail information, and the late-stage features have global semantic information; A third fusion module, configured to fuse the interaction features of the fused image features with the pre-and post-stage fusion features to obtain the final image features, the interaction features of the fused image features have fine-grained information between features; A result segmentation module, configured to perform image segmentation based on the final image features to obtain a segmentation result.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the high-precision image segmentation method based on feature interaction and pre-and post-stage feature fusion according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the high-precision image segmentation method based on feature interaction and pre-and post-stage feature fusion according to any one of claims 1-5.

Images