High-precision image segmentation method and device based on feature interaction and front and later feature fusion

Abstract

The invention provides a high-precision image segmentation method and device based on feature interaction and previous and later feature fusion, relates to the technical field of computer vision, and aims to capture global features and reserve detail information to realize high-precision image segmentation. The method comprises the steps that prompt features corresponding to prompt information are acquired, image features corresponding to a to-be-segmented image are acquired, and the prompt information is used for representing a segmented area; fusing the interaction feature of the prompt feature with the image feature to obtain a fused image feature; the fused image features are fused with the earlier-stage features and the later-stage features of the image features, earlier-stage and later-stage fused features are obtained, the earlier-stage features have spatial detail information, and the later-stage features have global semantic information; fusing the interaction features of the fused image features with the previous and later fusion features to obtain final image features; and performing image segmentation based on the final image features to obtain a segmentation result.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a high-precision image segmentation method and device based on feature interaction and front-end and rear-end feature fusion. Background Art

[0002] High-precision image segmentation tasks have important applications in the field of computer vision and are widely used in scenarios such as medical image processing, remote sensing image analysis, and target recognition. The key to achieving high-precision segmentation is to make full use of the global correlation and local detail information of features.

[0003] However, existing methods usually have difficulty in striking a balance between capturing long-range dependencies and preserving local details, resulting in segmentation results with missing details or blurred boundaries in complex scenes. Therefore, how to capture global features and retain detail information to achieve high-precision image segmentation is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide a high-precision image segmentation method and device based on feature interaction and front-end and post-stage feature fusion, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion is disclosed, the method comprising: Acquire a prompt feature corresponding to the prompt information, and acquire an image feature corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area; fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features; The fused image feature is fused with the early features and late features of the image feature to obtain the early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information; The interactive features of the fused image features are fused with the front and rear fused features to obtain the final image features, wherein the interactive features of the fused image features have fine-grained information between the features; Image segmentation is performed based on the final image features to obtain a segmentation result.

[0006] Optionally, the prompt features include first-category features and second-category features, the first-category features are used for image segmentation, and the second-category features are used for providing mask information; the method further includes: Performing attention interaction between the first type of features and the second type of features to obtain a first interaction feature; The first interactive feature is merged with the first category feature to obtain the first category feature containing mask information; The first type of features containing mask information and the second type of features are used as interactive features of the prompt features.

[0007] Optionally, the fused image features are third-category features and fourth-category features, the third-category features are used for image segmentation, and the fourth-category features are used for providing mask information; the method further comprises: Performing attention interaction between the third type of features and the fourth type of features to obtain a second interaction feature; Fusing the second interactive feature with the third type of feature to obtain a third type of feature containing mask information; The third type of features containing mask information and the fourth type of features are used as interactive features of the fused image features.

[0008] Optionally, the fused image feature is fused with the early feature and the late feature of the image feature to obtain the early and late fused feature, including: Fusing the fused image features with the previous features to obtain updated previous features; The updated early features are fused with the late features to obtain early and late fused features.

[0009] Optionally, the updating early feature is fused with the late feature to obtain the early and late fused features, including: Performing upsampling processing on the pre-update feature to obtain the pre-update feature having the same size as the post-update feature; The updated early features having the same size as the late features are fused with the late features to obtain early and late fused features.

[0010] Optionally, fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature includes: Inputting the interaction feature of the prompt feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature; The first attention interaction feature is input into the second attention interaction layer to obtain a fused image feature, and the second attention interaction layer is used for the interaction between the image feature and the prompt feature.

[0011] Optionally, the interactive features of the fused image features are fused with the front and rear fusion features to obtain the final image features, including: Processing the interactive features of the fused image features into the same dimension as the front and rear fusion features to obtain dimensionally processed features; After convolution downsampling the dimension processing features, they are fused with the front and rear fusion features to obtain the final image features.

[0012] In a second aspect of the embodiments of the present application, a high-precision image segmentation device based on feature interaction and front-end and rear-end feature fusion is disclosed, the device comprising: A first acquisition module is used to acquire a prompt feature corresponding to the prompt information and to acquire an image feature corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area; a first fusion module, used for fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features; A second fusion module is used to fuse the fused image features with the early features and late features of the image features to obtain early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information; A third fusion module is used to fuse the interactive features of the fused image features with the front and rear fusion features to obtain the final image features, wherein the interactive features of the fused image features have fine-grained information between features; The result segmentation module is used to perform image segmentation based on the final image features to obtain a segmentation result.

[0013] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the high-precision image segmentation method based on feature interaction and front- and late-stage feature fusion described in the first aspect of the embodiment of the present application are implemented.

[0014] A fourth aspect of an embodiment of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and front- and late-stage feature fusion described in the first aspect of the embodiment of the present application are implemented.

[0015] In a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the high-precision image segmentation method based on feature interaction and front- and late-stage feature fusion described in the first aspect of the embodiment of the present application.

[0016] The embodiments of the present application include the following advantages: In an embodiment of the present application, a prompt feature corresponding to the prompt information is obtained, and an image feature corresponding to the image to be segmented is obtained, wherein the prompt information is used to characterize the segmented area; then, the interactive feature of the prompt feature is fused with the image feature to obtain a fused image feature; the fused image feature is fused with the early feature and the late feature of the image feature to obtain a front-late fused feature, wherein the early feature has spatial detail information and the late feature has global semantic information; finally, the interactive feature of the fused image feature is fused with the front-late fused feature to obtain a final image feature; and image segmentation is performed based on the final image feature to obtain a segmentation result.

[0017] Since the interactive features of the prompt features have explicit correlation information between features, the interactive features of the prompt features are fused with the image features to enhance the interactive modeling capability of the input features, and the interactive features of the fused image features have fine-grained information between features, and the interactive features of the fused image features are fused with the front and rear fusion features to enhance the interactive modeling capability of the output features, thus solving the difficult problem of balancing global modeling and capturing local details, thereby improving the semantic consistency and boundary clarity of the segmentation results. At the same time, the fused image features are fused with the early and late features of the image features, making full use of the high-resolution spatial detail information in the early features and the global semantic information in the late features, effectively compensating for the loss of detail information caused by multiple downsampling, and improving the segmentation performance of the model for small targets and complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of the steps of a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion provided in an embodiment of the present application; Figure 2 It is a flowchart of the steps of another high-precision image segmentation method based on feature interaction and front- and rear-stage feature fusion provided in an embodiment of the present application.

[0020] Figure 3 It is a structural schematic diagram of an image segmentation model provided in an embodiment of the present application; Figure 4 It is a flow chart of an interactive feature for obtaining a prompt feature provided by an embodiment of the present application; Figure 5It is a flow chart of an interactive feature for obtaining fused image features provided by an embodiment of the present application; Figure 6 This is a flow chart of obtaining front-end and post-end fusion features provided by an embodiment of the present application; Figure 7 This is another flow chart for obtaining front-end and late-end fusion features provided in an embodiment of the present application; Figure 8 It is a structural schematic diagram of a high-precision image segmentation device based on feature interaction and front-end and rear-end feature fusion provided in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] The existing high-precision image segmentation methods include the following five: The first high-precision image segmentation method uses adapters and visual cues to achieve the purpose of adapting to different segmentation tasks without changing the main structure of the SAM model (Segment Anything Model, a model for high-precision image segmentation). The second high-precision image segmentation method uses a cue-driven adapter to optimize the sparse and dense cues of SAM, aiming to improve the quality of SAM's segmentation masks, so that the model's adaptability to specific tasks is enhanced. The third high-precision image segmentation method uses a two-level optimization strategy to fine-tune the SAM model, by optimizing the model's weight parameters and cue embedding on two different subsets of the training data set to prevent overfitting, thereby improving the model's performance on semantic segmentation tasks. The fourth high-precision image segmentation method improves the quality of mask prediction by introducing a learnable high-quality output Token in the SAM mask decoder, enabling the model to generate higher-quality segmentation masks, especially when dealing with objects with complex structures. The fifth high-precision image segmentation method combines a complementary convolutional neural network and the standard SAM network for medical image segmentation, by fine-tuning only the additional CNN network and the SAM decoder part to reduce training time and improve the model's performance on medical image segmentation tasks. Together, these methods have promoted the application and performance improvement of the SAM model in different fields.

[0023] Although the above methods can achieve high-precision image segmentation, they still face many challenges in capturing global features and retaining detail information. On the one hand, they are insufficient in global feature modeling and it is difficult to fully capture long-distance global dependencies, especially when dealing with complex scenes or large-scale targets. The problem of missing context information is prone to occur. On the other hand, due to multiple downsampling operations in the segmentation process, the high-resolution spatial detail information is severely weakened, and the results are often manifested as blurred boundaries or inaccurate segmentation of small targets. In addition, the decoder part of the existing methods lacks an explicit feature interaction mechanism, and the flow of generated feature information is limited, which makes it impossible for the model to fully tap the ability of potential feature expression, thereby limiting the overall accuracy and detail characterization ability of the segmentation results. Therefore, these methods have become the main bottleneck for the further development of existing segmentation methods under complex scenes and high-precision requirements.

[0024] In order to overcome the limitations of related technologies, the embodiments of the present application provide a high-precision image segmentation method based on feature interaction and fusion of front-end and late-stage features, which significantly enhances the segmentation accuracy and generalization ability by improving the model's ability to model the interaction between input features and output features; and, fully utilizes the high-resolution spatial detail information in the early features and the global semantic information in the late features, effectively compensates for the loss of detail information caused by multiple downsampling, and improves the model's segmentation performance for small targets and complex scenes.

[0025] The following is a detailed description of the high-precision image segmentation method based on feature interaction and front- and rear-stage feature fusion according to an embodiment of the present application in conjunction with the accompanying drawings.

[0026] Reference Figure 1 As shown, Figure 1 1 is a flowchart of the steps of a high-precision image segmentation method based on feature interaction and front-end and back-end feature fusion provided in an embodiment of the present application. Figure 1 As shown, the high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion may include steps S110 to S150: Step S110: Obtain prompt features corresponding to the prompt information, and obtain image features corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area.

[0027] In the embodiment of the present application, the prompt information may be prompt information in different forms, for example, the prompt information may be a box or a dot. Obtaining the prompt feature corresponding to the prompt information may be obtained through a prompt encoder, that is, the prompt information is converted into a prompt feature through the prompt encoder, and the prompt feature is used to represent the area to be segmented in the image.

[0028] The image features corresponding to the image to be segmented can be obtained through an image encoder, that is, the image to be segmented is converted into image features through an image encoder. Moreover, when the image features corresponding to the image to be segmented are obtained through an image encoder, multiple layers of features from shallow to deep can be obtained, wherein shallow features have higher resolution and can retain more spatial details, and deep features are often good at capturing global semantic information.

[0029] Step S120: Fusing the interactive features of the prompt features with the image features to obtain fused image features, wherein the interactive features of the prompt features have explicit association information between features.

[0030] In the embodiment of the present application, the interactive features of the prompt features refer to the interactive features obtained by cross-attention interaction of the prompt features. In order to improve the interactive modeling ability of the model for the input features, the interactive features of the prompt features are fused with the image features during mask decoding to obtain fused image features. The fused image features refer to image features containing mask information, that is, the mask probability of each image pixel can be generated later by fusion of the image features.

[0031] In some embodiments, the interactive feature of the prompt feature is fused with the image feature to obtain a fused image feature, including steps S120-1 to S120-2: Step S120 - 1 : Input the interaction feature of the prompt feature and the image feature into multiple first attention interaction layers for processing to obtain the first attention interaction feature.

[0032] Step S120-2: Input the first attention interaction feature into the second attention interaction layer to obtain a fused image feature. The second attention interaction layer is used for the interaction between the image feature and the prompt feature.

[0033] In the embodiment of the present application, the first attention interaction layer refers to the Transformer layer, the second attention interaction layer refers to the Token to image attn layer, and the fusion of the interactive features of the prompt features and the image features can be achieved through a mask decoder (Mask Decoder), and multiple first attention interaction layers and second attention interaction layers are all included in the mask decoder. When performing mask decoding, the interactive features and image features of the input prompt features are first processed using multiple layers of the first attention interaction layer to obtain the first attention interaction feature, and then the first attention interaction feature is processed using the second attention interaction layer.

[0034] Step S130: Fusing the fused image features with the early features and late features of the image features to obtain early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information.

[0035] In the embodiment of the present application, the image features include multiple layers of features from shallow to deep. Considering that in high-precision segmentation tasks, deep features are often good at capturing global semantic information, but are prone to losing local details, especially in the expression of the boundaries of segmented objects or small objects. On the contrary, shallow features can retain more spatial details due to their higher resolution, but lack the support of global context information.

[0036] In order to make up for this shortcoming, the present application implements the introduction of early features (shallow features) and late features (deep features) of image features to fuse the early and late features, that is, after obtaining the fused image features in step S120, the image is not segmented directly, but the fused image features are fused with the early and late features of the image features to obtain the early and late fused features. Among them, the early and late fused features are a feature that contains mask information and has global semantic information and spatial detail information. In this way, this method not only enhances the modeling ability of the mask decoder for global semantic information, but also significantly improves the retention effect of boundary details, thereby achieving more accurate and robust segmentation results.

[0037] Step S140: Fusing the interactive features of the fused image features with the front and rear fusion features to obtain final image features, wherein the interactive features of the fused image features have fine-grained information between features.

[0038] In the embodiment of the present application, the interactive features of the fused image features refer to the interactive features obtained by cross attention interaction of the fused image features. In order to further enhance the global modeling ability and detail expression ability of the features, after obtaining the front and rear fusion features, the interactive features with the fused image features are also used to fuse with the front and rear fusion features to obtain the final image features. Since the interactive features of the fused image features have fine-grained information between the features, the interactive modeling ability of the output features is enhanced, further solving the difficult problem of balancing global modeling and capturing local details.

[0039] Step S150: performing image segmentation based on the final image features to obtain a segmentation result.

[0040] In the embodiment of the present application, after obtaining the final image features, the mask probability of each image pixel can be generated based on the final image features, and then the segmentation result can be obtained according to the mask probability of each image pixel. For example, if the mask probability threshold is 0.6, the pixel area with a mask probability greater than 0.6 is determined as the mask area (or segmentation area).

[0041] By adopting the technical solution of the embodiment of the present application, since the interactive features of the prompt features have explicit correlation information between features, the interactive features of the prompt features are fused with the image features, thereby enhancing the interactive modeling capability of the input features, and the interactive features of the fused image features have fine-grained information between features, the interactive features of the fused image features are fused with the front and rear fusion features, thereby enhancing the interactive modeling capability of the output features, thus solving the difficult problem of balancing global modeling and capturing local details, thereby improving the semantic consistency and boundary clarity of the segmentation results. At the same time, the fused image features are fused with the early and late features of the image features, making full use of the high-resolution spatial detail information in the early features and the global semantic information in the late features, effectively compensating for the loss of detail information caused by multiple downsampling, and improving the segmentation performance of the model for small targets and complex scenes.

[0042] In combination with the above implementation, the embodiment of the present application provides a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion. In the method, the prompt feature includes a first type of feature and a second type of feature. The first type of feature is used for image segmentation, and the second type of feature is used to provide mask information. Before executing step S120, the interactive feature of the prompt feature is obtained by the following steps, specifically including: Perform attention interaction between the first category feature and the second category feature to obtain a first interaction feature; fuse the first interaction feature with the first category feature to obtain a first category feature containing mask information; and use the first category feature containing mask information and the second category feature as interaction features of the prompt feature.

[0043] In an embodiment of the present application, cross-attention interaction (i.e., crossattention) is performed between two types of features in the prompt features to establish an explicit association between the features, thereby better achieving high-precision image segmentation.

[0044] For example, performing attention interaction between the first type of features and the second type of features can be expressed as: , , in, represents the first interaction feature; Represents the first type of features, namely the newly introduced high-quality segmentation tokens; Represents the second type of features, namely the mask token of the SAM model; Represents feature dimension; , , Both represent weight parameters, Represents the query linear mapping layer, Represents the key linear mapping layer, Represents the query (Value) linear mapping layer.

[0045] Fusion of the first interactive feature with the first type of feature is equivalent to introducing the information of the mask token into the first type of feature, thereby obtaining the first type of feature containing the mask information. For example, fusion of the first interactive feature with the first type of feature can be expressed as: , in, Represents the first type of features containing mask information.

[0046] Finally, the first-category features and the second-category features containing mask information are used as interactive features of prompt features. Since the first-category features used for image segmentation contain mask information in the interactive features of prompt features, the interactive features based on prompt features can better realize image segmentation.

[0047] In a specific implementation, obtaining the interactive features of the prompt features can be implemented based on the first feature interaction module. Specifically, after obtaining the prompt features corresponding to the prompt information, the two types of features of the prompt features (i.e., the first type of features and the second type of features) are input into the first feature interaction module for cross attention to obtain the first type of features containing mask information, thereby using the first type of features and the second type of features containing mask information as the interactive features of the prompt features.

[0048] In combination with the above implementation, the embodiment of the present application provides a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion. In the method, the fused image features include third-category features and fourth-category features. The third-category features are used for image segmentation, and the fourth-category features are used to provide mask information. Before executing step S140, the interactive features of the fused image features are obtained by the following steps, specifically including: Perform attention interaction between the third category features and the fourth category features to obtain second interaction features; fuse the second interaction features with the third category features to obtain third category features containing mask information; and use the third category features containing mask information and the fourth category features as interaction features of the fused image features.

[0049] In an embodiment of the present application, cross attention is performed between two types of features in the fused image features to optimize the fine-grained interaction of the output, thereby better achieving high-precision image segmentation.

[0050] For example, performing attention interaction between the third type of features and the fourth type of features can be expressed as: , , in, represents the second interaction feature; Represents the third type of features, i.e., the output high-quality segmentation token; Represents the fourth type of feature, namely the mask token of the SAM model; Represents feature dimension; , , Both represent weight parameters, Represents the query linear mapping layer, Represents the key linear mapping layer, Represents the query (Value) linear mapping layer.

[0051] Fusion of the second interactive feature with the third feature is equivalent to introducing mask token information into the third feature to obtain the third feature containing mask information. For example, fusion of the second interactive feature with the third feature can be expressed as: , in, Represents the third type of features containing mask information.

[0052] Finally, the third and fourth features containing mask information are used as interactive features of fused image features. Since the third feature used for image segmentation contains mask information in the interactive features of the fused image features, the interactive features based on the fused image features can better realize image segmentation.

[0053] In a specific implementation, obtaining the interactive features of the fused image features can be realized based on the second feature interaction module. Specifically, after obtaining the fused image features, the two types of features of the fused image features (i.e., the third type of features and the fourth type of features) are input into the second feature interaction module for cross attention to obtain the third type of features containing mask information, thereby using the third type of features and the fourth type of features containing mask information as the interactive features of the fused image features.

[0054] In combination with the above implementation, the embodiment of the present application provides a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion. In the method, the above step S130 "fusing the fused image feature with the front-end feature and the rear-end feature of the image feature to obtain the front-end and rear-end fusion feature" specifically includes steps S130-1 to S130-2: Step S130 - 1 : Fusing the fused image features with the previous features to obtain updated previous features.

[0055] Step S130 - 2 : Fusing the updated early features with the late features to obtain early and late fused features.

[0056] In the embodiment of the present application, the early feature is recorded as early_feature ( ), the later feature is denoted as later_feature ( ), after obtaining the fused image features, the early features and the late features are fused with the fused image features.

[0057] First, step S130-1 is executed to update the previous features to obtain updated previous features. For example, the fusion of the fused image features and the previous features can be expressed as: , in, Indicates the update of previous features. Fusion of image features, Represents a convolution operation.

[0058] Next, step S130-2 is executed to fuse the pre-update feature with the post-update feature to obtain a pre-update feature of the same size as the post-update feature; the pre-update feature of the same size as the post-update feature is fused with the post-update feature to obtain a pre-update feature of the same size as the post-update feature.

[0059] Among them, the early update features can be upsampled to the same size as the late update features by transposed convolution, and the upsampling process of the early update features can be expressed as: , in, Represented as an updated early feature of the same size as the later feature, Represents a transposed convolution.

[0060] Finally, the updated early features of the same size as the late features are fused (added) with the late features to obtain the front and late fusion features, where the front and late fusion features It is expressed as: , In the embodiment of the present application, since the early features have spatial detail information and the late features have global semantic information, by fusing the fused image features with the early features and late features of the image features, the obtained fused features are a feature containing mask information and having global semantic information and spatial detail information. In this way, the modeling capability of the mask decoder for global semantic information is enhanced, and the effect of retaining boundary details is significantly improved, achieving a more accurate and robust segmentation result.

[0061] In a specific implementation, the fusion of the fused image features with the early features and late features of the image features can be achieved through a front-to-late fusion module. Specifically, after the fused image features are obtained, the fused image features, the early features and the late features are input into the front-to-late fusion module for feature fusion, thereby obtaining the front-to-late fusion features.

[0062] In combination with the above implementation, the embodiment of the present application provides a high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion. In the method, the above step S140 "fusing the interaction feature of the fused image feature with the front-end and rear-end fusion feature to obtain the final image feature" specifically includes steps S140-1 to S140-2: Step S140 - 1 : Processing the interactive features of the fused image features into the same dimension as the front and rear fusion features to obtain dimensionally processed features.

[0063] Step S140-2: After convolution down-sampling the dimension processing features, they are fused with the front and rear fusion features to obtain the final image features.

[0064] In the embodiment of the present application, the interactive features of the fused image features are processed into the same dimension as the front and rear fusion features by transposed convolution, and the dimension processing feature can be expressed as: , in, represents the interactive features of fused image features, represents the dimension processing feature, Represents a transposed convolution.

[0065] For example, the final image features It can be expressed as: , in, Represents a convolution operation (convolution downsampling).

[0066] Through the above process, the final image features are obtained based on the interactive features of the fused image features and the front and late fusion features. Since the interactive features of the fused image features are obtained by attention interaction between the two types of features contained in it, they have fine-grained information between the features, and the front and late fusion features are features that contain mask information and have global semantic information and spatial detail information. Therefore, the final image features contain rich detail information and global information, as well as mask information. Therefore, high-quality image segmentation can be achieved based on the final image features.

[0067] The following is a specific example to illustrate the high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion according to the embodiment of the application.

[0068] Reference Figure 2 As shown, Figure 2 It is a flowchart of another high-precision image segmentation method based on feature interaction and front-end and rear-end feature fusion provided by an embodiment of the present application, specifically, comprising the following steps S210 to S270: Step S210: Obtain prompt features corresponding to prompt information, and obtain image features corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area.

[0069] Step S220: The prompt feature includes a first category feature and a second category feature, and the first category feature and the second category feature are subjected to attention interaction to obtain a first interaction feature; the first interaction feature and the first category feature are fused to obtain a first category feature containing mask information; the first category feature containing mask information and the second category feature are used as interaction features of the prompt feature.

[0070] Step S230: Fusing the interactive features of the prompt features with the image features to obtain fused image features, wherein the interactive features of the prompt features have explicit association information between features.

[0071] Step S240: The third and fourth features of the fused image features are subjected to attention interaction between the third and fourth features to obtain second interaction features; the second interaction features are fused with the third features to obtain third features containing mask information; the third and fourth features containing mask information are used as interaction features of the fused image features.

[0072] Step S250: Fusing the fused image features with the early features and late features of the image features to obtain early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information.

[0073] Step S260: Fusing the interactive features of the fused image features with the front and rear fusion features to obtain final image features, wherein the interactive features of the fused image features have fine-grained information between features.

[0074] Step S270: performing image segmentation based on the final image features to obtain a segmentation result.

[0075] In an embodiment of the present application, by cross-attentionally interacting two types of features in the prompt features to establish explicit associations between the features, and by cross-attentionally interacting two types of features in the fused image features, the fine-grained interaction of the output is optimized, thereby enhancing the interactive modeling capabilities of the input features and the interactive modeling capabilities of the output features, thereby solving the difficult problem of balancing global modeling and capturing local details, thereby improving the semantic consistency and boundary clarity of the segmentation results.

[0076] In addition, the fused image features are fused with the early features and late features of the image features, which fully utilizes the high-resolution spatial detail information in the early features and the global semantic information in the late features, effectively compensates for the loss of detail information caused by multiple downsampling, and improves the model's segmentation performance for small targets and complex scenes.

[0077] In this way, this method not only makes full use of the detail advantages of early features and the global information of later features, but also improves the overall segmentation effect through feature interaction, providing a feasible implementation method for high-precision segmentation in complex scenes.

[0078] For example, the high-precision image segmentation method based on feature interaction and front-end and back-end feature fusion can be implemented based on an image segmentation model, such as Figure 3 As shown, Figure 3 It is a structural diagram of an image segmentation model provided by an embodiment of the present application. The image segmentation model includes an image encoder, a prompt encoder, and a mask decoder. When performing image segmentation, the prompt information and the image to be segmented are input into the image segmentation model, the prompt information is converted into a prompt feature by the prompt encoder, the image to be segmented is converted into an image feature by the image encoder, and then the prompt feature and the image feature are input into the mask decoder for decoding to obtain a segmentation result.

[0079] The mask decoder includes: a first feature interaction module, multiple first attention interaction layers (Transformer), a second attention interaction layer (Token to image attn), a second feature interaction module, a front and rear feature fusion module, and a transposed convolution layer (Trans Conv). The specific decoding process is: first, use the first feature interaction module to obtain the interactive features of the prompt features, such as Figure 4As shown, the prompt feature includes a first type of feature and a second type of feature. The two types of features of the prompt feature are input into the first feature interaction module for cross attention to obtain the first type of feature containing mask information, thereby using the first type of feature and the second type of feature containing mask information as the interactive features of the prompt feature.

[0080] Secondly, the interactive features of the prompt features and the image features are input into multiple first attention interaction layers for processing to obtain first attention interaction features, and the first attention interaction features are input into the second attention interaction layer to obtain fused image features.

[0081] Next, the second feature interaction module is used to obtain the interactive features of the fused image features, such as Figure 5 As shown, the fused image features include the third type of features and the fourth type of features, and then the two types of features of the fused image features are input into the second feature interaction module for cross attention to obtain the third type of features containing mask information, so that the third type of features and the fourth type of features containing mask information are used as the interactive features of the fused image features.

[0082] Afterwards, if Figure 6 As shown, the fused image features and the early features and late features of the image features are input into the early and late feature fusion module for feature fusion to obtain early and late fusion features. Finally, the interactive features of the fused image features are fused with the early and late fusion features to obtain the final image features, and image segmentation is performed based on the final image features to obtain the segmentation result.

[0083] Specifically, Figure 7 As shown in the figure, the process of fusing the front and rear fusion features and obtaining the final image features is as follows: fusing the fused image features with the front features to obtain the updated front features, upsampling the updated front features to obtain the updated front features of the same size as the rear features; fusing the updated front features of the same size as the rear features with the rear features to obtain the front and rear fusion features. Then, the interactive features of the fused image features are processed into the same dimension as the front and rear fusion features through transposed convolution to obtain the dimension processing features; and the dimension processing features are convoluted downsampled and fused with the front and rear fusion features to obtain the final image features.

[0084] In this way, the performance of high-precision image segmentation tasks is significantly improved through the innovative mask decoder structure and feature fusion method. The introduced feature interaction module (Token-to-Token Attention mechanism) effectively enhances the model's interactive modeling ability for input and output features, and solves the difficult problem of balancing global modeling and capturing local details, thereby improving the semantic consistency and boundary clarity of the segmentation results. At the same time, the front-end and rear-end feature fusion decoder combines the early features of the image encoder and the late deep semantic features, making full use of the high-resolution spatial information in the early features and the global semantic information in the late features, effectively compensating for the loss of detail information caused by multiple downsampling of the traditional decoder, and improving the model's segmentation performance for small targets and complex scenes.

[0085] The present application also provides a high-precision image segmentation device based on feature interaction and front-end and rear-end feature fusion, referring to Figure 8 As shown, Figure 8 : is a structural schematic diagram of a high-precision image segmentation device based on feature interaction and front-end and rear-end feature fusion provided in an embodiment of the present application, the device comprising: A first acquisition module 810 is used to acquire a prompt feature corresponding to the prompt information and to acquire an image feature corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area; A first fusion module 820, configured to fuse the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features; A second fusion module 830 is used to fuse the fused image feature with the early features and late features of the image feature to obtain a front-late fusion feature, wherein the early features have spatial detail information and the late features have global semantic information; A third fusion module 840 is used to fuse the interactive features of the fused image features with the front and rear fusion features to obtain final image features, wherein the interactive features of the fused image features have fine-grained information between features; The result segmentation module 850 is used to perform image segmentation based on the final image features to obtain a segmentation result.

[0086] In an optional embodiment, the prompt features include first-category features and second-category features, the first-category features are used for image segmentation, and the second-category features are used for providing mask information; The device also includes: The first interaction module is used to perform attention interaction between the first category feature and the second category feature to obtain a first interaction feature; fuse the first interaction feature with the first category feature to obtain a first category feature containing mask information; and use the first category feature containing mask information and the second category feature as interaction features of the prompt feature.

[0087] In an optional embodiment, the fused image features are third-category features and fourth-category features, the third-category features are used for image segmentation, and the fourth-category features are used for providing mask information; The device also includes: The first interaction module is used to perform attention interaction between the third type of features and the fourth type of features to obtain a second interaction feature; fuse the second interaction feature with the third type of features to obtain a third type of features containing mask information; and use the third type of features containing mask information and the fourth type of features as interaction features of the fused image features.

[0088] In an optional embodiment, the second fusion module includes: A first fusion submodule, used for fusing the fused image features with the previous features to obtain updated previous features; The second fusion submodule is used to fuse the updated early features with the late features to obtain early and late fused features.

[0089] In an optional embodiment, the second fusion submodule is further used to upsample the early update features to obtain early update features of the same size as the late update features; and fuse the early update features of the same size as the late update features with the late features to obtain front- and late-stage fusion features.

[0090] In an optional embodiment, the first fusion module includes: A first input module, used for inputting the interaction feature of the prompt feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature; The second input module is used to input the first attention interaction feature into the second attention interaction layer to obtain a fused image feature. The second attention interaction layer is used for the interaction between the image feature and the prompt feature.

[0091] In an optional embodiment, the third fusion module includes: A dimension processing module, used for processing the interactive features of the fused image features into the same dimension as the front and rear fusion features to obtain dimension processing features; The third fusion submodule is used to perform convolution downsampling on the dimension processing features and then fuse them with the front and rear fusion features to obtain the final image features.

[0092] The present application also provides an electronic device, referring to Fig. 9 , Fig. 9 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 900 includes: a memory 910 and a processor 920. The memory 910 and the processor 920 are connected via a bus communication. A computer program is stored in the memory 910. The computer program can be run on the processor 920 to implement the steps of the high-precision image segmentation method based on feature interaction and front- and late-stage feature fusion described in the embodiment of the present application.

[0093] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and front- and late-stage feature fusion described in the embodiment of the present application are implemented.

[0094] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the high-precision image segmentation method based on feature interaction and front- and rear-stage feature fusion described in the embodiment of the present application.

[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0096] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0100] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0101] The above is a detailed introduction to a high-precision image segmentation method and device based on feature interaction and front- and late-stage feature fusion provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A high-precision image segmentation method based on feature interaction and front-end and late-stage feature fusion, characterized in that: The method comprises: Acquire a prompt feature corresponding to the prompt information, and acquire an image feature corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area; fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features; The fused image feature is fused with the early features and late features of the image feature to obtain the early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information; The interactive features of the fused image features are fused with the front and rear fused features to obtain the final image features, wherein the interactive features of the fused image features have fine-grained information between the features; Image segmentation is performed based on the final image features to obtain a segmentation result.

2. The method according to claim 1, characterized in that The prompt features include first-type features and second-type features, the first-type features are used for image segmentation, and the second-type features are used to provide mask information; the method further includes: Performing attention interaction between the first type of features and the second type of features to obtain a first interaction feature; The first interactive feature is merged with the first category feature to obtain the first category feature containing mask information; The first type of features containing mask information and the second type of features are used as interactive features of the prompt features.

3. The method according to claim 1, characterized in that The third type of features and the fourth type of features of the fused image features, the third type of features are used for image segmentation, and the fourth type of features are used to provide mask information; the method further includes: Performing attention interaction between the third type of features and the fourth type of features to obtain a second interaction feature; Fusing the second interactive feature with the third type of feature to obtain a third type of feature containing mask information; The third type of features containing mask information and the fourth type of features are used as interactive features of the fused image features.

4. The method according to claim 1, characterized in that: The fused image feature is fused with the early features and late features of the image feature to obtain the early and late fused features, including: Fusing the fused image features with the previous features to obtain updated previous features; The updated early features are fused with the late features to obtain early and late fused features.

5. The method according to claim 4, characterized in that The updating early features are fused with the late features to obtain the early and late fused features, including: Performing upsampling processing on the pre-update feature to obtain the pre-update feature having the same size as the post-update feature; The updated early features having the same size as the late features are fused with the late features to obtain early and late fused features.

6. The method according to any one of claims 1 to 5, characterized in that: The interactive feature of the prompt feature is fused with the image feature to obtain a fused image feature, including: Inputting the interaction feature of the prompt feature and the image feature into a plurality of first attention interaction layers for processing to obtain a first attention interaction feature; The first attention interaction feature is input into the second attention interaction layer to obtain a fused image feature, and the second attention interaction layer is used for the interaction between the image feature and the prompt feature.

7. The method according to any one of claims 1 to 5, characterized in that: The interactive features of the fused image features are fused with the front and rear fusion features to obtain the final image features, including: Processing the interactive features of the fused image features into the same dimension as the front and rear fusion features to obtain dimensionally processed features; After convolution downsampling the dimension processing features, they are fused with the front and rear fusion features to obtain the final image features.

8. A high-precision image segmentation device based on feature interaction and front-end and back-end feature fusion, characterized in that: The device comprises: A first acquisition module is used to acquire a prompt feature corresponding to the prompt information and to acquire an image feature corresponding to the image to be segmented, wherein the prompt information is used to characterize the segmented area; a first fusion module, used for fusing the interactive feature of the prompt feature with the image feature to obtain a fused image feature, wherein the interactive feature of the prompt feature has explicit association information between the features; A second fusion module is used to fuse the fused image features with the early features and late features of the image features to obtain early and late fused features, wherein the early features have spatial detail information and the late features have global semantic information; A third fusion module is used to fuse the interactive features of the fused image features with the front and rear fusion features to obtain the final image features, wherein the interactive features of the fused image features have fine-grained information between features; The result segmentation module is used to perform image segmentation based on the final image features to obtain a segmentation result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the high-precision image segmentation method based on feature interaction and front- and rear-stage feature fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the high-precision image segmentation method based on feature interaction and front- and rear-stage feature fusion as described in any one of claims 1 to 7 are implemented.

Images