Multi-scale bridge joint network image segmentation method based on multiple attention mechanism
By introducing a multi-scale bridge joint network with multiple attention mechanisms in the image segmentation model, multimodal high-dimensional fusion of image and semantic features is solved, and the problem of insufficient information fusion in feature extraction is improved, and the performance and explanatory model are improved. sex.
Patent Information
- Application Number
- CN202411412840.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-10-11
AI Technical Summary
The existing image segmentation model lacks the fusion of image and semantic information during feature extraction, resulting in information loss and insufficient interpretability of the model, especially insufficient attention to image local information and multi-channel information.
Using a multi-scale bridge joint network based on multiple attention mechanisms, multi-modal feature extraction is performed through image channel and position attention mechanism, and feature cross-fusion is performed by combining image and semantic cross-attention mechanisms to enhance information relevance and model explanation sex.
The performance and accuracy of the image segmentation model are improved, visual information and semantic information are fully utilized, the interpretability of the model is enhanced, and the information fusion and interpretability defects of traditional models are compensated.
Smart Images
Figure CN118918333B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image segmentation, and in particular to a multi-scale bridge joint network image segmentation method based on a multiple attention mechanism. Background Art
[0002] In recent years, deep learning has been applied to the field of image segmentation, and many mask algorithms and models have been born. The traditional model uses a decoder and an encoder to extract features from images and semantics respectively, and then inputs them into the encoder for processing. Finally, the decoder outputs the corresponding segmentation mask. This method is undoubtedly lacking in interpretability. Without the interconnected features of images and semantics, the resulting model is not sufficient to meet the needs of practical applications.
[0003] Nowadays, deep learning image segmentation based on CNN and Transformer has basically matured. Most of them use image convolution and Transformer encoding and decoding structure for feature extraction and application, step by step to deeply explore the correlation between images and semantics, and finally obtain segmentation masks. However, there are also some defects: 1) The intermediate feature extraction part lacks the fusion of image and semantic information, which will lead to the loss of some fusion information; 2) Lack of attention to local information of the image, such as the pixel block position information of the image, and the multi-channel information of the image; 3) Lack of interpretability of the model.
[0004] Therefore, the present invention proposes a multi-scale bridge joint network image segmentation method based on a multiple attention mechanism to overcome the above technical defects. Summary of the invention
[0005] The purpose of the present invention is to provide a multi-scale bridge joint network image segmentation method based on a multiple attention mechanism, which abandons the independent structure of the traditional model, fuses the features of the picture and semantics in a multi-modal and high-dimensional manner, and outputs the encoder after feature fusion through bridge grafting and feature extraction, which is beneficial to the training of the decoder and the final effect output, so that the image segmentation model can make full use of the visual information and related semantic information of the image, and improve the performance and accuracy when processing image-related tasks; in addition, the present invention also increases the interpretability of the model and fills the gap in the interpretability of the model.
[0006] To achieve the above object, the present invention provides a multi-scale bridge joint network image segmentation method based on a multi-attention mechanism, comprising the following steps:
[0007] S1. Take image data from different categories and semantic data corresponding to the images as input to train and generate an image segmentation model;
[0008] S2. Based on the multi-attention mechanism, a multi-scale bridge joint network is constructed, including a multi-scale feature extraction structure based on image channel and position attention mechanism and a multi-scale position feature cross-fusion structure based on image and semantic cross-attention mechanism;
[0009] S3, a multi-scale feature extraction structure based on image channels and position attention mechanism, which performs multi-modal feature extraction on image data and semantic data respectively;
[0010] S4, a multi-scale position feature cross-fusion structure based on the image and semantic cross-attention mechanism, which performs cross-attention feature fusion on the image channel information and pixel block area position information with the semantic embedded feature vector;
[0011] S5. Input the fused features into the masker to segment and mask out the specific parts of the image corresponding to the semantic description.
[0012] Preferably, in step S2, the multiple attention mechanisms include self-attention, dual attention and cross attention;
[0013] The multi-scale bridge joint network based on multiple attention mechanisms includes: a multi-scale feature extraction structure based on image channel and position attention mechanism and a multi-scale position feature cross-fusion structure based on image and semantic cross-attention mechanism.
[0014] Preferably, in step S3, based on the multi-scale feature extraction structure of the image channel and the position attention mechanism, multi-modal feature extraction is performed on the image data and the semantic data respectively, including the following steps:
[0015] S31. Use the convolution layer and pooling layer to extract the regional features of the image and capture the local features of the image. The specific process is as follows:
[0016] S311, feature downsampling: The initial image feature vector is subjected to the feature downsampling operation of the convolutional layer to generate two downsampled feature maps, which reduce their resolution, capture the high-level abstract features of the image, and reduce unnecessary irrelevant information of the image to achieve image feature enhancement;
[0017] S312, position attention module: The position attention module extracts important information of pixel blocks at different positions in the image, and by calculating the importance of each pixel block relative to other pixel blocks, the image segmentation model can identify key areas in the image;
[0018] S313, channel attention module: screens and refines information on multi-channel feature maps, identifies and strengthens the most discriminative feature channels; the channel attention module enhances important features and improves the expressiveness of features by evaluating the feature responses of different channels;
[0019] S32. While extracting features from the image, natural language processing technology is used to extract semantic features corresponding to the image from the text description; combined with semantic self-attention distribution embedding, a word vector with fusion features is generated through the cross-attention module, which also includes semantic features related to the image.
[0020] Preferably, in step S4, based on the multi-scale position feature cross-fusion structure of the image and semantic cross-attention mechanism, the channel information of the image and the pixel block area position information are cross-attention feature fused with the feature vector embedded in the semantics; wherein the specific implementation process of feature fusion is as follows:
[0021] S41, double attention mechanism: before the image feature vector and word vector obtained in step S3 are subjected to feature fusion through the cross attention module, the double attention mechanism is used to further strengthen the features;
[0022] S42, Cross-Attention Module: Usually includes a self-attention mechanism, which allows the image segmentation model to consider semantic information while processing image features. The image feature vector and word vector are then fused through the cross-attention module to learn the relationship between image features and semantic features and convert them into fused features.
[0023] S43, feature fusion: After the image feature vector and word vector are fused and transformed by the cross attention module, they become feature representations with rich semantic information;
[0024] S44. The above fusion features are connected and combined to generate a final fusion image feature vector, thereby obtaining more comprehensive and effective image features to provide support for subsequent image segmentation.
[0025] Therefore, the present invention adopts the above-mentioned multi-scale bridge joint network image segmentation method based on multiple attention mechanism, abandons the independent structure of the traditional model, fuses the features of the picture and semantics in a multi-modal and high-dimensional manner, and outputs the encoder after feature fusion through bridge grafting and feature extraction, which is beneficial to the training of the decoder and the final effect output, so that the image segmentation model can make full use of the visual information and related semantic information of the image, and improve the performance and accuracy when processing image-related tasks; in addition, the present invention also increases the interpretability of the model and fills the gap in the interpretability of the model.
[0026] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a flow chart of a multi-scale bridge joint network image segmentation method based on a multiple attention mechanism of the present invention;
[0028] Figure 2 It is an algorithm structure diagram of the multi-scale bridge joint network image segmentation method based on the multi-attention mechanism of the present invention;
[0029] Figure 3 It is a multi-scale feature extraction structure based on image channel and position attention mechanism of the present invention;
[0030] Figure 4 It is a multi-scale position feature cross-fusion structure based on image and semantic cross-attention mechanism of the present invention. DETAILED DESCRIPTION
[0031] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0032] like Figure 1 As shown, the multi-scale bridge joint network image segmentation method based on the multi-attention mechanism of the present invention comprises the following steps:
[0033] S1. Take image data from different categories and semantic data corresponding to the images as input to train and generate an image segmentation model;
[0034] S2. Based on the multi-attention mechanism, a multi-scale bridge joint network is constructed, including a multi-scale feature extraction structure based on image channel and position attention mechanism and a multi-scale position feature cross-fusion structure based on image and semantic cross-attention mechanism;
[0035] S3, a multi-scale feature extraction structure based on image channels and position attention mechanism, which performs multi-modal feature extraction on image data and semantic data respectively;
[0036] S4, a multi-scale position feature cross-fusion structure based on the image and semantic cross-attention mechanism, which performs cross-attention feature fusion on the image channel information and pixel block area position information with the semantic embedded feature vector;
[0037] S5. Input the fused features into the masker to segment and mask out the specific parts of the image corresponding to the semantic description.
[0038] Example
[0039] like Figure 2As shown in the figure, in the process of feature extraction and information interaction, the picture is encoded by the image encoder (VIT), and the semantics is encoded by the text encoder. During the encoding process, a bridge is built through the Interator structure, and the correlation information of the interactive picture and semantics passes through the encoder. At the same time, the information generated after the interaction will be re-input back to their respective encoders for re-encoding. After passing through N encoders, the generated feature information is input into the decoder for decoding, and finally the output is generated to generate a mask corresponding to the input.
[0040] S1. Take image data from different categories and semantic data corresponding to the images as input, and train to generate an image segmentation model. The model can segment and mask specific areas of the image by understanding the semantics of the input text.
[0041] S2. Based on the multi-attention mechanism, a multi-scale bridge joint network is constructed.
[0042] Multiple attention mechanisms include: self-attention, dual attention, and cross attention.
[0043] The multi-scale bridge joint network based on the multi-attention mechanism includes: a multi-scale feature extraction structure based on the image channel and position attention mechanism and a multi-scale position feature cross-fusion structure based on the image and semantic cross-attention mechanism. The multi-scale bridge joint network based on the multi-attention mechanism fuses the multimodal information of image and semantics, enhances the information correlation between image and semantics, and reduces the cross-correlation loss of the mask.
[0044] S3, a multi-scale feature extraction structure based on image channels and position attention mechanism, performs multi-modal feature extraction on image data and semantic data respectively, such as Figure 3 shown.
[0045] S31. Use the convolution layer and pooling layer to extract the regional features of the image and capture the local features of the image. The specific process is as follows:
[0046] S311, Feature downsampling: The initial image feature vector is processed through the feature downsampling operation of the convolutional layer to generate two downsampled feature maps, reducing their resolution, capturing high-level abstract features of the image, while reducing unnecessary irrelevant information of the image to achieve image feature enhancement.
[0047] S312, Positional Attention Module: The positional attention module extracts important information of pixel blocks at different positions in the image. By calculating the importance of each pixel block relative to other pixel blocks, the image segmentation model can identify key areas in the image, such as edges, textures, or parts of specific objects.
[0048] S313, Channel Attention Module: It filters and refines information on multi-channel feature maps, identifies and strengthens the most discriminative feature channels. The channel attention module enhances important features and suppresses unimportant features by evaluating the feature responses of different channels, thereby improving the expressiveness of features.
[0049] S32. While extracting features from the image, use natural language processing (NLP) technology to extract semantic features corresponding to the image from the text description. Combined with the semantic self-attention distribution embedding, a word vector with fusion features is generated through the cross-attention module, which also includes semantic features related to the image.
[0050] Based on the features extracted above, the foundation is laid for subsequent feature fusion and segmentation.
[0051] S4, based on the multi-scale position feature cross-fusion structure of image and semantic cross-attention mechanism, cross-attention feature fusion is performed on the channel information of the image and the pixel block area position information with the feature vector embedded in the semantics, such as Figure 4 The specific implementation process of feature fusion is as follows:
[0052] S41, double attention mechanism: before the image feature vector and word vector obtained in step S3 are fused through the cross attention module, the double attention mechanism is used to further strengthen the features.
[0053] S42, Cross-Attention module: usually contains a self-attention mechanism, allowing the image segmentation model to consider semantic information while processing image features. Therefore, the image feature vector and word vector are fused through the cross-attention module to learn the relationship between image features and semantic features and convert them into fused features.
[0054] S43, Feature Fusion: After the image feature vector and word vector are fused and transformed through the cross-attention module, they become feature representations with rich semantic information.
[0055] S44, the above fusion features are connected and combined to generate the final fusion image feature vector, so as to obtain more comprehensive and effective image features and provide support for subsequent image segmentation. The functions and effects of the modules involved are as follows:
[0056] Position Attention (PA): Extracts the importance of features at different positions in the image and enhances the feature representation of key areas.
[0057] Channel Attention (CA): Filter and strengthen the most informative channels in the image to improve the discriminability of features.
[0058] Cross-Attention: Learn the relationship between image features and semantic features to generate fused features.
[0059] Dual Attention (DA): The dual attention mechanism is used to further refine and fuse features and improve the expressiveness of features.
[0060] Self-Attention: Used in the criss-cross attention module, it allows the model to consider features of one type while processing features of another type.
[0061] Word Embedding: Convert semantic information into vector form so that it can be integrated with image features.
[0062] Image Embedding: Convert image feature vectors into a form compatible with word vectors to facilitate feature fusion.
[0063] Therefore, the present invention adopts the above-mentioned multi-scale bridge joint network image segmentation method based on multiple attention mechanism, abandons the independent structure of the traditional model, fuses the features of the picture and semantics in a multi-modal and high-dimensional manner, and outputs the encoder after feature fusion through bridge grafting and feature extraction, which is beneficial to the training of the decoder and the final effect output, so that the image segmentation model can make full use of the visual information and related semantic information of the image, and improve the performance and accuracy when processing image-related tasks; in addition, the present invention also increases the interpretability of the model and fills the gap in the interpretability of the model.
[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A multi-scale bridge joint network image segmentation method based on a multi-attention mechanism, characterized in that: The following steps are involved: S1. Take image data from different categories and semantic data corresponding to the images as input to train and generate an image segmentation model; S2. Based on the multiple attention mechanisms, a multi-scale bridge joint network is constructed, including a multi-scale feature extraction structure based on image channel and position attention mechanisms and a multi-scale position feature cross-fusion structure based on image and semantic cross-attention mechanisms; wherein the multiple attention mechanisms include self-attention, dual attention and cross-attention; S3, a multi-scale feature extraction structure based on image channels and position attention mechanism, performs multi-modal feature extraction on image data and semantic data respectively, including the following steps: S31. Use the convolution layer and pooling layer to extract the regional features of the image and capture the local features of the image. The specific process is as follows: S311, feature downsampling: The initial image feature vector is subjected to the feature downsampling operation of the convolutional layer to generate two downsampled feature maps, which reduce their resolution, capture the high-level abstract features of the image, and reduce unnecessary irrelevant information of the image to achieve image feature enhancement; S312, position attention module: The position attention module extracts important information of pixel blocks at different positions in the image, and by calculating the importance of each pixel block relative to other pixel blocks, the image segmentation model can identify key areas in the image; S313, channel attention module: screens and refines information on multi-channel feature maps, identifies and strengthens the most discriminative feature channels; the channel attention module enhances important features and improves the expressiveness of features by evaluating the feature responses of different channels; S32. While extracting features from the image, use natural language processing technology to extract semantic features corresponding to the image from the text description; combine the semantic self-attention distribution embedding, and generate a word vector with fusion features through the cross attention module, which also includes semantic features related to the image; S4, based on the multi-scale position feature cross-fusion structure of the image and semantic cross-attention mechanism, cross-attention feature fusion is performed on the channel information of the image and the pixel block area position information with the feature vector embedded in the semantics; the specific implementation process of feature fusion is as follows: S41, double attention mechanism: before the image feature vector and word vector obtained in step S3 are subjected to feature fusion through the cross attention module, the double attention mechanism is used to further strengthen the features; S42, Cross Attention Module: Contains the self-attention mechanism, allowing the image segmentation model to consider semantic information while processing image features. The image feature vector and word vector are fused through the cross attention module to learn the relationship between image features and semantic features and convert them into fused features. S43, feature fusion: After the image feature vector and word vector are fused and transformed by the cross attention module, they become feature representations with rich semantic information; S44. The above fusion features are connected and combined to generate a final fusion image feature vector, thereby obtaining more comprehensive and effective image features to provide support for subsequent image segmentation.