Transform-based diagonal-global attention mechanism image segmentation method

By introducing diagonal-global attention mechanism and L-shaped patch embedding blocks into the image segmentation model, the attention mechanism is optimized, and the problem of low segmentation accuracy of small sample data sets is solved, efficient image segmentation is achieved, and improved Computational efficiency and accuracy.

CN118968047APending Publication Date: 2024-11-15HUAIYIN INSTITUTE OF TECHNOLOGY
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410929602.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing Vision Transformer has low accuracy and high computational volume in image segmentation of small sample datasets, making it difficult to improve segmentation efficiency.

Method used

A method of image segmentation based on Transformer's diagonal-global attention mechanism is proposed. Through diagonal-global attention mechanism and L-shaped patch embedding blocks, the attention mechanism is optimized, the calculation area is reduced, and the feature extraction efficiency is improved. , and delete the [cls] marker in the input sequence to simplify the model structure.

Benefits of technology

It significantly improves the segmentation accuracy of small sample data sets, reduces the number of parameters, improves the calculation efficiency and training speed of the model, and the accuracy rates reach 99.81% and 98.55%. Good experimental results were obtained on the IEEEDataPort platform and homemade data sets. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968047A_ABST
    Figure CN118968047A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, and discloses a transform-based diagonal-global attention mechanism image segmentation method, which is characterized in that a to-be-segmented image is segmented by using an image segmentation model, a self-attention mechanism of a backbone part is improved on the basis of an SETR model, a diagonal-global attention mechanism is adopted, and the self-attention mechanism of the backbone part is improved on the basis of the SETR model. On the basis of paying attention to diagonal patch blocks, part of the embedded blocks are selected to carry out a global attention mechanism, other embedded blocks selectively carry out attention mechanism calculation, the patch embedding mode is also modified, and a common rectangular embedding mode is changed into an L-shaped patch forming mode, so that the patch embedding efficiency is improved. While local detail information of the image is captured, broader context information is obtained. Compared with the prior art, the method can extract the image features as many as possible in the areas of interest as few as possible, and further improves the segmentation precision and reduces the parameter quantity.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image segmentation method based on a transformer-based diagonal-global attention mechanism. Background Art

[0002] Traditional segmentation methods based on manual features usually rely on manually extracted and designed features, which may not cover the diversity and complexity of image features, such as the flame field. In recent years, many researchers have applied deep learning technology to image segmentation, which achieves pixel-level segmentation of images by giving predefined semantic categories pixel by pixel, which is a key research topic in the field of computer vision.

[0003] The fully convolutional network (FCN) proposed by Long et al. borrowed some classic CNN network structures and replaced the fully connected layer with a convolutional layer to reduce the preprocessing work, but it had the problem of insufficient understanding of global information. Olaf et al. proposed Fully Convolutional Networks for Biomedical Image Segmentation (U-Net), which can effectively capture the detailed information of the image and is widely used in various directions of semantic segmentation. The Pyramid scene parsing network (PSPNet) model proposed by Zhao et al. later introduced multi-scale feature aggregation, which has a better effect on understanding complex scenes.

[0004] The Google team proposed Vision Transformer (ViT), which migrated the Transformer in the field of natural language modeling to the image field without modification. In the image segmentation task, all indicators exceeded the CNN model, but the amount of calculation was very large. Many papers have contributed to reducing the amount of calculation. The longformer proposed by Beltagy et al. introduced a global self-attention mechanism and a sparse attention mechanism, and Zhao et al. eliminated the interference of irrelevant information by using the Explicit Sparse Transformer network, making the attention focused. There are also some algorithms that combine CNN and Transformer, such as the Hierarchical Vision Transformer for Semantic Segmentation (SegFormer) proposed by Xie et al., which achieved excellent performance on various semantic segmentation benchmarks.

[0005] However, the above model encountered challenges in dealing with small sample segmentation and improving segmentation efficiency. The existing Vision Transformer currently has problems such as low segmentation accuracy of small sample data sets. How to improve the segmentation accuracy of small sample data sets is an urgent problem that needs to be solved. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides an image segmentation method based on the transformer diagonal-global attention mechanism, and proposes a new attention mechanism to minimize the focus area and extract as many image features as possible, thereby further improving the segmentation accuracy and reducing the number of parameters.

[0007] Technical solution: The present invention provides an image segmentation method based on a transformer-based diagonal-global attention mechanism, comprising the following steps:

[0008] Step 1: Get the image dataset to be processed, preprocess it, and randomly divide it into a training set and a validation set according to the proportion. During preprocessing, first crop the images in the image dataset into square images of size n×n, where n is the image size; and crop them into multiple small square images of size m×m, with each row and column of the image having small square pictures, m is the size of each small square picture, and the corresponding position information is matched for each small square picture;

[0009] Step 2: Build an image segmentation model. The image segmentation model improves the self-attention mechanism based on the SEgementationTRansformer model, and proposes a diagonal-global attention mechanism and an L-shaped patch embedding block;

[0010] The diagonal-global attention mechanism first selects the diagonal small square image area to focus on, then selects a small square image every s rows and s columns to focus on the row and all columns where the small square image is located, and finally selects u small square images on the left and right of the diagonal to focus on, where u is less than A positive integer of ;

[0011] The L-shaped patch embedding block splits the image selected by the diagonal-global attention mechanism into multiple k×k, (k+2)×(k+2), (k+2+2)×(k+2+2)···, p×p square images of different sizes as embedding blocks, where p is less than is a positive integer, k is a positive integer; except for the first k×k embedding block which is directly used for encoder operation, other pictures use the current square embedding block to subtract the previous embedding block to obtain an L-shaped embedding block with a length and width of 2 small square pictures for encoder operation;

[0012] Step 3: Use the improved image segmentation model to segment the image to be processed in step 1.

[0013] Furthermore, when performing encoder operation on the embedding block, a k×k embedding block and several L-shaped embedding blocks are pulled into a straight line and the position information of the corresponding blocks is matched to ensure that each small square image corresponds to a position information, and enters the TransformerLayer one by one for attention mechanism operation.

[0014] Furthermore, each straight-line embedded block enters the TransformerLayer one by one to perform attention mechanism calculation. The specific operation process is as follows:

[0015] Each embedded block pulled into a straight line passes through the normalization layer, the multi-head attention mechanism Multi-HeadAttention, the normalization layer, the fully connected layer mlp, and the normalization layer in turn; the formula of the multi-head attention mechanism is as follows:

[0016]

[0017] Among them, K represents the feature vector of each position in the input sequence, Q represents the feature vector of the current position, and V represents the actual value of the feature vector of each position in the input sequence;

[0018] Finally, the results of multiple embedding blocks that pass through the Transformer Layer individually are added together to form the encoder operation result.

[0019] Furthermore, after the encoder operation of the Transformer Layer, the image segmentation model uses a step-by-step upsampling algorithm to perform a decoder operation to restore the resolution of the original image and obtain the final image segmentation result.

[0020] Furthermore, the decoder operation gradually restores the image size by upsampling by a factor of 2 four times.

[0021] Furthermore, when the diagonal-global attention mechanism selects a small square image every s rows and s columns to pay attention to the row and all columns where the small square image is located, and when it overlaps with the already selected diagonal area, the attention is not repeated; when u small square images on the left and right of the diagonal are selected to pay attention to, when it overlaps with the selected diagonal area or overlaps with the row and all columns where a small square image is selected every s rows and s columns to pay attention to, the attention is not repeated.

[0022] Furthermore, in the diagonal-global attention mechanism, unselected small square images are masked.

[0023] Furthermore, when the L-shaped patch embedding block is split, if either the length or the width obtained by subtracting the previous embedding block from the last embedding block is less than the width of two small square images, it will be ignored and no encoder operation will be performed on it.

[0024] Furthermore, the learnable input sequence head [cls] embedding token of the linear projection sequence head of the SEgementation TRansformer model is deleted.

[0025] Beneficial effects:

[0026] 1. The model of the present invention aims to better solve the challenges of small sample segmentation and improve segmentation efficiency. The global-diagonal attention mechanism is adopted to accelerate processing and improve image utilization by calculating only some important elements. At the same time, in order to reduce the amount of parameters, L-shaped special-shaped embedding blocks are introduced. Each embedding block operates independently through the optimized attention mechanism, and the [cls] tag at the head of the input sequence is deleted to simplify the model structure. In order to verify the experimental results, the above-mentioned model was subjected to a large number of experiments on the official data set and the self-made data set from the IEEEDataPort platform, and good results were achieved, with accuracy rates of 99.81% and 98.55% respectively. The training speed of the model has increased slightly, which has great practical significance for actual production and life.

[0027] 2. The present invention can better capture the long-distance dependencies and global context information in the image by paying attention to the diagonal elements and the elements on the left and right of the diagonal. At the same time, the global attention calculation is performed in an interval selection manner, which can reduce the amount of calculation. The attention mechanism of the present invention pays attention to the small square pictures on the diagonal, so that the model is not only limited to the local area, but also pays more attention to the correlation between different positions in the image, and reduces the number of model parameters. Selecting u small square pictures on the left and right of the diagonal to pay attention to helps the model better capture the long-distance dependencies and global context information in the image. The present invention selects a small square picture to pay attention to every s rows and s columns, and pays attention to all the small square pictures in the row and column where the small square picture is located, so that the model can better pay attention to the global information. Selecting a part of the elements means that the model will not perform global attention calculations on each small square picture, and a part of the calculation amount can be reduced by selecting at intervals.

[0028] 3. The first conventional k×k embedding block of the present invention can capture the local detail information of the image, while the subsequent L-shaped embedding block can obtain broader contextual information within an extended range. Such a combination can capture local and global features at the same time, which helps to improve the model's understanding and representation of images. At the same time, since the length and width of the L-shaped patch block are small, the attention distribution that needs to be calculated is relatively small, which helps to improve the computational efficiency of the model.

[0029] 4. In order to better obtain global information in the traditional Transformer, an additional learnable [cls] embedding tag is added to the head of the linear projection sequence, that is, In the present invention, we choose to delete this embedded mark for the following reasons: First, the diagonal-global attention mechanism of the present invention has already paid attention to both global and local information, and this head is not very helpful to the model of the present invention. Second, this mark is suitable for uniformly distributed embedded blocks and is not suitable for the L-shaped embedded blocks proposed by the present invention. Third, a large number of experiments have proved that deleting [CLS] will not reduce accuracy, but will increase speed. Finally, the segmentation task of the present invention is different from the classification task. The result it outputs is a matrix of the same size as the original image, and classification is performed pixel by pixel, so this embedded mark is not needed. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is the overall structure flow chart of the present invention;

[0031] Figure 2 It is the SETR structure diagram of the present invention;

[0032] Figure 3 This is a flow chart of the self-attention mechanism structure of the present invention;

[0033] Figure 4 A structural diagram of selecting the attention area for the diagonal-global attention mechanism of the present invention;

[0034] Figure 5 A structural diagram of the L-shaped patch embedding block of the present invention when splitting and straightening an image and adding position information;

[0035] Figure 6 The segmentation result of the improved model on the data set 1 in the embodiment of the present invention;

[0036] Figure 7 Comparison of accuracy, loss rate and recall rate of different models on Dataset 1 for the embodiment of the present invention;

[0037] Figure 8 Comparison of the accuracy of different models of the embodiment of the present invention on a self-made data set (data set 2);

[0038] Fig. 9 This is a comparison diagram of the original image segmentation results and labels in Dataset 2 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0040] The present invention discloses an image segmentation method based on a transformer diagonal-global attention mechanism, which is applied to flame image segmentation in this embodiment, and includes the following steps:

[0041] Step 1: Get the image dataset to be processed, preprocess it, and randomly divide it into a training set and a validation set in proportion.

[0042] FLAME official dataset: The dataset used in this work is the flame dataset from the IEEE DataPort platform. In this experiment, the dataset is randomly divided into a training set and a validation set at a ratio of 70% and 30%, so that different models can be trained under the same experimental settings. It is called dataset 1.

[0043] Self-made dataset: In order to enhance the diversity and generalization ability of the dataset, 10% of non-frame flame images were randomly extracted from the original dataset and annotated using AnyLabeling software. This resulted in a total dataset containing 2,203 flame scenes under different weather and lighting conditions, called dataset 2.

[0044] During preprocessing, the images in the image dataset are first cropped into square images of size n×n, where n is the image size; and then cropped into multiple small square images of size m×m, with each row and column of the image having small square pictures, m is the size of each small square picture, and the corresponding position information is matched for each small square picture. In this embodiment, n=256, m=32 are selected, that is, the image is cropped into 256*256 small square pictures, 256*256 is divided into eight rows and eight columns, and the size of each small square picture is 32*32. And the corresponding position information is matched for each small square picture.

[0045] Step 2: Build an image segmentation model. The image segmentation model improves the self-attention mechanism based on the SEgementationTRansformer model and proposes a diagonal-global attention mechanism and an L-shaped patch embedding block.

[0046] In the field of semantic segmentation, SEgementation TRansformer (SETR) is a relatively representative semantic segmentation model with a pure Transformer structure, and the present invention is also improved based on this model.

[0047] The core architecture of SETR is still the Encoder-Decoder structure, but compared with the traditional CNN-dominated encoder structure, SETR uses Transformer instead. Figure 2 This is the overall architecture of SETR. You can see that the Encoder is composed of pure Transformer layers.

[0048] The Encoder process in SETR is consistent with the backbone ViT model. First, the input image is divided into blocks, and then each image block is embedded and position encoded. This process converts the image into a vector sequence. The vector sequence with matched position information is put into TransformerLayer for attention mechanism operation. TransformerLayer includes 24 Transformer layers, each of which is composed of MSA+MLP+LayerNorm+residual connection.

[0049] The decoder in SETR uses two methods:

[0050] One is the step-by-step upsampling algorithm, that is, Figure 2 (b) In order to avoid introducing noise and jagged edges, the encoding obtained by the encoder is gradually restored to its original size through 4 times of upsampling by a factor of 2 to obtain the final segmentation result.

[0051] One is the pyramid upsampling algorithm, that is, Figure 2 (c). Specifically, a feature is extracted from each 6th layer of Eoncder and then reshaped into The size is passed through a three-layer convolutional network of 1×1+3×3+3×3, where the first and third layers reduce the number of channels to half of the original, and the output Then use bilinear interpolation to upsample by 4 times to get These four features are then fused from top to bottom, and the top feature is concat with the three fused features to obtain The size of the network is increased to 4×, and finally the final h×w×c feature map is obtained through 4 times upsampling and convolution.

[0052] The improvement of the algorithm of the present invention is mainly based on the self-attention mechanism of the backbone part, which is different from the traditional attention mechanism in two main aspects. The diagonal-global attention mechanism is adopted. On the basis of paying attention to the diagonal small square pictures, some small square pictures are selected as embedding blocks for global attention mechanism, and other embedding blocks are selectively calculated for attention mechanism to achieve the requirement of paying attention to both the whole and the part at the same time. The embedding method is also modified, and the ordinary rectangular embedding method is changed to the form of L-shaped embedding blocks to ensure that while capturing the local details of the image, broader context information is obtained. The specific structure of the model is as follows: Figure 3 shown.

[0053] The diagonal-global attention mechanism first selects the diagonal small square image area to focus on, that is, Figure 4 Then select a small square image every s rows and columns and focus on the row and all columns where the small square image is located. Finally, select u small square images on the left and right of the diagonal to focus on, where u is less than A positive integer that is equal to . Figure 4 In this embodiment, s=4, u=3, that is, a small square image is selected every 4 rows and 4 columns, and the row and all columns where the small square image is located are focused on. That is, Figure 4 Select the three small squares on the left and right of the diagonal line to focus on, that is, Figure 4 The small squares in the light green area of ​​the image.

[0054] Diagonal-global attention mechanism selects a small square image every 4 rows and 4 columns to focus on the row and all columns where the small square image is located. When it overlaps with the selected diagonal area (dark green), it does not repeat the focus. When selecting 3 small square images on the left and right of the diagonal to focus on, it does not repeat the focus when it overlaps with the selected diagonal area or when it overlaps with the row and all columns where the small square image is selected every 4 rows and 4 columns. Mask processing is performed on the unselected small square images.

[0055] See also Figure 5 , the L-shaped patch embedding block splits the image selected by the diagonal-global attention mechanism into multiple k×k, (k+2)×(k+2), (k+2+2)×(k+2+2)···, p×p square images of different sizes as embedding blocks, where p is less than is a positive integer, k is a positive integer. Except for the first k×k embedding block, which is directly encoder-operated, the other images use the current square embedding block to subtract the previous embedding block to obtain an L-shaped embedding block with a length and width of 2 small square pictures for encoder operation. When the L-shaped patch embedding block is split, if the length and width of the last embedding block minus the previous embedding block are less than 2 small square pictures, it will be ignored and no encoder operation will be performed on it.

[0056] In this embodiment, the 256*256 image is cropped to become an image with 8 rows and 8 columns, and each small square image is 32*32. When splitting, k is 3, that is, it is split into: 3×3, 5×5, 7×7 square images. When splitting in this embodiment, there is an L-shaped embedded block with a width of 1 left at the end, which is not counted in the encoder operation, and p is directly 7.

[0057] When performing encoder operation on the embedding block, a k×k embedding block and several L-shaped embedding blocks are each pulled into a straight line and the position information of the corresponding blocks is matched, ensuring that each small square image corresponds to a position information, and enters the TransformerLayer one by one for attention mechanism operation. In this embodiment, a 3×3 embedding block, a 5×5-3×3 L-shaped embedding block, and a 7×7-5×5 L-shaped embedding block are each pulled into a straight line and the position information of the corresponding blocks is matched, ensuring that each small square image corresponds to a position information, and enters the TransformerLayer one by one for attention mechanism operation.

[0058] Each embedded block drawn into a straight line enters the TransformerLayer one by one to perform the attention mechanism operation. The specific operation process is as follows:

[0059] Each embedded block pulled into a straight line passes through the normalization layer, the multi-head attention mechanism Multi-HeadAttention, the normalization layer, the fully connected layer mlp, and the normalization layer in turn; the formula of the multi-head attention mechanism is as follows:

[0060]

[0061] Among them, K represents the feature vector of each position in the input sequence, Q represents the feature vector of the current position, and V represents the actual value of the feature vector of each position in the input sequence.

[0062] Finally, the results of multiple embedding blocks that pass through TransformerLayer separately are added together to form the encoder operation result.

[0063] After the encoder operation of TransformerLayer, the decoder operation is performed using a step-by-step upsampling algorithm to restore the original image resolution and obtain the final image segmentation result. The decoder operation gradually restores the image size by upsampling by a factor of 2 four times.

[0064] In order to better obtain global information in the traditional Transformer, an additional learnable [cls] embedding tag is added to the head of the linear projection sequence, that is, In the present invention, it is chosen to delete the input sequence header [cls] embedding tag.

[0065] Step 3: Use the improved image segmentation model to segment the image to be processed in step 1.

[0066] The following experimental verification:

[0067] Experimental environment: The network model was built using the PyTorch 1.10 deep learning framework, Python version 3.8, Ubuntu 20.04 system, the initial learning rate and weight decay were both 1e-5, and the image size was 256 × 256. The CPU was AMD EPYC 7742 with 30GB of memory, and the GPU was RTX 2080 Ti * 1.

[0068] Evaluation indicators: This model requires evaluation indicators such as Acc, Precision, F-score, Iou, and Loss for evaluation. Since this experiment is a binary classification, the accuracy can be directly represented by Acc.

[0069]

[0070] Among them, TP is the number of correct positive samples; FP is the number of negative samples that are positive; FN is the number of positive samples that are negative, and TN is the number of correct negative samples. The two evaluation indicators of Precision and Recall mentioned above are inversely proportional, so it is difficult to determine the experimental results simply by these two evaluation indicators. This experiment introduces F-score. F-score is the harmonic mean of Precision and Recall. Through this evaluation indicator, you can more intuitively see the quality of the experimental results.

[0071]

[0072] Experimental results analysis:

[0073] In order to meet the needs of actual production and life and improve the segmentation accuracy while increasing the segmentation speed, a large number of experiments were carried out on dataset 1 and dataset 2 to verify the feasibility of the model.

[0074] (1) FLAME official dataset

[0075] We selected the officially recognized dataset 1 for ablation experiments to verify the effectiveness of the model. The experiment compared the original model (SETR), model 1 with only L-shaped patch embedding blocks, model 2 with only diagonal-global attention mechanism, and the final model. The results show that both model 1 and model 2 outperform the SETR model in terms of Acc, IoU, and Loss. The final model has faster segmentation speed and is more practical while maintaining similar accuracy to model 2. Figure 6 The segmentation results of the final model on dataset 1 are shown, which are highly consistent with the labels, further confirming the effectiveness of the model.

[0076] Table 1 Ablation experiments on dataset 1

[0077]

[0078] according to Figure 7 The comparison results show that Model 1, Model 2 and the final model all perform better than the original model on Dataset 1, which proves that the improvement measures are effective and significantly improve the performance of the model on this dataset.

[0079] During rounds [2,3], the performance degradation of the green line indicates that the L-shaped embedding block may have caused overfitting. The L-shaped embedding block focuses on specific areas of the image, and the learned features are simple and limited. As the training progresses, the model's demand for complex features increases, but the features of the L-shaped embedding block are not sufficient to meet them, resulting in performance degradation. This shows that while the L-shaped embedding block reduces the number of parameters, it may bring instability and overfitting problems. Consider combining the L-shaped embedding block with other modules. At the same time, it is found that the global-local attention mechanism represented by the purple line is stable. This mechanism combined with the L-shaped embedding block can improve the uniqueness of the feature extraction of the L-shaped embedding block while maintaining its stability, thereby improving the model performance. The specific performance is shown by the red line.

[0080] This combination not only maintains the stability of the global-local attention mechanism, but also reduces the number of parameters through the L-shaped embedding block, speeds up the segmentation speed, improves the model efficiency and the accuracy of processing complex tasks. At the same time, limited by the cost issues in real scenarios, it is necessary to ensure the segmentation accuracy of the model, control the number of parameters as much as possible, and increase the segmentation speed. The segmentation accuracy of the two modules and the overall model proposed in the present invention is improved, the number of parameters does not increase too much, and the segmentation speed is improved, which proves the effectiveness of the model.

[0081] (2) Self-made dataset

[0082] Figure 7It shows that although the accuracy of the improved model on dataset 1 is 99%, the limitation of dataset 1 (based only on specific video frames) limits the generalization ability of the model. In order to evaluate the model more accurately, dataset 2 is selected to continue the experiment. Figure 8 The results show that the improved model performs more stably and has higher accuracy on Dataset 2, showing strong adaptability and robustness in different scenarios and data distributions. This not only verifies the stability and superiority of the model, but also proves its generalization ability when facing diverse data. The improvement effect and accuracy of the model are fully demonstrated through testing on different data sets.

[0083] The segmentation results and label comparisons on dataset 2 are as follows Fig. 9 The results show that the model performs well in flame segmentation overall, and most flame areas are accurately identified. However, there are also some areas that are incorrectly identified as flames. This shows that the model may make misjudgments in some cases and needs further optimization to improve accuracy.

[0084] Compared with other mainstream models:

[0085] In order to evaluate the performance of the improved flame segmentation model, three models with different design concepts and applicable scenarios were selected for comparison: U-Net, PSPNet and SegFormer. These models performed well in their respective fields, and the performance of these models was compared under the same experimental conditions.

[0086] (1) FLAME official dataset

[0087] The improved flame segmentation model was compared with other mainstream algorithms on Dataset 1. The results show that the algorithm of the present invention performs well in terms of accuracy, precision and IoU indicators. In particular, the algorithm of the present invention is significantly better than other algorithms in terms of accuracy and IoU, which verifies the effectiveness of the improved design. The algorithm of the present invention captures local and global information, such as the texture, edge, shape and position of the flame, through a sophisticated network structure, which improves the segmentation accuracy and shows a strong advantage in the flame segmentation task.

[0088] Table 2 Comparison results of mAP of different algorithms on the self-made flame smoke dataset

[0089]

[0090] (2) Self-made dataset

[0091] In order to verify the reliability of the algorithm of the present invention, a detailed comparative experiment was conducted on data set 2. The results are as follows: Fig. 9As shown in the figure, the model shows higher stability and efficiency in the flame segmentation task. It is worth mentioning that both the model and SegFormer are combined with Transformer, and their performance is significantly better than the other two pure convolutional models, which confirms the effectiveness of Transformer in improving accuracy. In addition, the model not only speeds up the training speed, but also further improves the final accuracy while maintaining the original accuracy, which further highlights the superiority of the improved algorithm of the present invention.

[0092] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A transformer-based diagonal-global attention mechanism image segmentation method, characterized in that: The steps include: Step 1: Get the image dataset to be processed, preprocess it, and randomly divide it into a training set and a validation set according to the proportion. During preprocessing, first crop the images in the image dataset into square images of size n×n, where n is the image size; and crop them into multiple small square images of size m×m, with each row and column of the image having small square pictures, m is the size of each small square picture, and the corresponding position information is matched for each small square picture; Step 2: Build an image segmentation model. The image segmentation model improves the self-attention mechanism of the backbone part based on the SEgementation TRansformer model, and proposes a diagonal-global attention mechanism and an L-shaped patch embedding block; The diagonal-global attention mechanism first selects the diagonal small square image area to focus on, then selects a small square image every s rows and s columns to focus on the row and all columns where the small square image is located, and finally selects u small square images on the left and right of the diagonal to focus on, where u is less than A positive integer of ; The L-shaped patch embedding block splits the image selected by the diagonal-global attention mechanism into multiple k×k, (k+2)×(k+2), (k+2+2)×(k+2+2)···, p×p square images of different sizes as embedding blocks, where p is less than is a positive integer, k is a positive integer; except for the first k×k embedding block which is directly used for encoder operation, other pictures use the current square embedding block to subtract the previous embedding block to obtain an L-shaped embedding block with a length and width of 2 small square pictures for encoder operation; Step 3: Use the improved image segmentation model to segment the image to be processed in step 1.

2. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 1, characterized in that: When performing encoder operation on the embedding block, a k×k embedding block and several L-shaped embedding blocks are pulled into a straight line and the position information of the corresponding blocks is matched to ensure that each small square image corresponds to a position information, and enter the Transformer Layer one by one for attention mechanism operation.

3. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 2, characterized in that: Each embedded block drawn into a straight line enters the Transformer Layer one by one to perform the attention mechanism operation. The specific operation process is as follows: Each embedded block pulled into a straight line passes through the normalization layer, the multi-head attention mechanism Multi-HeadAttention, the normalization layer, the fully connected layer mlp, and the normalization layer in turn; the formula of the multi-head attention mechanism is as follows: Among them, K represents the feature vector of each position in the input sequence, Q represents the feature vector of the current position, and V represents the actual value of the feature vector of each position in the input sequence; Finally, the results of multiple embedding blocks that pass through the Transformer Layer individually are added together to form the encoder operation result.

4. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 2 or 3, characterized in that: After the encoder operation of the Transformer Layer, the image segmentation model uses a step-by-step upsampling algorithm to perform a decoder operation to restore the resolution of the original image and obtain the final image segmentation result.

5. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 4, characterized in that: The decoder operation gradually restores the image size by upsampling by a factor of 2 four times.

6. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 1, characterized in that: When the diagonal-global attention mechanism selects a small square image every s rows and s columns to pay attention to the row and all columns where the small square image is located, when it overlaps with the already selected diagonal area, it will not pay attention repeatedly; when selecting u small square images on the left and right of the diagonal to pay attention to, when it overlaps with the selected diagonal area or overlaps with the row and all columns where a small square image is selected every s rows and s columns to pay attention to, it will not pay attention repeatedly.

7. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 1, characterized in that: In the diagonal-global attention mechanism, unselected small square images are masked.

8. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 1, characterized in that: When splitting the L-shaped patch embedding block, if either the length or width of the last embedding block minus the previous embedding block is less than the width of two small square images, it will be ignored and no encoder operation will be performed on it.

9. The image segmentation method based on the transformer diagonal-global attention mechanism according to claim 1, characterized in that: The learnable input sequence head [cls] embedding token of the SEgementation TRansformer model’s linear projection sequence head is removed.

Citation Information

Patent Citations

  • Image desensitization method and device, electronic equipment and storage medium

    CN114758145A

  • Method for constructing surface defect detection model based on RecTransform

    CN115578358A

  • Four-axis fusion method based on CNN and Transform

    CN116188928A

  • Brain tissue three-dimensional MRI image segmentation network and storage medium and processor thereof

    CN118072142A

  • Image processor and method

    JP2006229495A