Efficient Visual ViT Semantic Segmentation Method Based on Content Awareness and Token Sharing
By using efficient and efficient policy network based on content-awareness in semantic segmentation tasks, the problems of high computational complexity and token redundancy of ViT model are solved, and more efficient computing resource utilization is achieved, suitable for various types of applications. ViT model.
Patent Information
- Application Number
- CN202411146624.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-21
AI Technical Summary
The existing visual Transformer (ViT) model has high computational complexity, token redundancy and architectural complexity problems in semantic segmentation tasks, resulting in inefficiency.
Using an efficient policy network based on content awareness and token sharing, predict whether image blocks can share tokens, and realize the binary classification of hypertile by training the CNN model, reducing the number of tokens that need to be processed, and sharing functions through tokens and Distribution functions optimize token processing to reduce computational complexity.
It significantly improves the computing efficiency of semantic segmentation network, reduces resource consumption, and ensures segmentation quality. It is suitable for various ViT backbone networks and segmentation decoders, and is concise and versatile.
Smart Images

Figure CN118657948B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to an efficient visual ViT semantic segmentation method based on content perception and token sharing, and belongs to the technical field of image processing and computer vision. Background Art
[0002] In recent years, many studies have proposed using Vision Transformer (ViT) to replace convolutional neural networks (CNNs) to solve various computer vision tasks such as image classification, object detection, and semantic segmentation. ViT processes a set of tokens generated by fixed-size image patches through a multi-layer multi-head self-attention mechanism. The existing ViT model has reached the latest research level on multiple tasks, especially the excellent pre-training performance on large-scale datasets, which significantly improves the performance of downstream tasks.
[0003] The semantic segmentation task requires classification of each pixel in the image, which places higher demands on the computational power and efficiency of the model. Traditional CNN methods gradually extract image features through layer-by-layer convolution operations, but these methods usually rely on a large receptive field to capture global information. In contrast, ViT can directly capture the global dependencies of the image using its self-attention mechanism, which theoretically helps improve the accuracy of the segmentation task.
[0004] In practical applications, ViT has been used for a variety of dense prediction tasks, including semantic segmentation tasks, and has shown excellent performance in many benchmarks. For example, the ViT model has achieved the highest performance level on standard datasets such as ADE20K and Cityscapes. The semantic segmentation architecture based on ViT divides the input image into fixed-size image patches (Patch), which are fed into the Transformer encoder after linear embedding and position encoding. The encoder uses a multi-head self-attention mechanism to process the image patch sequence to capture global context information. Subsequently, the encoder output is converted into a two-dimensional feature map by the decoder, and then restored to the original image size after upsampling. Finally, the segmentation head (usually a convolutional layer) is used to generate a category prediction for each pixel to complete the semantic segmentation task. This architecture combines the advantages of Transformer in modeling long-distance dependencies and the ability of traditional convolutional networks in processing local features, effectively improving the performance of semantic segmentation. However, despite the excellent performance of ViT in these tasks, its computational complexity and efficiency issues remain challenges that need to be addressed. To address these challenges, researchers have proposed a series of improvements, such as:
[0005] (1) Hybrid architecture: Some studies combine the advantages of CNN and ViT and propose hybrid architectures. These methods usually use convolutional layers to extract local features in the early stage and then use Transformer to capture global features in the later stage;
[0006] (2) Hierarchical Transformer: Introducing a hierarchical structure, the image is decomposed into image blocks of different scales to reduce the computational burden. For example, the Swin Transformer performs self-attention calculations at different scales through a layered window approach, thereby reducing complexity;
[0007] (3) Dynamic token cropping: Dynamically adjust the number of tokens to be processed according to the image content to reduce the waste of computing resources. For example, researchers proposed a dynamic cropping strategy to decide whether to keep each token based on its importance, thereby improving efficiency.
[0008] Although these improvements have solved the efficiency problem of ViT in semantic segmentation tasks to a certain extent, there are still some problems that need to be solved in this task scenario, such as:
[0009] (1) High computational complexity: The computational complexity of ViT is proportional to the square of the number of input tokens, which becomes particularly inefficient when processing large images. The traditional ViT architecture needs to process a large number of tokens when performing semantic segmentation tasks, which greatly increases the computational burden;
[0010] (2) Token redundancy: In semantic segmentation tasks, multiple image patches may contain the same semantic class, so in some cases it is unreasonable for these image patches to share the same token. This redundancy is not fully utilized by traditional ViT methods;
[0011] (3) Architecture complexity: To improve efficiency, some studies have proposed new ViT architectures that combine global and local attention or introduce CNN-like pyramid structures. However, these new architectures are often complex and require additional design and optimization.
[0012] Therefore, improving the efficiency of ViT-based methods in semantic segmentation task scenarios and significantly reducing computational complexity and resource consumption are issues that need to be addressed urgently. Summary of the invention
[0013] The purpose of the present invention is to overcome the above-mentioned shortcomings and to provide an efficient visual ViT semantic segmentation method based on content perception and token sharing. By using an efficient strategy network to predict which image blocks can share tokens and reduce the number of tokens that need to be processed, the computational efficiency of the visual Transformer semantic segmentation network can be significantly improved while ensuring the segmentation quality.
[0014] The technical solution adopted by the present invention is:
[0015] An efficient visual ViT semantic segmentation method based on content perception and token sharing includes the following steps:
[0016] S1. Establish a token sharing strategy network p , used to predict which super-blocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of super-blocks share the same token;
[0017] S2. Training the token sharing policy network p , until the shared policy network p Convergence, the image I Entering a shared policy network p , get the shared strategy P ;
[0018] S3. Input image I Divide into N image blocks I patch ;
[0019] S4. Semantic segmentation architecture based on ViT, which divides image blocks I patch Sharing policies with tokens P Input Token Sharing Function t s , get a reduced set of tokens T’ ;
[0020] S5. Reduce the token set T’ Input the Transformer network to obtain a set of prediction tokens containing semantic information L ;
[0021] S6. Set the predicted tokens L and shared strategies P Input token distribution function t u , token distribution function t u The prediction token set L According to the sharing strategy P Distribute and upsample, and reorganize to obtain the predicted spatial features L spat ;
[0022] S7. Predict spatial features L spat Input the decoder to get the final semantic segmentation prediction result.
[0023] In the above method, the token sharing strategy network described in step S1 p Its function is achieved by training a CNN model, which predicts whether a super-block contains a single semantic category without predicting the specific category. If a super-block contains only one category, the image blocks in it can share the same token, and it only needs to implement binary classification for each super-block.
[0024] The token sharing strategy network described in step S2 p The training of the backbone network is separate and category-independent, and the token sharing policy network is trained p The required labels are directly generated from the segmentation labels of the semantic segmentation dataset. For each hyperpatch in the image, if it has only one semantic category in the segmentation label, the token sharing strategy network p The token sharing policy network is trained using the standard cross entropy loss using this binary label. p convergence.
[0025] Token sharing function described in step S4 t s The original image block I patch and its position encoding is mapped to a new mapping set I' = {I' 1 ,...,I' M}, where M < N, with corresponding positional encodings, the mapping is based on the predicted sharing strategy P To implement, share strategy P Specify which image blocks in the hyperblock should share tokens and which image blocks in the hyperblock should not share the original tokens. After obtaining the mapping set I', the token sharing function t s Projecting it to the token set T', this process can be formulated as:
[0026] ,
[0027] Among them, the token sharing function t s Based on the sharing strategy P Each hyper-block that can share a token is downsampled to the dimension of a single image block and linearly projected into a single token, thereby achieving token sharing for each image block in the hyper-block.
[0028] The Transformer network described in step S5 can be any semantic segmentation network based on the Transformer architecture, preferably a segmentation network Segmenter based on ViT.
[0029] The token distribution function described in step S6 t u First, follow the sharing strategy P The token dimension is restored through bilinear upsampling, that is, the token sharing of each image block is cancelled, and then the token is reorganized in space. The token is essentially a feature vector learned from each image block. The token is reorganized according to the position of the image block in the original image to obtain the predicted spatial features. L spat .
[0030] The decoder described in step S7 is preferably based on CNN's UPerNet as a decoder.
[0031] An efficient visual ViT semantic segmentation system based on content perception and token sharing includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing as described above is implemented.
[0032] The beneficial effects of the present invention are:
[0033] In view of the high computational complexity and token redundancy of ViT, the present invention uses an efficient policy network to predict which image blocks can share tokens, thereby reducing the number of tokens that need to be processed. This can significantly improve the computational efficiency of the semantic segmentation network based on the visual Transformer while ensuring the segmentation quality. Compared with the existing token-based lightweight Transformer network technology, this method uses a more reliable binary classification network to determine whether tokens need to be shared, thereby ensuring the reliability of the lightweight process, that is, the segmentation efficiency can be significantly improved without losing the segmentation effect. The method proposed in the present invention also has a plug-and-play feature, that is, it is suitable for various ViT backbone networks and segmentation decoders, and is compatible with the latest pre-training methods. The simplicity and versatility of this method make it easy to integrate into all existing ViT-based models, providing an effective method for the study and application of efficient architectures for dense prediction tasks based on Transformer. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 A network framework diagram of the method of the present invention;
[0035] Figure 2 Schematic diagram of image segmentation according to the present invention. DETAILED DESCRIPTION
[0036] The present invention is further described below in conjunction with specific embodiments and drawings.
[0037] Embodiment 1 An efficient visual ViT semantic segmentation method based on content perception and token sharing comprises the following steps (see Figure 1 )as follows:
[0038] S1. Establish a token sharing strategy network p , which is used to predict which hyperblocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of hyperblocks share the same token:
[0039] In the ViT-based semantic segmentation architecture, the input image I ∈R 3×H×W Divided into N image blocks I patch ={I 1 ,...,I N}, where H and W represent the height and width of image I respectively. These image patches are converted into a set of tokens by linear projection. I token ={T 1 ,...,T N}, and then add the corresponding learnable position encoding, which is then input into the Transformer. The total number of tokens N depends on the size of the input image and the predefined image patch size. The Transformer-based backbone network maps the input token Patch into a set of prediction tokens L={L 1 ,...,L N}. The Transformer model can be any architecture that uses global self-attention. The next step is to feed the output tokens L into the decoder. The decoder first rearranges the tokens into features according to their original positions , where E is the feature dimension, H L and W L is the score of the original H and W. Then the feature L spat Input to the decoder g to obtain the segmentation prediction result. The problem to be solved by the present invention is to identify image blocks that can share tokens without reducing Under the premise of semantic segmentation quality, the size of T is reduced to reduce computational complexity and resource consumption.
[0040] The present invention is based on an obvious assumption that there must be image blocks in the image that contain redundant information. This assumption is based on the phenomenon that pixels in a single object in the image are continuous. By analogy with the definition of superpixels, for example, 2×2 adjacent image blocks are defined as superblocks. The 2×2 superblocks that make up a single object only contain one semantic category. Therefore, there must be many semantically uniform areas in the image. It is assumed that by jointly processing these superblocks, efficiency can be improved without reducing segmentation quality. Based on this, the present invention proposes a content-aware token sharing method, which first predicts the input image I Which hyper-map blocks contain only one semantic category, and then let the image blocks in each group of 2×2 hyper-map blocks share the same token.
[0041] In order to make this method applicable to any Transformer-based model, the present invention designs a token sharing function t s , a token distribution function t u , and a token sharing strategy model p , token sharing function t s , Token Distribution Function t u And the token sharing strategy model p They are all implemented in a differentiable way so that they can be inserted into any current Transformer-based model for end-to-end training.
[0042] The role of the token sharing strategy network is to predict whether each superblock can share the same token without reducing the segmentation quality. This function is achieved by training an efficient CNN model, which predicts whether a superblock contains a single semantic category without predicting the specific category. If a superblock contains only one category, the image blocks in it can share the same token. The model is defined as P = p ( I ), where the model network p . It can be implemented with any lightweight network, which only needs to implement binary classification for each super-block. In the present invention, Lite0, the version with the smallest number of parameters in the EfficientNet series network, is used to implement content perception and token sharing. Based on the network p The output of S The token is shared by the super-blocks. S The four image blocks in a 2×2 super-block can share the same token, and finally retain M = N −3 Stokens, these shared tokens are fed into the Transformer network for learning, where N For images I The number of blocks the image is divided into.
[0043] S2. Training the token sharing policy network p , until the shared policy network p Convergence, the image I Entering a shared policy network p , get the shared strategy P :
[0044] The token sharing policy network is trained separately from the backbone network and is also class-independent, and the token sharing policy network is trained p The required labels can be directly generated from the segmentation labels of the semantic segmentation dataset. Specifically, for each hyper-patch in the image, if it has only one semantic category in the segmentation label, the token sharing strategy network p Using this binary label, the token sharing policy network is trained with the standard cross entropy loss. p . Using the converged strategy model p Perform follow-up operations.
[0045] The reason why the token sharing strategy model can be trained in this simple and efficient way is p , because the semantic segmentation task has realistic and available semantic labels. The semantic segmentation annotation provides a label for each pixel, which enables the present invention to directly transform the assumption about the redundancy of semantically similar image blocks into a binary label for each super-block. In addition, the token sharing strategy model p Category independence is the key to the network's efficiency. The binary classification task is much simpler than the multi-class segmentation task, so it can be performed by an efficient model, which is critical to achieving efficiency gains by reducing the number of tokens.
[0046] Image I Input the trained token sharing strategy model p , get the shared strategy P , shared strategy P Indicates whether the image blocks in each 2×2 superblock can share tokens.
[0047] S3. Input image I Divide into N image blocks I patch :
[0048] like Figure 2As shown, the Transformer-based backbone network receives an image block as input. The present invention first divides the input image into N patches and defines adjacent 2×2 image blocks as super-blocks.
[0049] S4. Semantic segmentation architecture based on ViT, which divides image blocks I patch Sharing policies with tokens P Input Token Sharing Function t s , get a reduced set of tokens T’ :
[0050] Token Sharing Function t s (That is, the token sharing module, which is the following set of operations) blocks the original image I patch and its positional encoding is mapped to a new mapping set I' = {I' 1 ,...,I' M}, where M < N, with corresponding positional encodings. This mapping is based on the predicted sharing strategy P To implement, share strategy P Specify which image blocks in the hyperblock should share tokens and which image blocks in the hyperblock should not share the original tokens. After obtaining the mapping set I', the token sharing function t s Project it to the token set T' to facilitate subsequent processing by the Transformer network. The above process can be formulated as:
[0051] ,
[0052] Among them, the token sharing function t s Based on the sharing strategy P Each super-block that can share a token is downsampled to the dimension of a single image block and linearly projected into a single token, thereby achieving token sharing for each image block in a 2×2 super-block.
[0053] S5. Reduce the token set T’ Input the Transformer network to obtain a set of prediction tokens containing semantic information L :
[0054] Get a reduced set of tokens T’ After that, it is input into the Transformer network f , perform feature learning, and obtain a set of prediction tokens L = {L 1 ,...,LM}, the process can be formulated as:
[0055] ,
[0056] Among them, the Transformer network f It can be any semantic segmentation network based on Transformer architecture. In the present invention, the classic model of Segmenter based on ViT is used as the network f To study.
[0057] S6. Set the predicted tokens L and shared strategies P Input token distribution function t u , token distribution function t u The prediction token set L According to the sharing strategy P Distribute and upsample, and reorganize to obtain the predicted spatial features L spat :
[0058] The prediction token set L and shared strategies P Input token distribution function t u , token distribution function t u (That is, the token distribution module, which is the following set of operations) First, according to the sharing strategy P The token dimension is restored by bilinear upsampling, that is, token sharing is cancelled across tiles, and then the tokens are spatially reorganized into predicted spatial features. L spat . Shared strategies in this process P Provide the index corresponding to the image block that has been token shared in step S4, and the token distribution function t u Upsampling is performed based on the index. The process can be formulated as:
[0059] .
[0060] S7. Predict spatial features L spat Input the decoder to get the final semantic segmentation prediction result:
[0061] For predicting spatial features L spat , input it into the decoder g to obtain the final semantic segmentation prediction result O , the process can be formulated as:
[0062] ,
[0063] Among them, the decoder g can adopt a variety of network structures. In the present invention, for efficiency considerations, CNN-based UPerNet is used as a decoder to decode the features.
[0064] Example 2: An efficient visual ViT semantic segmentation system based on content perception and token sharing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing as described in Example 1 is implemented.
[0065] The above is a further description of the present invention in combination with the embodiments, and the protection scope of the present invention is not limited thereto.
Claims
1. An efficient visual ViT semantic segmentation method based on content awareness and token sharing, characterized by: The steps include: S1. Establish a token sharing strategy network p , used to predict which super-blocks in the input image contain only one semantic category, group those containing one semantic category, and then let the image blocks in each group of super-blocks share the same token; S2. Training the token sharing policy network p , until the shared policy network p Convergence, the image I Entering a shared policy network p , get the shared strategy P ; S3. Input image I Divide into N image blocks I patch ; S4. Semantic segmentation architecture based on ViT, which divides image blocks I patch Sharing policies with tokens P Input Token Sharing Function t s , get a reduced set of tokens T’ ; S5. Reduce the token set T’ Input the Transformer network to obtain a set of prediction tokens containing semantic information L ; S6. Set the predicted tokens L and shared strategies P Input token distribution function t u , token distribution function t u The prediction token set L According to the sharing strategy P Distribute and upsample, and reorganize to obtain the predicted spatial features L spat ; S7. Predict spatial features L spat Input the decoder to get the final semantic segmentation prediction result.
2. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: The token sharing strategy network described in step S1 p Its function is achieved by training a CNN model, which predicts whether a super-block contains a single semantic category without predicting the specific category. If a super-block contains only one category, the image blocks in it can share the same token, and it only needs to implement binary classification for each super-block.
3. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: The token sharing strategy network described in step S2 p The training of the backbone network is separate and category-independent, and the token sharing policy network is trained p The required labels are directly generated from the segmentation labels of the semantic segmentation dataset. For each hyperpatch in the image, if it has only one semantic category in the segmentation label, the token sharing strategy network p The label is set to true if the class is set, otherwise it is set to false; using this binary label, the token sharing policy network is trained using the standard cross entropy loss p convergence.
4. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: Token sharing function described in step S4 t s The original image block I patch and their positional encodings are mapped to a new mapping set I' = {I'1,...,I' M }, where M < N, with corresponding positional encodings, the mapping is based on the predicted sharing strategy P To implement, share strategy P Specify which image blocks in the hyperblock should share tokens and which image blocks in the hyperblock should not share the original tokens. After obtaining the mapping set I', the token sharing function t s Projecting it to the token set T', this process can be formulated as: , Among them, the token sharing function t s Based on the sharing strategy P Each hyper-block that can share a token is downsampled to the dimension of a single image block and linearly projected into a single token, thereby achieving token sharing for each image block in the hyper-block.
5. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: The Transformer network described in step S5 can be any semantic segmentation network based on the Transformer architecture.
6. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: The token distribution function described in step S6 t u First, follow the sharing strategy P The token dimension is restored by bilinear upsampling, that is, the token sharing of each image block is cancelled, and then the token is reorganized according to the position of the image block in the original image to obtain the predicted spatial features. L spat .
7. The efficient visual ViT semantic segmentation method based on content perception and token sharing according to claim 1 is characterized in that: The decoder described in step S7 selects CNN-based UPerNet as the decoder.
8. An efficient visual ViT semantic segmentation system based on content perception and token sharing, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the efficient visual ViT semantic segmentation method based on content perception and token sharing is implemented as described in any one of claims 1-7.
