Weakly supervised localization method for guiding transformer across tokens

Abstract

The invention discloses a weakly supervised positioning method of a cross-token guided Transformer, comprising the following steps: collecting an image and preprocessing the image; constructing a Transformer encoder composed of a plurality of basic Transformer blocks and a plurality of position Transformer blocks connected in sequence, and processing the preprocessed image through the basic Transformer block and the position Transformer block in sequence; taking out and averaging the intermediate quantity of each position Transformer block to obtain the averaged intermediate quantity; constructing a filtering and regulating module FRM, and inputting the averaged intermediate quantity into the filtering and regulating module for optimization to obtain a final positioning map; the invention effectively alleviates the context deviation problem around the target area through the encoder, and further improves the clarity and semantic consistency of the positioning map through the filtering and regulating module.

Description

Technical Field

[0001] The present invention relates to the technical field of target positioning, and in particular to a weakly supervised positioning method of a cross-token guided Transformer. Background Art

[0002] In recent years, with the rapid improvement of computer performance and the rapid development of big data technology, visual information data has shown an explosive growth trend. Multimedia data comes in various forms, including static images, dynamic images, video files, and audio files, which are spreading on social media and the Internet at an unprecedented rate. As one of the core technologies in the field of computer vision, target detection technology has been widely used, covering many important fields such as target tracking, behavior recognition, human-computer interaction, autonomous driving, and face recognition. Since more than 80% of human information is obtained through vision, related technologies based on visual information processing have become one of the key research directions in the field of artificial intelligence. Among them, technologies such as face recognition, video surveillance, target detection, image content review, and biometric recognition have gradually become research hotspots and have been widely used in practical scenarios such as medical health, intelligent transportation, smart city management, and public safety, such as medical imaging diagnosis, posture estimation, station security, autonomous driving assistance systems, and behavior analysis in video surveillance.

[0003] Weakly Supervised Object Localization (WSOL) aims to locate the target object by training the model with image-level labels. Most existing technologies are based on Class Activation Mapping (CAM). The localization map generated by this method often only focuses on the salient area of ​​the target and is difficult to fully cover the entire target object. Although subsequent methods have tried to improve the problem of incomplete target activation through strategies such as data enhancement, adversarial learning, and the introduction of visual Transformer, these methods still have shortcomings in positioning performance due to limitations in context modeling capabilities and feature entanglement, especially the latest spatial token method. Although it effectively alleviates the optimization conflict between classification and positioning, diffuse activation is still common in the positioning map. This problem mainly stems from the entanglement of contextual elements, which causes the target area to be disturbed by the co-occurring background, reducing the accuracy and reliability of positioning. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a weakly supervised positioning method of a cross-token guided Transformer, which aims to solve the problems mentioned in the background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a weakly supervised localization method for guiding Transformer across tokens, comprising the following steps:

[0006] Step S1: collecting images and preprocessing them;

[0007] Step S2: Construct a Transformer encoder, which consists of multiple basic Transformer blocks and multiple position Transformer blocks connected in sequence. The preprocessed image is input into the Transformer encoder and processed by the basic Transformer block and the position Transformer block in sequence.

[0008] The overall structure of the basic Transformer block and the position Transformer block is the same, both consisting of a multi-head regular attention module and a multi-layer perceptron; the multi-head regular attention module in the basic Transformer block consists of an attention adjustment module ARM and a self-attention module; the multi-head regular attention module in the position Transformer block consists of an attention adjustment module ARM and a local attention module;

[0009] Step S3: extract the intermediate amount of the Transformer block at each position, average the extracted intermediate amounts, and obtain the averaged intermediate amount;

[0010] Step S4: construct a filtering and adjusting module FRM, input the averaged intermediate quantity into the filtering and adjusting module for optimization, and obtain a final positioning map.

[0011] Furthermore, the specific process of step S1 is as follows:

[0012] Step S1.1: Collect images and Perform preprocessing operations. represents the set of real numbers, Indicates height, Indicates the width. Divide into The image block is expanded and a sequence of image blocks is generated through linear transformation. , represents the total number of image blocks, Represents the dimension of the image block;

[0013] Step S1.2: In the image block sequence Based on this, add a learnable category token , location token and position embedding , and finally form the input sequence .

[0014] Furthermore, the Transformer encoder is configured with base Transformer blocks and Position Transformer blocks, represents the number of basic Transformer blocks in the Transformer encoder, Represents the sum of the base Transformer blocks and position Transformer blocks in the Transformer encoder;

[0015] The processing flow of the Transformer encoder is as follows: the input sequence is used as the input of the first basic Transformer block in the Transformer encoder, the output of the first basic Transformer block passes through the subsequent basic Transformer blocks in sequence, the output of the last basic Transformer block in the Transformer encoder is used as the input of the first position Transformer block in the Transformer encoder, and the output of the first position Transformer block passes through the subsequent position Transformer blocks in sequence.

[0016] Furthermore, the processing flow of the first basic Transformer block in the Transformer encoder is as follows: first, the input sequence is layer-normalized to obtain the normalized sequence ,Will The input is sent to the multi-head regular attention module, and after being processed by the attention adjustment module and the self-attention module, the output of the multi-head regular attention module is obtained. The output of the multi-head regular attention module and the input sequence are concatenated to obtain the intermediate feature. ,Will After layer normalization, the input is into the multi-layer perceptron, and the output and intermediate features of the multi-layer perceptron are Concatenate them to get the output sequence, which is the output of the first basic Transformer block.

[0017] Furthermore, the processing flow of the first position Transformer block in the Transformer encoder is as follows: first, the output of the last basic Transformer block in the Transformer encoder is layer-normalized, and the output after layer normalization is input into the multi-head regular attention module, and is processed by the attention adjustment module ARM and the local attention module in turn to obtain the output of the multi-head regular attention module, and the output of the multi-head regular attention module and the output of the last basic Transformer block in the Transformer encoder are spliced, and the spliced ​​output is layer-normalized and input into the multilayer perceptron, and the output of the multilayer perceptron and the spliced ​​output are spliced ​​to obtain the output of the first position Transformer block.

[0018] Furthermore, the attention adjustment module ARM in the multi-head regular attention module in the basic Transformer block and the position Transformer block has the same structure; the processing flow of the attention adjustment module ARM is as follows: first, the input is linearly transformed to generate a query matrix , key matrix , value matrix ,in, ,Will , , The shape is adjusted to , is the number of heads in the attention adjustment module ARM, is the dimension size of the head in the attention adjustment module ARM, equal to , and then from the shape adjusted , , Extract the location token The location query vector and with category tokens The category query vector , Category Key and category values ;

[0019] Using the position query vector With category key The inner product between them calculates the attention score , expressed as:

[0020] ;

[0021] In the formula, express The transpose of express function;

[0022] Using attention score right The position value in Optimize and get the optimized position value , expressed as:

[0023] ;

[0024] Will Replace with value matrix In the example above, we generate a class-aware value matrix .

[0025] Furthermore, the processing flow of the self-attention module is as follows: first, the query matrix output by the corresponding attention adjustment module ARM is , key matrix Perform matrix multiplication, then scale the result of the matrix multiplication and pass it The function calculates the weight distribution and finally compares the calculated weight distribution with the category-aware value matrix Multiply them together to generate the final output of the self-attention module, which corresponds to the output of the multi-head regular attention module in the basic Transformer block.

[0026] Furthermore, the processing flow of the local attention module is as follows: First, the position query vector output by the corresponding attention adjustment module ARM is and key matrix Compute position attention map , expressed as:

[0027] ;

[0028] In the formula, Represents a zoom operation; express The transpose of

[0029] Query vector by category and key matrix Compute category attention map , expressed as:

[0030] ;

[0031] Among them, the obtained position attention map and category attention map That is, the intermediate quantity of the position Transformer block;

[0032] The position attention map and category attention map Multiply the result and the category-aware value matrix Perform matrix multiplication to obtain the final output of the local attention module, which is the output of the multi-head regular attention module in the Transformer block at the corresponding position.

[0033] Further, the specific process of step S3 is: Position attention map Take the average to generate the final position attention map ; Through Class attention map Take the average to generate the final category attention map ;

[0034] Introducing regional loss function Adjust the final position attention map And the final category attention map The alignment with the target region is expressed as:

[0035] ;

[0036] In the formula, Represents the final position attention map Middle pixels and The value corresponding to the pixel point, Represents the final category attention map Middle pixels and The value corresponding to each pixel; represents a hyperparameter.

[0037] Furthermore, the specific process of step S4 is as follows: And the final category attention map The mean and variance are calculated as:

[0038] ;

[0039] ;

[0040] In the formula, Represents the local mean of the final position attention map; represents the local variance of the final class attention map; express The average pooling operation is Indicates the size of the pooling kernel; represents the local mean of the final category attention map;

[0041] Attention map of the final position And the final category attention map The covariance calculation is performed, expressed as:

[0042] ;

[0043] pass , , and Calculate linear coefficients and bias , expressed as:

[0044] ;

[0045] ;

[0046] In the formula, represents the smoothing factor;

[0047] Through the linear coefficient and bias Optimizing the final category attention map , and obtain the final positioning map , expressed as:

[0048] ;

[0049] Introducing binary cross entropy loss The output of the constrained final positioning map is expressed as:

[0050] ;

[0051] In the formula, express The average value of Represents the smoothing factor.

[0052] Compared with the existing technology, the present invention has the following beneficial effects:

[0053] (1) The present invention achieves remarkable results in weakly supervised target localization tasks through the uniquely designed attention adjustment module ARM and filter adjustment module FRM. The attention adjustment module ARM effectively alleviates the contextual bias problem around the target area by optimizing the spatial distribution of token activation, making the target localization map more focused and accurate. The filter adjustment module FRM enhances the semantic representation ability of the localization map through the local filtering mechanism guided by category tokens, which can highlight the significance of the target area while suppressing irrelevant background features, further improving the clarity and semantic consistency of the localization map.

[0054] (2) Compared with the existing weakly supervised target localization methods based on CAM or Transformer, the present invention innovatively solves the problem of diffuse activation caused by entanglement of context elements, makes up for the limitation of incomplete target activation in existing methods, and significantly improves the modeling ability of contextual relationships. It not only improves the accuracy and robustness of positioning, but also achieves a breakthrough in the bottleneck of positioning performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The present invention is a flow chart of the method. DETAILED DESCRIPTION

[0056] like Figure 1 As shown, the present invention provides a technical solution: a weakly supervised positioning method for guiding Transformer across tokens, comprising the following steps:

[0057] Step S1: Capture images and pre-process them.

[0058] Step S2: Construct a Transformer encoder. The Transformer encoder consists of multiple basic Transformer blocks and multiple position Transformer blocks connected in sequence. The preprocessed image is input into the Transformer encoder and processed by the basic Transformer block and the position Transformer block in sequence.

[0059] Among them, the Transformer encoder is set up with base Transformer blocks and Position Transformer blocks, represents the number of basic Transformer blocks in the Transformer encoder, Represents the sum of the base Transformer blocks and position Transformer blocks in the Transformer encoder.

[0060] Step S3: Take out the intermediate quantity of the Transformer block at each position, average the taken out intermediate quantity, and obtain the averaged intermediate quantity.

[0061] Step S4: construct a filtering and regulating module (FRM), input the averaged intermediate quantity into the filtering and regulating module for optimization, and obtain the final positioning map.

[0062] The specific process of step S1 is:

[0063] Step S1.1: Collect images and Perform preprocessing operations. represents the set of real numbers, Indicates height, Indicates the width. Divide into The image block is expanded and a sequence of image blocks is generated through linear transformation. , represents the total number of image blocks, Represents the dimension of the image patch.

[0064] Step S1.2: In the image block sequence Based on this, add a learnable category token , location token and position embedding , and finally form the input sequence .

[0065] The processing flow of the Transformer encoder is as follows: the input sequence is used as the input of the first basic Transformer block in the Transformer encoder, the output of the first basic Transformer block passes through the subsequent basic Transformer blocks in sequence, the output of the last basic Transformer block in the Transformer encoder is used as the input of the first position Transformer block in the Transformer encoder, and the output of the first position Transformer block passes through the subsequent position Transformer blocks in sequence.

[0066] Among them, the overall structure of the basic Transformer block and the position Transformer block is the same, both consisting of a multi-head regular attention module and a multi-layer perceptron; the multi-head regular attention module in the basic Transformer block consists of an attention regulation module (ARM) and a self-attention module; the multi-head regular attention module in the position Transformer block consists of an attention regulation module (ARM) and a local attention module.

[0067] The processing flow of the first basic Transformer block in the Transformer encoder is as follows: First, the input sequence is layer-normalized to obtain the normalized sequence ,Will The input is sent to the multi-head regular attention module, and after being processed by the attention adjustment module (ARM) and the self-attention module, the output of the multi-head regular attention module is obtained. The output of the multi-head regular attention module and the input sequence are concatenated to obtain the intermediate feature. ,Will After layer normalization, the input is into the multi-layer perceptron, and the output and intermediate features of the multi-layer perceptron are The output sequence is obtained by splicing, that is, the output of the first basic Transformer block. Since the structure of the basic Transformer block is the same, the processing flow of other basic Transformer blocks will not be described in detail here.

[0068] The processing flow of the first position Transformer block in the Transformer encoder is as follows: first, the output of the last basic Transformer block in the Transformer encoder is layer-normalized, and the output after layer normalization is input into the multi-head regular attention module. After being processed by the attention adjustment module (ARM) and the local attention module in turn, the output of the multi-head regular attention module is obtained. The output of the multi-head regular attention module and the output of the last basic Transformer block in the Transformer encoder are spliced, and the spliced ​​output is layer-normalized and input into the multi-layer perceptron. The output of the multi-layer perceptron and the spliced ​​output are spliced ​​to obtain the output of the first position Transformer block. Since the structure of the position Transformer blocks is the same, the processing flow of the Transformer blocks in other positions is not described in detail here.

[0069] Among them, the attention adjustment module (ARM) in the multi-head regular attention module in the basic Transformer block and the position Transformer block has the same structure; the processing flow of the attention adjustment module (ARM) is: first, the input is linearly transformed to generate a query matrix , key matrix , value matrix ,in, ,Will , , The shape is adjusted to , is the number of heads in the attention adjustment module ARM, is the dimension size of the head in the attention adjustment module ARM, equal to , and then from the shape adjusted , , Extract the location token The location query vector and with category tokens The category query vector , Category Key and category values .

[0070] Using the position query vector With category key The inner product between them calculates the attention score , expressed as:

[0071] (1);

[0072] In the formula, express The transpose of express function.

[0073] Using attention score right The position value in Optimize and get the optimized position value , expressed as:

[0074] (2).

[0075] Will Replace with value matrix In the example above, we generate a class-aware value matrix .

[0076] The processing flow of the self-attention module is as follows: first, the query matrix output by the corresponding attention adjustment module (ARM) , key matrix Perform matrix multiplication, then scale the result of the matrix multiplication and pass it The function calculates the weight distribution and finally compares the calculated weight distribution with the category-aware value matrix Multiply them to generate the final output of the self-attention module, which is the output of the multi-head regular attention module in the corresponding basic Transformer block, which is used as the input of the subsequent basic Transformer block or the input of the position Transformer block.

[0077] The processing flow of the local attention module is as follows: First, the position query vector output by the corresponding attention adjustment module (ARM) is and key matrix Compute position attention map , expressed as:

[0078] (3);

[0079] In the formula, Represents a zoom operation; express The transpose of .

[0080] Query vector by category and key matrix Compute category attention map , expressed as:

[0081] (4);

[0082] Among them, the obtained position attention map and category attention map That is, the intermediate quantity of the position Transformer block;

[0083] The position attention map and category attention map Multiply the result and the category-aware value matrix Perform matrix multiplication to obtain the final output of the local attention module, which is the output of the multi-head regular attention module in the Transformer block at the corresponding position, and is used as the input of the subsequent Transformer block.

[0084] The specific process of step S3 is: Since the position Transformer block is So we get the same number of position attention maps ; Through Position attention map Take the average to generate the final position attention map ; Through Class attention map Take the average to generate the final category attention map .

[0085] In order to adjust the alignment between the attention map and the target area, the regional loss function is introduced , expressed as:

[0086] (5);

[0087] In the formula, Represents the final position attention map Middle pixels and The value corresponding to the pixel point, Represents the final category attention map Middle pixels and The value corresponding to each pixel; represents a hyperparameter.

[0088] The specific process of step S4 is:

[0089] Attention map of the final position And the final category attention map The mean and variance are calculated as:

[0090] (6);

[0091] (7);

[0092] In the formula, Represents the local mean of the final position attention map; represents the local variance of the final class attention map; express The average pooling operation is Indicates the size of the pooling kernel; Represents the local mean of the final class attention map.

[0093] Attention map of the final position And the final category attention map The covariance calculation is performed, expressed as:

[0094] (8);

[0095] pass , , and Calculate linear coefficients and bias , expressed as:

[0096] (9);

[0097] (10);

[0098] In the formula, Represents the smoothing factor.

[0099] Through the linear coefficient and bias Optimizing the final category attention map , and obtain the final positioning map , expressed as:

[0100] (11);

[0101] Introducing binary cross entropy loss To constrain the output of the final localization map:

[0102] (12);

[0103] In the formula, express The average value of Represents the smoothing factor.

[0104] Total loss function of the positioning method of the present invention By classification loss function , regional loss function and binary cross entropy loss Composition, expressed as:

[0105] (13);

[0106] In the formula, is a weight parameter used to balance the loss terms.

[0107] Classification loss function Used to optimize the final position attention map And the final category attention map accuracy.

[0108] The proposed method and other methods are compared and evaluated on the CUB-200-2011 and ILSVRC datasets. Tables 1 and 2 show the positioning and classification results of the proposed method on the CUB-200-2011 and ILSVRC datasets, respectively. The evaluation indicators include the positioning accuracy (Loc.Acc.) of the Top-1 indicator, the Top-5 indicator, and the GT-known indicator.

[0109] Experimental results show that the positioning accuracy of the method of the present invention in the CUB-200-2011 dataset is 81.10% and 98.34% for the Top-1 index and GT-known index, respectively, and the accuracy of the Top-1 index and GT-known index in the ILSVRC dataset is 60.35% and 73.33%, respectively. This shows that the method of the present invention performs particularly well in positioning accuracy, which fully verifies the effectiveness of the present invention in weakly supervised target positioning tasks.

[0110] Table 1 Performance comparison of the proposed method and other methods on the CUB-200-2011 dataset

[0111]

[0112] Table 2 Performance comparison of the proposed method and various methods on the ILSVRC dataset

[0113]

[0114] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A weakly supervised localization method based on a cross-token guided Transformer, characterized in that: The steps include: Step S1: collect images and preprocess them; the preprocessing is to divide the images into image blocks, generate sequences from the image blocks, and add category tokens, position tokens and position embeddings based on the sequences; Step S2: Construct a Transformer encoder, which consists of multiple basic Transformer blocks and multiple position Transformer blocks connected in sequence. The preprocessed image is input into the Transformer encoder and processed by the basic Transformer block and the position Transformer block in sequence. The overall structure of the basic Transformer block and the position Transformer block is the same, both consisting of a multi-head regular attention module and a multi-layer perceptron; the multi-head regular attention module in the basic Transformer block consists of an attention adjustment module ARM and a self-attention module; the multi-head regular attention module in the position Transformer block consists of an attention adjustment module ARM and a local attention module; The attention adjustment module ARM in the multi-head regular attention module in the basic Transformer block and the position Transformer block has the same structure; the processing flow of the attention adjustment module ARM is: first, the input is linearly transformed to generate a query matrix , key matrix , value matrix ,in, ,Will , , The shape is adjusted to , represents the set of real numbers, represents the total number of image blocks, represents the dimension of the image block, is the number of heads in the attention adjustment module ARM, is the dimension size of the head in the attention adjustment module ARM, equal to , and then from the shape adjusted , , Extract the location token The location query vector and with category tokens The category query vector , Category Key and category values ; Using the position query vector With category key The inner product between them calculates the attention score , expressed as: ； In the formula, express The transpose of express function; Using attention score right The position value in Optimize and get the optimized position value , expressed as: ； Will Replace with value matrix In the example above, we generate a class-aware value matrix ; Step S3: The intermediate amount of each position Transformer block, i.e., the position attention map and category attention map Take out the intermediate amount, average it, and get the average intermediate amount, which is the final position attention map And the final category attention map ; Step S4: construct a filter adjustment module FRM, input the averaged intermediate quantity into the filter adjustment module for optimization, and obtain a final positioning map; The specific process of step S4 is: And the final category attention map The mean and variance are calculated as: ； ； In the formula, Represents the local mean of the final position attention map; represents the local variance of the final class attention map; express The average pooling operation is Indicates the size of the pooling kernel; represents the local mean of the final category attention map; Attention map of the final position And the final category attention map The covariance calculation is performed, expressed as: ； pass , , and Calculate linear coefficients and bias , expressed as: ； ； In the formula, represents the smoothing factor; Through the linear coefficient and bias Optimizing the final category attention map , and obtain the final positioning map , expressed as: ； In the formula, is the linear coefficient go through The smoothing coefficient obtained by the average pooling operation; Bias go through The smoothing bias obtained by the average pooling operation; For truncation operation.

2. The weakly supervised localization method of the cross-token guided Transformer according to claim 1, characterized in that: The specific process of step S1 is: Step S1.1: Collect images and Perform preprocessing operations. represents the set of real numbers, Indicates height, Indicates the width. Divide into The image block is expanded and a sequence of image blocks is generated through linear transformation. , represents the total number of image blocks, Represents the dimension of the image block; Step S1.2: In the image block sequence Based on this, add a learnable category token , location token and position embedding , and finally form the input sequence .

3. The weakly supervised localization method of the cross-token guided Transformer according to claim 2 is characterized by: The Transformer encoder has a total of base Transformer blocks and Position Transformer blocks, represents the number of basic Transformer blocks in the Transformer encoder, Represents the sum of the base Transformer blocks and position Transformer blocks in the Transformer encoder; The processing flow of the Transformer encoder is as follows: the input sequence is used as the input of the first basic Transformer block in the Transformer encoder, the output of the first basic Transformer block passes through the subsequent basic Transformer blocks in sequence, the output of the last basic Transformer block in the Transformer encoder is used as the input of the first position Transformer block in the Transformer encoder, and the output of the first position Transformer block passes through the subsequent position Transformer blocks in sequence.

4. The weakly supervised localization method of the cross-token guided Transformer according to claim 3 is characterized by: The processing flow of the first basic Transformer block in the Transformer encoder is as follows: First, the input sequence is layer-normalized to obtain the normalized sequence ,Will The input is sent to the multi-head regular attention module, and after being processed by the attention adjustment module and the self-attention module, the output of the multi-head regular attention module is obtained. The output of the multi-head regular attention module and the input sequence are concatenated to obtain the intermediate feature. ,Will After layer normalization, the input is into the multi-layer perceptron, and the output and intermediate features of the multi-layer perceptron are Concatenate them to get the output sequence, which is the output of the first basic Transformer block.

5. The weakly supervised localization method of the cross-token guided Transformer according to claim 4 is characterized by: The processing flow of the first position Transformer block in the Transformer encoder is as follows: first, the output of the last basic Transformer block in the Transformer encoder is layer-normalized, and the output after layer normalization is input into the multi-head regular attention module. After being processed by the attention adjustment module ARM and the local attention module in turn, the output of the multi-head regular attention module is obtained. The output of the multi-head regular attention module and the output of the last basic Transformer block in the Transformer encoder are spliced, and the spliced ​​output is layer-normalized and input into the multilayer perceptron. The output of the multilayer perceptron and the spliced ​​output are spliced ​​to obtain the output of the first position Transformer block.

6. The weakly supervised localization method of cross-token guided Transformer according to claim 5, characterized in that: The processing flow of the self-attention module is as follows: first, the query matrix corresponding to the output of the attention adjustment module ARM is , key matrix Perform matrix multiplication, then scale the result of the matrix multiplication and pass it The function calculates the weight distribution and finally compares the calculated weight distribution with the category-aware value matrix Multiply them together to generate the final output of the self-attention module, which corresponds to the output of the multi-head regular attention module in the basic Transformer block.

7. The weakly supervised localization method of cross-token guided Transformer according to claim 6, characterized in that: The processing flow of the local attention module is as follows: First, the position query vector corresponding to the output of the attention adjustment module ARM is and key matrix Compute position attention map , expressed as: ； In the formula, Represents a zoom operation; express The transpose of Query vector by category and key matrix Compute category attention map , expressed as: ； Among them, the obtained position attention map and category attention map That is, the intermediate quantity of the position Transformer block; The position attention map and category attention map Multiply the result and the category-aware value matrix Perform matrix multiplication to obtain the final output of the local attention module, which is the output of the multi-head regular attention module in the Transformer block at the corresponding position.

8. The weakly supervised localization method of cross-token guided Transformer according to claim 7, characterized in that: The specific process of step S3 is: Position Attention Map Take the average to generate the final position attention map ; Through Class attention map Take the average to generate the final category attention map ; Introducing regional loss function Adjust the final position attention map And the final category attention map The alignment with the target region is expressed as: ； In the formula, Represents the final position attention map Middle pixels and The value corresponding to the pixel point, Represents the final category attention map Middle pixels and The value corresponding to each pixel; Represents a hyperparameter.

9. The weakly supervised localization method of the cross-token guided Transformer according to claim 8, characterized in that: Introducing binary cross entropy loss The output of the constrained final positioning map is expressed as: ； In the formula, express The average value of Represents the smoothing factor.

Images