Bidirectional local-to-global attentions network system for visual representation and method thereof

Abstract

The present invention relates to a bidirectional local-global attention network system and method for visual representation. According to the present invention, a group attention unit that extracts global features of an image from an acquired group attention output value after performing cross-attention using a plurality of group tokens, which are grouped by a set number of image tokens constituting an input image, as a query (GG); a window attention unit that divides the plurality of image tokens into a plurality of windows and extracts local features by performing self-attention on the plurality of windows; and a bidirectional attention unit that performs global-local cross-attention using the local features of the plurality of windows as a query (Qw) and the global features of the image as a pair of key (KG) and value (VG), and performs local-global cross-attention using the global features of the image as a query (QG) and the local features of the plurality of windows as a pair of key (KW) and value (KW). and includes a data combining unit that sums the output values ​​of the global-local cross-attention and the local-global cross-attention together and then fuses them with the local features of a plurality of windows.

Description

Technology Field

[0001] The present invention relates to a bidirectional local-global attention network system and method for visual representation, and more specifically, to a bidirectional local-global attention network system and method for visual representation designed to capture local and global features and introduce a new bidirectional interaction between the two features. Background Technology

[0002] Transformer was initially designed for machine translation and demonstrated remarkable performance in both data modeling and efficiency.

[0003] As a transformer application technology in the field of vision, the DETR (Detection with Transformer) model integrates a transformer decoder into the detection head to model the relationship between image features and object queries. The ViT (Vision Transformer) model uses a transformer encoder for image classification and achieves more promising performance than ConvNets. Various methods are being proposed to improve the efficiency and accuracy of the ViT model.

[0004] As a core component of ViT, self-attention captures long-range dependencies from input tokens but has quadratic complexity depending on the token length. Furthermore, converting these ViT models to high-density prediction tasks incurs massive computational costs. In particular, when applying self-attention to visual tasks to treat pixels as tokens, memory access and computational costs increase due to high input resolution.

[0005] To mitigate these problems, spatial reduction attention and window self-attention have recently been proposed.

[0006] In the case of spatial reduction attention, each query focuses on down-sampled key and token values. Through this design, the computational cost is N 2 / r 2 It can be reduced by (N is the number of tokens, r is the reduction ratio), but relevant areas are ignored and unimportant areas are still maintained.

[0007] Window self-attention performs attention on non-overlapping windows and requires cyclic shift operations to communicate information between windows. Window-based vision transformers improve the efficiency of existing ViTs, where self-attention leads to linear complexity depending on token length.

[0008] Spatial reduction attention models interactions between tokens based on approximate features, whereas window self-attention captures local features within windows. Local and global information are both complementary. Therefore, a system is required that can enhance visual feature representation and modeling performance by utilizing these two features in model blocks.

[0009] The technology forming the background of the present invention is disclosed in Korean Published Patent No. 10-2019-0071527 (published June 24, 2019). The problem to be solved

[0010] The present invention aims to provide a bidirectional local-global attention network system and method capable of capturing local and global characteristics from an image and efficiently exchanging information between the two characteristics through bidirectional interaction to achieve better visual feature representation. means of solving the problem

[0011] The present invention queries (G) a plurality of group tokens grouped into a set number of image tokens constituting an input image. G ) and multiple image tokens as keys (K I) and Value(V I A group attention unit that performs cross attention using pairs of ) and extracts global features of an image from the acquired group attention output values; a window attention unit that divides the plurality of image tokens into a plurality of windows and performs self-attention on the plurality of windows to extract local features; and a query (Q) for the local features of the plurality of windows w ) and global features of the image as the key (K G ) and Value(V G Perform global-local cross-attention using ) pairs, and query the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K W A bidirectional local-global attention network system for visual representation is provided, comprising: a bidirectional attention unit that performs local-global cross-attention using pairs of; and a data combining unit that fuses the local features of a plurality of windows after summing the output values ​​of the global-local cross-attention and the local-global cross-attention.

[0012] In addition, the group attention unit can extract global features of an image by applying the group attention output value to a spatial MLP mixer.

[0013] In addition, the spatial MLP mixer may have a structure in which a GELU activation function is inserted between the first and second fully connected layers (FC Layers).

[0014] In addition, the window attention unit can divide the plurality of image tokens into a plurality of windows that do not overlap each other.

[0015] In addition, the bidirectional local-global attention network system may be implemented as a hierarchical bidirectional local-global attention network structure in which a bidirectional local-global attention block, including the group attention unit, the window attention unit, the bidirectional attention unit, and the combination unit, is stacked in four stages.

[0016] In addition, the image input to the above-mentioned hierarchical bidirectional local-global attention network can be output with the spatial dimension gradually downsampled as it passes through each stage and the channel dimension doubled as it passes through each stage.

[0017] In addition, the feature map output from the above-mentioned hierarchical bidirectional local-global attention network can be input to a CNN model for at least one of image classification, object detection within an image, and semantic segmentation.

[0018] And, the present invention relates to a bidirectional local-global attention execution method performed by a bidirectional local-global attention network system, wherein a plurality of group tokens, grouped by a set number of image tokens constituting an input image, are queried (G G ) and multiple image tokens as keys (K I ) and Value(V I A step of extracting global features of an image from the acquired group attention output value after performing cross-attention using a pair of ); a step of dividing the plurality of image tokens into a plurality of windows and performing self-attention on the plurality of windows to extract local features; and querying the local features of the plurality of windows (Q w ) and global features of the image as the key (K G ) and Value(V G Perform global-local cross-attention using ) pairs, and query the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K WThe method includes the step of performing local-global cross-attention using pairs of ); and the step of merging with the local features of a plurality of windows after summing the output values ​​of the global-local cross-attention and the local-global cross-attention. Effects of the invention

[0019] According to the present invention, better visual feature representation can be achieved by using a bidirectional local-global attention network that captures local and global characteristics from an image and efficiently exchanges information between the two characteristics through bidirectional interaction, thereby improving image analysis performance. Brief explanation of the drawing

[0020] FIG. 1 is a diagram showing the architecture of a hierarchical bidirectional local-global attention network system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing a bidirectional local-global attention network structure according to an embodiment of the present invention. Figure 3 is a diagram showing the bidirectional local-global network of Figure 2 in more detail. Figure 4 is a diagram illustrating a spatial MLP mixer spanning spatial and channel dimensions. FIG. 5 is a diagram illustrating a method for performing bidirectional local-global attention according to an embodiment of the present invention using the network of FIG. 3. Specific details for implementing the invention

[0021] Then, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification have been given similar reference numerals.

[0022] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected," but also cases where they are "electrically connected" with other components interposed between them. Furthermore, when a part is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0023] The present invention relates to a bidirectional local-to-global (BL2G) attention network system, and proposes a bidirectional attention network system capable of achieving better visual feature representation by extracting local and global features from an image and performing bidirectional interaction between the two features.

[0024] FIG. 1 is a diagram showing the architecture of a hierarchical bidirectional local-global attention (hierarchical BL2G) network system according to an embodiment of the present invention.

[0025] Figure 1 (a) shows the overall structure of a hierarchical BL2G attention network system, and (b) shows the detailed structure of a BL2G block included in each stage.

[0026] In FIG. 1, C, H, and W represent the number of base channels of the input feature, the height of the input feature, and the width of the input feature, respectively. FIG. 1 illustrates the input of a 3-channel image. Of course, 32 channels can be used as the base channels instead of 3. L represents the number of stacked BL2G blocks, and MLP (multi-layer perception) represents a multi-perceptron.

[0027] As shown in FIG. 1, a hierarchical BL2G attention network system according to an embodiment of the present invention can be implemented in a hierarchical structure in which four stages are stacked, and each stage can be configured to include the proposed bidirectional local-global attention block (BL2G block).

[0028] An image input to a hierarchical BL2G attention network system can be output with spatial dimensions gradually downsampled with strides of 4, 8, 16, and 32 as it passes through each stage, and channel dimensions can be doubled with respect to the base number of channels (C) as it passes through each stage.

[0029] In addition, the feature map output from the hierarchical BL2G attention network system can be input into a CNN model, through which image classification, object detection and object segmentation, and semantic segmentation can be performed.

[0030] Below, the basic structure of the bidirectional local-global attention (BL2G attention) network included in the BL2G block of Fig. 1 is described in detail.

[0031] FIG. 2 is a schematic diagram showing a bidirectional local-global attention network structure according to an embodiment of the present invention, and FIG. 3 is a diagram showing the bidirectional local-global network of FIG. 2 in more detail.

[0032] As shown in FIGS. 2 and 3, a bidirectional local-global attention network (100) according to an embodiment of the present invention includes a group attention unit (110), a window attention unit (120), a bidirectional attention unit (130), and a data combining unit (140). Here, the operation of each unit (110 to 140) and the data flow between each unit can be controlled by a control unit (not shown).

[0033] This bidirectional local-global attention network (100) can be physically configured and implemented as a computer device including a processor, memory, user interface input / output device and storage device, network input / output unit, etc., and can be implemented as an application program, source code, etc., executed on a computer device or user terminal.

[0034] The group attention unit (110) operates to extract global features of an image using multiple image tokens and group tokens for an input image.

[0035] This group attention unit (110) can obtain a group attention output value by performing cross attention using a plurality of image tokens and a plurality of group tokens for an input image, and can extract global features of the image from the group attention output value and provide them to a bidirectional attention unit (130).

[0036] More specifically, as shown in FIG. 3, the group attention unit (110) comprises a plurality of image tokens (images tokens, Multiple group tokens (11) (Group tokens, )(10) grouped into a set number Query )(G G Used as ) and multiple image tokens (10) Key(K I ) and Value(V I Group attention output value by performing cross-attention using a pair of ). You can obtain.

[0037] Then, the group attention unit (110) has a group attention output value Global features of an image can be extracted by applying it to a spatial MLP mixer (111). Here, the spatial MLP mixer may have a structure in which a GELU activation function is inserted between two fully connected layers (FC Layers).

[0038] The Group Attention process is explained in more detail as follows.

[0039] First, the given image tokens Regarding this, the features of the image can be grouped into G pre-set groups. Here, N is the number of image tokens (N = H × W), H is the height of the image, W is the width, and C is the number of channels of the image. The group attention unit (110) may directly perform the grouping of image tokens, but may also receive values ​​that have been performed in advance by a preprocessing unit, etc.

[0040] The group attention unit (110) has multiple group tokens query(G G Used as ) and multiple image tokens Eul Key(K I ) and Value(V I Group attention can be achieved by performing cross attention using pairs of ).

[0041] To output the attention map, the attention matrix A softmax function δ is applied to each row of. The group attention process can be summarized as shown in Equations 1 and 2.

[0042]

[0043]

[0044] of mathematical formula 1 is the output value h items to generate It represents multi-head group attention composed of is the group attention for the i-th head, and h represents the number of heads.

[0045] of mathematical formula 2 is a group query matrix updated with network parameters, and represents the key (K) matrix and the value (V) matrix. In this case, It represents a linear projection matrix that generates the query (Q), key (K), and value (V).

[0046] Group attention matrix This can mean that N image tokens are grouped into G tokens. The proposed group attention It captures long-range dependencies from image tokens while having a linear computational cost with respect to the number of image tokens N.

[0047] Group attention output value The global features between group tokens can be modified and exchanged in a spatial MLP mixer (111). The spatial MLP mixer (111) can achieve a balance between the number of parameters, gigaflops (GFLOP), and accuracy, and is very suitable for grouping tokens as it operates efficiently for a fixed number of input tokens.

[0048] The spatial MLP mixer (111) consists of two fully connected (FC1, FC2) layers, and An activation function can be inserted between these two fully connected layers to learn a non-linear function in high spatial dimensions. Output value of the spatial MLP mixer (111) It can be expressed as shown in mathematical formula 3 below.

[0049]

[0050] Here, is the group attention output value, FC1 and FC2 are the first and second fully connected layers applied across the spatial dimension, represents a linear projection matrix that mixes grouped tokens across the channel dimension (channel MLP).

[0051] FIG. 4 is a diagram illustrating a spatial MLP mixer across spatial and channel dimensions. Depending on the operation of this spatial ML mixer (111), each grouping token included in the group attention output value can be fully connected with all other tokens, and global information across the grouping tokens can be updated.

[0052] The window attention unit (120) operates to extract local features from multiple image tokens (10). Window self-attention can achieve higher efficiency compared to global self-attention compatible with high input resolution.

[0053] Specifically, as shown in FIG. 3, the window attention unit (120) can divide a plurality of image tokens into a plurality of non-overlapped windows, perform self-attention on each divided window to extract local features, and provide them to the bidirectional attention unit (130).

[0054] The output of window attention is local feature values As such, it can be expressed as shown in mathematical formula 4 below.

[0055]

[0056] Here, non-overlapping windows Represents the query, key, and value matrix projected from. Here, is the number of windows, represents the window area.

[0057] The bidirectional attention unit (130) performs bidirectional interaction between the global features of the image extracted from the group attention unit (110) and the local features of each window extracted from the window attention unit (120).

[0058] To this end, the bidirectional attention unit (130) may include a G2L (Global-to-Local Attention) attention unit (131) and an L2G (Local-to-Global Attention) attention unit (132).

[0059] Specifically, the G2L attention unit (131) queries the local features of multiple windows (Q w ) and global features of the image as the key (K G ) and Value(V G Global-local cross-attention is performed using a pair of ), and the L2G attention unit (132) queries the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K W Local-global cross-attention can be performed using pairs of ).

[0060] Here, in the G2L attention section (131), the local features used as a query Global features used as key-value pairs By participating, global information can be returned as image features. Global features generated by the spatial MLP mixer (111) It can be ungrouped through this cross-attention behavior.

[0061] The result of G2L attention (Global-to-Local Attention) can be calculated as shown in Equation 5 below.

[0062]

[0063] Here, are query, key, and value features, and represents the linear projection matrix.

[0064] In the L2G attention unit (132), the local features obtained from the window attention unit (120) This global feature By interacting with it, it allows local-global relationships to flow from the window attention branch to other branches. This is a local characteristic This global feature It can be achieved by being queried by.

[0065] The result of L2G attention (Local-to-Global Attention) can be calculated as shown in Equation 6 below.

[0066]

[0067] Here, are query, key, and value features, and represents the linear projection matrix.

[0068] Next, the data combining unit (140) is the output value of global-local cross-attention and the output value of local-global cross-attention After summing them together through the first combiner (141), the summed value is used in the second combiner (142) for the local characteristics of a plurality of windows By fusing through shortcut connections, a feature map with enhanced visual effects can be finally derived.

[0069] Based on the bidirectional local-global attention (BL2G attention) network shown in Fig. 2, a BL2G block as shown in Fig. 1 (b) can be obtained. The BL2G block consists of BL2G attention (spatial mixing), MLP (channel mixing), and two mixing ( It can have a structure including two residual connections inserted between them. Similar to a hierarchical backbone, the BL2G network is implemented by stacking BL2G blocks in 4 layers, and the spatial dimension can be downsampled through patch embedding. The number of BL2G blocks used in each of the 4 layers can be configured as {2, 2, 6, 6}, the number of attention heads as {2, 4, 8, 16}, and the MLP ratio value of 4 can be maintained without change across each of the 4 stages.

[0070] FIG. 5 is a diagram illustrating a method for performing bidirectional local-global attention according to an embodiment of the present invention using the network of FIG. 3.

[0071] First, the group attention unit (110) queries a plurality of group tokens for an input image (input feature) (G G ) and multiple image tokens as keys (K I ) and Value(V I A group attention output value is obtained by performing cross attention using a pair of ) (S510).

[0072] Then, the group attention unit (110) inputs the acquired group attention output value into the spatial MLP mixer (111) to extract global features of the image and transmits them to the bidirectional attention unit (130) (S520).

[0073] Next, the window attention unit (120) divides a plurality of image tokens into a plurality of windows, performs self-attention on each window to extract local features, and transmits them to the bidirectional attention unit (130) (S530).

[0074] Then, the bidirectional attention unit (130) performs global-local cross-attention and local-global cross-attention in both directions using the global features of the image and the local features of each window (S540).

[0075] Specifically, the bidirectional attention unit (130) queries the local features of multiple windows (Q w ) and global features of the image as the key (K G ) and Value(V G Perform global-local cross-attention using ) pairs, and query the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K W Local-global cross-attention can be performed using pairs of ). The result of the performance is provided to the data combining unit (140).

[0076] The data combining unit (140) sums the global-local cross-attention output value and the local-global cross-attention output value together and then fuses them with the local features of multiple windows (S550). Through this, a final feature map having a better visual feature representation can be generated.

[0077] According to the present invention as described above, better visual feature representation can be achieved by using a bidirectional local-global attention network that captures local and global characteristics from an image and efficiently exchanges information between the two characteristics through bidirectional interaction, thereby improving image analysis performance.

[0078] The present invention has been described with reference to embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0079] 100: Bidirectional Local-Global Attention Network 110: Group Attention Section 120: Window Attention Section 130: Bidirectional attention section 140: Data joining section

Claims

Claim 1 Query (G) multiple group tokens grouped by a set number of multiple image tokens that constitute the input image. G ) and multiple image tokens as keys (K I ) and Value(V I A group attention unit that performs cross attention using pairs of ) and extracts global features of an image from the acquired group attention output value; a window attention unit that divides the plurality of image tokens into a plurality of windows and performs self-attention on the plurality of windows to extract local features; and a query (Q) for the local features of the plurality of windows w ) and global features of the image as the key (K G ) and Value(V G Perform global-local cross-attention using ) pairs, and query the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K W A bidirectional local-global attention network system for visual representation comprising: a bidirectional attention unit that performs local-global cross-attention using pairs of; and a data combining unit that fuses with local features of a plurality of windows after summing the output values ​​of the global-local cross-attention and the local-global cross-attention. Claim 2 The bidirectional local-global attention network system for visual representation of claim 1, wherein the group attention unit applies the group attention output value to a spatial MLP mixer to extract global features of an image. Claim 3 In claim 2, the spatial MLP mixer is a bidirectional local-global attention network system for visual representation having a structure in which a GELU activation function is inserted between the first and second fully connected layers (FC Layer). Claim 4 In claim 1, the window attention unit is a bidirectional local-global attention network system that divides the plurality of image tokens into a plurality of windows that do not overlap each other. Claim 5 A bidirectional local-global attention network system according to claim 1, wherein the stages comprising the group attention unit, the window attention unit, the bidirectional attention unit, and the data combination unit are stacked in four layers, and the image input to the hierarchical bidirectional local-global attention network is output as a feature map, wherein the spatial dimension is gradually downsampled and the channel dimension is doubled as it passes through each stage. Claim 6 delete Claim 7 In claim 5, the feature map output from the hierarchical bidirectional local-global attention network is input to a CNN model for at least one of image classification, object detection within an image, and semantic segmentation, in a bidirectional local-global attention network system. Claim 8 In a bidirectional local-global attention execution method performed by a bidirectional local-global attention network system, a plurality of group tokens are grouped into a set number of image tokens constituting an input image and queried (G G ) and multiple image tokens as keys (K I ) and Value(V I A step of extracting global features of an image from the acquired group attention output value after performing cross-attention using a pair of ); a step of dividing the plurality of image tokens into a plurality of windows and performing self-attention on the plurality of windows to extract local features; and querying the local features of the plurality of windows (Q w ) and global features of the image as the key (K G ) and Value(V G Perform global-local cross-attention using ) pairs, and query the global features of the image (Q G ) and the local characteristics of multiple windows as a key (K W ) and Value(K W A method for performing bidirectional local-global attention for visual representation, comprising: a step of performing local-global cross-attention using a pair of ); and a step of fusing with the local features of a plurality of windows after summing the output values ​​of the global-local cross-attention and the local-global cross-attention.

Images