Convolutional neural network based on unified local-long-range contextual attention mechanism

Abstract

The present invention relates to the field of computers, and in particular to a convolutional neural network based on a unified local-long-range context attention mechanism, which performs self-attention calculations of different scales in multiple dimensions to learn local and global information of feature maps. The network embeds a unified spatial local-long-range context module and a unified channel local-long-range context module to model local context and long-range context from the spatial dimension and channel dimension, respectively. At the same time, an inverse proportional constraint is introduced to achieve implicit communication, and then explicit communication is further achieved by fusing local context and long-range context. This fusion operation ensures that the potential complementarity and synergy between local and global information are not ignored, and the local and global information of feature maps can be learned more effectively.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a convolutional neural network based on a unified local-long-range context attention mechanism. Background Art

[0002] Convolutional neural network is an important neural network in the computer field. At present, the attention mechanism plays an important role in the computer field. It allows the model to adaptively focus on important information and ignore unimportant information, and enhances the perception of discriminative information. Self-attention is a special attention mechanism that uses the dependency between pairs of pixels to capture context and adjust the weight of pixels. Due to the excellent performance of the self-attention mechanism in capturing context. Existing methods mainly explore its adaptability in the computer field from two aspects. On the one hand, the overall information of the feature map is learned by modeling the global context. Although global attention can capture dense long-range context under the global receptive field, its computational complexity is quadratic with the number of pixels. Therefore, another method calculates self-attention within the grid to capture sparse long-range context. Since the pixels within the grid are scattered across the entire feature map, the receptive field of the grid attention is global, which reduces the computational complexity of modeling the global context. On the other hand, the detailed information in the feature map is learned by modeling the local context, and the window attention is used to establish short-range connections between pixels. However, in order to capture the long-range context, they need to use complex operations such as shift window division to achieve cross-window communication. We can also restrict the receptive field of each target pixel to its nearest neighboring pixels by introducing an explicit sliding window attention mechanism similar to ordinary convolution.

[0003] Although the above methods model global context or local context, they implement the modeling of global context and local context independently in different modules, ignoring the potential complementary and synergistic effects between local and global information. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a convolutional neural network based on a unified local-long-range contextual attention mechanism.

[0005] The technical solution adopted by the present invention to solve the technical problem is:

[0006] A convolutional neural network based on a unified local-long-range contextual attention mechanism uses ResNet-50 as the backbone network, and uses the residual block before res_conv4_2 of the ResNet-50 network as the backbone of feature extraction on the entire network. After res_conv4_2, the network is divided into three branches: global feature module branch, branch one, and branch two.

[0007] The global feature module branch is used to extract global information of the feature map;

[0008] The branch 1 embeds a unified spatial local-long range context module ULL-S (Unifying Spatial Local-Long Range Context Module), and the ULL-S includes window attention and grid attention; the window attention divides the spatial dimension into multiple windows, and allows the target pixel to interact with other pixels in the same window where it is located, thereby establishing short-range connections between pixels within the window range to capture the spatial local context; the grid attention divides the feature map into multiple grids, and calculates the relationship matrix of each grid separately to establish interactions between paired pixels in the same grid, thereby establishing grid long-range connections to capture the spatial long-range context of the target pixel; establishes a spatial inverse proportional constraint, merges the window interaction and grid interaction of the target pixel to fuse the spatial local context and the long-range context, which not only directly expands the receptive field of the target pixel, but also realizes explicit communication between windows and explicit communication between grids, thereby improving the effectiveness of spatial global context modeling;

[0009] The branch two is embedded in a Unifying Channel Local-Long Range Context Module ULL-C, which includes interval attention and grid attention. The interval attention divides the channel dimension into multiple intervals, and calculates the relationship matrix of each interval separately to establish the interaction between two channels in the same interval, thereby establishing short-distance connections between channels within the interval range to capture the neighbor context of the target channel and fully learn the correlation between features in the local range. The grid attention divides the channel dimension into multiple grids, each grid has a number of channels selected at fixed intervals, and calculates the relationship matrix of each grid separately to establish the interaction between two channels in the same grid, thereby establishing a long-range sparse connection of the target channel to capture the long-range sparse context of the channel. A channel inverse proportional constraint is established, and the interval interaction and grid interaction are combined to fuse the neighbor context and long-range sparse context of the target channel, thereby broadening the feature perception range of the target channel and improving the effectiveness of channel global context modeling.

[0010] Furthermore, the specific process of the window attention capturing the local context of the space is:

[0011] Assume that the feature map input to ULL-S is X∈R H×W×C , where H, W, and C are the height, width, and number of channels of the feature map, respectively; first, the input tensor X is split into (H / P)×(W / P) non-overlapping windows of size P×P, and a tensor X with a shape of ((H / P)×(W / P), P×P, C) is obtained.’ ; Next, we use two different linear mappings to transform the tensor X ’ Convert to window query tensor X Q1 and the window key tensor X K1 , X Q1 and X K1 With X ’ have the same shape;

[0012] Then the relationship matrix of each window is calculated separately to establish the interaction between pairs of pixels in the same window; the relationship matrix of all windows A1∈R (HW / (P×P))×(P×P)×(P×P) The calculation process can be expressed as:

[0013] A1=X Q1 (X K1 ) T ;

[0014] in, represents matrix multiplication; the (i,j,k)th element in A1 represents the relationship between the jth element and the kth element in the i-th window, and the jth row of the i-th matrix in A1 represents the relationship between the jth element in the i-th window and other elements in the same window. Therefore, A1 simultaneously establishes the interaction between pixels in HW / (P×P) windows. Establishing short-range connections between pixels within the window range can capture the spatial local context of the target pixel, thereby fully learning the local information of the feature map.

[0015] Furthermore, the specific process of the grid attention capturing the spatial long-range context is:

[0016] Grid the input tensor X into a tensor X of shape (G×G, (H / G)×(W / G), C) ’’ , where the spatial dimension is evenly divided into (H / G)×(W / G) grids, and the size of each grid is G×G; to facilitate the calculation of the relationship between pixels in the grid, X ’’ Swap the first two axes and use two different linear mappings to get the grid query tensor X Q2 and the grid key tensor X K2 , X Q2 and X K2 The shape of is ((H / G)×(W / G), G×G, C); the relationship matrix of each grid is calculated separately to establish the interaction between pairs of pixels in the same grid; the relationship matrix of all grids A2∈R (HW / (G×G))×(G×G)×(G×G) The calculation process can be expressed as:

[0017] A2=X Q2 (X K2 ) T ;

[0018] The (i, j, k)th element in A2 represents the relationship between the jth element and the kth element in the i-th grid. A2 also establishes the interaction between the pixels of the (H / G)×(W / G) grids. The connection within the grid is a sparse long-range connection under the global receptive field. Therefore, the context of the target pixel captured by A2 is global and sparse, which reduces the computational complexity of spatial global context modeling.

[0019] Furthermore, in the first branch, the specific process of fusing the spatial local context and the long-range context is as follows:

[0020] Make the window division and grid division satisfy the spatial inverse proportional constraint HW / (P×P)=G×G, that is, the grid size is equal to the number of windows, and the window size is equal to the number of grids;

[0021] Swap the first and second axes of A2 of shape (HW / (G×G),G×G,G×G) to obtain a tensor A of shape (G×G,HW / (G×G),G×G) ’ 2, to align the target pixels in the window relationship matrix A1 and the grid relationship matrix A2;

[0022] Then, concatenate A1 and A ’ 2 Obtain the relation matrix A∈R that integrates local and long-range contexts (G ×G)×(P×P)×(P×P+G×G)) , the matrix A calculates the relationship between the target pixel and its window and any pixel in the grid, expressed as:

[0023] A = Cat(diag(A1), diag(A ’ 2));

[0024] Among them, Cat refers to the operation of concatenating two tensors along the column dimension; diag(•) is the target pixel autocorrelation deduplication operation.

[0025] Furthermore, the spatial local-long-range context is used to generate a spatial attention map:

[0026] First, average pooling and maximum pooling operations are applied to A along the column direction to obtain the context embedding of the target pixel;

[0027] Then, the two pooling results are concatenated, and convolution and Softmax operations are applied to obtain the spatial attention map A. S ∈R 1×H×W :

[0028] A S = Softmax(Conv2d(Cat(mean(A), max(A))));

[0029] Next, the spatial attention map A S Multiply the original feature X element by element, and make the residual of the product result and the original feature X;

[0030] Finally, Batchnorm normalization and GELU activation function are applied to get the output representation A ’ S :

[0031] A ’ S = GELU(Batchnorm(X+(X*A S )));

[0032] Where * is a broadcast multiplication operation.

[0033] Furthermore, the specific process of capturing the neighboring context of the target channel by interval attention is as follows:

[0034] The input tensor Y∈R H×W×C The channel dimension of is evenly divided into M non-overlapping intervals, resulting in a tensor Y with a shape of (M, C / M, H×W) ’ , where each interval contains C / M original index channels;

[0035] Use two different linear mappings to transform Y ’ Convert to interval query tensor Y Q1 and the interval-key tensor Y K1 , Y Q1 and Y K1 With Y ’ have the same shape;

[0036] The relationship matrix of each interval is calculated separately to establish the interaction between two channels in the same interval; the relationship matrix of all intervals E1∈R M×(C×M)×(C×M) :

[0037] E1=Y Q1 (Y K1 ) T ;

[0038] The (i, j, k)th element in E1 represents the relationship between the jth channel and the kth channel in the i-th interval, and the jth row of the i-th matrix in E1 represents the relationship between the jth channel in the i-th interval and other channels in the same interval. Therefore, E1 simultaneously establishes the interaction between channels in M ​​intervals. Establishing short-distance connections between channels within an interval can capture the neighboring context of the target channel, thereby fully learning the correlation between features within a local range.

[0039] Furthermore, the specific process of the grid attention capturing the long-range sparse context of the channel is:

[0040] Reshape the input tensor Y into (C / N, N, H×W) and swap the first two axes to get a tensor Y with shape (N,C / N, H×W) ’’ , where the channel dimension is divided into N grids, each grid has C / N channels selected at fixed intervals;

[0041] Use linear mapping to get the grid query tensor Y Q2 and the grid key tensor Y K2 , Y Q2 and Y K2 The shape and Y ’’ same;

[0042] The relationship matrix of each grid is calculated separately to establish the interaction between two channels in the same grid; the relationship matrix of all grids E2∈R N×(C×N)×(C×N) :

[0043] E2= Y Q2 (Y K2 ) T ;

[0044] The (i, j, k)th element in E2 represents the relationship between the jth channel and the kth channel in the i-th grid, and establishes the relationship between the channels of the N grids. E2 establishes long-range sparse connections for the target channels, reducing the computational complexity of channel global context modeling.

[0045] Furthermore, in the second branch, the specific process of fusing the neighbor context and the long-range sparse context of the target channel is as follows:

[0046] The inverse proportional constraint of the design channel is C / M=N, that is, the interval size is equal to the number of grids, and the grid size is equal to the number of intervals;

[0047] Swap the first and second axes of E2 of shape (N, C / N, C / N) to obtain a tensor E of shape (C / N,N,C / N) ’ 2;

[0048] Then, splice E1 and E ’ 2 Obtain the relationship matrix E∈R that integrates the neighbor context and the remote context M×N×(N+M)) , the matrix E calculates the dependency between the target channel and its interval and any channel in the grid, expressed as:

[0049] E = Cat(diag(E1), diag(E ’ 2));

[0050] Where diag(•) is the target channel autocorrelation deduplication operation.

[0051] Furthermore, the channel-local-long-range context is used to generate the channel-wise attention map:

[0052] First, apply average pooling and maximum pooling operations to E along the column direction;

[0053] Then, the two pooling results are concatenated, and convolution and Softmax operations are applied to obtain the channel attention map E C ∈R C×1×1 :

[0054] E C = Softmax(Conv1d(Cat(mean(E), max(E))));

[0055] Next, the channel attention map E C Multiply the original feature X element by element, and make the residual of the product result and the original feature X;

[0056] Finally, the Batchnorm normalization operation and the GELU activation function are applied to obtain the enhanced feature representation E that incorporates the channel context information. ’ C :

[0057] E ’ C = GELU(Batchnorm(X+(X*F C )));

[0058] Where * is a broadcast multiplication operation.

[0059] Furthermore, the res_conv5_1 residual block of the ResNet-50 network does not perform downsampling operations in order to retain more spatial information without adding additional parameters.

[0060] Technical effects of the present invention:

[0061] Compared with the prior art, the present invention proposes a unified local-long range context attention mechanism to effectively and efficiently learn the local and global information of feature maps. The attention mechanism is implemented as a unified spatial local-long range context module (ULL-S) and a unified channel local-long range context module (ULL-C) in the spatial and channel dimensions respectively. The unified spatial local-long range context module consists of window attention and grid attention. Window attention captures the spatial local context of the target pixel by establishing short-range connections between pixels. Grid attention captures the long-range context of the target pixel by establishing long-range connections between pixels. In order to make window attention and grid attention highly complementary, the present invention proposes a spatial inverse proportional constraint to limit the inverse proportional relationship between the number of windows and the number of grids. Through this constraint, the grid attention naturally realizes cross-window communication, and the window attention also realizes cross-grid communication accordingly, which gives the ULL-S module the ability to capture spatial global dense context. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The structure diagram of the convolutional neural network based on the unified local-long-range context attention mechanism of the present invention;

[0063] Figure 2 It is the structural block diagram of ULL-S of the present invention;

[0064] Figure 3 This is a structural block diagram of the ULL-C of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention is clearly and completely described below in conjunction with the accompanying drawings.

[0066] Example:

[0067] This embodiment involves a convolutional neural network based on a unified local-long-range context attention mechanism, and its overall structure is as follows: Figure 1As shown in the figure, the network uses ResNet-50 as the backbone network, and uses the residual block before res_conv4_2 of the ResNet-50 network as the backbone part of the feature extraction on the entire network. After res_conv4_2, the network is divided into three branches: global feature module branch, branch one, and branch two. At the same time, the res_conv5_1 residual block of the ResNet-50 network does not perform downsampling operations in order to retain more spatial information without adding additional parameters. Among them, the global feature module branch is used to extract the global information of the input feature map, and attention combination is performed on the branch one and branch two, so that the network can pay attention to local context information and global context information in both channels and space. Specifically, the branch one is embedded in the Unifying Spatial Local-Long Range Context Module ULL-S (Unifying Spatial Local-Long Range Context Module), and the branch two is embedded in the Unifying Channel Local-Long Range Context Module ULL-C (Unifying Channel Local-Long Range Context Module), aiming to complement each other from two different dimensions to help the network more effectively enhance the perception of discriminative information. Next, the three branches perform global average pooling, and then perform dimensionality reduction to reduce the feature dimension to 256 dimensions; in the process of back propagation, the 256-dimensional feature vector after dimensionality reduction and the feature vector output by the fully connected layer are used to calculate the triple loss function and the cross entropy loss respectively, and the parameters are updated according to the gradient information to minimize the loss function. In the test phase, the 256-dimensional feature vectors output by the three branches are connected to obtain the final feature embedding.

[0068] 1.ULL-S

[0069] The structure of the ULL-S is as follows Figure 2 As shown in Figure 1, the ULL-S includes window attention and grid attention. The window attention divides the spatial dimension into multiple windows and allows the target pixel to interact with other pixels in the same window, thereby establishing short-range connections between pixels within the window range to capture the spatial local context. Specifically, let the feature map input to ULL-S be X∈R H×W×C , where H, W, and C are the height, width, and number of channels of the feature map, respectively; first, the input tensor X is split into (H / P)×(W / P) non-overlapping windows of size P×P, and a tensor X with a shape of ((H / P)×(W / P), P×P,C) is obtained. ’ ; Next, we use two different linear mappings to transform the tensor X ’ Convert to window query tensor X Q1 and the window key tensor X K1 , XQ1 and X K1 With X ’ have the same shape; then calculate the relationship matrix of each window separately to establish the interaction between pairs of pixels in the same window; the relationship matrix A1∈R of all windows (HW / (P×P))×(P×P)×(P×P) The calculation process can be expressed as:

[0070] A1=X Q1 (X K1 ) T ;

[0071] in, represents matrix multiplication; the (i,j,k)th element in A1 represents the relationship between the jth element and the kth element in the i-th window, and the jth row of the i-th matrix in A1 represents the relationship between the jth element in the i-th window and other elements in the same window. Therefore, A1 simultaneously establishes the interaction between pixels in HW / (P×P) windows. Establishing short-range connections between pixels within the window range can capture the spatial local context of the target pixel, thereby fully learning the local information of the feature map.

[0072] ULL-S not only captures the spatial local context of the target pixel through window short-range connections, but also captures the spatial long-range context of the target pixel through grid long-range connections. For the grid attention, the feature map is divided into multiple grids, and the relationship matrix of each grid is calculated separately to establish the interaction between pairs of pixels in the same grid, thereby establishing grid long-range connections to capture the spatial long-range context of the target pixel. Specifically, the input tensor X is first gridded into a tensor X with a shape of (G×G, (H / G)×(W / G), C) ’’ , where the spatial dimension is evenly divided into (H / G)×(W / G) grids, and the size of each grid is G×G; to facilitate the calculation of the relationship between pixels in the grid, X ’’ Swap the first two axes and use two different linear mappings to get the grid query tensor X Q2 and the grid key tensor X K2 , X Q2 and X K2 The shape of is ((H / G)×(W / G), G×G, C); the relationship matrix of each grid is calculated separately to establish the interaction between pairs of pixels in the same grid; the relationship matrix of all grids A2∈R (HW / (G×G))×(G×G)×(G×G) The calculation process can be expressed as:

[0073] A2=X Q2 (X K2 ) T ;

[0074] The (i, j, k)th element in A2 represents the relationship between the jth element and the kth element in the i-th grid. A2 also establishes the interaction between the pixels of the (H / G)×(W / G) grids. The connection within the grid is a sparse long-range connection under the global receptive field. Therefore, the context of the target pixel captured by A2 is global and sparse, which reduces the computational complexity of spatial global context modeling.

[0075] When modeling the window local context and the grid long-range context separately, the window size P and the grid size G can be set independently without affecting each other. However, in order to make the window context and the grid context have strong complementarity, the window division and the grid division satisfy the spatial inverse proportional constraint HW / (P×P)=G×G, that is, the grid size is equal to the number of windows, and the window size is equal to the number of grids. The above setting has two advantages: on the one hand, the pixels in the grid are sparsely distributed on the entire feature map, and the interval between adjacent pixels is the window size P. Obviously, the pixels in a grid are composed of pixels at the same position of all windows. In this way, ULL-S connects different windows and realizes global communication between windows. Compared with local self-attention, this method realizes global communication between windows with a simpler strategy. On the other hand, the elements in a single window are composed of the elements at the same position of all grids, so the interaction of the elements in the window connects different grids, giving ULL-S the ability to capture spatial global dense context.

[0076] The window interaction and grid interaction of the target pixel are combined to fuse the spatial local context and the long-range context. Specifically, in order to align the target pixel in the window relationship matrix A1 and the grid relationship matrix A2, the first axis and the second axis of A2 with the shape of (HW / (G×G), G×G, G×G) are swapped to obtain a tensor A with the shape of (G×G,HW / (G×G), G×G) ’ 2. Obviously, A ’ The jth row in the i-th matrix in 2 represents the relationship between the i-th pixel in the j-th grid and other pixels in the same grid; since the window division and grid division satisfy the spatial inverse proportional constraint HW / (P×P)=G×G, the i-th pixel in the j-th grid is the same pixel as the j-th pixel in the i-th window; therefore, A ’ The j-th row in the i-th matrix in 2 represents the relationship between the j-th pixel in the i-th window and other elements in the same grid, which means that the relationship matrix A ’ 2 is aligned with A1. Then, splice A1 and A ’ 2 Obtain the relation matrix A∈R that integrates local and long-range contexts (G ×G)×(P×P)×(P×P+G×G)) , the matrix A calculates the relationship between the target pixel and its window and any pixel in the grid, expressed as:

[0077] A = Cat(diag(A1), diag(A ’ 2));

[0078] Among them, Cat refers to the operation of splicing two tensors along the column dimension; diag(•) is the target pixel autocorrelation deduplication operation, which subtracts ln2 from the matrix diagonal elements to eliminate the impact of repeated autocorrelation calculations.

[0079] Using spatial local-long-range context to generate a spatial attention map has lower computational complexity. Specifically, firstly, average pooling and maximum pooling operations are applied to A along the column direction to obtain the context embedding of the target pixel, where average pooling helps to retain overall information and reduce the impact of noise, while maximum pooling focuses on retaining key context information. Then, the results of the two pooling operations are concatenated, and convolution and Softmax operations are applied to obtain the spatial attention map A. S ∈R 1 ×H×W :

[0080] A S = Softmax(Conv2d(Cat(mean(A), max(A)))).

[0081] Next, the spatial attention map A S Multiply the original feature X element by element, and make a residual of the product result and the original feature X; finally, apply the Batchnorm normalization operation and the GELU activation function to get the output representation A ’ S :

[0082] A ’ S = GELU(Batchnorm(X+(X*A S )));

[0083] Where * is a broadcast multiplication operation.

[0084] 2. ULL-C

[0085] The structure of the ULL-C is as follows Figure 3 As shown in the figure, in order to dynamically adjust the weights of different channels to highlight important features, the channel recalibration mechanism should effectively and efficiently capture the dependencies between the target channel and other channels, that is, the context of the target channel. To this end, the ULL-C module uses interval short-range interactions and grid long-range interactions to capture the neighbor context and long-range sparse context of the target channel respectively, and then unifies the modeling of the two contexts through channel inverse proportional constraints and alignment operations, thereby achieving strong complementarity between them.

[0086] Specifically, the ULL-C includes interval attention and grid attention. The interval attention divides the channel dimension into multiple intervals, and calculates the relationship matrix of each interval separately to establish the interaction between two channels in the same interval, thereby establishing short-distance connections between channels within the interval range to capture the neighboring context of the target channel and fully learn the correlation between features in the local range. Specifically, the input tensor Y∈R H×W×C The channel dimension of is evenly divided into M non-overlapping intervals, resulting in a tensor Y with a shape of (M, C / M, H×W) ’ , where each interval contains C / M original index channels. Then two different linear mappings are used to transform Y ’ Convert to interval query tensor Y Q1 and the interval-key tensor Y K1 , Y Q1 and Y K1 With Y ’ Next, the relationship matrix of each interval is calculated separately to establish the interaction between two channels in the same interval; the relationship matrix of all intervals E1∈R M×(C×M)×(C×M) :

[0087] E1=Y Q1 (Y K1 ) T ;

[0088] The (i, j, k)th element in E1 represents the relationship between the jth channel and the kth channel in the i-th interval, and the jth row of the i-th matrix in E1 represents the relationship between the jth channel in the i-th interval and other channels in the same interval. Therefore, E1 simultaneously establishes the interaction between channels in M ​​intervals. Establishing short-distance connections between channels within an interval can capture the neighboring context of the target channel, thereby fully learning the correlation between features within a local range.

[0089] ULL-C not only captures the immediate context of the target channel through interval short-range interactions, but also captures the long-range context of the target channel through grid long-range interactions. For grid attention, the channel dimension is divided into multiple grids, each grid has several channels selected at fixed intervals, and the relationship matrix of each grid is calculated separately to establish the interaction between two channels in the same grid, thereby establishing a long-range sparse connection of the target channel to capture the long-range sparse context of the channel. Specifically, the input tensor Y is transformed into (C / N, N, H×W), and the first two axes are exchanged to obtain a tensor Y with a shape of (N,C / N, H×W) ’’ , where the channel dimension is divided into N grids, each grid has C / N channels selected at fixed intervals. Then, a linear mapping is used to obtain the grid query tensor Y Q2 and the grid key tensor YK2 , Y Q2 and Y K2 The shape and Y ’’ Next, the relationship matrix of each grid is calculated separately to establish the interaction between two channels in the same grid; the relationship matrix of all grids E2∈R N ×(C×N)×(C×N) :

[0090] E2= Y Q2 (Y K2 ) T ;

[0091] The (i, j, k)th element in E2 represents the relationship between the jth channel and the kth channel in the i-th grid, and establishes the relationship between the channels of the N grids. E2 establishes long-range sparse connections for the target channels, reducing the computational complexity of channel global context modeling.

[0092] In order to make the interval neighbor context and the grid remote context have strong complementarity, similar to the ULL-S module, the channel inverse proportional constraint C / M=N is also set in ULL-C, that is, the interval size is equal to the number of grids, and the grid size is equal to the number of intervals. Since the channels in a grid are composed of the channels at the same index of all intervals, the communication between intervals is established by the interaction within the grid. Conversely, since the channels in the interval are composed of the channels at the same index of all grids, the communication between grids is established by the interaction within the interval. This constraint cleverly realizes indirect communication between all channels.

[0093] The interval interaction and grid interaction are combined to fuse the neighboring context and long-range sparse context of the target channel, thereby broadening the feature perception range of the target channel and improving the effectiveness of channel global context modeling. Specifically, in order to align the target channel in the interval relationship matrix E1 and the grid relationship matrix E2, the first axis and the second axis of E2 with a shape of (N, C / N, C / N) are swapped to obtain a tensor E with a shape of (C / N, N,C / N) ’ 2; Obviously, E ’ The jth row in the i-th matrix in 2 represents the relationship between the i-th channel in the j-th grid and other channels in the same grid; due to the constraint condition C / M=N, the i-th channel in the j-th grid and the j-th channel in the i-th interval are the same channel; therefore, E ’ The jth row in the i-th matrix in 2 represents the relationship between the j-th channel in the i-th interval and other channels in the same grid, which means that the relationship matrix E ’ 2 is aligned with E1. Then, splice E1 and E ’ 2 Obtain the relationship matrix E∈R that integrates the neighbor context and the remote contextM×N×(N+M)) , the matrix E calculates the dependency between the target channel and its interval and any channel in the grid, expressed as:

[0094] E = Cat(diag(E1), diag(E ’ 2));

[0095] Where diag(•) is the target channel autocorrelation deduplication operation.

[0096] After obtaining the relationship matrix E, the channel local-long-range context is used to generate the channel-wise attention map E C ∈R C×1×1 :

[0097] E C = Softmax(Conv1d(Cat(mean(E), max(E)))).

[0098] Among them, the maximum pooling can capture the maximum response area in the channel attention map, and the average pooling can reflect the average feature strength of the entire channel; the results of the two pooling are spliced, and convolution and Softmax operations are applied to obtain the channel attention map E C Next, the channel attention map E C Multiply the original feature X element by element, and make a residual of the product result and the original feature X. Finally, apply the Batchnorm normalization operation and the GELU activation function to obtain the enhanced feature representation E that integrates the channel context information ’ C :

[0099] E ’ C = GELU(Batchnorm(X+(X*F C ))).

[0100] 3. Loss Function

[0101] The present invention chooses to use cross entropy loss and triplet loss in the training stage. Among them, cross entropy loss focuses on improving the distinction between categories, while triplet loss focuses on optimizing the distance metric in the feature space. By minimizing these two losses at the same time, the model can learn a more robust feature representation.

[0102] When implementing window attention and grid attention in series or in parallel, the spatial inverse proportional constraint enables cross-window communication and cross-grid communication to be cleverly implemented, so that the two types of attention have strong complementarity, and unify the spatial local context and the spatial long-range context, greatly reducing the amount of calculation. The present invention combines window interaction and grid interaction through alignment operation and target pixel autocorrelation deduplication operation to fuse the local context and long-range context of the target pixel, wherein the spatial inverse proportional constraint enables the alignment operation to be easily implemented. Fusion of local and long-range context not only directly expands the receptive field of the target pixel, but also realizes explicit communication between windows and explicit communication between grids.

[0103] In addition, channel attention and spatial attention are equally important because they enrich the expressive power of features in the channel dimension and spatial dimension respectively. The ULL-C of the present invention uses interval attention and grid attention to capture channel local context and channel long-range sparse context respectively, and then unifies the modeling of these two contexts through channel inverse proportional constraints, alignment operations and target channel autocorrelation deduplication operations. This module captures the context of the target channel in a manner similar to the unified spatial local-long-range context module, effectively and efficiently modeling the attention of the channel dimension. The present invention designs a convolutional neural network with unified local-long-range context based on the unified spatial local-long-range context module and the unified channel local-long-range context module, which can effectively extract comprehensive feature representations of feature maps in spatial dimensions and channel dimensions.

Claims

1. A processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism, characterized in that: The convolutional neural network uses ResNet-50 as the backbone network, and uses the residual block before res_conv4_2 of the ResNet-50 network as the backbone part of feature extraction on the entire network. After res_conv4_2, the network is divided into three branches: Global feature module branch, branch one, branch two; The global feature module branch is used to extract global information of the feature map; The branch 1 embeds a unified spatial local-long-range context module ULL-S, which includes window attention and grid attention; the window attention divides the spatial dimension into multiple windows and allows the target pixel to interact with other pixels in the same window, thereby establishing short-range connections between pixels within the window range to capture the spatial local context; The grid attention divides the feature map into multiple grids, and calculates the relationship matrix of each grid separately to establish the interaction between pairs of pixels in the same grid, thereby establishing grid long-range connections to capture the spatial long-range context of the target pixel; Establish a spatial inverse proportional constraint so that the window division and grid division meet the spatial inverse proportional constraint HW / (P×P)=G×G, merge the window interaction and grid interaction of the target pixel to fuse the spatial local context and long-range context, and generate a spatial attention map based on the fusion result; The second branch embeds a unified channel local-long-range context module ULL-C, which includes interval attention and grid attention; the interval attention divides the channel dimension into multiple intervals, and calculates the relationship matrix of each interval separately to establish the interaction between two channels in the same interval, thereby establishing short-distance connections between channels within the interval to capture the neighboring context of the target channel; The grid attention divides the channel dimension into multiple grids, each grid has a number of channels selected at fixed intervals, and calculates the relationship matrix of each grid separately to establish the interaction between two channels in the same grid, thereby establishing a long-range sparse connection of the target channel to capture the long-range sparse context of the channel; Establish channel inverse proportional constraint and design channel inverse proportional constraint as C / M=N; Combine interval interaction and grid interaction to fuse the neighbor context and long-range sparse context of the target channel, and generate a channel-wise attention map based on the fusion result; The specific process of capturing the spatial local context by window attention is: Assume that the feature map input to ULL-S is X∈R H×W×C , where H, W, and C are the height, width, and number of channels of the feature map, respectively; first, the input tensor X is split into (H / P)×(W / P) non-overlapping windows of size P×P, and a tensor X with a shape of ((H / P)×(W / P), P×P, C) is obtained. ’ ; Next, we use two different linear mappings to transform the tensor X ’ Convert to window query tensor X Q1 and the window key tensor X K1 , X Q1 and X K1 With X ’ have the same shape; Then the relationship matrix between each window query tensor and key tensor is calculated separately to establish the interaction between pairs of pixels in the same window; The specific process of grid attention capturing spatial long-range context is: Grid the input tensor X into a tensor X of shape (G×G, (H / G)×(W / G), C) ’’ , where the spatial dimension is evenly divided into (H / G)×(W / G) grids, and the size of each grid is G×G; X ’’ Swap the first two axes and use two different linear mappings to get the grid query tensor X Q2 and the grid key tensor X K2 , X Q2 and X K2 The shape of is ((H / G)×(W / G), G×G, C); the relationship matrix between each grid query tensor and the key tensor is calculated separately to establish the interaction between pairs of pixels in the same grid; The specific process of capturing the neighboring context of the target channel by interval attention is as follows: The input tensor Y∈R H×W×C The channel dimension of is evenly divided into M non-overlapping intervals, resulting in a tensor Y with a shape of (M, C / M, H×W) ’ , where each interval contains C / M original index channels; Use two different linear mappings to transform Y ’ Convert to interval query tensor Y Q1 and the interval-key tensor Y K1 , Y Q1 and Y K1 With Y ’ have the same shape; The relationship matrix between the query tensor and the key tensor of each interval is calculated separately to establish the interaction between two channels in the same interval; The specific process of the grid attention capturing the long-range sparse context of the channel is: Reshape the input tensor Y into (C / N, N, H×W) and swap the first two axes to get a tensor Y with shape (N,C / N, H×W) ’’ , where the channel dimension is divided into N grids, each grid has C / N channels selected at fixed intervals; Use linear mapping to get the grid query tensor Y Q2 and the grid key tensor Y K2 , Y Q2 and Y K2 The shape and Y ’’ same; The relationship matrix between each grid query tensor and key tensor is calculated separately to establish the interaction between two channels in the same grid.

2. The processing method of the convolutional neural network based on the unified local-long-range contextual attention mechanism according to claim 1 is characterized in that: The relationship matrix A1∈R of all windows (HW / (P×P))×(P×P)×(P×P) The calculation process is: A1=X Q1 (X K1 ) T ; in, Represents matrix multiplication.

3. The processing method of the convolutional neural network based on the unified local-long-range contextual attention mechanism according to claim 2 is characterized in that: The relationship matrix A2∈R of all grids (HW / (G×G))×(G×G)×(G×G) The calculation process is: A2=X Q2 (X K2 ) T 。 4. The processing method of the convolutional neural network based on the unified local-long-range contextual attention mechanism according to claim 3 is characterized in that: In the first branch, the specific process of fusing the spatial local context and the long-range context is as follows: Make the window division and grid division satisfy the spatial inverse proportional constraint HW / (P×P)=G×G; Swap the first and second axes of A2 of shape (HW / (G×G),G×G,G×G) to obtain a tensor A of shape (G×G,HW / (G×G),G×G) ’ 2, to align the target pixels in the window relationship matrix A1 and the grid relationship matrix A2; Then, concatenate A1 and A ’ 2 Obtain the relation matrix A∈R that integrates local and long-range contexts (G×G)×(P×P)×(P×P+G×G)) : A= Cat(diag(A1), diag(A ’ 2)); Among them, Cat refers to the operation of concatenating two tensors along the column dimension; diag(•) is the target pixel autocorrelation deduplication operation.

5. The processing method of the convolutional neural network based on the unified local-long-range contextual attention mechanism according to claim 4 is characterized in that: Use the spatial local-long range context to generate a spatial attention map: First, average pooling and maximum pooling operations are applied to A along the column direction to obtain the context embedding of the target pixel; Then, the two pooling results are concatenated, and convolution and Softmax operations are applied to obtain the spatial attention map A. S ∈R 1×H×W : A S = Softmax(Conv2d(Cat(mean(A), max(A)))); Next, the spatial attention map A S Multiply the original feature X element by element, and make the residual of the product result and the original feature X; Finally, Batchnorm normalization and GELU activation function are applied to get the output representation A ’ S : A ’ S = GELU(Batchnorm(X+(X*A S ))); Where * is a broadcast multiplication operation.

6. The processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism according to claim 1, characterized in that: The relation matrix E1∈R for all intervals M×(C×M)×(C×M) : E1=Y Q1 (Y K1 ) T 。 7. The processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism according to claim 6, characterized in that: The relation matrix E2∈R of all grids N×(C×N)×(C×N) : E2= Y Q2 (Y K2 ) T 。 8. The processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism according to claim 7, characterized in that: In the second branch, the specific process of fusing the neighbor context and the long-range sparse context of the target channel is as follows: Design channel inverse proportional constraint as C / M=N; Swap the first and second axes of E2 of shape (N, C / N, C / N) to obtain a tensor E of shape (C / N, N,C / N) ’ 2; Then, splice E1 and E ’ 2 Obtain the relationship matrix E∈R that integrates the neighbor context and the remote context M×N×(N+M)) , the matrix E calculates the dependency between the target channel and its interval and any channel in the grid, expressed as: E = Cat(diag(E1), diag(E ’ 2)); Where diag(•) is the target channel autocorrelation deduplication operation.

9. The processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism according to claim 8, characterized in that: Use channel-local-long-range context to generate channel-wise attention maps: First, average pooling and maximum pooling operations are applied to E along the column direction; Then, the two pooling results are concatenated, and convolution and Softmax operations are applied to obtain the channel attention map E C ∈R C×1×1 : E C = Softmax(Conv1d(Cat(mean(E), max(E)))); Next, the channel attention map E C Multiply the original feature X element by element, and make the residual of the product result and the original feature X; Finally, the Batchnorm normalization operation and the GELU activation function are applied to obtain the enhanced feature representation E that incorporates the channel context information. ’ C : It is ’ C = GELU(Batchnorm(X+(X*F C ))); Where * is a broadcast multiplication operation.

10. The processing method of a convolutional neural network based on a unified local-long-range contextual attention mechanism according to any one of claims 1 to 9, characterized in that: The res_conv5_1 residual block of the ResNet-50 network does not perform downsampling.

Images