Transformer module processing method with multi-scale pooling

Abstract

The application discloses a transformer module processing method with multi-scale pooling, which can be used in a living body detection model based on a deep neural network to assist in training a CNN branch in a training stage, can provide global context information for the CNN branch, and enhances the modeling capability of the model on long-distance dependence in an image. In addition, by introducing multi-scale pooling into a multi-head self-attention module of the transformer module, the length of a feature sequence can be effectively compressed, the required computing resources of the training model are reduced, extremely discriminative feature representations are obtained, and the performance of the model is further improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for processing transformer modules with multi-scale pooling. Background Technology

[0002] The Transformer is a deep learning model architecture widely used in fields such as natural language processing and computer vision. Key features of the Transformer model include: an attention mechanism-based approach that flexibly weights different parts of the input sequence to better capture long-range dependencies; and strong parallel computing capabilities, significantly improving training and inference efficiency. The emergence of the Transformer architecture has greatly propelled the development of artificial intelligence, providing an efficient and powerful method for processing sequential data.

[0003] While the standard transformer module can perform long-distance dependency modeling and global information exchange, this also leads to high computational complexity. For longer input sequences, the computational and storage overhead increases further.

[0004] In view of this, the present invention was developed to address the problems of high computational complexity and large storage overhead of existing standard transformer modules. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a transformer module processing method with multi-scale pooling that can effectively compress the length of feature sequences and reduce the computational resources required for training models.

[0006] To achieve the above objectives, the solution of the present invention is: A method for processing transformer modules with multi-scale pooling includes the following steps: Step B1: The input features of shape [C,H,W] are serialized into a tensor of shape [(H / P)×(W / P),C], where C is the channel, H is the height, W is the width, and P is the size of the patch, which can also be understood as the downsampling factor. The tensors from steps B2 and B1 are processed by a multi-head self-attention module with multi-scale pooling. The multi-head self-attention module utilizes pooling layers of different scales to obtain contextual information at different scales and reduce [the impact of these differences]. Dimensions; Step B3: Add the outputs of Step B1 and Step B2 together and perform layer normalization. Step B4: Feed the output of step B3 into the feedforward neural network. The feedforward neural network consists of convolution and activation functions, which perform nonlinear transformations and mappings to obtain two-dimensional nearest neighbor information and enhance the representation ability of the model. Step B5: Add the outputs of Step B3 and Step B4 together and perform layer normalization. Step B6: Perform feature deserialization on the output of step B5 to transform the output features into tensors of shape [C, H / P, W / P].

[0007] Furthermore, in step B1, the feature serialization processing steps include: Step C1: The input features pass through the convolution module; Step C2: Flatten the output of the convolution module; Step C3: Dimensional Transformation; Step C4: Use layer normalization to process the features to obtain the output features.

[0008] Furthermore, in step B2, let the input features be... The shape is [B, H × W, C], where B is the batch size. The processing steps of the multi-head self-attention module with multi-scale pooling include: Step D1: Reshape input features The shape is changed to [B,C,H,W]; the input feature is then processed using average pooling layers of different scales, P1, P2, P3, and P4. The following characteristics are obtained. , , , The shapes are: [B,C,H / P1,W / P1], [B,C,H / P2,W / P2], [B,C,H / P3,W / P3], and [B,C,H / P4,W / P4].

[0009]

[0010]

[0011]

[0012] Step D2: Reshape the four features output from step D1 , , , This transforms their shapes into [B,C,(H / P1)×(W / P1)], [B,C,(H / P2)×(W / P2)], [B,C,(H / P3)×(W / P3)], and [B,C,(H / P4)×(W / P4)]. Combining these features, the shape becomes [B,C,S].

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] in, Indicates feature reshaping, This represents the concatenation function. S represents the concatenated output features; S represents the feature. The length of the last dimension; Step D3: Reconstruct the spliced ​​output features from step D2 and perform layer normalization to obtain the features. The shape becomes [B,S,C].

[0019] in, Presentation layer normalization processing, The function represents the permutation operation; Step D4: Analyze the features output in step D3. The feature is fed into a fully connected layer (FC) to obtain the feature. The shape becomes [B,S,2C]; subsequently, the features are reshaped to obtain the features. The shape becomes [2, B, N, S, C / N], where N is the number of heads in the multi-head self-attention module, used to control the number of attention heads in the attention mechanism.

[0020]

[0021] in, Functions are used to change the shape of dimensions; Step D5: Segment the features output from step D4 The segmented features are as follows and All shapes are [B, N, S, C / N]:

[0022] Step D6: Input features Send into the fully connected layer Given a shape of [B, H × W, C], reshape the features to obtain the feature. Its shape is [B, N, H × W, C / N].

[0023] Step D7: Reshape Features The shape is obtained as a feature with shape [B, N, C / N, S]. ;right and Performing matrix multiplication, the resulting feature shape becomes [B, N, H]. [W,S]; Perform feature analysis along the last dimension. Processing to obtain features :

[0024]

[0025] in, Indicates matrix multiplication. Used as the output layer of a neural network, it converts the network's output into the probability of each category appearing. Step D8: The features output in step D7 With features Perform matrix multiplication to obtain the features The shape is [B,N,H] [W,C / N]; Reshaping features , to obtain features The shape becomes [B,H] W,C]:

[0026]

[0027] in, Indicates transpose; Step D9: The features output in step D8 Send into the fully connected layer To obtain the final output features The shape is [B,H] W,C]: .

[0028] Furthermore, in step B4, the fully connected layers in the feedforward neural network are replaced with local convolutions, and a connection is made after the local convolutions. Activation function The activation function provides a non-linear transformation when the input value is between -3 and 3, which alleviates the gradient vanishing problem while enhancing the expressive power of the model.

[0029] Furthermore, the feedforward neural network processing steps include: Step E1: Input Features The tensor, with shape [H] W,C], for input features Perform feature reshaping to obtain features The shape becomes [C,H,W]:

[0030] Step E2: After passing through two convolutional layers and an activation layer consecutively, the features are obtained. :

[0031] in, The formula for calculating the activation function is as follows:

[0032]

[0033] Step E3: Obtain features after passing through the convolution module. ,feature After reshaping and transposing, the output features are obtained. The shape is [H] W,C],

[0034] .

[0035] Furthermore, in step B6, feature deserialization is equivalent to feature reshaping, the purpose of which is to change the shape of the input features from [H]... [W,C] is reshaped into [C,H,W]. .

[0036] By adopting the above scheme, the transformer module processing method with multi-scale pooling of the present invention can effectively compress the length of the feature sequence and reduce the computational resources required for training the model by introducing multi-scale pooling into the multi-head self-attention module of the transformer module, while obtaining highly discriminative feature representations, thereby further improving the performance of the model. Attached Figure Description

[0037] Figure 1 This is a flowchart of the transformer module with multi-scale pooling in this invention.

[0038] Figure 2 This is a structural diagram of the feature serialization module of the present invention.

[0039] Figure 3 This is a flowchart of the multi-head self-attention module with multi-scale pooling of the present invention.

[0040] Figure 4 The flowchart of the feedforward neural network of this invention. Detailed Implementation

[0041] To further explain the technical solution of the present invention, the present invention will be described in detail below through specific embodiments.

[0042] like Figure 1 As shown, this invention discloses a transformer module with multi-scale pooling, featuring a multi-head self-attention module with multi-scale pooling. Multiple transformer modules constitute transformer branches, used to assist in training CNN branches during the training phase. This provides global contextual information to the CNN branches, enhancing the model's ability to model long-range dependencies in images. Furthermore, by introducing multi-scale pooling into the multi-head self-attention module of the transformer module, the length of the feature sequence can be effectively compressed, reducing the computational resources required for training the model, while simultaneously obtaining highly discriminative feature representations.

[0043] While standard transformer modules can perform long-distance dependency modeling and global information interaction, this also leads to high computational complexity. For longer input sequences, the computational and storage overhead increases further. Therefore, this invention introduces multi-scale pooling into the transformer module to form a transformer module with multi-scale pooling. This effectively compresses the length of feature sequences, reduces the computational resources required to train the model, and simultaneously obtains highly discriminative feature representations.

[0044] The processing methods of the transformer module with multi-scale pooling include: Step B1: The input features of shape [C,H,W] are serialized into a tensor of shape [(H / P)*(W / P),C], where C is the channel, H is the height, W is the width, and P is the size of the patch, which can also be understood as the downsampling factor. Step B2: The tensor passes through a multi-head self-attention module with multi-scale pooling. The multi-head self-attention module utilizes pooling layers of different scales to obtain contextual information at different scales and reduce [the impact of these limitations]. (key) and The size of (value); Step B3: Add the outputs of Step B1 and Step B2 together and perform layer normalization. Step B4: Feed the output of step B3 into the feedforward neural network. The feedforward neural network consists of convolution and activation functions, which perform nonlinear transformations and mappings to obtain two-dimensional nearest neighbor information and enhance the representation ability of the model. Step B5: Add the outputs of Step B3 and Step B4 together and perform layer normalization. Step B6: Perform feature deserialization on the output of step B5 to transform the output features into tensors of shape [C, H / P, W / P].

[0045] like Figure 2 As shown, in step B1, the feature serialization processing steps include: Step C1: The input features are processed by a convolution module with a stride of P (dividing the image into N small blocks of size P). Step C2: Flatten the output of the convolution module; Step C3: Dimensional Transformation; Step C4: Use layer normalization to process the features to obtain the output features.

[0046] like Figure 3 As shown, in step B2, let the input features be... The shape is [B, H × W, C], where B is the batch size. The processing steps of the multi-head self-attention module with multi-scale pooling include: Step D1: Reshape the input feature to [B, C, H, W]. Process the feature using average pooling layers of different scales (P1, P2, P3, P4) to obtain the following features. , , , The shapes are: [B,C,H / P1,W / P1], [B,C,H / P2,W / P2], [B,C,H / P3,W / P3], and [B,C,H / P4,W / P4].

[0047]

[0048]

[0049]

[0050] Step D2: Reshape the four features output from step D1 , , , This transforms their shapes into [B,C,(H / P1)×(W / P1)], [B,C,(H / P2)×(W / P2)], [B,C,(H / P3)×(W / P3)], and [B,C,(H / P4)×(W / P4)], respectively. By combining these features, the shape becomes [B,C,S].

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in, Indicates feature reshaping, This represents the concatenation function. This indicates the concatenated output features, where S represents the feature. The length of the last dimension.

[0057] Step D3: Reconstruct the spliced ​​output features from step D2 and perform layer normalization to obtain the features. The shape becomes [B,S,C].

[0058] in, Presentation layer normalization processing, The function represents a permutation operation.

[0059] Step D4: Analyze the features output in step D3. The feature is fed into a fully connected layer (FC) to obtain the feature. The shape becomes [B,S,2C]. Then, the features are reshaped to obtain the feature set. The shape becomes [2, B, N, S, C / N], where N is the number of heads in the multi-head self-attention module, used to control the number of attention heads in the attention mechanism.

[0060]

[0061] in, Functions are used to change the shape of dimensions; Step D5: Segment the features output from step D4 The segmented features are as follows (key) and (value), all with shapes [B, N, S, C / N]:

[0062] Step D6: Input features Send into the fully connected layer The shape is [B, H × W, C]. Reshape the features to obtain the desired features. (query), with shape [B,N,H×W,C / N],

[0063] Step D7 / Reshaping Features The shape is obtained as a feature with shape [B, N, C / N, S]. .right and Perform matrix multiplication; the resulting feature shape becomes [B, N, H*W, S]. Then, perform matrix multiplication on the features along the last dimension. Processing to obtain features :

[0064]

[0065] in, Indicates matrix multiplication. Used as the output layer of a neural network, it converts the network's output into the probability of each category appearing.

[0066] Step D8: The features output in step D7 With features Perform matrix multiplication to obtain the features The shape is [B,N,H] [W,C / N]. Reshaping features , to obtain features The shape becomes [B,H] W,C]:

[0067]

[0068] in, This indicates transpose.

[0069] Step D9: The features output in step D8 Send into the fully connected layer Then the final output features can be obtained. The shape is [B,H] W,C]: .

[0070] like Figure 4 As shown, in step B4, this invention replaces the fully connected layers in the feedforward neural network with local convolutions, aiming to help the model acquire two-dimensional nearest neighbor information while reducing the model's computational load. A connection is then placed after the local convolution. Activation function The activation function provides a non-linear transformation when the input value is between -3 and 3, alleviating the vanishing gradient problem while enhancing the model's expressive power. Furthermore, the computation of this activation function is relatively simple, involving only multiplication and addition.

[0071] Ignoring batch size dimension, input features The shape is [H The feedforward neural network processing steps include: [W,C] Step E1: Input features Perform feature reshaping to obtain features The shape becomes [C,H,W]:

[0072] Step E2: After passing through two convolutional layers and an activation layer consecutively, the features are obtained. :

[0073] in, The formula for calculating the activation function is as follows:

[0074]

[0075] Step E3: Obtain features after passing through the convolution module. ,feature After reshaping and transposing, the output features are obtained. The shape is [H] W,C],

[0076] .

[0077] In step B6, feature deserialization is equivalent to feature reshaping, the purpose of which is to change the shape of the input features from [H] [W,C] is reshaped into [C,H,W]. .

[0078] This invention introduces a transformer module with multi-scale pooling for use in deep neural network-based liveness detection models. It assists in training CNN branches during the training phase, providing global contextual information to the CNN branches and enhancing the model's ability to model long-range dependencies in images. Furthermore, by introducing multi-scale pooling into the multi-head self-attention module of the transformer module, the length of the feature sequence can be effectively compressed, reducing the computational resources required for training the model, while simultaneously obtaining highly discriminative feature representations, further improving model performance.

[0079] The above embodiments and figures are not intended to limit the product form and style of the present invention. Any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims

1. A method for processing transformer module with multi-scale pooling, applied to a living body detection model based on deep neural network, used to assist training of a CNN branch in a training stage, to provide global context information for the CNN branch, and to enhance the modeling ability of the model for long-distance dependencies in an image, characterized in that, Includes the following steps: Step B1: The input features of shape [C,H,W] are serialized into a tensor of shape [(H / P)×(W / P),C], where C is the channel, H is the height, W is the width, and P is the size of the patch, which is the downsampling factor. Step B2, the tensor of Step Bl passes through a multi-head self-attention module with multi-scale pooling, which uses different scale pooling layers to obtain different scale context information and reduce the size of the tensor ; Step B3: Add the outputs of Step B1 and Step B2 and perform layer normalization. Step B4: Feed the output of step B3 into the feedforward neural network. The feedforward neural network consists of convolution and activation functions, which perform nonlinear transformations and mappings to obtain two-dimensional nearest neighbor information and enhance the representation ability of the model. Step B5: Add the outputs of Step B3 and Step B4 together and perform layer normalization. Step B6: Perform feature deserialization on the output of step B5 to transform the output features into tensors of shape [C, H / P, W / P]. 2.The method of claim 1, wherein: In step B1, the feature serialization processing steps include: Step C1: The input features pass through the convolution module; Step C2: Flatten the output of the convolution module; Step C3: Dimensional Transformation; Step C4: Use layer normalization to process the features to obtain the output features.

3. The processing method of the transformer module with multi-scale pooling as described in claim 1, characterized in that: In step B2, let the input features be... The shape is [B, H × W, C], where B is the batch size. The processing steps of the multi-head self-attention module with multi-scale pooling include: Step D1: Reshape input features The shape is changed to [B,C,H,W]; the input feature is then processed using average pooling layers of different scales, P1, P2, P3, and P4. The following characteristics are obtained. , , , The shapes are: [B,C,H / P1,W / P1], [B,C,H / P2,W / P2], [B,C,H / P3,W / P3], and [B,C,H / P4,W / P4]. Step D2: Reshape the four features output from step D1 , , , This transforms their shapes into [B,C,(H / P1)×(W / P1)], [B,C,(H / P2)×(W / P2)], [B,C,(H / P3)×(W / P3)], and [B,C,(H / P4)×(W / P4)], respectively. Combining these features, the shape becomes [B,C,S]. in, Indicates feature reshaping, This represents the concatenation function. S represents the concatenated output features; S represents the feature. The length of the last dimension; Step D3: Reconstruct the spliced ​​output features from step D2 and perform layer normalization to obtain the features. The shape becomes [B,S,C]. in, Presentation layer normalization processing, The function represents the permutation operation; Step D4: Analyze the features output in step D3. The feature is fed into a fully connected layer (FC) to obtain the feature. The shape becomes [B,S,2C]; subsequently, the features are reshaped to obtain the features. The shape becomes [2, B, N, S, C / N], where N is the number of heads in the multi-head self-attention module, used to control the number of attention heads in the attention mechanism. in, Functions are used to change the shape of dimensions; Step D5: Segment the features output from step D4 The segmented features are as follows and All shapes are [B, N, S, C / N]: Step D6: Input features Send into the fully connected layer Given a shape of [B, H × W, C], reshape the features to obtain the feature. Its shape is [B, N, H × W, C / N]. Step D7: Reshape Features The shape is obtained as a feature with shape [B, N, C / N, S]. ;right and Performing matrix multiplication, the resulting feature shape becomes [B, N, H]. [W,S]; Perform feature analysis along the last dimension. Processing to obtain features : in, This represents matrix multiplication. Used as the output layer of a neural network, it converts the network's output into the probability of each category appearing. Step D8: The features output in step D7 With features Perform matrix multiplication to obtain the features The shape is [B,N,H] [W,C / N]; Reshaping features , to obtain features The shape becomes [B,H] W,C]: in, Indicates transpose; Step D9: The features output in step D8 Send into the fully connected layer To obtain the final output features The shape is [B,H] W,C]: 。 4. The processing method of the transformer module with multi-scale pooling as described in claim 1, characterized in that: In step B4, the fully connected layers in the feedforward neural network are replaced with local convolutions, and a connection is made after the local convolutions. Activation function The activation function provides a non-linear transformation when the input value is between -3 and 3, which alleviates the gradient vanishing problem while enhancing the expressive power of the model.

5. The processing method of the transformer module with multi-scale pooling as described in claim 4, characterized in that: The processing steps of a feedforward neural network include: Step E1: Input Features The tensor, with shape [H] W,C], for input features Perform feature reshaping to obtain features The shape becomes [C,H,W]: Step E2: After passing through two convolutional layers and an activation layer consecutively, the features are obtained. : in, The formula for calculating the activation function is as follows: Step E3: Obtain features after passing through the convolution module. ,feature After reshaping and transposing, the output features are obtained. The shape is [H] W,C], 。 6. The processing method of the transformer module with multi-scale pooling as described in claim 1, characterized in that: In step B6, feature deserialization is equivalent to feature reshaping, the purpose of which is to change the shape of the input features from [H] [W,C] is reshaped into [C,H,W]. 。

Images