Hierarchical information fusion system and fusion method based on CNN-Transformer

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a hierarchical information fusion system and a fusion method based on CNN-Transformer, including a local convolution module LPM, a global and local information feature fusion module GLAF-Mixer and a serial multi-scale forward feedback network SMFFN, wherein the local convolution module LPM uses an inverted residual block to introduce an inductive bias, the global and local information feature fusion module GLAF-Mixer uses a parallel lightweight global attention module and lightweight local information to extract features, combines global modeling information with local information, and the serial multi-scale forward feedback network SMFFN uses multiple 3×3 depth separable convolution serial structures to extract multi-scale features, and fuses multiple depth convolution outputs. The ability to capture multi-scale features and the accuracy of the model are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a hierarchical information fusion system and a fusion method based on CNN-Transformer. Background Art

[0002] In recent years, deep learning technology has been widely used in many fields. As an important milestone in the field of deep learning, convolutional neural network (CNN) has achieved remarkable results in many visual tasks such as image classification, object detection, and image segmentation. With the continuous deepening of deep learning research, an innovative framework called Transformer has emerged, and it initially shined in the field of natural language processing. As an extension of the Transformer architecture, VisionTransformer (ViT) was developed and its application scope was expanded to the field of computer vision. The emergence of ViT provides new perspectives and methods for the processing of visual tasks, further promoting the development of deep learning in the field of visual recognition.

[0003] In visual tasks, CNN can effectively extract local features in images through convolution operations, and can capture geometric and topological information well with its translation invariance and local connectivity. However, the local convolution operation of CNN makes it weak in perceiving long-distance pixel relationships, lacks a global understanding of the image, and ignores the correlation between the whole and the parts; in addition, CNN is sensitive to position translation, which makes the model not very robust to transformations such as translation and rotation. Compared with traditional CNN, ViT is more suitable for processing complex visual tasks. Its good global feature processing ability has achieved breakthrough success in many fields such as image classification, image segmentation and object detection, especially when the data scale is large. The MHSA of the Transformer-Encoder structure of the ViT network achieves global perception coverage by focusing on building and capturing long-term dependencies between tokens, but ignores the spatial local information of the image and lacks the local perception and weight sharing capabilities inherent in the CNN model, resulting in weak inductive bias ability and prone to overfitting problems. Moreover, the computational and storage costs of the MHSA operation in ViT are quadratically related to the input image resolution. When the input image resolution is high, it will inevitably lead to problems such as insufficient GPU memory, low computational efficiency, and increased model parameters.

[0004] In visual tasks, how to combine these two architectures to fully utilize their respective advantages has become one of the hot topics in the field of computer vision research. This fusion strategy is particularly important in complex visual tasks that require understanding both local features and global context. In order to address the limitations of ViT, Swin Transformer introduces inductive bias and reduces the computational cost of MHSA by proposing shifted window self-attention. However, Swin Transformer has a limited receptive field of view due to its window-based local attention. Based on the problem of limited receptive field of view of local windows, HaloNet and Focal Transformer use Expanding to expand the receptive field of view, while ShuffleTransformer introduces Shuffle operation to complete the interaction between local windows.

[0005] Therefore, there is an urgent need to solve the technical problems that the local convolution operation of the convolutional neural network in the existing technology makes it have weak perception of long-distance pixel relationships, lacks a global understanding of the image, ignores the correlation between the whole and the parts; and is sensitive to translation, making the model not robust to transformations such as translation and rotation. Summary of the invention

[0006] The purpose of the present invention is to provide a hierarchical information fusion system and fusion method based on CNN-Transformer, so as to solve the problems in the prior art that the local convolution operation of the convolutional neural network makes it weak in the perception of long-distance pixel relationships, lacks a global understanding of the image, ignores the correlation between the whole and the parts; and is sensitive to translation, so that the model does not have good robustness to transformations such as translation and rotation.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0008] First, a hierarchical information fusion system based on CNN-Transformer is provided, including a local convolution module LPM, a global and local information feature fusion module GLAF-Mixer and a serial multi-scale feedforward network SMFFN. The local convolution module LPM uses an inverted residual block to introduce an inductive bias, the global and local information feature fusion module GLAF-Mixer uses a parallel lightweight global attention module and lightweight local information to extract features, combining global modeling information with local information, and the serial multi-scale feedforward network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features, and fuses multiple depth-convolution outputs.

[0009] Preferably, the overall structure is set to a hierarchical structure, which includes a convolution initial layer ConvStem, 4 stages stage, a global average pooling layer Avgpool, a convolution layer Conv and a classification layer Classifier;

[0010] The convolution initial layer Conv Stem includes 2 convolution layers, wherein the convolution kernel size of one convolution layer is 3×3 and the step size is 2; the convolution kernel size of the other convolution layer is 3×3 and the step size is 1;

[0011] The four stages are used to generate feature maps of different scales, respectively. Each stage includes a convolution layer with a convolution kernel size of 3×3 and a stride of 2 and several stacked residual modules DH-Former Block. The convolution layer in each stage is used to reduce the size of the feature map, reduce the width and height of the feature map by 2 times, and expand the channel dimension by 2 times. The local and long-distance dependencies are captured by the stacked residual modules DH-Former Block to extract and fuse global information and local information.

[0012] Preferably, the specific formula for extracting local information and introducing translation invariance in the local convolution module LPM is as follows:

[0013] ;

[0014] Among them, x is the input image, the dimension is H×W×C, H, W are the width and height of the current feature map, C is the number of channels, Conv is a regular convolution, DwConv is a deep convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the change of the channel before and after the convolution, that is,

[0015] The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, and finally the residual connection is used to add the output to the original input x. The meaning is the original input x, and the output of the local convolution module LPM is obtained.

[0016] Preferably, the global and local information feature fusion module GLAF-Mixer includes a lightweight multi-head self-attention module LWMHSA, a local feature extraction module LFEM and an information enhancement module IEM;

[0017] The working process of the global and local information feature fusion module GLAF-Mixer is as follows:

[0018] The input feature map of size H×W×C is split along the channel into two parts of size H×W×C / 2, respectively. and ;

[0019] The two parts of size H×W×C / 2 are respectively sent to the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM to generate corresponding feature maps, both of which are of size H×W×C / 2;

[0020] Concatenate two feature maps of size H×W×C / 2 along the channel;

[0021] The information enhancement module IEM is used to fuse the global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM with the local information.

[0022] Preferably, the lightweight multi-head self-attention module LWMHSA includes a k×k depth separation convolution DwConv with a step size of k and a self-attention module.

[0023] In the original self-attention module, the input will be converted linearly into a query ,key Sum , where n = H × W represents the number of blocks into which the image or feature map is divided, and d, d k and d v Represents the dimensions of input, key, query and value respectively. The calculation formula is as follows:

[0024] ;

[0025] Where K T The meaning is the transpose of the key matrix K. T is the transposition symbol, QK T is the dot product of the transpose of the query matrix Q and the key matrix K, which calculates the similarity between each query and all keys; d k is the dimension of the key vector, Used to prevent the dot product result from being too large, causing the gradient to disappear or explode; the output of the Softmax activation function is multiplied by the value matrix V to obtain a weighted value vector. Attn(Q,K,V) means the self-attention output obtained when the query matrix Q, key matrix K and value matrix V are input.

[0026] The process of calculating self-attention of the lightweight multi-head self-attention module LWMHSA is:

[0027] First, a depthwise separable convolution DwConv with a step size of k and a kernel size of k×k is used to reduce the spatial size of K and V. After the depthwise separable convolution, the key K and value V become and , and add the learnable relative position B to each attention module. The corresponding lightweight formula is as follows:

[0028] ;

[0029] in The meaning is that the key matrix is ​​reduced after the deep separation convolution reduces the spatial size The transpose of T is the transpose symbol, is the query matrix Q and the key matrix after the depthwise separation convolution reduces the spatial size The dot product of the transpose of , calculates the similarity between each query and all keys; d k is the dimension of the key vector, It is used to prevent the dot product result from being too large, causing the gradient to disappear or explode; the output of the Softmax activation function is convolved with the depth to reduce the spatial size of the value matrix Multiply them together to get a weighted value vector, The meaning is that when the input is the query matrix Q, the key matrix Sum Matrix The resulting lightweight self-attention output.

[0030] The specific formula for using the information enhancement module IEM to fuse the global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM with the local information is as follows:

[0031] .

[0032] in, This means that when the input image of H×W×C is split along the channel into a size of H×W×C / 2 As input, the output obtained by the lightweight multi-head self-attention module LWMHSA, It means that when the input image is split along the channel As input, the output obtained by the local feature extraction module LFEM; ConCat means the local feature extraction module and Splicing is performed along the channel dimension; the IEM module means sending the spliced ​​feature map to the information enhancement module to obtain the output after feature fusion.

[0033] Preferably, the lightweight multi-head self-attention module LWMHSA divides the input data into multiple different subspaces, i.e., multiple heads, in the multi-head self-attention, and each head performs self-attention calculation independently; for each head, the LightAttn lightweight attention function is applied, and the output of each head is a sequence of size n×d / h, and the sequences output from different heads are finally spliced ​​together to form a final output sequence of size n×d; the local feature extraction module LFEM captures the spatial local information of the image by using 1 3×3DwConv and 2 1×1Conv, and the formula for capturing the spatial local information of the image is as follows:

[0034] .

[0035] Among them, x is the input image split along the channel As input, DwConv is a deep convolution, Conv is a regular convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a depthwise convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a convolution kernel size of 1×1, and the input channel is , the output channel is c, and finally the residual connection is used to add the output to the input x. The meaning of LFEM(x) is the input x, and the output of the local feature extraction module LFEM is obtained.

[0036] Preferably, the information enhancement module IEM is provided with two branches, one of which includes one convolution layer with a convolution kernel size of 1×1, two convolution layers with a convolution kernel size of 3×3, a Sigmoid activation function and element-by-element multiplication, and the other branch includes one convolution layer with a convolution kernel size of 1×1, a global average pooling layer Avgpool, two convolution layers with a convolution kernel size of 1×1, a Sigmoid activation function and channel-by-channel multiplication.

[0037] Preferably, the information enhancement module IEM fuses the global information generated by the lightweight multi-head self-attention module LWMHSA module and the local information generated by the local feature extraction module LFEM as follows:

[0038] The information enhancement module IEM first uses Conv with a convolution kernel size of 1×1 to fuse channel information for the two branches, and the number of channels c remains unchanged. Then, the obtained features are respectively captured by channel attention and spatial attention to capture channel and spatial dependencies, and the features of the two enhanced branches are further fused. The specific formula for fusion is as follows:

[0039] Among them, x is the feature map of the concatenation of the output of the lightweight multi-head self-attention module LWMHSA and the output of the local feature extraction module LFEM, Conv is a conventional convolution, the subscript 1×1 of the conventional convolution represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a kernel size of 1×1, an input channel of c, an output channel of c, and y i is the output after 1×1Conv, i = 0, 1, y0 is used as the input of the spatial attention branch, and y1 is used as the input of the channel attention branch.

[0040] Then based on branch 1, first use two consecutive convolution kernels of size 3×3 Conv on the input y0, then use the Sigmoid activation function to generate a spatial attention map, and finally multiply it with the original feature map to enhance the network's understanding and representation of the input image by emphasizing important spatial positions. The formula is as follows:

[0041]

[0042] Among them, y0 is the 1×1Conv output of one of the branches, which is used as the input of the spatial attention branch. Conv is a regular convolution. The subscript 3×3 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. That is, The meaning is to use a convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. is the Sigmoid activation function, and s(y0) means the feature map obtained by using the spatial attention branch when the input is y0.

[0043] For branch 2, first use AvgPool2d to directly compress the feature map containing global information into a 1×1 feature vector for input y1, and then use two 1×1 Convs to learn the nonlinear relationship between channels. Then use the Sigmoid activation function to multiply the vector weights obtained by input y1 channel by channel to enhance important feature channels. The formula is as follows:

[0044] ;

[0045] Among them, y1 is the output of another branch 1×1Conv as the input of the channel attention branch, Conv is a regular convolution, the subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, is the Sigmoid activation function, AvgPool2d is the 2d global average pooling, and c(y1) means the feature map obtained by the channel attention branch when the input is y1.

[0046] The features of the two enhanced branches are added and fused to get the output , the specific formula of fusion is as follows:

[0047] ;

[0048] Among them, y0 is the input of the spatial attention branch, and y1 is the input of the channel attention branch, s(y0) is the feature map obtained after the spatial attention branch is enhanced, and c(y1) is the feature map obtained after the channel attention branch is enhanced. To fuse the enhanced features of the two branches.

[0049] Preferably, the serial multi-scale feedforward network SMFFN module first uses a convolution kernel with a size of 1×1 to reduce the channel dimension to The output y0 is obtained, and then three deep convolution serial structures with a convolution kernel size of 3×3 are used to extract features from y0, and the outputs y1, y2, and y3 are obtained respectively. The obtained y0, y1, y2, and y3 are added and fused, and finally a Conv with a convolution kernel size of 1×1 is used to increase the channel dimension to c. The specific formula is as follows:

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] .

[0055] Among them, x is the output of the information enhancement module IEM, Conv is a conventional convolution, DwConv is a deep convolution, the subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, y0 is the output after 1×1Conv, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , y1 is the output of the first depth convolution using a convolution kernel size of 3×3, y2 is the output of the second depth convolution using a convolution kernel size of 3×3, and y3 is the output of the third depth convolution using a convolution kernel size of 3×3. The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, and the meaning of SMFFN(x) is the feature map obtained after the features are fused through the serial multi-scale forward feedback network when the input x is the output of the information enhancement module IEM.

[0056] In a second aspect, a hierarchical information fusion method based on CNN-Transformer is provided, which is implemented based on any of the hierarchical information fusion systems based on CNN-Transformer, and includes the following steps:

[0057] S1: The local convolution module LPM uses an inverted residual block to introduce an inductive bias to extract local information and introduce translation invariance;

[0058] S2: Global and local information feature fusion module GLAF-Mixer uses parallel lightweight global attention modules and lightweight local information extraction features to combine global modeling information with local information;

[0059] S3: The serial multi-scale feedforward network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features and fuse multiple depth-wise convolution outputs.

[0060] The beneficial effects of the present invention include:

[0061] The present invention provides a hierarchical information fusion system and fusion method based on CNN-Transformer, comprising a local convolution module LPM, a global and local information feature fusion module GLAF-Mixer and a serial multi-scale forward feedback network SMFFN. The local convolution module LPM uses an inverted residual block to introduce an inductive bias, the global and local information feature fusion module GLAF-Mixer uses a parallel lightweight global attention module and lightweight local information to extract features, combining global modeling information with local information, and the serial multi-scale forward feedback network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features, and fuses multiple depth-convolution outputs.

[0062] Firstly, the local convolution module (LPM) is used to extract local information and introduce translation invariance to prevent the destruction of the translation invariance of the image, and effectively extract the local relationship and structural information within the block.

[0063] Secondly, the corresponding feature maps are obtained through the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM, and the two feature maps are spliced ​​along the channel. Finally, the global information learned by LWMHSA and LFEM is fused with the local information using the information enhancement module IEM, realizing the fusion of global information and local details. At the same time, a large receptive field and strong inductive bias are injected to improve the perception ability of a wider range of contextual information.

[0064] Again, the lightweight multi-head self-attention module LWMHSA uses a k×k depth-wise separable convolution with a step size of k before the attention operation to reduce the spatial size of K and V, and adds a learnable relative position B to each attention module after the depth-wise separable convolution, effectively reducing the computational overhead.

[0065] Again, the lightweight multi-head self-attention module LWMHSA divides the input data into multiple different subspaces in the multi-head self-attention, and each head performs self-attention calculations independently. For each head, a lightweight attention function is applied, and the output sequences from different heads are finally spliced ​​together to form a final output sequence of size n×d. By using 1 3×3DwConv and 2 1×1Conv to capture the spatial local information of the image, and combining the information enhancement module IEM with 1×1Conv to fuse channel information, and then using channel attention and spatial attention to capture the channel and space dependencies of the obtained features, the features of the two enhanced branches are further fused, realizing the fusion and feature enhancement of global and local information, making up for the shortcomings of CNN and vision-Transformer that can only focus on a single feature respectively.

[0066] Finally, the SMFFN module, a serial multi-scale feedforward network, extracts multi-scale features by using multiple 3×3 depthwise separable convolution serial structures and fuses multiple deep convolution outputs, effectively solving the limitation that traditional FFN cannot fully capture multi-scale features. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the overall composition structure diagram of the hierarchical information fusion system based on CNN-Transformer of the present invention.

[0068] Figure 2 It is a composition structure diagram of the global and local information feature fusion module GLAF-Mixer of the present invention.

[0069] Figure 3 This is a structural diagram of the serial multi-scale forward feedback network SMFFN of the present invention. DETAILED DESCRIPTION

[0070] The following is combined with Figure 1~Figure 3 The present invention is further described in detail:

[0071] First, the relevant abbreviations involved in the present invention are introduced:

[0072] Transformer: Transformer is a deep learning model proposed by Google researchers in 2017. It is mainly used to process sequence data. It uses the Multi-Head Self-Attention (MHSA) mechanism to enhance the expressiveness of the model and has quickly become a core technology in the field of NLP.

[0073] ViT: The full name is Vision Transformer, which is a model that applies the Transformer architecture to computer vision tasks.

[0074] CNN: The full name is Convolutional Neural Network, which is a deep learning model widely used in computer vision tasks such as image recognition, image classification, and object detection.

[0075] Transformer-Encoder: Transformer-Encoder is a core component in the Transformer model, which is mainly composed of multiple encoder layers stacked together.

[0076] MHSA: It is a key component in the Transformer model and an extension of Self-Attention. It decomposes the concept of Self-Attention into multiple "heads", each of which has its own set of parameters and can learn different representations of sequences.

[0077] Self-Attention: Also known as Scaled Dot-Product Attention, it is an attention mechanism that allows the model to take into account how each element in the sequence is related to each other when generating a representation of a sequence. Its core idea is that given a sequence, the output of each position is the weighted sum of all positions in the sequence, and the weight (attention weight) is calculated based on the correlation between the current position and each position in the sequence.

[0078] Token: refers to an element in the input sequence. In the field of NLP, this element is a single word. In the field of vision, an image is divided into a series of fixed-size image patches, and each image patch is regarded as a "token".

[0079] Shuffle: refers to the rearrangement or randomization of data.

[0080] Kernel: In convolutional neural networks, a convolution kernel is a small matrix that is used to perform convolution operations on images to extract image features.

[0081] Depthwise Convolution: A special convolution operation in which each convolution kernel is responsible for only one input channel. It is mainly used to reduce the number of model parameters and the amount of calculation.

[0082] MBConv: The "inverted bottleneck" structure convolution block, proposed in the MobileNetV2 network, allows most of the calculations to be performed in a lower-dimensional feature space, thereby improving efficiency.

[0083] Concat: concatenate, concatenation is a commonly used operation used to merge multiple tensors along a specific dimension.

[0084] FFN: "Feed-Forward Network", which is a feed-forward network. It further extracts features and learns representations of the output of the self-attention layer through two fully connected layers to capture more complex features.

[0085] ResidualConnection: refers to the residual connection method of ResNet, which allows information in the network to be directly transmitted across certain layers through identity mapping, that is, directly adding the input to the output of the subsequent layer. This connection helps solve the gradient vanishing problem in deep neural network training, allowing the network to successfully train deeper layers.

[0086] LCM: This patent proposes an efficient and scalable local perception component that hierarchically fuses CNN and ViT structure attention blocks, introducing translation invariance.

[0087] GLAF-Mixer: This patent proposes an efficient and scalable feature fusion component that hierarchically fuses CNN and ViT structure attention blocks to fuse global information and local information features.

[0088] SMFFN: This patent proposes an efficient and scalable hierarchical fusion of CNN and ViT structure attention blocks to fuse multi-scale information.

[0089] LWMHSA: One of the components of GLAF-Mixer, lightweight multi-head self-attention module.

[0090] LFEM: One of the components of GLAF-Mixer, local feature extraction module.

[0091] IEM: One of the components of GLAF-Mixer, Information Enhancement Module.

[0092] Example 1

[0093] See attached Figure 1 As shown in the figure, the hierarchical information fusion system based on CNN-Transformer includes a local convolution module LPM, a global and local information feature fusion module GLAF-Mixer and a serial multi-scale feedforward network SMFFN. The local convolution module LPM uses an inverted residual block to introduce an inductive bias. The global and local information feature fusion module GLAF-Mixer uses a parallel lightweight global attention module and a lightweight local information to extract features, combining global modeling information with local information. The serial multi-scale feedforward network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features and fuse multiple depth-convolution outputs.

[0094] The local convolution module LPM introduces inductive bias by using an inverted residual block, aiming to solve the limitation that ViT cannot introduce translation invariance due to the absolute position encoding. The global and local information feature fusion module GLAF-Mixer combines global modeling information with local information by using parallel lightweight global attention modules and lightweight local information to extract features, effectively solving the problem that layered fusion structures such as MixFormer, CMT, and MaxVit cannot fully aggregate global and local information, and improves the limitation of inductive bias. The serial multi-scale feedforward network SMFFN extracts multi-scale features by using multiple 3×3 depth-separable convolution serial structures and fuses multiple deep convolution outputs, greatly improving the ability to capture multiple scales, and optimizing the problem that the FFN modules of ViT, CMT, and TransXNet are insufficient in capturing multi-scales. This module is plug-and-play and can improve the accuracy of the model.

[0095] The overall structure of the hierarchical information fusion system based on CNN-Transformer is set as a hierarchical structure, which includes a convolution initial layer Conv Stem, 4 stages, 1 global average pooling layer Avgpool, 1 convolution layer Conv and 1 classification layer Classifier. The convolution initial layer Conv Stem includes 2 convolution layers, one of which has a convolution kernel size of 3×3 and a step size of 2; the other has a convolution kernel size of 3×3 and a step size of 1. Compared with PatchEmbedding, it effectively overcomes the limitation that PatchEmbedding is divided into non-overlapping blocks and ignores the local information inside the blocks. The four stages are used to generate feature maps of different scales. Each stage includes a convolution layer with a convolution kernel size of 3×3 and a stride of 2 and several stacked residual modules DH-Former Block. The convolution layer in each stage is used to reduce the size of the feature map, reducing the width and height of the feature map by 2 times and expanding the channel dimension by 2 times. The stacked residual module DH-Former Block captures local and long-distance dependencies to extract and fuse global and local information.

[0096] Example 2

[0097] Based on Example 1, the specific formula for extracting local information and introducing translation invariance by the local convolution module LPM is as follows:

[0098] ;

[0099] Among them, x is the input image, the dimension is H×W×C, H, W are the width and height of the current feature map, C is the number of channels, Conv is a regular convolution, DwConv is a deep convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the change of the channel before and after the convolution, that is,

[0100] The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, and finally the residual connection is used to add the output to the original input x. The meaning is the original input x, and the output of the local convolution module LPM is obtained.

[0101] Since the absolute position encoding used in Transformer was originally designed to utilize the order of tokens, which will destroy the translation invariance in the image, Transformer ignores the local relationship and structural information within the block. Therefore, this embodiment extracts local information through the local convolution module LPM and introduces translation invariance, which effectively solves the problem of Transformer destroying translation invariance.

[0102] In this embodiment, the global and local information feature fusion module GLAF-Mixer includes a lightweight multi-head self-attention module LWMHSA, a local feature extraction module LFEM and an information enhancement module IEM.

[0103] The working process of the global and local information feature fusion module GLAF-Mixer is as follows:

[0104] The input feature map of size H×W×C is split along the channel into two parts of size H×W×C / 2, respectively. and ; The two parts of size H×W×C / 2 are respectively sent to the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM to generate corresponding feature maps, both of which are of size H×W×C / 2; The two feature maps of size H×W×C / 2 are spliced ​​along the channel; The global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM is fused with the local information using the information enhancement module IEM.

[0105] The lightweight multi-head self-attention module LWMHSA consists of a k×k depth-wise separable convolution DwConv with a stride of k and a self-attention module.

[0106] In the original self-attention module, the input will be converted linearly into a query ,key Sum , where n = H × W represents the number of blocks into which the image or feature map is divided, and d, d k and d v Represents the dimensions of input, key, query and value respectively. The calculation formula is as follows:

[0107] ;

[0108] Where K T The meaning is the transpose of the key matrix K. T is the transposition symbol, QK T is the dot product of the transpose of the query matrix Q and the key matrix K, which calculates the similarity between each query and all keys; d k is the dimension of the key vector, Used to prevent the dot product result from being too large, causing the gradient to disappear or explode; the output of the Softmax activation function is multiplied by the value matrix V to obtain a weighted value vector. Attn(Q,K,V) means the self-attention output obtained when the query matrix Q, key matrix K and value matrix V are input.

[0109] The process of calculating self-attention of the lightweight multi-head self-attention module LWMHSA is:

[0110] First, a depthwise separable convolution DwConv with a step size of k and a kernel size of k×k is used to reduce the spatial size of K and V. After the depthwise separable convolution, the key K and value V become and , and add the learnable relative position B to each attention module. The corresponding lightweight formula is as follows:

[0111] ;

[0112] in The meaning is that the key matrix is ​​reduced after the deep separation convolution reduces the spatial size The transpose of T is the transpose symbol, is the query matrix Q and the key matrix after the depthwise separation convolution reduces the spatial size The dot product of the transpose of , calculates the similarity between each query and all keys; d k is the dimension of the key vector, It is used to prevent the dot product result from being too large, causing the gradient to disappear or explode; the output of the Softmax activation function is convolved with the depth to reduce the spatial size of the value matrix Multiply them together to get a weighted value vector, The meaning is that when the input is the query matrix Q, the key matrix Sum Matrix The resulting lightweight self-attention output.

[0113] The specific formula for using the information enhancement module IEM to fuse the global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM with the local information is as follows:

[0114] .

[0115] in, This means that when the input image of H×W×C is split along the channel into a size of H×W×C / 2 As input, the output obtained by the lightweight multi-head self-attention module LWMHSA, It means that when the input image is split along the channel As input, the output obtained by the local feature extraction module LFEM; ConCat means the local feature extraction module and Splicing is performed along the channel dimension; the IEM module means sending the spliced ​​feature map to the information enhancement module to obtain the output after feature fusion.

[0116] The lightweight multi-head self-attention module LWMHSA divides the input data into multiple different subspaces, i.e., multiple heads, in the multi-head self-attention. Each head performs self-attention calculation independently. For each head, the LightAttn lightweight attention function is applied. The output of each head is a sequence of size n×d / h. The sequences from different head outputs are finally spliced ​​together to form a final output sequence of size n×d. The local feature extraction module LFEM captures the spatial local information of the image by using 1 3×3DwConv and 2 1×1Conv. The formula for capturing the spatial local information of the image is as follows:

[0117] .

[0118] Among them, x is the input image split along the channel As input, DwConv is a deep convolution, Conv is a regular convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a depthwise convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, and finally the residual connection is used to add the output to the input x. The meaning of LFEM(x) is the input x, and the output of the local feature extraction module LFEM is obtained.

[0119] Example 3

[0120] Based on Example 1 or Example 2, the information enhancement module IEM is provided with two branches, one of which includes one convolution layer with a convolution kernel size of 1×1, two convolution layers with a convolution kernel size of 3×3, a Sigmoid activation function and element-by-element multiplication, and the other branch includes one convolution layer with a convolution kernel size of 1×1, a global average pooling layer Avgpool, two convolution layers with a convolution kernel size of 1×1, a Sigmoid activation function and channel-by-channel multiplication.

[0121] The information enhancement module IEM fuses the global information generated by the lightweight multi-head self-attention module LWMHSA and the local information generated by the local feature extraction module LFEM as follows:

[0122] The information enhancement module IEM first uses Conv with a convolution kernel size of 1×1 to fuse channel information for the two branches, and the number of channels c remains unchanged. Then, the obtained features are respectively captured by channel attention and spatial attention to capture channel and spatial dependencies, and the features of the two enhanced branches are further fused. The specific formula for fusion is as follows:

[0123] ;

[0124] Among them, x is the feature map of the concatenation of the output of the lightweight multi-head self-attention module LWMHSA and the output of the local feature extraction module LFEM, Conv is a conventional convolution, the subscript 1×1 of the conventional convolution represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a kernel size of 1×1, an input channel of c, an output channel of c, and y i is the output after 1×1Conv, i = 0, 1, y0 is used as the input of the spatial attention branch, and y1 is used as the input of the channel attention branch.

[0125] Then based on branch 1, first use two consecutive convolution kernels of size 3×3 Conv on the input y0, then use the Sigmoid activation function to generate a spatial attention map, and finally multiply it with the original feature map to enhance the network's understanding and representation of the input image by emphasizing important spatial positions. The formula is as follows:

[0126]

[0127] Among them, y0 is the 1×1Conv output of one of the branches, which is used as the input of the spatial attention branch. Conv is a regular convolution. The subscript 3×3 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. That is, The meaning is to use a convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. is the Sigmoid activation function, and s(y0) means the feature map obtained by using the spatial attention branch when the input is y0.

[0128] For branch 2, first use AvgPool2d to directly compress the feature map containing global information into a 1×1 feature vector for input y1, and then use two 1×1 Convs to learn the nonlinear relationship between channels. Then use the Sigmoid activation function to multiply the vector weights obtained by input y1 channel by channel to enhance important feature channels. The formula is as follows:

[0129] ;

[0130] Among them, y1 is the output of another branch 1×1Conv as the input of the channel attention branch, Conv is a regular convolution, the subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a kernel size of 1×1 and an input channel of , the output channel is c, is the Sigmoid activation function, AvgPool2d is the 2d global average pooling, and c(y1) means the feature map obtained by the channel attention branch when the input is y1.

[0131] The features of the two enhanced branches are added and fused to get the output , the specific formula of fusion is as follows:

[0132] ;

[0133] Among them, y0 is the input of the spatial attention branch, y1 is the input of the channel attention branch, s(y0) is the feature map obtained after the spatial attention branch is enhanced, and c(y1) is the feature map obtained after the channel attention branch is enhanced. To fuse the enhanced features of the two branches.

[0134] Example 4

[0135] On the basis of Example 1 or Example 2 or Example 3, the serial multi-scale forward feedback network SMFFN module first uses a convolution kernel with a size of 1×1 to reduce the channel dimension to The output y0 is obtained, and then three deep convolution serial structures with a convolution kernel size of 3×3 are used to extract features from y0, and the outputs y1, y2, and y3 are obtained respectively. The obtained y0, y1, y2, and y3 are added and fused, and finally a Conv with a convolution kernel size of 1×1 is used to increase the channel dimension to c. The specific formula is as follows:

[0136] ;

[0137] ;

[0138] ;

[0139] ;

[0140] .

[0141] Among them, x is the output of the information enhancement module IEM, Conv is a conventional convolution, DwConv is a deep convolution, the subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, y0 is the output after 1×1Conv, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , y1 is the output of the first depth convolution using a convolution kernel size of 3×3, y2 is the output of the second depth convolution using a convolution kernel size of 3×3, and y3 is the output of the third depth convolution using a convolution kernel size of 3×3. The meaning of is to use a convolution with a convolution kernel size of 1×1, and the input channel is , the output channel is c, and the meaning of SMFFN(x) is the feature map obtained after the features are fused through the serial multi-scale forward feedback network when the input x is the output of the information enhancement module IEM.

[0142] The hierarchical information fusion method based on CNN-Transformer is implemented based on any one of the hierarchical information fusion systems based on CNN-Transformer, comprising the following steps:

[0143] S1: The local convolution module LPM uses an inverted residual block to introduce an inductive bias to extract local information and introduce translation invariance;

[0144] S2: Global and local information feature fusion module GLAF-Mixer uses parallel lightweight global attention modules and lightweight local information extraction features to combine global modeling information with local information;

[0145] S3: The serial multi-scale feedforward network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features and fuse multiple depth-wise convolution outputs.

[0146] In order to adapt to different computational costs and different application requirements, several different variants are designed by increasing the number of DH-Former blocks and channel sizes in each stage, as shown in Table 1, including DH-Former-T (Tiny), DH-Former-S (small), and DH-Former-B (Base). The expansion and shrink rates of all models SE are 4 and 0.25 by default. See Table 1 for a summary of the architecture configuration of the DH-Former variants, where B is the number of DH-Former-Blocks, C is the number of output channels of each stage, and H is the size of the attention head of the attention block.

[0147] Table 1 Architecture configuration of the hierarchical information fusion system (DH-Former) based on CNN-Transformer

[0148]

[0149] Based on the classification task, on the ImageNet-1K dataset, with an input size of 224×224, the Top-1 Acc of different models is shown in Table 2. The top1-1 acc of DH-Former-T is 83.3%, the top1-1 acc of DH-Former-S is 84.2%, and the top1-1 acc of DH-Former-B can reach 85.01%. As shown in Table 2, DH-Former-B is 0.41% higher than the top1-1 acc of TransXNet-B-224, 1.21% higher than ConvNeXt-B-224, 0.06% higher than MaxVit-B-224, 1.21% higher than MixFormer-B6-224, 0.51% higher than Swin-B-384, and 0.51% higher than CMT-B-256. In summary, the DH-Former stacked by DH-Former Block is an efficient general Vision Transformer. The LPM, GLAF-Mixer, and SMFFN modules proposed by DH-Former Block alleviate the problem that the existing hybrid network structure cannot fully aggregate global and local information, and all improve the accuracy of the model. Experiments show that the DH-Former model performs well in image classification and various downstream visual tasks.

[0150] Table 2 Verification of the performance of DH-Former-B on the ImageNet1K dataset

[0151]

[0152] In summary, the hierarchical information fusion system and fusion method based on CNN-Transformer provided by the present invention extracts local information and introduces translation invariance through the local convolution module LPM to prevent the destruction of the translation invariance of the image, and effectively extracts the local relationship and structural information within the block. The corresponding feature map is obtained through the multi-head self-attention module LWMHSA and the local feature extraction module LFEM, and the two feature maps are spliced ​​along the channel. Finally, the global information learned by LWMHSA and LFEM is fused with the local information using the information enhancement module IEM, realizing the fusion of global information and local details, while injecting a large receptive field and a strong inductive bias to improve the perception of a wider range of contextual information.

[0153] The LWMHSA module uses a k×k depth-separable convolution with a step size of k before the attention operation to reduce the spatial size of K and V, and adds a learnable relative position B to each attention module after the depth-separable convolution, which effectively reduces the computational overhead. In the LWMHSA module, the input data is divided into multiple different subspaces in the multi-head self-attention, and each head performs self-attention calculations independently. For each head, a lightweight attention function is applied, and the output sequences from different heads are finally spliced ​​together to form a final output sequence of size n×d. The local feature extraction module LFEM captures the spatial local information of the image by using 1 3×3DwConv and 2 1×1Conv, and combines the IEM module with 1×1Conv to fuse channel information. Then, the obtained features are respectively captured by channel attention and spatial attention. The features of the two enhanced branches are further fused to achieve the fusion of global information and local information and feature enhancement, which makes up for the shortcomings of CNN and vision-Transformer that can only focus on a single feature. The SMFFN module, a serial multi-scale feedforward network, processes channels using multiple 3×3 depthwise separable convolution serial structures and fuses multiple depthwise convolution outputs, effectively solving the limitation that traditional FFN cannot fully capture multi-scale features.

Claims

1. A hierarchical information fusion system based on CNN-Transformer, characterized by: It includes a local convolution module LPM, a global and local information feature fusion module GLAF-Mixer and a serial multi-scale forward feedback network SMFFN. The local convolution module LPM uses an inverted residual block to introduce an inductive bias. The global and local information feature fusion module GLAF-Mixer uses a parallel lightweight global attention module and a lightweight local information to extract features, combining global modeling information with local information. The serial multi-scale forward feedback network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features and fuse multiple depth convolution outputs. The local convolution module LPM, the global and local information feature fusion module GLAF-Mixer and the serial multi-scale forward feedback network SMFFN are connected in sequence. The local convolution module LPM is fused with the global and local information feature fusion module GLAF-Mixer and then input into the serial multi-scale forward feedback network SMFFN. Among them, the global and local information feature fusion module GLAF-Mixer includes a lightweight multi-head self-attention module LWMHSA, a local feature extraction module LFEM and an information enhancement module IEM; The working process of the global and local information feature fusion module GLAF-Mixer is as follows: The input image of size H×W×C is split into two parts of size H×W×C / 2 along the channel, and the obtained and ; The two parts of size H×W×C / 2 are respectively sent to the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM to generate corresponding feature maps, both of which are of size H×W×C / 2; Concatenate two feature maps of size H×W×C / 2 along the channel; Use the information enhancement module IEM to fuse the global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM with the local information; The information enhancement module IEM is provided with two branches, one of which includes one convolution layer with a convolution kernel size of 1×1, two convolution layers with a convolution kernel size of 3×3, a Sigmoid activation function and element-by-element multiplication, and the other branch includes one convolution layer with a convolution kernel size of 1×1, a global average pooling layer Avgpool, two convolution layers with a convolution kernel size of 1×1, a Sigmoid activation function and channel-by-channel multiplication.

2. The hierarchical information fusion system based on CNN-Transformer according to claim 1, characterized in that: The overall structure is set to a hierarchical structure, which includes a convolutional initial layer Conv Stem, 4 stages, a global average pooling layer Avgpool, a convolutional layer Conv and a classification layer Classifier; The convolution initial layer Conv Stem includes 2 convolution layers, wherein the convolution kernel size of one convolution layer is 3×3 and the step size is 2; the convolution kernel size of the other convolution layer is 3×3 and the step size is 1; The four stages are used to generate feature maps of different scales. Each stage includes a convolution layer with a convolution kernel size of 3×3 and a stride of 2 and several stacked residual modules DH-Former Block. The convolutional layer in each stage is used to reduce the size of the feature map, reducing the width and height of the feature map by 2 times and expanding the channel dimension by 2 times. The stacked residual module DH-Former Block captures local and long-distance dependencies to extract and fuse global and local information.

3. The hierarchical information fusion system based on CNN-Transformer according to claim 1, characterized in that: The specific formula of the local convolution module LPM to extract local information and introduce translation invariance is as follows: ； Among them, x is the input image, the dimension is H×W×C, H, W are the width and height of the current feature map, C is the number of channels, Conv is a regular convolution, DwConv is a deep convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the change of the channel before and after the convolution, that is, The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , The meaning of is to use a convolution with a kernel size of 1×1, and the input channel is , the output channel is c, and finally the residual connection is used to add the output to the original input x. The meaning is the original input x, and the output of the local convolution module LPM is obtained.

4. The hierarchical information fusion system based on CNN-Transformer according to claim 3, characterized in that: The lightweight multi-head self-attention module LWMHSA includes a k×k deep separation convolution DwConv with a step size of k and a self-attention module; In the original self-attention module, the input will be converted linearly into a query ,key Sum , where n = H × W represents the number of blocks into which the image or feature map is divided, and d, d k and d v Represents the dimensions of input, key, query and value respectively; the calculation formula is as follows: ； Where K T The meaning is the transpose of the key matrix K. T is the transpose symbol, QK T is the dot product of the transpose of the query matrix Q and the key matrix K, which calculates the similarity between each query and all keys; d k is the dimension of the key vector, It is used to prevent the dot product result from being too large, causing the gradient to disappear or explode. The output of the Softmax activation function is multiplied by the value matrix V to obtain a weighted value vector. Attn(Q,K,V) means the self-attention output obtained when the query matrix Q, key matrix K and value matrix V are input. The process of calculating self-attention of the lightweight multi-head self-attention module LWMHSA is: First, a depthwise separable convolution DwConv with a step size of k and a kernel size of k×k is used to reduce the spatial size of K and V. After the depthwise separable convolution, the key K and value V become and , and add the learnable relative position B to each attention module. The corresponding lightweight formula is as follows: ； in The meaning is that the key matrix is ​​reduced after the deep separation convolution reduces the spatial size The transpose of T is the transpose symbol, is the query matrix Q and the key matrix after the depthwise separation convolution reduces the spatial size The dot product of the transpose of , calculates the similarity between each query and all keys; d k is the dimension of the key vector, It is used to prevent the dot product result from being too large, causing the gradient to disappear or explode; the output of the Softmax activation function is convolved with the depth to reduce the spatial size of the value matrix Multiply them together to get a weighted value vector, The meaning is that when the input is the query matrix Q, the key matrix Sum Matrix The obtained lightweight self-attention output; The specific formula for fusing the global information learned by the lightweight multi-head self-attention module LWMHSA and the local feature extraction module LFEM with the local information using the information enhancement module IEM is as follows: ； in, This means that when the input image of H×W×C is split along the channel into a size of H×W×C / 2 As input, the output obtained by the lightweight multi-head self-attention module LWMHSA, It means that when the input image is split along the channel As input, the output obtained by the local feature extraction module LFEM; ConCat means the local feature extraction module and Splicing is performed along the channel dimension; the IEM module means sending the spliced ​​feature map to the information enhancement module to obtain the output after feature fusion.

5. The hierarchical information fusion system based on CNN-Transformer according to claim 4, characterized in that: The lightweight multi-head self-attention module LWMHSA divides the input data into multiple different subspaces, i.e., multiple heads, in the multi-head self-attention. Each head performs self-attention calculation independently. For each head, the LightAttn lightweight attention function is applied. The output of each head is a sequence of size n×d / h. The sequences from different head outputs are finally spliced ​​together to form a final output sequence of size n×d. The local feature extraction module LFEM captures the spatial local information of the image by using 1 3×3DwConv and 2 1×1Conv. The formula for capturing the spatial local information of the image is as follows: ； Among them, x is the input image split along the channel As input, DwConv is a deep convolution, Conv is a regular convolution, the subscripts 1×1 and 3×3 of the regular convolution and deep convolution represent the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a depthwise convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a kernel size of 1×1, and the input channel is , the output channel is c, and finally the residual connection is used to add the output to the input x. The meaning of LFEM(x) is the input x, and the output of the local feature extraction module LFEM is obtained.

6. The hierarchical information fusion system based on CNN-Transformer according to claim 1, characterized in that: The information enhancement module IEM fuses the global information generated by the lightweight multi-head self-attention module LWMHSA and the local information generated by the local feature extraction module LFEM as follows: The information enhancement module IEM first uses Conv with a convolution kernel size of 1×1 to fuse channel information for the two branches, and the number of channels c remains unchanged. Then, the obtained features are respectively captured by channel attention and spatial attention to capture channel and spatial dependencies, and the features of the two enhanced branches are further fused. The specific formula for fusion is as follows: ； Among them, x is the feature map of the concatenation of the output of the lightweight multi-head self-attention module LWMHSA and the output of the local feature extraction module LFEM, Conv is a conventional convolution, the subscript 1×1 of the conventional convolution represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a kernel size of 1×1, an input channel of c, an output channel of c, and y i is the output after 1×1Conv, i = 0, 1, y0 is used as the input of the spatial attention branch, and y1 is used as the input of the channel attention branch; Then based on branch 1, first use two consecutive convolution kernels of size 3×3 Conv on the input y0, then use the Sigmoid activation function to generate a spatial attention map, and finally multiply it with the original feature map to enhance the network's understanding and representation of the input image by emphasizing important spatial positions; the formula is as follows: ； Among them, y0 is the 1×1Conv output of one of the branches, which is used as the input of the spatial attention branch. Conv is a regular convolution. The subscript 3×3 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. That is, The meaning is to use a convolution with a kernel size of 3×3, an input channel of c, and an output channel of c. is the Sigmoid activation function, and s(y0) means the feature map obtained by using the spatial attention branch when the input is y0; For branch 2, first use AvgPool2d to directly compress the feature map containing global information into a 1×1 feature vector for the input y1, and then use two 1×1 Convs to learn the nonlinear relationship between channels; then use the Sigmoid activation function, and multiply the obtained vector weights with the input y1 channel by channel to enhance the important feature channels; the formula is as follows: ； Among them, y1 is the output of another branch 1×1Conv, which is used as the input of the channel attention branch. Conv is a regular convolution. The subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution. The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, The meaning of is to use a convolution with a kernel size of 1×1, and the input channel is , the output channel is c, is the Sigmoid activation function, AvgPool2d is the 2d global average pooling, and c(y1) means the feature map obtained by the channel attention branch when the input is y1; The features of the two enhanced branches are added and fused to get the output , the specific formula of fusion is as follows: ； Among them, y0 is the input of the spatial attention branch, y1 is the input of the channel attention branch, s(y0) is the feature map obtained after the spatial attention branch is enhanced, and c(y1) is the feature map obtained after the channel attention branch is enhanced. To fuse the enhanced features of the two branches.

7. The hierarchical information fusion system based on CNN-Transformer according to claim 1, characterized in that: The serial multi-scale feedforward network (SMFFN) module first uses a convolution kernel with a size of 1×1 to reduce the channel dimension to The output y0 is obtained, and then three deep convolution serial structures with a convolution kernel size of 3×3 are used to extract features from y0, and the outputs y1, y2, and y3 are obtained respectively. The obtained y0, y1, y2, and y3 are added and fused, and finally a Conv with a convolution kernel size of 1×1 is used to increase the channel dimension to c; the specific formula is as follows: ； ； ； ； ； Among them, x is the output of the information enhancement module IEM, Conv is a conventional convolution, DwConv is a deep convolution, the subscript 1×1 represents the size of the convolution kernel used, and the superscript represents the channel change before and after the convolution, that is, The meaning is to use a convolution with a convolution kernel size of 1×1, the input channel is c, and the output channel is , r is the channel reduction factor, y0 is the output after 1×1Conv, The meaning is to use a depth convolution with a convolution kernel size of 3×3, and the input channel is , the output channel is , y1 is the output of the first depth convolution using a convolution kernel size of 3×3, y2 is the output of the second depth convolution using a convolution kernel size of 3×3, and y3 is the output of the third depth convolution using a convolution kernel size of 3×3. The meaning of is to use a convolution with a kernel size of 1×1, and the input channel is , the output channel is c, and the meaning of SMFFN(x) is the feature map obtained after the features are fused through the serial multi-scale forward feedback network when the input x is the output of the information enhancement module IEM.

8. A CNN-Transformer-based hierarchical information fusion method, implemented based on the CNN-Transformer-based hierarchical information fusion system according to any one of claims 1 to 7, characterized in that: The following steps are involved: S1: The local convolution module LPM uses an inverted residual block to introduce an inductive bias to extract local information and introduce translation invariance; S2: Global and local information feature fusion module GLAF-Mixer uses parallel lightweight global attention modules and lightweight local information extraction features to combine global modeling information with local information; S3: The serial multi-scale feedforward network SMFFN uses multiple 3×3 depth-separable convolution serial structures to extract multi-scale features and fuse multiple depth-wise convolution outputs.

Images