Image processing method based on double-branch parallel neural network

Abstract

The application relates to the technical field of computer vision image classification, in particular to an image processing method based on a double-branch parallel neural network. The method comprises the following steps: S1: inputting an image into a first basic convolution module and a feature embedding module to respectively perform feature extraction and spatial vector mapping, so as to obtain features X0 and a spatial vector Z0; S2: inputting X0 and Z0 into a bidirectional bridging module Block to output X block0 and Z block0 ; the step S2 comprises the following steps: S2-1: inputting X0 and Z0 into a first fusion module to output Z hid ; S2-2: inputting Z hid into a Transformer network module to learn and extract low-frequency global features in image features and output Z block0 ; S2-3: inputting X0 and Z block0 into a Flexible CNN to obtain local features in image features and output X hid ; S2-4: inputting X hid and Z block0 into a second fusion module to output X block0 ; S3: taking the output of the last bidirectional bridging module Block as the input of the next bidirectional bridging module Block, repeating the step S2 for several times, and finally outputting X block and Z block .

Description

Technical Field

[0001] This invention relates to the field of computer vision image classification technology, and specifically to an image processing method based on a dual-branch parallel neural network. Background Technology

[0002] Neural networks, as powerful machine learning models, have achieved excellent results in various visual tasks such as image classification, object detection, and semantic segmentation by learning complex features and functions from massive amounts of data. To enable networks to learn sufficiently rich data features and meet the needs of various complex scenarios, the depth and width of neural networks are constantly increasing, accompanied by a surge in computational cost and the number of parameters. Lightweight neural networks aim to build a lightweight network model with fewer parameters and lower computational resource consumption through better module combinations, while still meeting performance requirements. This fundamentally addresses the severe storage and computational resource consumption problem of deep neural networks, promoting the widespread adoption and practical application of deep learning. Most existing lightweight neural networks only optimize the convolutional module, breaking down ordinary convolution into a combination of depthwise convolution that changes the feature map size and 1×1 convolution that expands the number of feature map channels, in order to reduce the FLOPs (floating-point operations) of the neural network. Although good progress has been made, most of the work only optimizes the convolutional module. The basic convolution operation calculates the pixels within the rectangular area around the convolution kernel each time. No matter how the depth and width of the network change, the feature extraction in the network is always limited to the local range and cannot establish long-distance connections in the global image. On the other hand, the capacity of the neural network is limited, and the network benefits brought by increasing the number of data samples will gradually decrease.

[0003] To address the limitations of convolutional neural networks in capturing global information due to their limited receptive field, BotNet (Bottleneck Transformers for Visual Recognition) combines Transformers and convolutional neural networks. It integrates the self-attention mechanism from Transformers into the backbone of traditional convolutional neural networks, proposing a new module called BotBottleNeck. Using ResNet-50 as the base network, it replaces the 3×3 convolutions in the last three BottleNeck layers with a Multi-Head-Self-Attention (MHSA) module, similar to that in Transformers, while keeping the other modules unchanged. The network takes a traditional image as input, extracts features through the original backbone network to obtain a 2048-channel feature map, and inputs it into BotBottleNeck. First, a 1×1 convolution is performed to reduce the number of channels, and then it is fed into the MHSA layer. Unlike the multi-head self-attention layer in traditional Transformers, BotNet uses two vectors as spatial attention in the horizontal and vertical dimensions for the position encoding part. The two vectors are added together and multiplied by the query matrix to obtain the content position. The content position is then multiplied by the content encoding obtained from the query matrix and the key matrix to obtain the spatial similarity feature. Thus, when the multi-head self-attention module focuses on a more suitable region, the feature map is fed back into the 1×1 convolution to spread the channel.

[0004] However, existing techniques combining convolutional neural networks and Transformers either use convolutions at the beginning or embed them into each Transformer module, resulting in a cascaded design paradigm for the entire network. This concatenated approach to the two network structures leads to strong interdependencies between modules, requiring careful consideration of the connections and insertion points between different structures, making network modification difficult. Furthermore, existing solutions do not consider the cost-benefit ratio between the computational complexity and the resulting performance improvements, resulting in network models with FLOPs reaching several gigabytes and high deployment costs. Summary of the Invention

[0005] The present invention aims to provide an image processing method based on a dual-branch parallel neural network, which can solve the problem of high network deployment cost when combining convolutional neural networks and Transformers.

[0006] The basic solution provided by this invention is an image processing method based on a dual-branch parallel neural network, comprising the following steps:

[0007] S1: The input image is fed into the first basic convolutional module and the feature embedding module to perform feature extraction and spatial vector mapping respectively, so as to obtain feature X0 and spatial vector Z0.

[0008] S2: Input X0 and Z0 into the bidirectional bridging module Block, which includes a parallel convolutional neural network module Flexible CNN, a Transformer network module, a first fusion module, and a second fusion module;

[0009] S2 includes the following steps:

[0010] S2-1: X0 and Z0 are fed into the first fusion module, which is used to fuse local and global features and output Z0. hid ;

[0011] S2-2: Z hid The input is fed into the Transformer network module, which learns and extracts low-frequency global features from the image features, and outputs Z. block0 ;

[0012] S2-3: Set X0 and Z... block0 Inputting data into a Flexible CNN extracts local features from the image features and outputs X. hid ;

[0013] S2-4: X hid and Z block0 The input is to the second fusion module, which is used to fuse local and global features again, and outputs X. block0 ;

[0014] S3: Convert the X output from the previous bidirectional bridging module Block. block0 and Z block0 As the input to the next bidirectional bridging module Block, step S2 is repeated several times, finally outputting X. block and Z block .

[0015] The principle and advantages of this invention are as follows: The key aspect is adjusting the network structure to a dual-branch parallel structure, combining the relatively efficient computational operations of convolutional neural networks with the global feature extraction capabilities of the Transformer model. The network consists of lightweight convolutional branches and modular Transformer branches. Each branch only needs to focus on its own feature extraction and module construction, reducing the dependency between the two different networks. Simultaneously, a bidirectional bridging module is constructed with reference to the self-attention mechanism. This bridging module receives features from both the convolutional and Transformer branches, performs feature fusion within the module, and outputs the results, enabling information interaction between different branches. Ultimately, a dual-branch parallel network structure is obtained, thus solving the problem of high network deployment costs.

[0016] By constructing convolutional neural networks (CNNs) and Transformer neural networks in parallel, the parallel CNN branches and Transformer branches extract input features separately, thus decoupling the two types of neural networks. Simultaneously, a bidirectional bridging module is constructed, referencing the self-attention mechanism, to bidirectionally fuse and interact the features extracted by the two branches. This allows the network to fully leverage the advantages of CNNs in feature capture and Transformers in global information modeling.

[0017] In existing technologies, the network structure is designed in a serial shape, introducing Transformer modules at the head, middle, or tail of the convolutional neural network. This approach means that the relationships between preceding and subsequent modules must be considered when constructing each module, resulting in insufficient network flexibility. This invention re-examines the limitations of existing schemes combining convolutional neural networks and Transformer networks, redesigns the network structure, and adopts a parallel approach to make the modules in the network easier to modify.

[0018] Furthermore, it also includes the following steps:

[0019] S4: X block The input is fed into the second basic convolutional module for channel expansion, and the output is X. tail ;

[0020] S5: X tail In the input adaptive pooling module, X tail Let Input be the size of the input and Output be the expected size of the output. The adaptive pooling module calculates the pooling kernel size and stride based on Input and Output, performs dimensionality reduction on the features, and outputs X. pool .

[0021] To ensure the consistency of the output after pooling, an adaptive pooling operation is introduced to replace the conventional pooling. During the pooling process, the pooling kernel size and stride size can be automatically adjusted, reducing the network's limitation on the size of the input image, so as to adapt to input images of different sizes and improve the flexibility of the network.

[0022] Furthermore, S5 includes the following steps:

[0023] S501: Determine whether Input can divide Output. If Input can divide Output, execute S502. If Input cannot divide Output, execute step S503.

[0024] S502: Calculate the pooling kernel size K and step size S using the following formula;

[0025]

[0026] S503: Calculate the pooling kernel size K using the following formula. i and step size S i ;

[0027]

[0028] i represents the i-th pooling operation on a feature map, calculated starting from 0. ceil represents rounding up, and floor represents rounding down.

[0029] When Input cannot divide Output, the step size and pooling kernel size of each pooling operation change dynamically. The pooling kernel size and step size are calculated by rounding up and rounding down.

[0030] Furthermore, it also includes the following steps:

[0031] S6: X pool and Z block Concatenate along the channel dimension to output X. cat ;

[0032] S7: X cat The input is fed into a two-layer fully connected neural network to obtain the final output X. out .

[0033] Furthermore, S3 includes the following steps:

[0034] S301: Before outputting the previous output of the bidirectional bridging module Block to the bidirectional bridging module Block again, adjust the parameters in the convolutional neural network module, including the channels and stride.

[0035] By adjusting the parameters, the number of feature map channels in the final output of each step gradually increases while the feature map size gradually decreases. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the dual-branch network structure in an embodiment of the image processing method based on a dual-branch parallel neural network of the present invention.

[0037] Figure 2 This is a visualization of the layer-by-layer category activation map of the dual-branch parallel neural network, as an embodiment of the image processing method based on the dual-branch parallel neural network of the present invention. Detailed Implementation

[0038] The following detailed description illustrates the specific implementation method:

[0039] The basic implementation examples are as follows: Figure 1 and attached Figure 2 As shown:

[0040] The image processing method based on a dual-branch parallel neural network includes the following steps:

[0041] S1: The input image is fed into the first basic convolution module and the feature embedding module to perform feature extraction and spatial vector mapping, respectively, to obtain feature X0 and spatial vector Z0.

[0042] In this embodiment, S1 further includes the following steps:

[0043] S101: The input image is fed into the Base Conv module. The Base Conv module consists of three pure convolution operations. The first convolution operation is called Stem Conv, and the second and third convolution operations are called Neck Conv. It mainly performs basic feature extraction on the input image and obtains an output X0 for this step.

[0044] S2: Input X0 and Z0 into the bidirectional bridge module Block, and output X. block0 and Z block0 The bidirectional bridging module Block includes a parallel convolutional neural network module Flexible CNN, a Transformer network module, a first fusion module, and a second fusion module;

[0045] S2 includes the following steps:

[0046] S2-1: X0 and Z0 are fed into the first fusion module, which is used to fuse local and global features and output Z0. hid In this embodiment, the first fusion module is: Figure 2The CNN to Former module in the Transformer class performs multiple matrix multiplication and addition operations. Referring to the standard self-attention module in the Transformer, the implementation process is as follows, with all operations described below involving matrix multiplication:

[0047] (1) Multiply Z0 with the query matrix obtained from training;

[0048] (2) Multiply the result obtained in (1) by X0;

[0049] (3) Input the result obtained in (2) into the Softmax function;

[0050] (4) Multiply the result obtained in (3) by X0;

[0051] (5) Multiply the result obtained in (4) with the transformation matrix obtained from training;

[0052] (6) Add the result obtained in (5) to Z0 to obtain the final output Z. hid

[0053] S2-2: Z hid The input is fed into the Transformer network module, which learns and extracts low-frequency global features from the image features, and outputs Z. block0 ;

[0054] S2-3: Set X0 and Z... block0 Inputting data into a Flexible CNN extracts local features from the image features and outputs X. hid Specifically, through convolution and inductive bias, local small-domain contextual information of the features is obtained, resulting in the output X of this step. hid .

[0055] S2-4: X hid and Z block0 The input is to the second fusion module, which is used to fuse local and global features again, and outputs X. block0 ;

[0056] S3: Use the output of the previous bidirectional bridging module Block as the input of the next bidirectional bridging module Block, repeat step S2 several times, and finally output X. block and Z block The second fusion module is Figure 2 The Former to CNN module enables the further interaction and fusion of global and local features.

[0057] Specifically, X0 and Z0 are input into the Block-0 module, and X is output. block0 and Z block0 Then Xblook0 and Z block0 Input into Block-1 module, repeat step S2, and output X. block1 and Z block1 This process repeats, adjusting the block module's parameters, including channels and stride, before each new block's output becomes the input. During network construction, the corresponding configuration file `config.py` is read, and different networks are built based on the parameters set for each layer. This is a common technique in convolutional neural networks. Each channel can be considered a different feature. As the network's learned features become increasingly abstract, it's necessary to increase the number of channels to allow the network to represent and cover more key features, thereby improving classification accuracy. Feature map shrinking aims to reduce the number of parameters in the network, avoiding overfitting, and simultaneously improving inference speed.

[0058] This process gradually increases the number of channels in the final output feature map while gradually decreasing the feature map size at each step. In this embodiment, this is repeated eight times, from Block-0 to Block-7, with Block-7 ultimately used as the final output X. block7 and Z block7 The visualization results of the hierarchical category activation map from Block-0 to Block-7 are as follows: Figure 2 As shown.

[0059] S4: X block The input is fed into the second basic convolutional module for channel expansion, and the output is X. tail ;

[0060] S5: X tail In the input adaptive pooling module, X tail Let Input be the size of the input and Output be the expected size of the output. The adaptive pooling module calculates the pooling kernel size and stride based on Input and Output, performs dimensionality reduction on the features, and outputs X. pool The code implementation for this step will directly provide the size of the output feature map, and then adjust the step size and pooling kernel size for each pooling process based on the divisibility relationship between the input and output feature maps. Steps S501 and S502 provide the calculation method for the above two parameters for each pooling.

[0061] S501: Determine whether Input can divide Output. If Input can divide Output, execute S502. If Input cannot divide Output, execute step S503.

[0062] S502: Calculate the pooling kernel size K and step size S using the following formula;

[0063]

[0064] S503: When Input cannot divide Output, the step size and pooling kernel size of each pooling operation change dynamically. The pooling kernel size K is calculated using the following formula. i and step size S i ;

[0065]

[0066] i represents the i-th pooling operation on a feature map, calculated starting from 0. ceil represents rounding up, and floor represents rounding down.

[0067] Assuming adaptive pooling is max pooling, with an input size of 14 and a desired output size of 4, the adaptive pooling operation divides the above pooling steps into four intervals.

[0068] Input: [0,1,2,3,4,5,6,7,8,9,10,11,12,13]

[0069] Output: [3,6,10,13]

[0070] Interval 1: Pooling kernel size = 4 [0, 1, 2, 3] → [3]

[0071] Step size is 3

[0072] Interval 2: Pooling kernel size = 4[3,4,5,6] → [6]

[0073] Step size is 4

[0074] Interval 3: Pooling kernel size = 4 [7,8,9,10] →

[10]

[0075] Step size is 3

[0076] Interval 4: Pooling kernel size = 4[10,11,12,13] →

[13] .

[0077] S6: X pool and Z block Concatenate along the channel dimension to output X. cat ;

[0078] S7: X cat The input is fed into a two-layer fully connected neural network to obtain the final output X. out .

[0079] The above are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An image processing method based on a double-branch parallel neural network, characterized in that: The method comprises the following steps: S1: input the input image into the first base convolution module and the feature embedding module to respectively perform feature extraction and spatial vector mapping, and obtain feature and spatial vector ; S2: Will and Input is fed into the bidirectional bridging module Block, and output is... and The bidirectional bridging module Block includes a parallel convolutional neural network module Flexible CNN, a Transformer network module, a first fusion module, and a second fusion module; The S2 comprises the following steps: S2-1: send into a first fusion module, the first fusion module is used for completing fusion of local features and global features, output and send into a first fusion module, the first fusion module is used for completing fusion of local features and global features, output ; S2-2: inputting the image feature into the Transformer network module, learning and extracting low-frequency global features in the image feature, and outputting ;​ S2-3: obtaining the local features in the image features in the input Flexible CNN, and outputting the local features and in the input Flexible CNN, and outputting the local features ; S2-4: inputting the local feature and the global feature into a second fusion module, and outputting a second fusion result and a second fusion module for fusing the local feature and the global feature again, and outputting a second fusion result ; S3: taking the output of the last bidirectional bridging module Block as the input of the next bidirectional bridging module Block, repeating step S2 for several times, and finally outputting and ; The method further comprises the following steps: S4: inputting the first feature map into a second basic convolution module, performing channel expansion, and outputting a second feature map ; and S4: inputting the first feature map into a second basic convolution module, performing channel expansion, and outputting a second feature map S5: the input size is denoted as Input, the expected output size is denoted as Output, the adaptive pooling module calculates the pooling kernel size and the step size of the pooling operation according to Input and Output, reduces the dimension of the feature, and outputs in the adaptive pooling module, ; the input size of the adaptive pooling module is denoted as Input, the expected output size is denoted as Output, the adaptive pooling module calculates the pooling kernel size and the step size of the pooling operation according to Input and Output, reduces the dimension of the feature, and outputs ; The method further comprises the following steps: S6: concatenating and in the channel dimension, output ; S7: obtaining the final output by inputting the two-layer fully connected neural network .​ 2. The image processing method based on the double-branch parallel neural network according to claim 1, wherein: The S5 comprises the following steps: S501: judge whether Input can be divided by Output, when Input can be divided by Output, execute S502, when Input cannot be divided by Output, execute step S503; S502: Calculate the size of the pooling kernel by the following formula and the size of the step ; S503: Calculate the size of the pooling kernel by the following formula and the size of the step ; i represents the i-th pooling operation in a feature map, and is calculated from 0, ceil represents rounding up, and floor represents rounding down.

3. The dual-branch parallel neural network based image processing method of claim 1, wherein: The S3 comprises the following steps: S301: adjusting parameters in the convolutional neural network module before the output of the last bidirectional bridging module Block is output to the bidirectional bridging module Block again, the parameters comprising a channel and a step length.

Images