Automatic driving 3D target detection method based on local feature coding and decoding

Abstract

The invention relates to an automatic driving 3D target detection method based on local feature coding and decoding, and belongs to the technical field of automatic driving road vehicle detection. The method comprises the following steps: S1, inputting road point cloud data into an Input network module for preprocessing; s2, performing three-dimensional feature extraction on the preprocessed data through a three-dimensional backbone network module; the three-dimensional backbone network module comprises a deep down-sampling convolutional network, a self-attention coding module and a cross attention decoding module; s3, inputting the three-dimensional features into a two-dimensional backbone network module, and projecting the three-dimensional features to a BEV view angle to obtain multi-scale two-dimensional features; the two-dimensional backbone network module comprises a self-attention position coding module based on Swin-Transform and an enhanced feature fusion module; s4, the three-dimensional features and the two-dimensional features are sent to a Head network module for fusion, classification and regression results are obtained, and detection information of the vehicles, the pedestrians and the riders of all categories is output.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving road vehicle detection, and relates to an autonomous driving 3D target detection method based on local feature encoding and decoding. Background Art

[0002] As the number of cars increases, vehicle-related traffic problems are becoming increasingly serious, such as road congestion and frequent accidents, causing immeasurable losses to personal safety and the economy.

[0003] In recent years, deep learning algorithms have continued to develop. With the excellent feature extraction capabilities of convolutional neural networks, deep learning methods have swept the field of target detection. Research on autonomous driving target detection based on deep learning has become a hot research area. To achieve autonomous driving, it is necessary to make instant decisions based on environmental information. Usually, the surrounding environment, including roads, vehicles, pedestrians, traffic signs, etc., is first perceived through various sensors (such as cameras, lidar, etc.). Then, a three-dimensional (3D) target detection method is used to generate a 3D digital representation of the vehicle's surrounding environment (including the shape, position and category of objects, etc.) through data processing and algorithm analysis.

[0004] Although the detection algorithm based on deep learning has achieved good results, there are still the following problems: the limitation of the receptive field leads to unsatisfactory detection results, the attention paid to foreground points is not higher than that to background points, and there is no good association between features, which makes it difficult to extract feature information and the detection performance of some categories is poor. Therefore, a new 3D object detection method for autonomous driving is urgently needed to solve the above problems. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an autonomous driving 3D target detection method based on local feature encoding and decoding. Aiming at the two problems of multi-level feature extraction and multi-scale feature context dependency in autonomous driving 3D target detection, a point-voxel detection framework is adopted to propose a local feature encoding and decoding network LFED-RCNN (Local Feature Encode-Decode RCNN) that integrates multiple technologies. It can accurately identify the information of vehicles, pedestrians and cyclists on the road in the collected point cloud data, so as to achieve fast and accurate detection effect.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A method for 3D target detection in autonomous driving based on local feature encoding and decoding, specifically comprising the following steps:

[0008] S1: Input the road point cloud data into the Input network module for preprocessing;

[0009] S2: The pre-processed road point cloud data is subjected to 3D feature extraction through a 3D backbone network module to obtain corresponding 3D features; the 3D backbone network module includes an Extra Downsampling Convolutional Network (EDSNet) and a Transformer-based local feature encoding and decoding module (LEDNet); the Transformer-based local feature encoding and decoding module (LEDNet) includes a self-attention encoding module (SA Encoder) and a cross-attention decoding module (CA Decoder);

[0010] S3: Input the three-dimensional features into the two-dimensional backbone network module, and project them to the BEV (Birds Eye View) perspective to obtain multi-scale two-dimensional features; the two-dimensional backbone network module includes a self-attention position encoding module (Position Encode, PE) based on Swin-Transformer and an enhanced feature fusion module;

[0011] S4: The 3D features and 2D features are fed into the Head network module to fuse and obtain the classification and regression results, and output the detection information of each category of vehicles, pedestrians and cyclists.

[0012] Furthermore, in step S1, the preprocessing is to perform data enhancement on the road point cloud data, specifically: randomly scaling, randomly cropping, randomly arranging and randomly flipping the road point cloud data.

[0013] Further, in step S2, the receptive field size calculation formula of the deep downsampling convolutional network (EDSNet) is:

[0014]

[0015] Among them, f k-1 is the size of the receptive field corresponding to the k-1th layer, l k is the convolution kernel size of the kth layer, s i is the step size of the i-th layer; it can be seen from the above formula that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, the receptive field can be expanded by increasing the depth of the convolutional network to obtain deeper features and fuse multi-level 3D features.

[0016] Furthermore, in step S2, the self-attention encoding module (SA Encoder) is designed based on the multi-head self-attention (MSA) mechanism, which consists of a multi-head self-attention (W hw _MSA) and D-window-based multi-head self-attention (Wd _MSA) and connected in a residual way; the HW window focuses on the association between HW dimensions, such as a tensor of (-1,H*W,C), and the D window focuses on the association on the D dimension, such as a tensor of (-1,D,C);

[0017] The implementation details are as follows: the input 3D convolutional feature map is divided into small blocks of equal size along the HW dimension, and encoded in the HW direction by means of a sliding window; then, the output features are reconstructed to restore the shape of the initial features; it is divided into small blocks again along the D dimension, encoded by means of a sliding window in the D direction, and finally, the encoded feature map is reconstructed to the initial shape and size; at the same time, in order to prevent overfitting, a Dropout layer is added, and then the initial features and the encoded features are concatenated by means of a residual connection; the local feature encoding process can be expressed as follows:

[0018] F l =W d _MSA(W hw _MSA{F S})

[0019] Among them, F S is the input feature, F l is the encoded feature, W d _MSA is the multi-head attention mechanism of D window, W hw _MSA is the multi-head attention mechanism for HW windows.

[0020] In general, SA Encoder divides the input features into small blocks and encodes the information in the small blocks along different dimensions in sequence, thus maintaining the continuity of 3D local area features and helping to provide more detailed information for foreground targets.

[0021] Further, in step S2, the cross attention decoding module (CA Decoder) specifically includes: the input feature map is the encoded multi-scale feature F l With F l `; First, perform a 3D convolution operation on the input features and reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is as follows:

[0022] Conv3d(x)=σ(w*x+b)

[0023] Among them, Conv3d(x) is a 3D convolution operation, x is the input feature map size, w is the convolution kernel size, b is the bias term, and σ is the activation function; after obtaining the same two features, first perform a 1*1*1 convolution to generate the initial query (Q), key (K), and value (V) respectively; reconstruct the space of Q, K, and V into R HW*D*C, where HW represents the size of the HW window, D represents the size of the D window, and C is the number of channels. The cross-attention weight of the HW dimension is calculated by matrix multiplication; then, the output feature is sent to the 1*1*1 convolution again to produce a new K and a new V; the initial Q, new K and new V are combined into a new query pair, and the space is reshaped into R D *HW*C , the D-dimensional cross attention weight is calculated by matrix multiplication. Through the cross query method, the two input features are combined, which greatly improves the representation ability of the output features.

[0024] Furthermore, in step S3, the specific implementation details of the position encoding module are as follows: the input features are first normalized by a LayerNorm layer, and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed (qkv) to obtain the query, key and value, and then the attention weight is calculated by the softmax function, and they are multiplied by the value to obtain the representation of the self-attention size; after the attention weight size is calculated, the original input and the output of the attention mechanism are added through a residual connection, and finally processed by the LayerNorm and MLP modules and the residual connection is used again to fuse the features. LayerNorm is a technology for standardizing input data, which helps to accelerate training and improve the generalization ability of the model. After the above operations, adding position association and rich context information to the 2D features helps to improve the quality of 2D features.

[0025] The beneficial effects of the present invention are as follows: the present invention adopts a point-voxel detection framework and constructs a local feature encoding and decoding network (LFED-RCNN) that integrates multiple technologies, which can accurately identify the information of vehicles, pedestrians, and cyclists on the road in the collected point cloud data, and achieve fast and accurate detection effects. Specifically, in the three-dimensional feature extraction stage, a three-dimensional CED backbone combining a convolutional network and a Transformer encoding and decoding structure is proposed, in which a deep downsampling convolutional network (EDSNet) extracts multi-level three-dimensional features, and a local encoding and decoding network (LEDNet) models the association between features and fuses deep and shallow features, thereby improving the model's ability to acquire features of foreground targets under complex backgrounds; secondly, a position encoding module is designed to perform position encoding of two-dimensional features from the BEV perspective, establish long-term dependencies, and improve detection accuracy.

[0026] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0028] Figure 1 It is a structural schematic diagram of the autonomous driving 3D target detection model based on local feature encoding and decoding of the present invention;

[0029] Figure 2 Schematic diagram of the deep downsampling convolutional network (EDSNet);

[0030] Figure 3 It is a schematic diagram of the local feature encoding structure (SA Encoder);

[0031] Figure 4 Schematic diagram of local feature cross decoding structure (CA Decoder);

[0032] Figure 5 It is the position encoding module in the two-dimensional backbone from the BEV perspective;

[0033] Figure 6 This is a performance comparison chart from the perspective of BEV;

[0034] Figure 7 This is a comparison chart of the recall rates of various methods. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0036] See also Figure 1 to Figure 7 The present invention provides an autonomous driving 3D target detection method based on local feature encoding and decoding, and an autonomous driving 3D target detection model based on local feature encoding and decoding. The target detection model includes an Input network module, a three-dimensional backbone network module, a two-dimensional backbone network module and a Head network module.

[0037] The three-dimensional backbone network module includes a deep downsampling convolutional network (EDSNet) and a Transformer-based local feature encoding and decoding module (LEDNet).

[0038] The formula for calculating the receptive field size of the deep downsampling convolutional network (EDSNet) is:

[0039]

[0040] Among them, f k-1 is the size of the receptive field corresponding to the k-1th layer, l k is the convolution kernel size of the kth layer, s i is the step size of the i-th layer. It can be seen from the above formula that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, the receptive field can be expanded by increasing the depth of the convolutional network to obtain deeper features. To this end, the present invention increases the depth of the downsampling convolutional network, obtains higher-level features, and fuses multi-level 3D features, such as Figure 2 In the 16× downsampling layer and the 32× downsampling layer, feature maps of size 100*88*3 and 50*44*3 are obtained respectively. At the same time, the choice of adding only 16 and 32 downsampling layers in the network design is based on several considerations. First, the purpose of adding downsampling layers is to perform effective feature extraction in higher network layers and increase the receptive field, which helps to improve the model's abstract ability and ability to handle complex patterns.

[0041] In the Transformer-based local feature encoding and decoding module (LEDNet), the self-attention encoding module (SAEncoder) is designed based on the multi-head self-attention (MSA) mechanism, such as Figure 3 As shown in the figure. The self-attention encoding module (SAEncoder) consists of a HW window-based (W hw _MSA) and D-window-based (W d _MSA) and connected in a residual way. The HW window focuses on the association between the HW dimensions, such as a tensor of (-1,H*W,C), while the D window focuses on the association on the D dimension, such as a tensor of (-1,D,C).

[0042] The implementation details are to divide the input 3D convolutional feature map into small blocks of equal size along the HW dimension, and encode it in the HW direction using a sliding window. Then, the output features are reconstructed to restore the shape of the initial features; they are divided into small blocks again along the D dimension, and encoded using a sliding window in the D direction, and finally the encoded feature map is reconstructed to the initial shape and size. At the same time, in order to prevent overfitting, a Dropout layer is added, and then the initial features and the encoded features are spliced ​​using a residual connection method. The local feature encoding process can be expressed as follows:

[0043] F l =W d _MSA(W hw _MSA{F S})

[0044] Among them, F S is the input feature, F l is the encoded feature.

[0045] In general, SA Encoder divides the input features into small blocks and encodes the information in the small blocks along different dimensions in sequence, thus maintaining the continuity of 3D local area features and helping to provide more detailed information for foreground targets.

[0046] In the Transformer-based local feature encoding and decoding module (LEDNet), the cross attention decoding module (CADecoder) is as follows Figure 4 As shown, the input feature map is the encoded multi-scale feature F l With F l First, perform a 3D convolution operation on the input features and reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is shown in formula (3):

[0047] Conv3d(x)=σ(w*x+b)

[0048] Among them, x is the input feature map size, w is the convolution kernel size, b is the bias term, and σ is the activation function. After obtaining the same two features, first perform a 1*1*1 convolution to generate the initial query (Q), key (K), and value (V) respectively. Reconstruct the space of Q, K, and V into R HW*D*C , the cross-attention weights of the HW dimension are calculated by matrix multiplication. After that, the output features are fed into the 1*1*1 convolution again to produce a new K and a new V. The initial Q, new K, and new V are combined into a new query pair, and the space is reshaped into R D *HW*C , the D-dimensional cross attention weight is calculated by matrix multiplication. Through the cross query method, the two input features are combined, which greatly improves the representation ability of the output features.

[0049] The two-dimensional backbone strengthens feature fusion based on the Swin-Transformer self-attention mechanism and outputs the layer to be predicted to the Head network module.

[0050] The complexity calculation formulas for Transformer and Swin-Transformer are as follows:

[0051] Ω T =4hwC 2 +2(hw) 2 C

[0052] Ω ST =4hwC 2 +2(M)2 HkDJ

[0053] In the above formula, the first is the complexity of Transformer, the second is the complexity of Swin-Transformer, hw represents the size of the input feature map, and M is the size of the sliding window. According to the above formula, it can be seen that the complexity of Swin-Transformer is only linearly related to the size of the feature map. When the sequence length is long, it is significantly lower than the quadratic linear relationship of Transformer. Position Encode (PE) from the perspective of BEV is as follows Figure 5 As shown. The specific implementation details are as follows: the input features are first normalized by a LayerNorm layer, and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed (qkv) to obtain the query, key and value, and then the attention weight is calculated by the softmax function, and they are multiplied by the value to obtain the representation of the self-attention size. After the attention weight size is calculated, the original input and the output of the attention mechanism are added through a residual connection, and finally processed by the LayerNorm and MLP modules and the residual connection is used again to fuse the features. LayerNorm is a technology that standardizes the input data, which helps to accelerate training and improve the generalization ability of the model. After the above operations, adding position association and rich context information to the 2D features helps to improve the quality of 2D features.

[0054] The head network module classifies and regresses the prediction layer to obtain detection information of vehicles, pedestrians, and cyclists on the road.

[0055] Verification experiment:

[0056] 1) Calculation and standards of indicators

[0057] Calculation of IoU:

[0058]

[0059] We mainly examine three categories of difficulty. For cars, the IoU is 0.7, 0.7, and 0.7, and for pedestrians and cyclists, the IoU is 0.5, 0.5, and 0.5.

[0060] Calculation of precision and recall:

[0061]

[0062] When evaluating proposals in the network, all proposals are first sorted by confidence score, and the proposal with the highest score is compared with the ground truth box by intersection and union, and compared with the specified intersection and union threshold. If it is greater than the threshold, it is marked as a true positive (TP), otherwise it is marked as a false positive (FP), and the precision of the detection result is calculated using the first formula. Foreground objects that are not detected are marked as false negatives (FN), and true negatives (TN) are not counted. The second formula is used to calculate the recall of all detection results.

[0063] Calculation of average precision:

[0064]

[0065] In the evaluation phase, different intersection-over-union ratios will result in different precision and recall results. As the recall increases, the precision gradually decreases. The PR curve is drawn with the recall as the horizontal axis and the precision as the vertical axis. If the horizontal axis calculation formula uses the equally divided points with values ​​of [0, 0.025, ...., 0.975, 1] ​​and the average value of the vertical axis is calculated, the result is as shown in the above formula.

[0066] 2) Experimental results

[0067] (1) Experimental data and settings

[0068] The experiment uses two NVIDIA GeForce RTX 3090 graphics cards, Pytorch version 1.10.0, distributed training, batch size set to 8, and training rounds of 80. Rotation and cropping methods are used for data augmentation. The number of channels of the sparse convolution backbone is 16, 32, 64, 64, 64, 64, respectively, the point set abstract radius is (0.4m, 0.8m), and training is performed at a learning rate of 0.01.

[0069] In order to evaluate the method and verify the feasibility of the method, the most advanced detection methods were followed and experiments were conducted on the KITTI dataset and the ONCE dataset. The KITTI dataset was jointly established by the Karlsruhe Institute of Technology in Germany and Toyota America Technical Research Institute to evaluate the performance of computer vision technologies such as stereo, optical flow, visual odometry, 3D object detection and 3D tracking in a vehicle-mounted environment. The dataset covers multiple real scenes such as urban areas, rural areas and highways. Each image contains up to 15 vehicles and 30 pedestrians, divided into different degrees of occlusion and truncation. It is currently one of the most widely used open source datasets in the field of 3D object detection. The dataset covers different weather, lighting and traffic conditions, provides a variety of training and testing scenarios, and can effectively test the performance of the model in different environments. It contains 3712 training samples and 3769 verification samples. To further reflect the generalization of the model, the method of the present invention was also experimented on the ONCE dataset. To accelerate the development of autonomous driving technology, Huawei Noah's Ark Lab, together with Huawei Autonomous Driving Cloud Service, Sun Yat-sen University, and the Chinese University of Hong Kong, released the ONCE dataset of one million 3D scene autonomous driving. The ONCE dataset contains a variety of driving scenarios, including different time periods (day / night), different weather (sunny / cloudy / rainy days), and different regions (urban / highway / tunnel). The ONCE dataset contains 1 million frames of 3D point cloud scenes, of which 15,000 frames of point clouds are annotated with detection boxes. This method uses training samples to train the model and uses verification samples to verify the performance of the model.

[0070] (2) Results and analysis

[0071] The results on the KITTI dataset and the comparison with other methods are shown in Table 1, the results on the ONCE dataset are shown in Table 2, and the ablation experiment is shown in Table 3.

[0072] Table 1 Experimental results on the KITTI dataset

[0073]

[0074] Table 2 Experimental results on the ONCE dataset

[0075]

[0076] Table 3 Ablation experiments of each module

[0077]

[0078] It can be seen from Table 1 and Table 2 that the method of the present invention has good effects in various categories and has good generalization. From Table 3, it can be concluded that the effectiveness of each module proposed by the method of the present invention.

[0079] In summary, in order to solve the problem of low detection accuracy in autonomous driving target detection, the present invention proposes an autonomous driving 3D target detection method based on local feature encoding and decoding. Through the strong feature extraction capabilities of the three-dimensional backbone and the two-dimensional backbone, higher quality features are obtained and effectively fused, the feature representation capability is enhanced, and the detection accuracy is improved, especially for categories that are difficult to detect and in complex road conditions.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A 3D target detection method for autonomous driving based on local feature encoding and decoding, characterized in that: The method specifically comprises the following steps: S1: Input the road point cloud data into the Input network module for preprocessing; S2: The pre-processed road point cloud data is subjected to 3D feature extraction through a 3D backbone network module to obtain corresponding 3D features; the 3D backbone network module includes a deep downsampling convolutional network and a Transformer-based local feature encoding and decoding module; the Transformer-based local feature encoding and decoding module includes a self-attention encoding module and a cross-attention decoding module; S3: Input the three-dimensional features into the two-dimensional backbone network module, and project them to the BEV perspective to obtain multi-scale two-dimensional features; the two-dimensional backbone network module includes a self-attention position encoding module based on Swin-Transformer and an enhanced feature fusion module; S4: The 3D features and 2D features are fed into the Head network module to fuse and obtain the classification and regression results, and output the detection information of each category of vehicles, pedestrians and cyclists.

2. The method for 3D target detection in autonomous driving according to claim 1, characterized in that: In step S1, the preprocessing is to perform data enhancement on the road point cloud data, specifically: randomly scaling, randomly cropping, randomly arranging and randomly flipping the road point cloud data.

3. The method for 3D target detection in autonomous driving according to claim 1, characterized in that: In step S2, the receptive field size calculation formula of the deep downsampling convolutional network is: Among them, f k-1 is the size of the receptive field corresponding to the k-1th layer, l k is the convolution kernel size of the kth layer, s i is the step size of the i-th layer; it can be seen from the above formula that the receptive field of the convolutional network increases layer by layer, showing an iterative relationship. Therefore, by increasing the depth of the convolutional network, the receptive field is expanded to obtain deep features and fuse multi-level 3D features.

4. The method for 3D target detection in autonomous driving according to claim 1, characterized in that: In step S2, the self-attention encoding module is designed based on a multi-head self-attention mechanism, and is composed of a multi-head self-attention based on an HW window and a multi-head self-attention based on a D window, which are connected in a residual manner; the HW window focuses on the association between HW dimensions, and the D window focuses on the association on the D dimension; The implementation details are as follows: the input 3D convolutional feature map is divided into small blocks of equal size along the HW dimension, and encoded in the HW direction by means of a sliding window; then, the output features are reconstructed to restore the shape of the initial features; it is divided into small blocks again along the D dimension, encoded by means of a sliding window in the D direction, and finally, the encoded feature map is reconstructed to the initial shape and size; at the same time, in order to prevent overfitting, a Dropout layer is added, and then the initial features and the encoded features are concatenated by means of a residual connection; the local feature encoding process is expressed as follows: F l =W d _MSA(W hw _MSA{F S }) Among them, F S is the input feature, F l is the encoded feature, W d _MSA is the multi-head attention mechanism of D window, W hw _MSA is the multi-head attention mechanism for HW windows.

5. The method for detecting 3D objects in an autonomous driving process according to claim 4, wherein: In step S2, the cross attention decoding module specifically includes: the input feature map is the encoded multi-scale feature F l With F l `; First, perform a 3D convolution operation on the input features and reconstruct the multi-scale 3D feature maps into the same size to generate key-value pairs. The specific calculation is as follows: Conv3d(x)=σ(w*x+b) Among them, Conv3d(x) is a 3D convolution operation, x is the input feature map size, w is the convolution kernel size, b is the bias term, and σ is the activation function; after obtaining the same two features, first perform a 1*1*1 convolution to generate the initial query Q, key K, and value V respectively; reconstruct the space of Q, K, and V into R HW*D*C , where HW represents the size of the HW window, D represents the size of the D window, and C is the number of channels. The cross-attention weight of the HW dimension is calculated by matrix multiplication; then, the output feature is sent to the 1*1*1 convolution again to produce a new K and a new V; the initial Q, new K and new V are combined into a new query pair, and the space is reshaped into R D*HW*C , the D-dimensional cross-attention weights are calculated by matrix multiplication.

6. The method for 3D target detection in autonomous driving according to claim 1, characterized in that: In step S3, the specific implementation details of the position encoding module are as follows: the input features are first normalized by a LayerNorm layer, and then passed to the WindowAttention layer for self-attention weight calculation. The input tensor is linearly transformed to obtain the query, key and value, and then the attention weights are calculated by the softmax function, and they are multiplied by the value to obtain the representation of the self-attention size; after the attention weight size is calculated, the original input and the output of the attention mechanism are added through a residual connection, and finally processed by the LayerNorm and MLP modules and the features are fused again using the residual connection.

Images