An image fusion module, a fusion system and a fusion method

Abstract

The application discloses an image fusion module, which comprises a modality weight network for generating modality weights, a channel weight network for generating channel weights, a weighted output network, an aggregation module and a feedback module connecting the aggregation module and an encoder, the weighted output network combines the modality weights with corresponding input modality features after multiplying the modality weights with the corresponding input modality features, and then multiplies the modality weights with the channel weights; the aggregation module refines the weighted features; and the feedback module feeds the fusion features back to the encoder to guide the extraction of modality features in the next stage. The application further discloses an image fusion network and an image fusion method. According to the image fusion module network and the fusion method, the weights are decomposed into modality weights and channel weights, multi-modality data are fused, the essence of the weight difference, i.e., the modality difference and the channel difference, can be focused on more than the prior art, the feedback module is used to enhance the selection ability of the encoder to useful information, and more efficient feature fusion is achieved.

Description

Technical Field

[0001] This invention belongs to the field of software technology and relates to image recognition technology, specifically to an image fusion module, fusion system and fusion method. Background Technology

[0002] Infrared thermal imaging offers advantages such as good penetration, long visible distance, sensitivity to biological temperature (when the organism is within a specific temperature range), and active imaging, making it a good complement to visible light imaging. With the advancement of infrared sensor industrialization, its cost has gradually approached that of visible light sensors, and in recent years, sensing technologies based on the fusion of visible light and infrared images have gained increasing attention.

[0003] Efficient feature fusion can enhance the complementary information between visible and infrared light while suppressing noise interference caused by modal failure. Therefore, effective feature fusion methods play an important role in visual tasks such as target detection and recognition based on infrared and visible light. Modal data exhibit both complementary and interference-suppressing effects. During fusion, it is desirable to strengthen the complementary positive information and suppress the interfering noise information. Previous infrared and visible light fusion schemes have used infrared features plus visible light features or channel merging of the two features. However, simple addition or channel parallelization is not the optimal approach for multimodal data fusion, as it does not explicitly represent the complementary relationship between the two modalities. Furthermore, traditional fusion schemes do not utilize the fused features to further guide feature extraction in the backbone network. The fused features tend to retain effective information. This tendency is beneficial for the backbone network to retain effective information and improve its selective extraction capability of effective information. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention discloses an image fusion module, a fusion system, and a fusion method.

[0005] The image fusion module of this invention includes a modality weighting network for generating different modality weights, a channel weighting network for generating different channel weights, a weighted output network, and an aggregation module connected to the weighted output network. The function of the weighted output network is as follows:

[0006] The modal weights output by the modal weighting network are multiplied by the corresponding input signals and then combined, and then multiplied by the channel weights output by the channel weighting network.

[0007] Preferably, the aggregation module is composed of multiple convolutional residual layers connected in series.

[0008] Preferably, the modal weighting network includes two convolutional layers and a converter connected to the outputs of the two convolutional layers. The converter includes a convolutional residual layer, a convolutional layer, and a sigmoid function layer connected in sequence.

[0009] Preferably, the channel weight network includes two average pooling layers and an extractor connected to the outputs of the two average pooling layers. The extractor includes a linear layer, a nonlinear layer, a linear layer, and a sigmoid function layer connected in sequence.

[0010] Preferably, the nonlinear layer is a SiLU function layer.

[0011] The present invention also discloses an image fusion system, including a first encoder to a Kth encoder for processing images of different modalities, and a decoder. The first encoder to the Kth encoder includes multiple encoding stages. The image fusion system further includes a fusion unit, which includes multiple fusion modules. Each fusion module is connected to the encoding stage corresponding to the first encoder to the Kth encoder, and the output of each fusion module is connected to the decoder.

[0012] Preferably, the first encoder to the Kth encoder are all feature encoders based on convolutional neural networks or feature encoders based on conversion modules.

[0013] Preferably, it also includes a feedback module connected between adjacent encoding stages of each encoder, wherein the first input terminal of the feedback module is connected to the output terminal of the previous encoding stage, the second input terminal of the feedback module is connected to the output terminal of the fusion module corresponding to the previous encoding stage, and the output terminal of the feedback module is connected to the input terminal of the next encoding stage.

[0014] The feedback module includes a weight generation network connected to the second input terminal of the feedback module. The output terminal of the weight generation network and the first input terminal of the feedback module are connected to a multiplication calculator. The output terminal of the multiplication calculator is connected to a feature extraction network. The output terminal of the feature extraction network and the first input terminal of the feedback module are connected to an addition calculator. The addition calculator is connected to the output terminal of the feedback module.

[0015] This invention also discloses an image fusion method, comprising the following steps:

[0016] Step 1. Use K encoders corresponding to different modal inputs to extract the input features in stages; perform all the following steps before the decoding step on the K modal features extracted in stages;

[0017] Step 2. The K modal features extracted from the corresponding stage of the encoder are fed into the convolutional layer of the modal weight network in the fusion module to extract the features. Then, the output features are non-linearly transformed using the Sigmoid function layer to obtain the modal weights from the 1st to the (K-1)th modal weights. The Kth modal weight is equal to 1 minus the sum of the modal weights from the 1st to the (K-1)th modal weights.

[0018] Step 3. Multiply the obtained modal weights by the modal features respectively, and then perform channel merging to obtain preliminary fused features;

[0019] Step 4. Reduce the spatial resolution of modal features using the average pooling layer in the channel weight network, then extract features sequentially using linear and nonlinear layers, and finally generate channel weights using the Sigmoid function layer.

[0020] Step 5. Multiply the channel weights obtained in Step 4 by the preliminary fusion features obtained in Step 3 to obtain the weighted fusion features;

[0021] Step 6. Aggregate the weighted fusion features and output the aggregated stage features;

[0022] Decoding steps: Input the features of each stage into the decoder for decoding.

[0023] Preferably, steps 7 to 8 are also included between step 6 and the decoding step;

[0024] Step 7. Using the aggregated stage features obtained in Step 6, input them into the feedback module and perform feedback fusion with the image features generated in Step 1 to generate feedback features;

[0025] Step 8. Input the feedback features obtained in Step 7 into the next stage of the encoder.

[0026] The image fusion module, fusion system, and fusion method described in this invention separate the weights into modal space weights and channel weights for feature fusion. By utilizing a cross-modal space weighting mechanism, multimodal data is fused. Compared with existing technologies, this approach can focus more on the essence of weight differences, namely modal and channel differences. At the same time, the feedback module enhances the encoder's ability to select useful information, thereby achieving more efficient feature extraction and fusion. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating a specific embodiment of the image fusion module described in this invention;

[0028] Figure 2 This is a schematic diagram of a specific embodiment of the image fusion system described in this invention;

[0029] Figure 3This is a schematic diagram of another specific embodiment of the image fusion system described in this invention;

[0030] Figure 4 This is a schematic diagram of a specific embodiment of the feedback module described in this invention;

[0031] Figure 5 This is a schematic diagram comparing a specific embodiment of the present invention with the prior art;

[0032] Figures 1 to 4 The reference numerals in the attached diagram are as follows: 1-merging calculator, 2-multiplication calculator, IN1-first input terminal of the image fusion system, IN2-second input terminal of the image fusion system, IN3-first input terminal of the feedback module, IN4-second input terminal of the feedback module, R-first input terminal of the image fusion module, T-second input terminal of the image fusion module, F-output terminal of the image fusion module. Detailed Implementation

[0033] The specific embodiments of the present invention will be described in further detail below.

[0034] This invention discloses an image fusion method, which can be based on, for example, Figure 2 The image fusion system shown is implemented as follows: Figure 1 The specific implementation of the image fusion module in the example is as follows: Figure 1 As shown, for ease of description, the input fusion is described using two modes, namely infrared and visible light, i.e., K=2:

[0035] The method includes the following steps:

[0036] Step 1. Use two encoders corresponding to different modal inputs to extract the infrared and visible light features of the input image in stages;

[0037] Common image encoders can be used, such as residual network-based encoders and transformation-based feature encoders; the corresponding feature extraction stages and the channel dimension of the features can be adjusted according to the specific task.

[0038] The encoder extracts features from the input image in stages, with the resolution gradually decreasing and the number of encoding channels increasing in each stage. The dimensions of the infrared and visible light features extracted in each stage differ. The specific parameters of each submodule in the image fusion module connected to each stage are determined based on the specific parameters of the input and output features for that stage.

[0039] like Figure 1 As shown, the infrared and visible light features extracted in each stage of the encoder are processed in steps 2 to 6 respectively.

[0040] Step 2. The encoder feeds the extracted infrared and visible light features into the convolutional layer of the modality weight network in the image fusion module to extract the features. Then, the Sigmoid function layer is used to perform nonlinear transformation on the output features to obtain the visible light modality weight. The infrared modality weight is equal to 1 minus the visible light modality weight.

[0041] Figure 1 In the image fusion module shown, infrared features and visible light features are input from the first input terminal R and the second input terminal T of the image fusion module, respectively. The convolutional layer extracts the features, and after stitching, they are sequentially input into the convolutional residual layer and the output convolutional layer for processing. The convolutional residual layer performs further feature extraction on the stitched features, transforming the image features from the original feature space to a weighted feature space that represents the effectiveness of the features. The output convolutional layer compresses the channels to 1.

[0042] Step 3. Multiply the obtained visible light mode weights and infrared mode weights by the visible light features and infrared features respectively, and then merge the channels to obtain preliminary fusion features; this step can be completed by the weighted output network in the image fusion system.

[0043] By multiplying the visible light mode weights and infrared mode weights by the visible light features and infrared features respectively and merging them, a preliminary fusion feature is obtained. The preliminary fusion performs a weighted summation of the modal features with different effectiveness in different spatial regions, namely infrared and visible light features, which can suppress the interference of ineffective modal features on other modal features and enhance the expression of effective features.

[0044] Step 4. Reduce the spatial resolution of infrared and visible light features using the average pooling layer in the channel weight network, then extract features sequentially using linear and nonlinear layers, and finally generate channel weights using the Sigmoid function layer.

[0045] Since the final output channel weight spatial resolution needs to be consistent with the intermediate image feature resolution, the output channel weight resolution is relatively low, at 1*1, while the input image feature resolution is relatively high, at M*N, where M is the width and N is the height, both greater than 1. First, an average pooling layer is used to reduce the spatial resolution of the input image features. Specifically, the average of the image features in the width and height dimensions is calculated to compress the spatial resolution of the M*N image features to 1*1, which facilitates the subsequent conversion of the image features into channel weights.

[0046] Features are extracted sequentially using linear and nonlinear layers, transforming the features from the image feature space to the channel weight feature space.

[0047] The nonlinear layer can be a SiLU nonlinear layer, which is an existing technology in deep learning. The linear layer is a feature integration unit commonly used in deep learning.

[0048] Step 5. Multiply the channel weights obtained in Step 4 by the preliminary fusion features obtained in Step 3 to obtain the weighted fusion features;

[0049] Channel weighting can enhance the expression of effective channel features and suppress the expression of ineffective channels, thereby improving the expressive power of features.

[0050] Step 6. Aggregate the weighted fusion features and output the aggregated stage features from the image fusion module output terminal F;

[0051] Figure 1 In the specific implementation of the image fusion module shown, multiple convolutional residual units are used to further aggregate features. Aggregation can further extract and refine features, and the ability to abstract and extract features is enhanced by using multiple convolutional residual units for aggregation. Convolutional residual units are existing technology and can be composed of convolutional layers, batch normalization layers, nonlinear layers, and residual structures.

[0052] The infrared and visible light features extracted at each stage of the encoder are processed according to steps 2 to 6 above, as shown in Figure 2. F1...FN represent the fusion results output by different image fusion modules after fusion at different stages. Finally, the decoding step is performed, and the features at each stage are input to the decoder for decoding.

[0053] The image fusion module of this invention includes a modality weighting network for generating different modality weights, a channel weighting network for generating different channel weights, a weighted output network, and an aggregation module connected to the weighted output network. The function of the weighted output network is as follows:

[0054] The modal weights output by the modal weighting network are multiplied by the corresponding input signals and then combined, and then multiplied by the channel weights output by the channel weighting network.

[0055] The image fusion system of the present invention includes a first encoder and a second encoder for processing images of different modalities, and a decoder. The first encoder and the second encoder include multiple encoding stages. The image fusion system also includes a fusion unit, which includes multiple fusion modules. Each fusion module is connected to the encoding stage corresponding to the first encoder and the second encoder, and the output of each fusion module is connected to the decoder.

[0056] The image fusion method can be implemented using the image fusion module and image fusion system described above.

[0057] Figure 3 and Figure 4 Another preferred implementation of the image fusion system is given, which uses a feedback module for feedback calculation.

[0058] Figure 3 The image fusion system shown also includes a feedback module connected between adjacent encoding stages of each encoder. The first input terminal IN3 of the feedback module is connected to the output terminal of the previous encoding stage, the second input terminal IN4 of the feedback module is connected to the output terminal of the corresponding fusion module of the previous encoding stage, and the output terminal of the feedback module is connected to the input terminal of the next encoding stage.

[0059] like Figure 4 As shown, the feedback module includes a weight generation network connected to the second input of the feedback module, the output of the weight generation network and the first input of the feedback module are connected to a multiplication calculator, the output of the multiplication calculator is connected to a feature extraction network, the output of the feature extraction network and the first input of the feedback module are connected to an addition calculator, and the addition calculator is connected to the output of the feedback module.

[0060] based on Figure 3 The image fusion method of the image fusion system shown can continue with the following steps after steps 1 to 6 above:

[0061] Step 7. Using the aggregated stage features obtained in Step 6, input them into the feedback module and perform feedback fusion with the image features generated in Step 1 to generate feedback features;

[0062] Step 8. Input the feedback features obtained in Step 7 into the next stage of the encoder;

[0063] The encoder repeats steps 1 to 8 at each encoding stage, and finally inputs the features of each stage output from each image fusion module into the decoder for decoding.

[0064] The weight generation network in the feedback module can be composed of convolutional layers, batch normalization layers, residual structures, and nonlinear layers (i.e., convolutional residual layers) as used in existing technologies. Considering that the fused features output by the fusion module contain useful information, we can use the fused features to generate weight templates and extract useful information from the modal features output by the previous encoding stage. The weight generation network generates weights based on the fused features output by the input fusion module, and multiplies them with the modal features output by the previous encoding stage. Multiplication is an operation of enhancing similarity and suppressing dissimilarity; it can be seen as retaining similar components in the modal features output by the previous encoding stage and the fused features, while eliminating dissimilar components. This product is then processed by a feature extraction network for feature extraction, and added to the modal features before output. The feature extraction network consists of convolutional residual layers, whose main function is to perform further feature extraction, generating more advanced features. The function of adding the modal features output by the previous encoding stage is similar to that of the residual structure, primarily to make the backpropagation of network gradients more efficient and avoid gradient saturation or vanishing gradient problems. By outputting feedback features to the next encoder stage through the aforementioned feedback module, the image fusion system can be more inclined to select features that are similar to the fused features for encoding, thus possessing a more efficient feature extraction capability. Specific Implementation

[0065] Feature extraction is performed on an image with an initial resolution of 1024*768.

[0066] An encoder based on a residual network is adopted. In this embodiment, ResNet50 is used as the encoder network as the first encoder and the second encoder. The extraction is set to be performed in four stages: the first stage has a resolution of 256*192 and 256 channels; the second stage has a resolution of 128*96 and 512 channels; the third stage has a resolution of 64*48 and 1024 channels; and the fourth stage has a resolution of 32*24 and 2048 channels.

[0067] In the first stage of the image fusion module, the specific parameters of the convolutional layer in the modality weighting network are set as follows: 256 input channels, 3*3 convolutional kernels, and 64 output channels.

[0068] The specific parameters of the convolutional residual layer are: 128 input channels, 3x3 kernel, and 64 output channels. Batch normalization and ReLU non-linear processing are used. The specific operation of the ReLU function is: RELU(x) = max(x, 0). The specific parameters of the output convolutional layer are...

[0069] The input channel is 64, the convolution kernel is 1*1, and the output channel is 1.

[0070] The specific form of the Sigmoid function layer is S(x) = 1 / (1 + exp(-x)).

[0071] The specific parameters of the convolutional residual layer used in the aggregation stage are 512 input channels, 3*3 convolutional kernels, 256 output channels, batch normalization, and ReLU function nonlinear processing.

[0072] The specific operation of setting the average pooling layer in the channel weight network is as follows:

[0073] x represents the input pixels, M is the image pixel width, and N is the image pixel length.

[0074] The first linear layer has the following parameters: 512 input channels and 32 output channels.

[0075] The specific operations of the SiLU function layer are as follows:

[0076] S iLU (x) = x / (1 + exp(-x)), where the specific parameters of the second linear layer are 32 input channels and 512 output channels.

[0077] The Sigmoid function layer is specifically as follows:

[0078] S(x) = 1 / (1 + exp(-x)).

[0079] The operation performed by the feedback module is X. OUT =G(W(F in )*X in )+X in , where X in F is the output of the previous encoding stage. in The fusion features are represented by W, which represents the weight generation network, implemented by convolutional residual layers. Its specific parameters are: 256 input channels, 1*1 convolutional kernel, 256 output channels, batch normalization, and the Sigmoid non-linear function to generate weights. G represents the feature extraction network, which is composed of convolutional residual layers. Its specific parameters are: 256 input channels, 1*1 convolutional kernel, 256 output channels, batch normalization, and ReLU non-linear processing.

[0080] In the second-stage image fusion module, the specific parameters of the convolutional layer in the modal weight network are set as follows: 512 input channels, 3*3 convolutional kernels, and 128 output channels. The specific parameters of the convolutional residual layer are set as follows: 256 input channels, 3*3 convolutional kernels, and 128 output channels. Batch normalization and ReLU function nonlinear processing are used. The specific parameters of the output convolutional layer are set as follows: 128 input channels, 1*1 convolutional kernel, and 1 output channel.

[0081] The specific parameters of the convolutional residual layer used in the aggregation stage are: 1024 input channels, 3*3 convolutional kernels, 512 output channels, batch normalization, and ReLU function nonlinear processing.

[0082] The specific parameters of the first linear layer in the channel weight network are set as follows: 1024 input channels and 64 output channels. The specific parameters of the second linear layer are 64 input channels and 1024 output channels.

[0083] The parameters of the feedback module are as follows: the weight generation network has 512 input channels, 3*3 convolutional kernels, 512 output channels, batch normalization, and a Sigmoid non-linear function; the feature extraction network has 512 input channels, 3*3 convolutional kernels, 512 output channels, batch normalization, and a ReLU non-linear function.

[0084] In the third-stage image fusion module, the specific parameters of the convolutional layer in the modal weight network are set as follows: input channels 1024, convolutional kernel 3*3, output channels 256. The specific parameters of the convolutional residual layer are set as follows: input channels 512, convolutional kernel 3*3, output channels 256. Batch normalization and ReLU function nonlinear processing are used. The specific parameters of the output convolutional layer are set as follows: input channels 256, convolutional kernel 1*1, output channel 1.

[0085] The specific parameters of the convolutional residual layer used in the aggregation stage are as follows: 2048 input channels, 3*3 convolutional kernels, 1024 output channels, batch normalization, and ReLU function non-linear processing.

[0086] The specific parameters of the first linear layer in the channel weight network are set as follows: 2048 input channels and 128 output channels. The specific parameters of the second linear layer are set as follows: 128 input channels and 2048 output channels.

[0087] The parameters of the feedback module are as follows: for the weight generation network, the specific parameters are 1024 input channels, 3*3 convolutional kernels, 1024 output channels, batch normalization, and non-linear processing using the Sigmoid function; for the feature extraction network, the specific parameters are 1024 input channels, 3*3 convolutional kernels, 1024 output channels, batch normalization, and non-linear processing using the ReLU function.

[0088] In the fourth stage of the image fusion module, the specific parameters of the convolutional layer in the modal weight network are set as follows: input channels 2048, convolutional kernel 3*3, output channels 512; the specific parameters of the convolutional residual layer are set as follows: input channels 1024, convolutional kernel 3*3, output channels 512; batch normalization and ReLU function nonlinear processing are used; and the specific parameters of the output convolutional layer are set as follows: input channels 512, convolutional kernel 1*1, output channel 1.

[0089] The specific parameters of the convolutional residual layer used in the aggregation stage are: 4096 input channels, 3*3 convolutional kernels, 2048 output channels, batch normalization, and ReLU function non-linear processing.

[0090] The specific parameters of the first linear layer in the channel weight network are set as follows: 4096 input channels and 256 output channels. The specific parameters of the second linear layer are set as follows: 256 input channels and 4096 output channels.

[0091] The parameters of the feedback module are as follows: the weight generation network has 2048 input channels, 3*3 convolutional kernels, and 2048 output channels, using batch normalization and sigmoid function for non-linear processing; the feature extraction network has 2048 input channels, 3*3 convolutional kernels, and 2048 output channels, using batch normalization and ReLU function for non-linear processing.

[0092] The image fusion method described in this invention is compared with methods such as Simple Addition (ADD), Channel Merging (CONCAT), and Direct Weight Matrix Generation (SSMA). Using the same training strategy, it was trained four times, and the results are as follows. Figure 5 As shown, ME, ADD, CONCAT, and SSMA represent the method described in this invention, the simple addition method, the channel merging method, and the direct generation of the weight matrix method, respectively.

[0093] mIoU (mean Intersection-Over-Union) and mAcc (mean Accuracy) are the intersection-over-union ratio and accuracy, respectively, in the evaluation metrics of semantic segmentation algorithms. Higher values ​​indicate better model performance. It can be seen that the proposed scheme in this invention has better feature extraction and fusion efficiency compared to other schemes, further improving the performance of downstream semantic segmentation tasks.

[0094] The foregoing descriptions are preferred embodiments of the present invention. Unless there is a clear contradiction between the preferred embodiments or a prerequisite for a particular preferred embodiment, the preferred embodiments can be arbitrarily combined and used. The embodiments and specific parameters described are only for clearly illustrating the inventor's invention verification process and are not intended to limit the scope of patent protection of the present invention. The scope of patent protection of the present invention shall still be determined by its claims. Similarly, any equivalent structural changes made based on the description and drawings of the present invention shall also be included within the scope of protection of the present invention.

Claims

1. An image fusion system, comprising a first encoder to a Kth encoder for processing images of different modalities, and a decoder, wherein the first encoder to the Kth encoder comprises multiple encoding stages, characterized in that, The image fusion system further includes a fusion unit, which includes multiple image fusion modules. Each fusion module is connected to the encoding stage corresponding to the first encoder to the Kth encoder, and the output of each fusion module is connected to the decoder. It also includes a feedback module connected between adjacent encoding stages of each encoder. The first input of the feedback module is connected to the output of the previous encoding stage, the second input of the feedback module is connected to the output of the corresponding fusion module connected to the previous encoding stage, and the output of the feedback module is connected to the input of the next encoding stage. The feedback module includes a weight generation network connected to the second input terminal of the feedback module, the output terminal of the weight generation network and the first input terminal of the feedback module are connected to a multiplication calculator, the output terminal of the multiplication calculator is connected to a feature extraction network, the output terminal of the feature extraction network and the first input terminal of the feedback module are connected to an addition calculator, and the addition calculator is connected to the output terminal of the feedback module. The image fusion module includes a modality weighting network for generating different modality weights, a channel weighting network for generating different channel weights, a weighted output network, and an aggregation module connected to the weighted output network. The function of the weighted output network is as follows: The modal weights output by the modal weighting network are multiplied by the corresponding input signals and then combined, and then multiplied by the channel weights output by the channel weighting network.

2. The image fusion system as described in claim 1, characterized in that, The first encoder to the Kth encoder are all feature encoders based on convolutional neural networks or feature encoders based on conversion modules.

3. An image fusion method, characterized in that, Includes the following steps: Step 1. Use K encoders corresponding to different modal inputs to extract the input features in stages; Before the decoding step, all of the following steps are performed on the K modal features extracted in stages; Step 2. The K modal features extracted from the corresponding stage of the encoder are fed into the convolutional layer of the modal weight network in the fusion module to extract the features. Then, the output features are non-linearly transformed using the Sigmoid function layer to obtain the modal weights from the 1st to the (K-1)th modal weights. The Kth modal weight is equal to 1 minus the sum of the modal weights from the 1st to the (K-1)th modal weights. Step 3. Multiply the obtained modal weights by the modal features respectively, and then perform channel merging to obtain preliminary fused features; Step 4. Reduce the spatial resolution of modal features using the average pooling layer in the channel weight network, then extract features sequentially using linear and nonlinear layers, and finally generate channel weights using the Sigmoid function layer. Step 5. Multiply the channel weights obtained in Step 4 by the preliminary fusion features obtained in Step 3 to obtain the weighted fusion features; Step 6. Aggregate the weighted fusion features and output the aggregated stage features; Decoding steps: Input the features of each stage into the decoder for decoding; It also includes steps 7 and 8 between step 6 and the decoding step; Step 7. Using the aggregated stage features obtained in Step 6, input them into the feedback module and perform feedback fusion with the image features generated in Step 1 to generate feedback features; Step 8. Input the feedback features obtained in Step 7 into the next stage of the encoder.

Images