A multi-scale feature fusion method based on Hadamard product

Abstract

The invention relates to a multi-scale feature fusion method based on Hadamard product, which belongs to the technical field of data processing. The method comprises: obtaining a feature image based on an underwater image, performing feature extraction to obtain a feature image S3, a feature image S4 and a feature image S5; encoding the feature image S5 to obtain a feature image F5, inputting the feature image F5 into an FPN encoding structure, performing two-stage upsampling on a feature image U1 obtained by convolution processing, and fusing the feature image S4 and the feature image S3 step by step, and obtaining a feature image U2 and a feature image O1 in turn; performing two-stage downsampling on the feature image O1 after convolution processing, and fusing the feature image U2 and the feature image U1 step by step, and then fusing the feature image O1 with the feature image O1 to obtain a fusion result; and multiple fusions can be performed on feature images of different scales through Hadamard product to prevent information loss of target features during extraction and fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a multi-scale feature fusion method based on Hadamard product. Background Art

[0002] Effective feature extraction of input images is the main factor in determining whether the model has good detection results. The stacking of ordinary convolutions will cause feature information to be gradually lost as the depth increases. Therefore, the concept of residual network is introduced to splice shallow features with deep features, alleviating the problem of feature information loss. Therefore, feature fusion helps extract features from the backbone layer. The network neck layer structure serves as a connection between the backbone and the detection head, and directly determines the detection effect of the detection head. Therefore, feature fusion of the neck layer is crucial.

[0003] FPN is introduced for the neck layer. First, upsampling is performed to increase the size of the small-scale feature map and fuse it with the features of the previous level. Then downsampling is used and reverse secondary fusion is performed, during which three scales of output are completed: large, medium and small. However, the existing feature fusion method structure, such as CN114694003A, discloses that the feature map C5 is fused with the feature map C4 after upsampling to obtain P4_2, and P4_2 is fused with the feature map C3 after upsampling to obtain P3_2, and finally the feature map P3_2, feature map P4_2 and feature map P5_2 fused in the spatial dimension are obtained; the size of the feature map P3_2, the size of the feature map P4_2 and the size of the feature map P5_2 are not equivalent, and the above patent has the problem of only performing feature fusion of the spatial and channel dimensions on feature images of different scales, which is easy to affect the fusion efficiency, and there is a problem that rich and effective target features cannot be captured, resulting in information loss of target features during extraction and fusion. Summary of the invention

[0004] In view of the shortcomings existing in the related art, the purpose of the present invention is to provide a multi-scale feature fusion method based on the Hadamard product to solve the technical problems that the prior art only performs feature fusion of spatial and channel dimensions on feature images of different scales once, which easily affects the fusion efficiency and fails to capture rich and effective target features, resulting in information loss of target features during extraction and fusion.

[0005] The present invention provides a multi-scale feature fusion method based on Hadamard product, comprising the following steps:

[0006] Image processing step: acquiring an underwater image, and scaling the underwater image to obtain a feature image; wherein the underwater image is scaled to a preset reference size by an adaptive image scaling algorithm, and when the underwater image is smaller than the reference size, the underwater image is scaled to the reference size by an interpolation method;

[0007] Feature extraction step: extracting features from the feature image to obtain a multi-scale feature image, wherein the multi-scale feature image includes: feature image S3, feature image S4 and feature image S5;

[0008] Feature fusion step: encoding the feature image S5 through a multi-head attention mechanism to obtain a feature image F5, inputting the feature image F5 into an FPN coding structure, performing two-stage upsampling on the feature image U1 obtained by convolution processing and fusing the feature image S4 and the feature image S3 step by step, obtaining the feature image U2 and the feature image O1 in turn, performing two-stage downsampling on the feature image O1 after convolution processing and fusing the feature image U2 and the feature image U1 step by step, obtaining the feature image O2 and the feature image O3 in turn, fusing the feature image O1, the feature image O2, and the feature image O3 to obtain a fusion result;

[0009] The FPN coding structure includes: a three-level coding model SSEF and a Hadamard model HMF, and adjacent coding models SSEF are connected through a first convolutional layer and a second convolutional layer, and the output of each level of coding model SSEF is connected to the Hadamard model HMF, and the three-level coding model SSEF includes a first coding model, a second coding model, and a third coding model;

[0010] Wherein, the feature fusion step includes:

[0011] The first-level upsampling and fusion step is to convolve the feature image F5 through the first convolution layer to obtain the feature image U1, and the feature image U1 is input into the second coding model. The feature image U1 is upsampled by the second coding model, and the feature image is fused with the feature image S4, and then convolved through the first convolution layer to obtain the feature image U2;

[0012] A secondary upsampling and fusion step, wherein the feature image U2 is input into a third coding model, and the feature image U2 is upsampled by the third coding model and then feature-fused with the feature image S3 to obtain a feature image O1;

[0013] A first-level downsampling and fusion step, convolving the feature image O1 through a second convolutional layer to obtain a feature image D1, inputting the feature image D1 into a second coding model, downsampling the feature image D1 through the second coding model, and performing feature fusion with the feature image U2 to obtain a feature image O2;

[0014] In the secondary downsampling and fusion step, the feature image O2 is convolved through the second convolutional layer to obtain the feature image D2, and the feature image D2 is input into the first coding model. The feature image D2 is downsampled through the first coding model and feature fused with the feature image U1 to obtain the feature image O3.

[0015] The embodiment of the present invention helps to understand how the model processes and fuses features by fusing feature images of different scales. Multiple fusions through Hadamard products can provide better performance and more effective fusion, and can prevent information loss of target features during extraction and fusion. Multiple fusions can help capture richer information.

[0016] In some embodiments of the present invention, the coding model SSEF includes: a built-in Hadamard model, a first convolution branch and a second convolution branch connected to the output of the built-in Hadamard model, the first convolution branch includes a third convolution layer, the second convolution branch includes a fourth convolution layer and a network block structure RepBlock, and the first convolution branch and the second convolution branch process the feature image and then add the features and output them;

[0017] Among them, after the feature image is input into the encoding model SSEF, the feature image is upsampled or downsampled by the built-in Hadamard model and then feature fusion is performed, and the feature image after feature fusion is processed in parallel by the first convolution branch and the second convolution branch in the encoding model SSEF, wherein the first convolution branch encodes the channel semantics through the 1×1 convolution of the third convolution layer, and the second convolution branch encodes the channel semantics through the 1×1 convolution of the fourth convolution layer, and then the channel dimension and the spatial dimension are unified by superposition of multiple layers of 3×3 convolutions of the network block structure RepBlock, and the features are added;

[0018] The Hadamard model HMF is used to fuse the feature image O1, the feature image O2, and the feature image O3 to obtain a third fusion result, including:

[0019] The input feature image is unified in spatial dimension, the channel semantics is encoded through the channel coding model CSC, and the feature fusion is completed by performing point-by-point multiplication on the feature image after spatial dimension unification through the Hadamard model;

[0020] The channel coding model CSC includes: a first depth-separable convolutional layer, a second depth-separable convolutional layer, a connection layer, and a reorganization layer connected in sequence.

[0021] The scale of the feature image input to the Hadamard model HMF is unified through the pooling operation, and the unified feature image is divided into two equal channels to generate the first equal channel and the second equal channel. The first equal channel is convolved twice by the first depthwise separable convolution layer and the second depthwise separable convolution layer in sequence, and then spliced ​​with the second equal channel through the connection layer. The spliced ​​feature image is split into tensor blocks by the recombination layer splitting method and then output.

[0022] The embodiment of the present invention can ensure the correct encoding of feature information by refining the semantic information of the unified features; through the parallel processing of the two convolution branches in the encoding model SSEF, feature images of different scales can be considered at the same time, which helps the model to capture rich features at different levels, and extracts and regularizes features from the channel and spatial dimensions respectively, which helps to enrich the final feature representation and improve the performance of the model.

[0023] In some embodiments of the present invention, the method further comprises:

[0024] Output decoding step: input the fusion result into the prediction model, select the pre-selected box of the fusion result through the mapping module, predict the prediction model by combining the denoising and denoising of the real box, and generate the target space and target category information.

[0025] The embodiment of the present invention can complete the detection task of underwater targets based on mapping and prediction of the image fused with the above features. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0027] Figure 1 A flowchart of a multi-scale feature fusion method based on Hadamard product provided in an embodiment of the present invention;

[0028] Figure 2 An application flow chart of a multi-scale feature fusion method based on Hadamard product provided in an embodiment of the present invention;

[0029] Figure 3 A flow chart of a feature fusion step S3 provided in an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of the structure of a coding model SSEF provided in an embodiment of the present invention;

[0031] Figure 5A schematic diagram of the structure of a channel coding model CSC provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0035] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0036] As attached Figure 1 As shown, the present invention provides a multi-scale feature fusion method based on Hadamard product, comprising the following steps:

[0037] Image processing step S1: acquiring an underwater image, and scaling the underwater image to obtain a feature image; wherein the underwater image is scaled to a preset reference size by an adaptive image scaling algorithm, and when the underwater image is smaller than the reference size, the underwater image is scaled to the reference size by an interpolation method; optionally, the reference size is 640x640 pixels;

[0038] Feature extraction step S2: extracting features from the feature image to obtain a multi-scale feature image, the multi-scale feature image comprising: feature image S3, feature image S4 and feature image S5; optionally, feature image S3 is an 8×8 scale feature image, feature image S4 is a 16×16 scale feature image, and feature image S5 is a 32×32 scale feature image; optionally, through the convolution module of the network Backbone, Resnet50 is used to perform feature extraction through a series of convolution residual stacking;

[0039] Feature fusion step S3: encoding the feature image S5 through a multi-head attention mechanism to obtain a feature image F5, inputting the feature image F5 into an FPN coding structure, performing two-stage upsampling on the feature image U1 obtained by convolution processing and fusing the feature image S4 and the feature image S3 step by step, obtaining the feature image U2 and the feature image O1 in turn, performing two-stage downsampling on the feature image O1 after convolution processing and fusing the feature image U2 and the feature image U1 step by step, obtaining the feature image O2 and the feature image O3 in turn, fusing the feature image O1, the feature image O2 and the feature image O3 to obtain the fusion result; optionally, encoding the feature image S5 through a multi-head attention mechanism and then performing weighted processing to follow the processing principle of the RT-DETR real-time target detection model;

[0040] Combined with Figure 2 As shown, the FPN coding structure includes: a three-level coding model SSEF and a Hadamard model HMF, adjacent coding models SSEF are connected through a first convolutional layer and a second convolutional layer, the output of each level of coding model SSEF is connected to the Hadamard model HMF, and the three-level coding model SSEF includes a first coding model, a second coding model, and a third coding model;

[0041] Among them, combined with Figure 3 As shown, the feature fusion step S3 includes:

[0042] First-level upsampling and fusion step S31: convolve the feature image F5 through the first convolution layer to obtain a feature image U1, and input the feature image U1 into the second coding model. The feature image U1 is upsampled by the second coding model, and feature fused with the feature image S4, and then convolved through the first convolution layer to obtain a feature image U2;

[0043] Secondary upsampling and fusion step S32: inputting the feature image U2 into the third coding model, upsampling the feature image U2 through the third coding model, and performing feature fusion with the feature image S3 to obtain the feature image O1;

[0044] First-level downsampling fusion step S33: convolve the feature image O1 through the second convolutional layer to obtain the feature image D1, input the feature image D1 into the second coding model, downsample the feature image D1 through the second coding model, and perform feature fusion with the feature image U2 to obtain the feature image O2;

[0045] Secondary downsampling and fusion step S34: The feature image O2 is convolved through the second convolutional layer to obtain the feature image D2, and the feature image D2 is input into the first coding model. The feature image D2 is downsampled through the first coding model and feature fused with the feature image U1 to obtain the feature image O3.

[0046] Through the above method, the fusion of feature images of different scales helps to understand how the model processes and fuses features. Multiple fusions through Hadamard products can provide better performance and more effective fusion, which can prevent the loss of information of target features when extracting and fusion. Multiple fusions can help capture richer information.

[0047] Combined with Figure 4 As shown, the coding model SSEF includes: a built-in Hadamard model, a first convolution branch and a second convolution branch connected to the output of the built-in Hadamard model, the first convolution branch includes a third convolution layer, the second convolution branch includes a fourth convolution layer and a network block structure RepBlock, and the first convolution branch and the second convolution branch process the feature image and then add the features and output them;

[0048] After the feature image is input into the encoding model SSEF, the feature image is upsampled or downsampled by the built-in Hadamard model and then feature fused. The first convolution branch and the second convolution branch in the encoding model SSEF process the fused feature image in parallel, wherein the first convolution branch encodes the channel semantics through the 1×1 convolution of the third convolution layer, and the second convolution branch encodes the channel semantics through the 1×1 convolution of the fourth convolution layer. The channel dimension and the spatial dimension are unified by superposition of multiple layers of 3×3 convolutions of the network block structure RepBlock, and the features are added.

[0049] By refining the semantic information of the unified features, the correct encoding of the feature information can be guaranteed; through the parallel processing of the two convolution branches in the encoding model SSEF, feature images of different scales can be considered at the same time, which helps the model capture rich features at different levels, extract and regularize features from the channel and spatial dimensions respectively, which helps to enrich the final feature representation and improve the performance of the model.

[0050] The Hadamard model HMF is used to fuse the feature image O1, the feature image O2, and the feature image O3 to obtain a third fusion result, including:

[0051] The input feature image is unified in spatial dimension, the channel semantics is encoded through the channel coding model CSC, and the feature fusion is completed by performing point-by-point multiplication on the feature image after spatial dimension unification through the Hadamard model;

[0052] Combined with Figure 5 As shown, the channel coding model CSC includes: a first depth-separable convolutional layer, a second depth-separable convolutional layer, a connection layer, and a reorganization layer connected in sequence.

[0053] The scale of the feature image input to the Hadamard model HMF is unified through a pooling operation, and the unified feature image is divided into two equal channels to generate a first equal channel and a second equal channel. After the first equal channel is convolved twice through a first depth-separable convolution layer and a second depth-separable convolution layer, it is spliced ​​with the second equal channel through a connection layer, and the spliced ​​feature image is split into tensor blocks through a reorganization layer splitting method and then output; preferably, the depth-separable convolution applied in the channel coding model CSC replaces the traditional 3×3 convolution in the Hadamard model HMF to alleviate the problem of an increase in the number of model parameters and thus reducing the detection efficiency of the model;

[0054] In the above process, when the feature image is fused by each level coding model SSEF, the channel coding model CSC is used to perform channel coding on the unified feature image, and finally the Hadamard model HMF is used to multiply the features of each dimension point by point to complete the feature fusion. The calculation formula of the coding model SSEF is:

[0055]

[0056]

[0057] in, i is the number of fused feature images; For the i Characteristics of the level; It is a feature image with non-uniform spatial size; It is the feature image after the spatial size is unified; It is the feature image processed by the channel coding model CSC; is the input of the encoding model SSEF, is a unified parameter for the space size, which is used as input Unify the size of spatial dimensions, For the pooling operation, p It is the symbolic representative letter of Pooling; M is the maximum number of fused feature images; is the fusion result of feature fusion.

[0058] Through the above method, the depth-wise separable convolution accumulates the burden of a small amount of parameters, which can ensure the correct encoding of feature information, thereby refining the semantic information of the unified features.

[0059] In order to be able to complete the underwater target detection task according to the mapping and prediction of the above-mentioned feature fused image, the feature fusion method also includes: an output decoding step, inputting the fusion result into the prediction model, selecting a pre-selected box for the fusion result through a mapping module, predicting the prediction model by combining denoising and denoising of the real box, and generating target space and target type information; optionally, the mapping module is used to select a fixed number of features from the fusion result output by the FPN encoding structure as constructor object queries, which are mapped to confidence and bounding box by the prediction head after passing through the decoding structure, and the original real label of the data set is subjected to the process of adding noise and denoising in the decoding structure, and the added and subtracted noise tensors are used as a prediction hint to correct the accuracy of the mapping module prediction and reduce the training time cost.

[0060] It should be noted that the above is a reference method of the multi-scale feature fusion method based on the Hadamard product, and the present invention is not limited thereto.

[0061] The embodiments of the present invention realize the fusion of feature images of different scales, which helps to understand how the model processes and fuses features. Multiple fusions through Hadamard products can provide better performance and more effective fusion, and can prevent the loss of information of target features when they are extracted and fused. Multiple fusions can help capture richer information, and solve the technical problems that the prior art only performs feature fusion of spatial and channel dimensions on feature images of different scales once, which easily affects the fusion efficiency and has the problem of being unable to capture rich and effective target features, resulting in the loss of information of target features when they are extracted and fused.

[0062] Finally, it should be noted that: the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0063] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present invention, which should be included in the scope of the technical solution for protection of the present invention.

Claims

1. A multi-scale feature fusion method based on Hadamard product, characterized in that: The steps include: Image processing step: acquiring an underwater image, and scaling the underwater image to obtain a feature image; Feature extraction step: extracting features from the feature image to obtain a multi-scale feature image, wherein the multi-scale feature image includes: feature image S3, feature image S4 and feature image S5; Feature fusion step: the feature image S5 is encoded through a multi-head attention mechanism to obtain a feature image F5, the feature image F5 is input into an FPN encoding structure, the feature image U1 obtained by convolution processing is upsampled in two stages and the feature image S4 and the feature image S3 are fused step by step to obtain the feature image U2 and the feature image O1 in turn, the feature image O1 is downsampled in two stages after convolution processing and the feature image U2 and the feature image U1 are fused step by step to obtain the feature image O2 and the feature image O3 in turn, and the feature image O1, the feature image O2 and the feature image O3 are fused to obtain the fusion result.

2. The multi-scale feature fusion method based on Hadamard product according to claim 1, characterized in that: The FPN coding structure includes: a three-level coding model SSEF and a Hadamard model HMF, adjacent coding models SSEF are connected through a first convolutional layer and a second convolutional layer, the output of each level of coding model SSEF is connected to the Hadamard model HMF, and the three-level coding model SSEF includes a first coding model, a second coding model, and a third coding model.

3. The multi-scale feature fusion method based on Hadamard product according to claim 2, characterized in that: The feature fusion step includes: The first-level upsampling and fusion step is to convolve the feature image F5 through the first convolution layer to obtain the feature image U1, and the feature image U1 is input into the second coding model. The feature image U1 is upsampled by the second coding model, and the feature image is fused with the feature image S4, and then convolved through the first convolution layer to obtain the feature image U2; In the second level upsampling and fusion step, the feature image U2 is input into the third coding model, and the feature image U2 is upsampled by the third coding model to be fused with the feature image S3 to obtain the feature image O1.

4. The multi-scale feature fusion method based on Hadamard product according to claim 3, characterized in that: The feature fusion step is specifically as follows: A first-level downsampling and fusion step, convolving the feature image O1 through a second convolutional layer to obtain a feature image D1, inputting the feature image D1 into a second coding model, downsampling the feature image D1 through the second coding model, and performing feature fusion with the feature image U2 to obtain a feature image O2; In the secondary downsampling and fusion step, the feature image O2 is convolved through the second convolutional layer to obtain the feature image D2, and the feature image D2 is input into the first coding model. The feature image D2 is downsampled through the first coding model and feature fused with the feature image U1 to obtain the feature image O3.

5. The multi-scale feature fusion method based on Hadamard product according to claim 4, characterized in that: The coding model SSEF includes: a built-in Hadamard model, a first convolution branch and a second convolution branch connected to the output of the built-in Hadamard model, the first convolution branch includes a third convolution layer, the second convolution branch includes a fourth convolution layer and a network block structure RepBlock, and the first convolution branch and the second convolution branch process the feature image and then add the features and output them.

6. The multi-scale feature fusion method based on Hadamard product according to claim 5, characterized in that: After the feature image is input into the encoding model SSEF, the feature image is upsampled or downsampled through the built-in Hadamard model and then feature fusion is performed. The first convolution branch and the second convolution branch in the encoding model SSEF process the feature fused feature image in parallel, wherein the first convolution branch encodes the channel semantics through the 1×1 convolution of the third convolution layer, and the second convolution branch encodes the channel semantics through the 1×1 convolution of the fourth convolution layer. After that, the channel dimension and the spatial dimension are unified through the superposition of multiple layers of 3×3 convolutions of the network block structure RepBlock, and the features are added.

7. The multi-scale feature fusion method based on Hadamard product according to claim 2, characterized in that: The Hadamard model HMF is used to fuse the feature image O1, the feature image O2, and the feature image O3 to obtain a third fusion result, including: The spatial dimension of the input feature image is unified, the channel semantics is encoded through the channel coding model CSC, and then the feature fusion is completed by performing point-by-point multiplication of the feature image after the spatial dimension is unified through the Hadamard model.

8. The multi-scale feature fusion method based on Hadamard product according to claim 7, characterized in that: The channel coding model CSC comprises: a first depth-separable convolutional layer, a second depth-separable convolutional layer, a connection layer, and a recombinant layer connected in sequence. The scale of the feature image input to the Hadamard model HMF is unified through the pooling operation, and the unified feature image is divided into two equal channels to generate the first equal channel and the second equal channel. The first equal channel is convolved twice by the first depthwise separable convolution layer and the second depthwise separable convolution layer in sequence, and then spliced ​​with the second equal channel through the connection layer. The spliced ​​feature image is split into tensor blocks by the recombination layer splitting method and then output.

9. The multi-scale feature fusion method based on Hadamard product according to any one of claims 1 to 8, characterized in that: Also includes: Output decoding step: input the fusion result into the prediction model, select the pre-selected box of the fusion result through the mapping module, predict the prediction model by combining the denoising and de-noising of the real box, and generate the target space and target category information.

10. The multi-scale feature fusion method based on Hadamard product according to any one of claims 1 to 8, characterized in that: The image processing step of scaling the underwater image is specifically as follows: The underwater image is scaled to a preset reference size by an adaptive picture scaling algorithm. When the underwater image is smaller than the reference size, the underwater image is scaled to the reference size by an interpolation method.

Images