Multi-scale feature fusion method and device and computer readable storage medium

Abstract

The invention discloses a multi-scale feature fusion method and device and a computer readable storage medium, and the method comprises the steps: extracting a first feature map corresponding to a plurality of feature extraction stages based on a to-be-processed image, the feature extraction stages comprising a first feature extraction stage, a second feature extraction stage and a third feature extraction stage; performing up-sampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage; fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage; and performing down-sampling on the basis of the third feature map and the first feature map corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage, thereby improving the accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a multi-scale feature fusion method, device and computer-readable storage medium. Background Art

[0002] In the task of target detection, multi-scale feature fusion plays a very critical role. Since deep feature maps have low resolution and large receptive fields, they contain high-level semantic information and are suitable for target classification. Shallow feature maps have high resolution and small receptive fields, and contain texture detail information of the target, which is suitable for target positioning. Multi-scale feature fusion is used to fuse feature images of different scales together. The generated features contain not only target semantic information such as the target category, but also the target positioning information, thereby improving the accuracy of image detection.

[0003] In related technologies, FPN networks are used for target detection, deep features are fused with shallow features, and targets are detected in feature maps of different scales. However, the spatial information in shallow features is continuously lost during the layer-by-layer convolution process, resulting in a lack of spatial position information in deep feature maps, which reduces the accuracy of target detection. Summary of the invention

[0004] The embodiments of the present application aim to improve the accuracy of target detection by providing a multi-scale feature fusion method, device and computer-readable storage medium.

[0005] The embodiment of the present application provides a multi-scale feature fusion method, the multi-scale feature fusion method comprising:

[0006] Extracting first feature graphs corresponding to a plurality of feature extraction stages based on the image to be processed, wherein the feature extraction stages include a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage;

[0007] Perform upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage;

[0008] Fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage;

[0009] Downsampling is performed based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage.

[0010] Optionally, the step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage includes:

[0011] After upsampling the first feature map corresponding to the first feature extraction stage by bilinear interpolation, the first feature map is fused with the first feature map corresponding to the second feature extraction stage to obtain a second feature map corresponding to the second feature extraction stage;

[0012] After upsampling the second feature map corresponding to the second feature extraction stage by bilinear interpolation, it is fused with the first feature map corresponding to the third feature extraction stage to obtain the second feature map corresponding to the third feature extraction stage.

[0013] Optionally, the step of performing downsampling based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage includes:

[0014] After downsampling the first feature map corresponding to the third feature extraction stage by a maximum pooling method, the third feature map corresponding to the second feature extraction stage and the first feature map are fused to obtain a fused feature map corresponding to the second feature extraction stage;

[0015] After downsampling the fused feature map corresponding to the second feature extraction stage by means of maximum pooling, the fused feature map is fused with the first feature map corresponding to the first feature extraction stage to obtain the fused feature map corresponding to the first feature extraction stage.

[0016] Optionally, the step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage includes:

[0017] During the upsampling process, the first feature map passes through the attention mechanism module in the upsampling path, and performs convolution operation through the convolution layer of the attention mechanism module in the upsampling path to obtain the second feature map corresponding to each feature extraction stage, wherein the upsampling path includes a first upsampling path from the first feature extraction stage to the second feature extraction stage, and a second upsampling path from the second feature extraction stage to the third feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first upsampling path and the second upsampling path are different.

[0018] Optionally, the step of performing downsampling based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage includes:

[0019] During the downsampling process, the third feature map and the first feature map corresponding to each feature extraction stage pass through the attention mechanism module in the corresponding downsampling path, and perform convolution operation through the convolution layer of the attention mechanism module in the downsampling path to obtain a fused feature map corresponding to each feature extraction stage, wherein the downsampling path includes a first downsampling path from the third feature extraction stage to the second feature extraction stage, and a second downsampling path from the second feature extraction stage to the first feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first downsampling path and the second downsampling path are different.

[0020] Optionally, before the step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage, the step further includes:

[0021] Performing a convolution operation on the first feature map corresponding to each feature extraction stage based on the first convolution unit to obtain a first sub-feature map corresponding to each feature extraction stage;

[0022] The step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage comprises:

[0023] Upsampling is performed according to the first sub-feature map corresponding to each feature extraction stage to obtain a second feature map corresponding to each feature extraction stage.

[0024] Optionally, the step of fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage includes:

[0025] Fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a second sub-feature map;

[0026] Based on the second convolution unit, a convolution operation is performed on the second sub-feature map to obtain a third feature map corresponding to the second feature extraction stage.

[0027] Optionally, the step of extracting first feature maps corresponding to multiple feature extraction stages based on the image to be processed includes:

[0028] The image to be processed is subjected to feature extraction through multiple feature extraction stages of the RepVGG backbone network to obtain first feature maps corresponding to the multiple feature extraction stages.

[0029] In addition, to achieve the above-mentioned purpose, the present invention also provides a multi-scale feature fusion device including: a memory, a processor, and a multi-scale feature fusion program stored in the memory and executable on the processor, wherein the multi-scale feature fusion program implements the steps of the above-mentioned multi-scale feature fusion method when executed by the processor.

[0030] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a multi-scale feature fusion program is stored, and when the multi-scale feature fusion program is executed by a processor, the steps of the above-mentioned multi-scale feature fusion method are implemented.

[0031] A technical solution of a multi-scale feature fusion method, device and computer-readable storage medium provided in the embodiments of the present application adds a bottom-up fusion channel from low-level features to high-level features in the feature fusion network. Through downsampling operations, information such as the spatial position of shallow features can be effectively fused into deep feature maps, so that feature maps at different levels can obtain spatial information of targets of different sizes. And a horizontal connection is added to fuse feature maps of the same level. Since the path from the input feature map of the current layer to the fused feature map is very long during the feature fusion process, some features may be lost during the transmission process. Therefore, in the second feature extraction stage, a fast channel is introduced from the input feature map to the fused feature map, so that more feature information of this layer is included, the fusion effect is enhanced, and the accuracy of target detection is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the first embodiment of the multi-scale feature fusion method of the present invention;

[0033] Figure 2 A schematic diagram of the structure of the hardware operating environment involved in the embodiment of the present invention;

[0034] Figure 3 This is the ECA-FPN network structure diagram of the present invention;

[0035] Figure 4 The ECANet network structure of the present invention;

[0036] Figure 5 Schematic diagram of the structure of the RepVGG-block module corresponding to the feature extraction stage of the present invention.

[0037] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. The above-mentioned drawings are only an embodiment diagram, not the entire invention. DETAILED DESCRIPTION

[0038] At present, the FPN network is used for target detection. The core idea of ​​the FPN network algorithm is to use a feature pyramid to extract feature information of different scales in the image, thereby improving the accuracy and efficiency of image segmentation. In traditional convolutional neural networks, each convolution layer will reduce the input image by half, so the resolution of the feature map will be relatively low in the deeper layers of the network, which will cause smaller details to be ignored when performing image segmentation. To solve this problem, the FPN algorithm introduces a feature pyramid structure, which consists of multiple feature maps of different resolutions, each of which corresponds to a different scale of the input image. In this way, the FPN algorithm can simultaneously consider feature information of different scales, thereby improving the accuracy and efficiency of image segmentation. However, there is a problem that the deep features are fused into the low-level features from top to bottom, and the fusion direction is unidirectional, because the shallow features are not fused into the deep features from bottom to top. The spatial information in the shallow features is continuously lost during the layer-by-layer convolution process, resulting in a lack of spatial position information in the deep feature map, which in turn reduces the accuracy of target detection.

[0039] Based on the above problems, the present application proposes a new multi-scale feature fusion method. The technical solution adopted in the present application includes: extracting first feature maps corresponding to multiple feature extraction stages based on the image to be processed, and the feature extraction stages include a first feature extraction stage, a second feature extraction stage and a third feature extraction stage; upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage; fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage; downsampling based on the third feature map and the first feature maps corresponding to each of the feature extraction stages to obtain a fused feature map corresponding to each feature extraction stage.

[0040] This application adds a bottom-up fusion channel from low-level features to high-level features in the feature fusion network. Through downsampling operations, the spatial position and other information of shallow features can be effectively fused into the deep feature map, so that feature maps at different levels can obtain spatial information of targets of different sizes. And a horizontal connection is added to fuse feature maps at the same level. Since the path from the input feature map of the current layer to the fused feature map is very long during the feature fusion process, some features may be lost during the transmission process. Therefore, in the second feature extraction stage, a fast channel is introduced from the input feature, that is, the first feature map to the fused feature map, so as to include more feature information of this layer, enhance the fusion effect, and improve the accuracy of target detection.

[0041] In addition, since there is target information of different sizes in the shallow feature map, large targets, medium targets and small targets are mixed together, resulting in interference between targets of different scales, which increases the difficulty of multi-scale target detection, especially small targets. In order to solve this problem, this application adds an attention mechanism module in the FPN upsampling and downsampling process. ECANet is an attention mechanism network with excellent performance and low parameter count, which is very suitable for embedding in the upsampling and downsampling paths of the feature fusion network. During the feature fusion process, different attention weights are assigned according to the importance of different feature maps. By suppressing interfering features and strengthening the attention of useful features, the importance of different feature maps is automatically adjusted, so as to further extract effective feature information.

[0042] In addition, the currently commonly used feature fusion network has a complex structure, a large number of parameters and a large amount of calculation, which is not conducive to deployment on AR glasses with limited computing resources. The upsampling of FPN is achieved through deconvolution, but deconvolution will increase the number of parameters of the feature fusion network, which is not conducive to the lightweight model. In order to solve this problem, the present application reduces the number of parameters of the model. In the process of top-down and bottom-up bidirectional fusion, upsampling is achieved through bilinear interpolation, and downsampling operations are achieved through the maximum pooling layer, so as to achieve matching between feature maps of different scales and reduce the number of model parameters.

[0043] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0044] like Figure 1As shown, in the first embodiment of the present application, the multi-scale feature fusion method of the present application is applied to a mobile device, and the mobile device can be smart glasses such as VR glasses, MR glasses, etc., unmanned vehicles, drones and other devices. The mobile device has a processor and a memory, and the processor can process the multi-scale feature fusion program stored in the memory to implement the steps of the multi-scale feature fusion method of the present application. In other embodiments, the multi-scale feature fusion method of the present application can also be applied to an intermediate processing device, and the intermediate processing device has a processor and a memory, and the processor can process the multi-scale feature fusion program stored in the memory. The intermediate processing device is connected to the image acquisition device and the smart glasses, and is used to receive the image to be processed acquired by the image acquisition device, perform feature fusion processing on the image to be processed, and send the image after feature fusion processing to the mobile device for display. This method transplants the main processing part of the image processing to the intermediate processing device for processing, reduces the image processing pressure of the mobile device, but at the same time increases the image processing cost. This application takes the mobile device as smart glasses and performs multi-scale feature fusion processing on the smart glasses as an example. The multi-scale feature fusion method of the present application includes the following steps:

[0045] Step S110, extracting first feature maps corresponding to multiple feature extraction stages based on the image to be processed, wherein the feature extraction stages include a first feature extraction stage, a second feature extraction stage and a third feature extraction stage.

[0046] Optionally, when wearing smart glasses, the image acquisition device on the smart glasses acquires images, and the acquired images are used as images to be processed, and feature extraction processing is performed through multiple feature extraction stages of the lightweight multi-scale feature fusion network of the present application.

[0047] Optionally, the image to be processed is subjected to feature extraction through multiple feature extraction stages of the RepVGG backbone network to obtain a first feature map corresponding to multiple feature extraction stages. Among them, the RepVGG backbone network includes multiple feature extraction stages, each feature extraction stage is used to extract feature information of different scales for subsequent feature fusion. The feature extraction stages of the present application include but are not limited to the first feature extraction stage, the second feature extraction stage and the third feature extraction stage, and may also include more feature extraction stages. When the number of feature extraction stages is more, the extracted features are richer. Among them, the depth of the first feature extraction stage, the second feature extraction stage and the third feature extraction stage gradually decreases, that is, the depth of the first feature extraction stage is the highest, the depth of the second feature extraction stage is second, and the depth of the third feature extraction stage is the lowest. The third feature extraction stage is used to extract shallow features, and the first feature extraction stage is used to extract deep features. Among them, the shallow feature receptive field is small, and more attention is paid to detail information; the overlapping area of ​​the receptive field corresponding to each pixel point is small, which ensures that the network can capture more details, and the shallow features have higher resolution; the spatial information is rich. The feature extraction process of the present application is specifically described in the third embodiment.

[0048] Step S120: upsampling is performed based on the first feature map to obtain a second feature map corresponding to each feature extraction stage.

[0049] Optionally, the first feature extraction stage has the highest depth, and upsampling is performed starting from the first feature extraction stage. The first feature map extracted in the first feature extraction stage is upsampled to obtain the second feature map corresponding to each feature extraction stage. After upsampling, the present application integrates the deep features into the shallow features.

[0050] Optionally, refer to Figure 3 , the present application upsamples the first feature map corresponding to the first feature extraction stage to obtain the second feature map corresponding to the second feature extraction stage, and upsamples the second feature map corresponding to the second feature extraction stage to obtain the second feature map corresponding to the third feature extraction stage. Specifically, the first feature map corresponding to the first feature extraction stage is upsampled and fused with the first feature map corresponding to the second feature extraction stage to obtain the second feature map corresponding to the second feature extraction stage; the second feature map corresponding to the second feature extraction stage is upsampled and fused with the first feature map corresponding to the third feature extraction stage to obtain the second feature map corresponding to the third feature extraction stage. In addition, the present application can directly use the first feature map corresponding to the first feature extraction stage as the second feature map corresponding to the first feature extraction stage. Through top-down upsampling, the deep features are fused with the shallow features to obtain the second feature map corresponding to each feature extraction stage.

[0051] Furthermore, the present application performs upsampling by bilinear interpolation to obtain a second feature map corresponding to each feature extraction stage. Specifically, the present application performs upsampling on the first feature map corresponding to the first feature extraction stage by bilinear interpolation, and then fuses it with the first feature map corresponding to the second feature extraction stage to obtain the second feature map corresponding to the second feature extraction stage; the second feature map corresponding to the second feature extraction stage is upsampled by bilinear interpolation, and then fuses it with the first feature map corresponding to the third feature extraction stage to obtain the second feature map corresponding to the third feature extraction stage. Among them, bilinear interpolation is used to improve the resolution of the image. Its principle is to first find the pixel position of the target image pixel on the source image, and then find the four upper, lower, left and right nearest neighbor pixels corresponding to the target image pixel for calculating interpolation, thereby achieving the effect of magnifying the image.

[0052] Step S130, fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage.

[0053] Optionally, the present application fuses the first feature map corresponding to the second feature extraction stage and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage.

[0054] Step S140, downsampling is performed based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage.

[0055] Optionally, the present application also adds a bottom-up fusion channel from low-level features to high-level features in the feature fusion network. Through the downsampling operation, the spatial position and other information of the shallow features can be effectively fused into the deep feature map, so that feature maps at different levels can obtain the spatial information of targets of different sizes.

[0056] Optionally, refer to Figure 3 , because the path from the input feature map of the current layer to the fused feature map is very long during the feature fusion process, some features may be lost during the transmission process. Therefore, a fast channel is introduced from the input feature map to the fused feature map in the second feature extraction stage, so as to include more feature information of this layer, enhance the fusion effect, and improve the accuracy of target detection. Optionally, in addition to introducing a fast channel in the second feature extraction stage, corresponding fast channels can also be introduced in the first feature extraction stage and the third feature extraction stage according to actual needs.

[0057] Optionally, the third feature extraction stage has the lowest depth, and upsampling is performed starting from the third feature extraction stage, and downsampling is performed on the first feature map processed in the third feature extraction stage, so as to obtain a fused feature map corresponding to each feature extraction stage. After upsampling, the spatial information of the shallow features is fused into the deep features. Specifically, the present application downsamples the first feature map corresponding to the third feature extraction stage to obtain a fused feature map corresponding to the second feature extraction stage, and downsamples the fused feature map corresponding to the second feature extraction stage to obtain a fused feature map corresponding to the first feature extraction stage. Specifically, the first feature map corresponding to the third feature extraction stage is downsampled, and the third feature map and the first feature map corresponding to the second feature extraction stage are fused to obtain a fused feature map corresponding to the second feature extraction stage; the fused feature map corresponding to the second feature extraction stage is downsampled, and the fused feature map corresponding to the first feature extraction stage is fused to obtain a fused feature map corresponding to the first feature extraction stage. In addition, the present application can directly use the first feature map corresponding to the third feature extraction stage as the fused feature map corresponding to the third feature extraction stage. Through the bottom-up downsampling operation, the shallow features are fused with the deep features to obtain the fused feature map corresponding to each feature extraction stage.

[0058] Further, the present application performs downsampling by maximum pooling to obtain a fused feature map corresponding to each feature extraction stage. Specifically, after the first feature map corresponding to the third feature extraction stage is downsampled by the maximum pooling method, the third feature map and the first feature map corresponding to the second feature extraction stage are fused to obtain the fused feature map corresponding to the second feature extraction stage; after the fused feature map corresponding to the second feature extraction stage is downsampled by the maximum pooling method, it is fused with the first feature map corresponding to the first feature extraction stage to obtain the fused feature map corresponding to the first feature extraction stage. Among them, maximum pooling is to divide the input matrix into several rectangular areas and output the maximum value for each sub-area. For the maximum pooling operation, only the maximum value in each rectangular area is selected to enter the next layer, and other elements will not enter the next layer. Therefore, the part with the strongest response in the maximum pooling extraction feature map enters the next layer. This method discards a large amount of redundant information in the network, making the network easier to optimize. At the same time, this operation mode often loses some detail information in the feature map, so the maximum pooling retains more texture information of the image.

[0059] The above upsampling operation is achieved through deconvolution, but deconvolution will increase the number of parameters of the feature fusion network, which is not conducive to the lightweight model. In order to reduce the number of model parameters, in the top-down and bottom-up bidirectional fusion process, upsampling operations are achieved through bilinear interpolation, and downsampling operations are achieved through the maximum pooling layer to achieve matching between feature maps of different scales and reduce the number of model parameters.

[0060] In addition, it should be noted that in the feature fusion process of the second feature extraction stage of the present application, the first feature map corresponding to the third feature extraction stage is downsampled, and in addition to being fused with the third feature map corresponding to the second feature extraction stage, it is also fused with the first feature map corresponding to the second feature extraction stage to obtain a fused feature map corresponding to the second feature extraction stage, so that the fused feature map corresponding to the second feature extraction stage contains more feature information of this layer, thereby enhancing the fusion effect and improving the accuracy of target detection.

[0061] In one embodiment, first feature maps corresponding to multiple feature extraction stages are extracted based on the image to be processed; convolution operation is performed on the first feature map corresponding to each feature extraction stage based on the first convolution unit to obtain a first sub-feature map corresponding to each feature extraction stage; upsampling is performed according to the first sub-feature map corresponding to each feature extraction stage to obtain a second feature map corresponding to each feature extraction stage; the first feature map and the second feature map corresponding to the second feature extraction stage are fused to obtain a third feature map corresponding to the second feature extraction stage; downsampling is performed based on the third feature map and the first feature maps corresponding to each of the feature extraction stages to obtain a fused feature map corresponding to each feature extraction stage. Among them, the first convolution unit is preferably a 1*1 convolution. By adding a 1*1 convolution, the present application can reduce the number of input channels, reduce the convolution kernel parameters and the computational complexity. The first sub-feature map corresponding to each feature extraction stage is equivalent to Figure 3 P2-P5 in.

[0062] In one embodiment, first feature maps corresponding to multiple feature extraction stages are extracted based on the image to be processed, and the feature extraction stages include a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage; upsampling is performed based on the first feature map to obtain a second feature map corresponding to each feature extraction stage; the first feature map and the second feature map corresponding to the second feature extraction stage are fused to obtain a second sub-feature map; a convolution operation is performed on the second sub-feature map based on a second convolution unit to obtain a third feature map corresponding to the second feature extraction stage; downsampling is performed based on the third feature map and the first feature maps corresponding to each of the feature extraction stages to obtain a fused feature map corresponding to each feature extraction stage.

[0063] In one embodiment, a convolution operation is performed on the first feature map corresponding to each feature extraction stage based on the first convolution unit to obtain the first sub-feature map corresponding to each feature extraction stage; upsampling is performed according to the first sub-feature map corresponding to each feature extraction stage to obtain the second feature map corresponding to each feature extraction stage; the first feature map and the second feature map corresponding to the second feature extraction stage are fused to obtain the second sub-feature map; a convolution operation is performed on the second sub-feature map based on the second convolution unit to obtain the third feature map corresponding to the second feature extraction stage; downsampling is performed based on the third feature map and the first feature map corresponding to each of the feature extraction stages to obtain the fused feature map corresponding to each feature extraction stage. Wherein, the second feature map is the feature map of the second feature extraction stage obtained after upsampling. After performing 3*3 convolution on the second feature map corresponding to the second feature extraction stage, the second sub-feature map corresponding to the second feature extraction stage is obtained. Wherein, the second convolution unit is preferably a 3*3 convolution. By adding 3*3 convolution, the present application can represent more complex functions and extract small features because the nonlinearity of 3*3 convolution is higher.

[0064] by Figure 3 Taking this as an example, the technical solution of this application will be specifically described below:

[0065] This application proposes an ECA-FPN multi-scale feature fusion network, which mainly includes three parts: a top-down upsampling feature fusion channel, a bottom-up downsampling feature fusion channel, and a feature fusion channel that horizontally connects the first sub-feature map and the fusion feature map. Among them, the attention mechanism module ECAnet is embedded in the top-down upsampling feature fusion channel and the bottom-up downsampling feature fusion channel.

[0066] (1) The top-down feature fusion channel uses the output of the C2, C3, C4, and C5 layers of the backbone network RepVGG as the first feature map of ECA-FPN. The first feature map is first adjusted by 1×1 convolution to output the first sub-feature maps P2, P3, P4, and P5. The number of channels from P2 to P5 remains consistent at 256, and the feature map sizes are 160*160, 80*80, 40*40, and 20*20, respectively, with strides of 4, 8, 16, and 32, respectively. Bilinear interpolation upsampling is then used to enlarge the scale of the feature map of the previous layer to the same as that of the next layer. The resolution and number of channels of the two layers of feature maps are consistent, so the feature maps can be fused by element-by-element addition. Then, the fused new feature maps are Figure 1 The above operation is performed until the first sub-feature map P2 of the lowest layer is fused. The top-down feature fusion process is shown in formulas (1) and (2):

[0067]

[0068]

[0069] Where P2, P3, P4, and P5 are the first sub-feature maps with the same number of channels after the input feature map is processed by 1×1 convolution. Where Up(·) represents the bilinear interpolation upsampling operation. 2 up , P 3 up , P 4 up It is the second feature map after fusion.

[0070] (2) Bottom-up feature fusion channel: the second feature map generated after top-down fusion is P 2 up , P 3 up , P 4 up First, bilinear interpolation downsampling is used to reduce the size of the next layer of feature maps to the same size as the previous layer. Since the resolution and number of channels of the two layers of feature maps are the same, the feature maps are directly fused by element-by-element addition. The bottom-up feature fusion process is shown in formula (3):

[0071]

[0072] Here, Down(·) represents the bilinear interpolation downsampling operation, which uses bilinear interpolation to halve the resolution of the feature map so that it is consistent with the resolution of the feature map of the previous layer. Figure 1 The top-down channels and bottom-up channels in are asymmetric. The P5 layer undergoes an upsampling operation and the P2 layer undergoes a downsampling operation. Because the input nodes of the P5 and P2 layers have only one fusion process, the impact on the fusion features is small. Therefore, the above input nodes are deleted to reduce the amount of model calculation.

[0073] (3) The feature fusion channel of the horizontal connection adds a horizontal connection to fuse the feature maps of the same level. Since the path from the first sub-feature map of the current layer to the fused feature map is very long during the feature fusion process, some features may be lost during the transmission process. Therefore, the introduction of the horizontal connection can provide a fast channel from the first sub-feature map to the fused feature map, thereby including more feature information of the current layer and enhancing the fusion effect. The feature fusion channel process of the horizontal connection is expressed as formula (4):

[0074]

[0075] Where P 4 down ,P 3 downIt is the intermediate feature map output by the bottom-up fusion channel, which strengthens the feature information of this layer through lateral connections. Finally, the feature map output after top-down, bottom-up and lateral connections not only strengthens the original feature information of this layer, but also integrates the feature information of other layers, including the target semantic information and target spatial information, and improves the multi-scale target detection effect.

[0076] According to the above technical solution, this embodiment adds a bottom-up fusion channel from low-level features to high-level features in the feature fusion network. Through downsampling operations, the spatial position and other information of shallow features can be effectively fused into the deep feature map, so that feature maps of different levels can obtain spatial information of targets of different sizes. And a horizontal connection is added to fuse feature maps of the same level. Since the path from the input feature map of the current layer to the fused feature map is very long during the feature fusion process, some features may be lost during the transmission process. Therefore, in the second feature extraction stage, a fast channel is introduced from the input feature, that is, the first feature map to the fused feature map, so as to include more feature information of this layer, enhance the fusion effect, and improve the accuracy of target detection.

[0077] Second embodiment

[0078] Based on the first embodiment, in a second embodiment of the present application, the multi-scale feature fusion method of the present application further includes the following steps:

[0079] Step S110, extracting first feature maps corresponding to multiple feature extraction stages based on the image to be processed, wherein the feature extraction stages include a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage;

[0080] Optionally, step S120 further includes the following steps:

[0081] Step S121, during the upsampling process, the first feature map passes through the attention mechanism module in the upsampling path, and performs convolution operation through the convolution layer of the attention mechanism module in the upsampling path to obtain a second feature map corresponding to each feature extraction stage, wherein the upsampling path includes a first upsampling path from the first feature extraction stage to the second feature extraction stage, and a second upsampling path from the second feature extraction stage to the third feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first upsampling path and the second upsampling path are different;

[0082] Step S130, fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage;

[0083] Optionally, step S140 further includes the following steps:

[0084] Step S241, during the downsampling process, the third feature map and the first feature map corresponding to each feature extraction stage pass through the attention mechanism module in the corresponding downsampling path, and perform convolution operation through the convolution layer of the attention mechanism module in the downsampling path to obtain a fused feature map corresponding to each feature extraction stage, wherein the downsampling path includes a first downsampling path from the third feature extraction stage to the second feature extraction stage, and a second downsampling path from the second feature extraction stage to the first feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first downsampling path and the second downsampling path are different.

[0085] Reference Figure 3 , this application adds attention mechanism modules in both FPN upsampling and downsampling processes. ECANet is an attention mechanism network with excellent performance and low parameter count, which is very suitable for embedding into the upsampling and downsampling paths of feature fusion networks. During the feature fusion process, different attention weights are assigned according to the importance of different feature maps. By suppressing interfering features and strengthening the attention of useful features, the importance of different feature maps is automatically adjusted, thereby further extracting effective feature information.

[0086] The network structure of the attention mechanism module is as follows Figure 4 shown.

[0087] The network structure of the attention mechanism module includes a one-dimensional convolution layer, the convolution kernel size of the convolution layer is k, and after the one-dimensional convolution operation, it is fused with the input feature map to generate the final 1×1×C feature map. Among them, the input feature map can be the third feature map, the first feature map corresponding to each of the feature extraction stages, etc.

[0088] In this embodiment, according to the above technical solution, since there is target information of different sizes in the shallow feature map, large targets, medium targets and small targets are mixed together, resulting in interference between targets of different scales, which increases the difficulty of multi-scale target detection, especially small targets. In order to solve this problem, the present application adds an attention mechanism module in the FPN upsampling and downsampling process. ECANet is an attention mechanism network with excellent performance and low parameter count, which is very suitable for embedding in the upsampling and downsampling paths of the feature fusion network. In the feature fusion process, different attention weights are assigned according to the importance of different feature maps. By suppressing interfering features, strengthening the attention of useful features, and automatically adjusting the importance of different feature maps, effective feature information can be further extracted.

[0089] Third embodiment

[0090] Based on the first embodiment, in the third embodiment of the present application, the specific implementation of step S110 of the present application is as follows:

[0091] Optionally, the image to be processed is input into each feature extraction stage of the RepVGG backbone network for corresponding feature extraction to obtain the first feature map corresponding to each feature extraction stage. After obtaining the first feature map corresponding to each feature extraction stage, the feature map corresponding to each feature extraction stage is input into the feature fusion network for feature fusion to obtain fused features. The purpose of feature fusion is to merge the features extracted from the image into a feature that has better discriminative power than the input feature. This application performs target detection based on the fused feature, which is more conducive to the judgment of target detection.

[0092] Optionally, the present application may use the RepVGG-A0 lightweight network in the RepVGG series to perform feature extraction at different feature extraction stages on the processed image. The RepVGG backbone network is divided into a training stage and an inference stage. The training stage and the inference stage use different network architectures. The training stage focuses more on accuracy, while the inference stage focuses more on speed. In the training stage, the present application adds a Conv1x1 convolution branch to the RepVGG backbone network, which forms an asymmetric convolution structure with the original Conv3x3 convolution branch to enhance the feature extraction capability; at the same time, the residual branch is introduced to make the model easier to converge and solve the gradient disappearance problem. In the inference stage, all network layers are converted to Conv3*3 through the Op fusion strategy, which reduces the number of parameters and calculations, improves the detection speed and facilitates deployment. RepVGG-A0 has a simple structure, is easy to design and implement, and has high efficiency in terms of model size and running speed. Its model size is only 5.5MB, and it can run quickly in resource-limited scenarios such as smart glasses.

[0093] Reference Figure 5 In each feature extraction stage, the RepVGG backbone network adopts a multi-branch structure in the training stage and a single-branch structure in the inference stage. The multi-branch structure in the training stage can be converted into a single-branch structure of a 3×3 convolution unit and ReLU combination in the inference stage with the help of structural reparameterization technology to improve the inference speed of the model. The multi-branch structure includes a first convolution branch, a second convolution branch, and an identity residual branch. Among them, the first convolution branch is a Conv3x3 convolution branch, and the second convolution branch is a Conv1x1 convolution branch.

[0094] Optionally, for the deep feature extraction stage, such as the first feature extraction stage. The present application may also embed a multi-scale attention mechanism module in the residual branch, and the attention mechanism module adopts the MSECANet network. The present application improves the lightweight attention mechanism ECANet, proposes a multi-scale attention mechanism MSECANet, and adds it to the deep feature extraction stage, automatically adjusting the weights between channels to highlight key features, and solving the problem of insufficient feature information extraction of lightweight networks.

[0095] According to the above technical solution, this embodiment adopts a lightweight RepVGG backbone network to replace the backbone network of the FCOS anchor-free target detection algorithm, reduces the number of model parameters and calculations to achieve real-time target detection, and adds an attention mechanism in the deep feature extraction stage to improve detection accuracy. While maintaining the detection accuracy, the model can run efficiently on mobile devices.

[0096] The embodiments of the present invention provide embodiments of the multi-scale feature fusion method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown here.

[0097] like Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the hardware operating environment of the multi-scale feature fusion device involved in the embodiment of the present invention. The multi-scale feature fusion device may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen, and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory, such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0098] Those skilled in the art will understand that Figure 2 The multi-scale feature fusion device structure shown in the figure does not constitute a limitation on the multi-scale feature fusion device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0099] like Figure 2As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a multi-scale feature fusion program. The operating system is a program that manages and controls the hardware and software resources of the multi-scale feature fusion device, the operation of the multi-scale feature fusion program and other software or programs.

[0100] exist Figure 2 In the multi-scale feature fusion device shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used for the background server and communicates data with the background server; the processor 1001 can be used to call the multi-scale feature fusion program stored in the memory 1005.

[0101] In this embodiment, the multi-scale feature fusion device includes: a memory 1005, a processor 1001, and a multi-scale feature fusion program stored in the memory and executable on the processor, wherein:

[0102] When the processor 1001 calls the multi-scale feature fusion program stored in the memory 1005, the following operations are performed:

[0103] Extracting first feature graphs corresponding to a plurality of feature extraction stages based on the image to be processed, wherein the feature extraction stages include a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage;

[0104] Perform upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage;

[0105] Fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage;

[0106] Downsampling is performed based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage.

[0107] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, which stores a multi-scale feature fusion program. When the multi-scale feature fusion program is executed by a processor, it implements the various steps of the multi-scale feature fusion method described above and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0108] Since the storage medium provided in the embodiments of the present application is the storage medium used to implement the method of the embodiments of the present application, based on the method introduced in the embodiments of the present application, the person skilled in the art can understand the specific structure and deformation of the storage medium, so it is not repeated here. All storage media used in the methods of the embodiments of the present application belong to the scope of protection of this application.

[0109] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0110] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, TV, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0112] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multi-scale feature fusion method, characterized in that: The multi-scale feature fusion method comprises: Extracting first feature graphs corresponding to a plurality of feature extraction stages based on the image to be processed, wherein the feature extraction stages include a first feature extraction stage, a second feature extraction stage, and a third feature extraction stage; Perform upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage; Fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a third feature map corresponding to the second feature extraction stage; Downsampling is performed based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage.

2. The multi-scale feature fusion method according to claim 1, characterized in that: The step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage comprises: After upsampling the first feature map corresponding to the first feature extraction stage by bilinear interpolation, the first feature map is fused with the first feature map corresponding to the second feature extraction stage to obtain a second feature map corresponding to the second feature extraction stage; After upsampling the second feature map corresponding to the second feature extraction stage by bilinear interpolation, it is fused with the first feature map corresponding to the third feature extraction stage to obtain the second feature map corresponding to the third feature extraction stage.

3. The multi-scale feature fusion method according to claim 1 or 2, characterized in that: The step of performing downsampling based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage comprises: After downsampling the first feature map corresponding to the third feature extraction stage by means of maximum pooling, the third feature map corresponding to the second feature extraction stage and the first feature map are fused to obtain a fused feature map corresponding to the second feature extraction stage; After downsampling the fused feature map corresponding to the second feature extraction stage by means of maximum pooling, the fused feature map is fused with the first feature map corresponding to the first feature extraction stage to obtain the fused feature map corresponding to the first feature extraction stage.

4. The multi-scale feature fusion method according to claim 1, characterized in that: The step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage comprises: During the upsampling process, the first feature map passes through the attention mechanism module in the upsampling path, and performs convolution operation through the convolution layer of the attention mechanism module in the upsampling path to obtain the second feature map corresponding to each feature extraction stage, wherein the upsampling path includes a first upsampling path from the first feature extraction stage to the second feature extraction stage, and a second upsampling path from the second feature extraction stage to the third feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first upsampling path and the second upsampling path are different.

5. The multi-scale feature fusion method according to claim 1 or 4, characterized in that: The step of performing downsampling based on the third feature map and the first feature maps corresponding to each feature extraction stage to obtain a fused feature map corresponding to each feature extraction stage comprises: During the downsampling process, the third feature map and the first feature map corresponding to each feature extraction stage pass through the attention mechanism module in the corresponding downsampling path, and perform convolution operation through the convolution layer of the attention mechanism module in the downsampling path to obtain a fused feature map corresponding to each feature extraction stage, wherein the downsampling path includes a first downsampling path from the third feature extraction stage to the second feature extraction stage, and a second downsampling path from the second feature extraction stage to the first feature extraction stage, and the attention weights corresponding to the attention mechanism modules embedded in the first downsampling path and the second downsampling path are different.

6. The multi-scale feature fusion method according to claim 1, characterized in that: Before the step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage, the method further includes: Performing a convolution operation on the first feature map corresponding to each feature extraction stage based on the first convolution unit to obtain a first sub-feature map corresponding to each feature extraction stage; The step of performing upsampling based on the first feature map to obtain a second feature map corresponding to each feature extraction stage comprises: Upsampling is performed according to the first sub-feature map corresponding to each feature extraction stage to obtain a second feature map corresponding to each feature extraction stage.

7. The multi-scale feature fusion method according to claim 1 or 6, characterized in that: The step of fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain the third feature map corresponding to the second feature extraction stage includes: Fusing the first feature map and the second feature map corresponding to the second feature extraction stage to obtain a second sub-feature map; Based on the second convolution unit, a convolution operation is performed on the second sub-feature map to obtain a third feature map corresponding to the second feature extraction stage.

8. The multi-scale feature fusion method according to claim 1, characterized in that: The step of extracting first feature maps corresponding to multiple feature extraction stages based on the image to be processed includes: The image to be processed is subjected to feature extraction through multiple feature extraction stages of the RepVGG backbone network to obtain first feature maps corresponding to the multiple feature extraction stages.

9. A multi-scale feature fusion device, characterized in that: The multi-scale feature fusion device includes: a memory, a processor, and a multi-scale feature fusion program stored in the memory and executable on the processor. When the multi-scale feature fusion program is executed by the processor, the steps of the multi-scale feature fusion method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-scale feature fusion program, and when the multi-scale feature fusion program is executed by a processor, the steps of the multi-scale feature fusion method described in any one of claims 1 to 8 are implemented.

Images