Embedded feature pyramid network system for object detection and instance segmentation

Abstract

The application discloses an embedded feature pyramid network system for target detection and instance segmentation, which comprises a feature re-extraction module, which is a local pooling pyramid structure, and is used for extracting detailed information of large-scale feature maps and semantic information of small-scale feature maps in each stage of a backbone network, so that the representation of multi-scale feature information in the FPN is enhanced; a feature aggregation module connected with the feature re-extraction module, which is used for fusing multi-scale feature information at different levels in the FPN, and improving the utilization rate of the multi-scale feature information; and a feature selection module connected with the feature aggregation module, which is used for selecting feature maps more conducive to improving the performance of the model, and increasing the receptive field of the network model. The FRM module composed of the local pooling pyramid structure efficiently transmits low-level details and high-level semantic information to each level of the FPN, and makes up for the deficiencies of the FPN in global semantic information, local detail spatial details and position information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly to an embedded feature pyramid network system for target detection and instance segmentation. BACKGROUND

[0002] As a basic visual task, target detection and instance segmentation have attracted much attention due to their importance in many computer vision tasks, such as image retrieval, pedestrian counting, video segmentation and target tracking. In recent years, compared with traditional methods relying on hand-crafted features, convolutional neural networks (CNNs) can greatly promote the development of target detection and instance segmentation by extracting high-level semantics and low-level details in multi-scale space. Due to the ability to simply and efficiently construct rich multi-scale feature maps, the feature pyramid network (FPN) structure has attracted widespread interest, in which the feature maps output by the shallower layers usually have larger spatial sizes and maintain complex and detailed bottom information. In contrast, the feature maps from the deeper layers represent high-level semantic information and location information.

[0003] Although the above type of FPN has achieved good performance, there is still much room for improvement. A typical drawback of the FPN structure is that there is a significant difference in scale information between the global semantic information collected at the top layer and the local detailed information collected at the bottom layer, which leads to the problem of possible lack of multi-scale information, which damages the ability of the FPN to accurately detect and segment each part of the object. Another drawback is that the receptive field of the CNN model cannot grow proportionally to its layer depth, which will make the output layer lack sufficient high-level semantic information to determine the location of the target. SUMMARY

[0004] The present application aims to overcome the above technical deficiencies and proposes an embedded feature pyramid network system for target detection and instance segmentation, which solves the technical problems of lack of multi-scale information and inability of the model's receptive field to grow proportionally to its layer depth in the prior art.

[0005] To achieve the above technical purpose, in a first aspect, the technical solution of the present application provides an embedded feature pyramid network system for target detection and instance segmentation, comprising:

[0006] The feature re-extraction module is a local pooling pyramid structure, and is used to extract detailed information of large-scale feature maps and semantic information of small-scale feature maps at each stage of the backbone network, so as to enhance the multi-scale feature information representation in the FPN; the feature aggregation module is connected with the feature re-extraction module, and is used to fuse multi-scale feature information at different levels in the FPN, so as to improve the utilization rate of the multi-scale feature information;

[0007] The feature selection module is connected with the feature aggregation module, and the feature selection module is used for selecting a feature map which is more beneficial to improving model performance and increasing the receptive field of the network model.

[0008] The feature re-extraction module (FRM), the feature aggregation module (FAM) and the feature selection module (FSM) are used for efficiently transmitting low-level details and high-level semantic information to each level of the FPN by using the FRM module composed of a local pooling pyramid structure, and the deficiencies of the FPN in global semantic information and local detail spatial details and position information are made up.

[0009] According to some embodiments of the present application, the feature re-extraction module comprises three sub-branches, the first sub-branch is an original image size feature extraction layer, and the second sub-branch and the third sub-branch are feature extraction layers using different sizes of pooling kernels.

[0010] According to some embodiments of the present application, the pooling kernel of the second sub-branch is 4, and the pooling kernel of the third sub-branch is 8.

[0011] According to some embodiments of the present application, the feature aggregation module comprises:

[0012] The FPN generates a feature hierarchy diagram, which comprises a feature map P2, a feature map P3, a feature map P4 and a feature map P5, and the enhancement path starts from the lowest level P2 to the highest level P5.

[0013] The feature map N2, the feature map N3, the feature map N4 and the feature map N5 are generated according to the feature map P2, the feature map P3, the feature map P4 and the feature map P5, and the new feature pyramid network structure is generated according to the feature map N2, the feature map N3, the feature map N4 and the feature map N5.

[0014] According to some embodiments of the present application, the feature map N2, the feature map N3, the feature map N4 and the feature map N5 are generated according to the feature map P2, the feature map P3, the feature map P4 and the feature map P5, and the new feature pyramid network structure is generated according to the feature map N2, the feature map N3, the feature map N4 and the feature map N5.

[0015] The feature map N2 is the feature map P2 without processing, and the feature map Ni is obtained by performing feature extraction through a 3*3 convolution layer with a step of 2, and generates a down-sampling feature map;

[0016] The feature map Pi+1 is fused with the down-sampling feature map through a transverse connection to obtain a fused feature map, and another 3*3 convolution layer is used to process the fused feature map to generate the feature map Ni+1, and the value range of i is [2, 3, 4, 5].

[0017] According to some embodiments of the present application, the feature selection module comprises a global pooling layer and two fully connected layers connected in sequence.

[0018] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for an embodiment of the present application;

[0021] Figure 2 A feature re-extraction module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application;

[0022] Figure 3 A feature aggregation module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application;

[0023] Figure 4 A feature selection module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0025] It should be noted that although the functional modules are divided in the system schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0026] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0027] Referring to Figures 1 to 4 , Figure 1 a schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for an embodiment of the present application; Figure 2 a feature re-extraction module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application; Figure 3 a feature aggregation module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application; Figure 4 a feature selection module schematic diagram of an embedded feature pyramid network for target detection and instance segmentation provided for another embodiment of the present application.

[0028] In an embodiment, the embedded feature pyramid network for target detection and instance segmentation includes: a feature re-extraction module, which is a local pooling pyramid structure, the feature re-extraction module is used to extract detailed information of large-scale feature maps and semantic information of small-scale feature maps in each stage of the backbone network, so as to enhance the multi-scale feature information representation in the FPN; a feature aggregation module connected with the feature re-extraction module, the feature aggregation module is used to fuse multi-scale feature information of different levels in the FPN, and improve the utilization rate of multi-scale feature information; a feature selection module connected with the feature aggregation module, the feature selection module is used to select feature maps more conducive to improving the performance of the model, and increase the receptive field of the network model.

[0029] The embedded feature pyramid network (EFPN) is composed of three modules: a feature re-extraction module (FRM), a feature aggregation module (FAM), and a feature selection module (FSM). The FRM, as a local pooling pyramid structure, can effectively extract multi-scale information at each level. The FRM is composed of three sub-branches for capturing the context information of the input image. In the FRM, short connections between different branch structures are introduced to establish an internal information flow mechanism. The first sub-branch is the original image size feature extraction layer, and the remaining two sub-branches use different size pooling kernels. The original feature map is converted into different scale feature maps through the pooling operation, and the pool kernel size of 4 and 8 is selected. After extracting the features through convolution, the feature maps of different branches are upsampled to the original image size. After upsampling, all these branches are combined by summation and then input into the convolution layer to ensure that they match the input feature map. At the same time, short connections between adjacent branches are added to the FRM, which further down-samples the larger scale feature maps and fuses them into smaller scale feature maps. This connection explicitly establishes internal communication between adjacent branches, so that the output features contain more rich feature information. Given the characteristics of the local pool pyramid structure, FRM is used to help extract detailed information of large-scale feature maps and semantic information of small-scale feature maps at each level in the backbone network to enhance the multi-scale feature information representation in the FPN.

[0030] Although the FRM is used to transmit more rich detailed information and more effective global semantic information from the backbone network to different levels of FPN, it is necessary to seamlessly fuse the multi-scale feature maps in the FRM with the feature maps of different pyramid scales. Given that the current FPN structure mainly uses a top-down path to fully transmit top-level semantic information for full utilization, a bottom-up path enhancement method is adopted to further fully utilize the bottom-level detail information. Therefore, FAM is used to achieve this goal by adding a bottom-up enhancement path. The feature maps [P2, P3, P4, P5] represent the feature levels generated by the FPN. The enhancement path starts from the lowest level P2 to the highest level P5, and [N2, N3, N4, N5] is used to represent the newly generated feature maps corresponding to [P2, P3, P4, P5], N2 is P2 without any processing. Each feature map Ni first goes through a 3x3 convolution layer with a step of 2 to implement the feature extraction process and generate a down-sampled feature map. Then, the feature map Pi+1 is fused with the down-sampled feature map through a lateral connection, and the fused feature map is processed through another 3x3 convolution layer to generate Ni+1 for the subsequent network. Finally, a new feature pyramid network structure (i.e., [N2, N3, N4, N5]) is generated according to the new feature map.

[0031] The FAM is used to better fuse the multi-scale feature information of different levels in the FPN, and improve the utilization rate of multi-scale feature information by taking advantage of the feature fusion advantage of the enhanced path.

[0032] Although the FPN provides a classic architecture for combining multi-level features from the classification backbone, the problem with this architecture is that the actual receptive field of the CNN is much smaller than the theoretical receptive field. In order to improve the problem of insufficient receptive field of FPN, and further alleviate the aliasing effect in the feature fusion process, FSM is introduced to improve the accuracy of model target detection and segmentation.

[0033] The FSM is composed of a global pooling layer and a fully connected layer. First, the channel relationship is established using a global average pooling layer and two fully connected layers. Then, the spatial pixel relationship is established using a global average pooling layer and a global maximum pooling, which is used to make each level of feature map understand the target position as much as possible. With the characteristics of global pooling, FSM module is used to focus on calibrating the weight of the feature map, which can effectively select the feature map that is more conducive to improving the performance of the model, and further increase the receptive field of the network model.

[0034] Experimental process:

[0035] A. Experimental setup

[0036] Implementation details: The framework proposed in this application is implemented based on the PyTorch repository. By default, the model is trained for 12 epochs using 4 NVIDIA RTX-1080Ti GPUs (2 images per GPU). The initial learning rate is set to 0.02, and it is reduced by a factor of 0.1 after the 8th and 11th epochs, respectively. All other hyperparameters in this paper remain unchanged. ResNet-50 or ResNet-101 is used as the predefined CNN for all experiments [41, 44]. Faster R-CNN, Mask R-CNN or Cascade Mask R-CNN [41, 44] is also used as the baseline architecture of the EFPN model.

[0037] In order to evaluate the performance of the framework, experiments were conducted on three commonly used datasets, including PASCAL VOC 2007+2012, HRSID and MS COCO datasets.

[0038] B. Ablation study

[0039] This experiment conducts experiments on different module design options and network configurations to illustrate the effectiveness of each component in this application. By default, unless otherwise specified, the ablation experiment is based on ResNet-50 and the MS COCO dataset.

[0040] FRM-only: The addition of FRM improves the performance of AP, AP, APm, AP1 on MS COCO dataset in all aspects compared to the FPN baseline. The multi-scale feature information generated by FRM enables the network to pay more attention to the integrity and edge contour of the target, greatly improving the quality of the generated feature map. As shown in Table 1, simply adding FAM to the baseline FPN has a performance gain of about 1.7 AP (39.5 v.s. 37.8) in target detection and a performance increment of about 1.1 AP (34.5 v.s. 35.6) in instance segmentation on the MS COCO dataset.

[0041] FAM-only: Simply embedding FAM into the FPN baseline helps almost target detection and instance segmentation. For example, adding FAM improves target detection and instance segmentation on the MS COCO dataset by 0.5 AP and 0.4 AP, respectively, compared to the results without introducing FAM. This is because the bottom-top path inside FAM also better fuses feature maps of different scales. In addition, the FPN baseline containing FRM still needs to merge feature maps of different levels, and the further improvement after adding FAM shows the effectiveness of FAM.

[0042] FSM-only: Simply adding FSM to the FPN baseline helps almost target detection and instance segmentation. For example, adding FSM improves object detection and instance segmentation on the MS COCO dataset by 0.5 AP and 0.4 AP, respectively, compared to the results without incorporating FSM. Because FSM helps the model to effectively help the model to select key fusion of low-level details and high-level semantic information at each level, and the pooling operation inside FSM also expands the receptive field of the entire network. In addition, the FPN baseline incorporating FRM still needs to select key feature maps from different levels, and the further improvement after adding FSM shows the effectiveness of FSM.

[0043] FRM & FAM & FSM: By introducing FRM, FAM and FSM into the FPN baseline, the performance of AP, AP, APm, AP1 on the MS COCO dataset can be further enhanced in all aspects compared to the model incorporating FRM or FAM or FSM. This phenomenon shows that FRM, FAM and FSM are three complementary modules. First, the use of the proposed FAM enables the present application to have a strong ability to accurately discover objects and keep objects more complete. Then, FAM can help refine the details of the discovered objects. In addition, FSM highlights strongly related features while compensating for the receptive field defects of existing networks.

[0044] Pooling operation type. Pooling techniques play a fundamental role in the proposed method. To investigate the influence of the pooling operation type on improving model performance, the full-channel adaptive average pooling operation in FRM is replaced by the adaptive max pooling operation. Alternatively, the channel-global average pool operation in FSM is replaced by the channel-global max pool to see the difference in model performance. It can be seen that after using max pooling in FRM and FSM, the performance of object detection and instance segmentation is reduced by about 1.5 AP and 1.0 AP, respectively. The change of average pooling in FRM brings more performance improvement than FSM. Therefore, average pooling is used in EFPN to obtain better model performance. This may be because, unlike max pooling, average pooling can better extract the background information of objects, which is beneficial to object classification and contour positioning. In addition, when all the pooling branches are combined by using short connections, the results can be maximized. It also proves the effectiveness of integrating cross-scale feature representations. These experiments show that both pyramid pooling modules and short connections play an important role in EFPN, and without any of them, the performance of the present application will be impaired.

[0045]

[0046] Table 1 Ablation analysis of FRM, FAM, FSM and short connection (S-C) to be used. It can be seen that each component in the architecture plays an important role and contributes to the performance, the best performance in each column is highlighted in bold.

[0047]

[0048] Table 2: Influence ablation analysis of different types of pooling operations used in FRM and FSM, it can be observed that when using average pooling operations in FRM and FSM, better overall performance can be achieved, the best performance in each column is highlighted in bold.

[0049]

[0050] Table 3: Quantitative object detection results on the MS COCO dataset. The best results of different trunks are highlighted in bold, respectively. It can be seen that the present application achieves the best results on almost all datasets and indicators.

[0051] C. Comparison with advanced algorithms

[0052] In this subsection, EFPN is compared with the previous 10 most advanced methods. In order to make a fair comparison, the results produced by these methods are generated by the original code published by the corresponding authors or directly provided by them. All prediction results are evaluated using the same evaluation code and environment.

[0053] Target detection results on the MS COCO dataset, EFPN is compared with the previous 10 most advanced detection methods, and the results of ResNet-50 and ResNet-101 backbone networks are reported. By replacing FPN with EFPN, several classic target detection models consistently achieve better performance. To ensure the effectiveness of EFPN, to make a fair comparison, the corresponding baseline methods of FPN, HRFPN and CARAFE are re-deployed on Mask R-CNN, and the results of re-deploying these public models are mostly better than the official results. In this case, EFFN still achieves the best performance, and the performance of EFPN is 2.0AP, 1.7AP and 0.6AP higher than FPN, HRFPN and CARAFE respectively, which proves the superiority of the present application.

[0054] In addition, even if a more powerful detector is used, EFPN can always bring non-negligible performance. For example, when using cascaded Mask R-CNN as a target detector, EFPN still improves the performance by 1.2AP. The performance improvement of the above three different detectors proves the robustness of the present application.

[0055] Table 4: Quantitative instance segmentation results on the MS COCO dataset. The best results of different trunks are highlighted in bold respectively. It can be seen that the present application achieves the best results on almost all datasets and indicators.

[0056]

[0057] Table 5: Quantitative object detection results on the PASCAL VOC2007+2012 dataset. The best results of different trunks are highlighted in bold respectively. It can be seen that the present application achieves the best results on almost all datasets and indicators.

[0058] Table 6: Quantitative target detection and instance segmentation results on high-resolution SAR images of the HRSID dataset, the results of different trunks are highlighted in bold respectively, and the present application achieves the best results on almost all datasets and indicators.

[0059] Instance segmentation results on the MS COCO dataset. The generalization ability of EFPN is studied by applying it to the instance segmentation task. EFPN is compared with the previous 9 most advanced instance segmentation methods. As shown in Table 4, the results are comparable to the most advanced models, and even better than most of the previous classic models, which means that EFPN is also beneficial to the instance segmentation task.

[0060] Specifically, in order to ensure the effectiveness of the EFPN, in order to make a fair comparison, the corresponding baseline methods are re-deployed on Mask R-CNN equipped with FPN, HRFPN, CARAFE. As shown in Table 4, the results of re-deploying these public models are mostly better than the official results. In this case, EFPN still achieves the best performance, and the performance of EFPN is 1.5AP, 1.5AP and 0.6AP higher than FPN, HRFPN and CARAFE respectively, which proves the superiority of the present application. In addition, even using a more powerful model, EFPN can always bring negligible performance. For example, when using Cascade Mask R-CNN as the basic model, the present application still improves the performance by 0.9AP. The above two examples of different detector performance improvements prove the robustness of the present application.

[0061] Detection results of PASCAL VOC 07+12 dataset. In order to further evaluate, whether EFPN is suitable for other datasets, namely PASCAL VOC 2007+2012, is tested. EFPN is compared with the previous 11 most advanced detection methods. Obviously, as shown in Table 5, the performance of EFPN is better than 0.8AP, reaching the most advanced performance.

[0062] Detection and segmentation results on HRSID dataset. In order to further evaluate, whether EFPN is suitable for other datasets, namely the high-resolution SAR image dataset HRSID, is tested. EFPN is compared with the previous 8 most advanced detection methods. Obviously, as shown in Table 5, for example, based on Mask R-CNN, Cascade Mask R-CNN, EFPN is better than 1.7AP, 3.8AP respectively on the target detection task, and 1.1AP, 4.8AP respectively on the instance segmentation task.

[0063] The device embodiments described above are only schematic, and units described as separate components can or can not be physically separate, and can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0064] The above is a specific description of the preferred embodiment of the present application, but the present application is not limited to the above-mentioned embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

[0065] The above description of the specific embodiments of the present application is not intended to limit the scope of the present application. Any other corresponding changes and modifications made according to the technical concept of the present application should be included in the scope of protection of the claims of the present application.

Claims

1. An embedded feature pyramid network system for object detection and instance segmentation, characterized in that, include: The feature re-extraction module is a local pooling pyramid structure. The feature re-extraction module is used to extract detailed information of large-scale feature maps and semantic information of small-scale feature maps at each stage of the backbone network to enhance the multi-scale feature information representation in FPN. The feature aggregation module is connected to the feature re-extraction module. The feature aggregation module is used to fuse multi-scale feature information at different levels in FPN to improve the utilization rate of multi-scale feature information. The feature selection module is connected to the feature aggregation module. The feature selection module is used to select feature maps that are beneficial to improving model performance and increasing the receptive field of the network model. The feature re-extraction module includes three sub-branches: the first sub-branch is a feature extraction layer of the original image size, and the second and third sub-branches are feature extraction layers using pooling kernels of different sizes.

2. The embedded feature pyramid network system for target detection and instance segmentation according to claim 1, characterized in that, The pooling kernel of the second sub-branch is 4, and the pooling kernel of the third sub-branch is 8.

3. The embedded feature pyramid network system for target detection and instance segmentation according to claim 1, characterized in that, The feature aggregation module includes: The feature hierarchy maps generated by FPN include: feature map P2, feature map P3, feature map P4 and feature map P5, with the enhancement path starting from the lowest level P2 and going up to the highest level P5; Based on feature maps P2, P3, P4, and P5, generate feature maps N2, N3, N4, and N5. Based on feature maps N2, N3, N4, and N5, generate a new feature pyramid network structure.

4. The embedded feature pyramid network system for target detection and instance segmentation according to claim 3, characterized in that, Generate feature maps N2, N3, N4, and N5 based on feature maps P2, P3, P4, and P5, including the following steps: Feature map N2 is feature map P2 in the unprocessed state. Feature map Ni is extracted through a 3×3 convolutional layer with a stride of 2, and a downsampled feature map is generated. The feature map Pi+1 is fused with the downsampled feature map by lateral connection to obtain a fused feature map, and the fused feature map is processed by another 3×3 convolutional layer to generate feature map Ni+1, where the value of i is in the range of [2, 3, 4, 5].

5. The embedded feature pyramid network system for target detection and instance segmentation according to claim 1, characterized in that, The feature selection module includes: a global pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence.

Images