Feature fusion network containing FPN and PAN structures
By fusing the feature fusion network of FPN and PAN structures in image processing, the problems of small field of view and low resolution in the prior art are solved, and effective fusion and enhancement of multi-scale features are achieved.
Patent Information
- Application Number
- CN202411327121.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-07
AI Technical Summary
During the image feature extraction process of existing FPN and PAN convolution neural networks, the FPN has a small field of view from deep to shallow convolution, and the PAN has a low resolution from shallow to deep convolution, and lacks object position information and cannot be effective. Fusion of multi-scale features.
Design a feature fusion network containing FPN and PAN structures, build a backbone network through YOLOv5 algorithm, fuse FPN and PAN neural networks, extract feature layers of different depths, and pass equivalent replication and semantic information to from deep to shallow Convolution with shallow layer to deep layer to form multi-scale expression results.
The construction of feature pyramids is realized, providing complete target position information and high spatial resolution, and enhancing the fusion of feature information and multi-scale expression.
Smart Images
Figure CN119274180A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, in particular to a feature fusion network containing FPN and PAN structures. Background Art
[0002] After the feature backbone network extracts image features, the feature map needs to be processed again to obtain more accurate image features, so as to provide better feature layer resources for subsequent image processing. The existing reprocessing method generally uses a single ResNet-50 or ResNet-101 fused with FPN or PAN convolutional neural network to process the image. However, the FPN convolutional neural network performs convolution from deep to shallow layers, but has a small field of view. The PAN convolutional neural network performs convolution from shallow to deep layers, but has low resolution and lacks object location information.
[0003] Therefore, it is necessary to design a feature fusion network containing FPN and PAN structures. Summary of the invention
[0004] The purpose of the present invention is to provide a feature fusion network containing FPN and PAN structures to solve the problems raised in the above background technology.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0006] A feature fusion network containing FPN and PAN structures, including an FPN neural network and a PAN neural network, comprises the following steps:
[0007] S100, build a backbone network based on the YOLOv5 algorithm, and then fuse the FPN neural network and the PAN neural network based on the backbone network to form a feature fusion network with two branches;
[0008] S200, extracting three feature layers of different depths, importing them into a feature fusion network, and equivalently copying the three feature layers to form two groups;
[0009] S300, FPN neural network transfers semantic information from deep layer to shallow layer of one set of three feature layers and parses the feature layers;
[0010] S400 and PAN neural networks transfer semantic information from shallow layers to deep layers of another set of three feature layers and parse the feature layers;
[0011] S500, fusing the semantic information and feature layer analysis results obtained in S300 with the semantic information and feature layer analysis results obtained in S400 to obtain a multi-scale expression result.
[0012] According to the above technical solution, the specific method of S100 is as follows:
[0013] Based on YOLOv5, FPN neural network and PAN neural network are added as two independently extended network branch structures, and then they are simultaneously converged to the YOLOv5 backbone network. The backbone guidance part based on YOLOv5 contains a replication structure, and the FPN neural network and PAN neural network are branches of the same level, and their information processing is carried out synchronously.
[0014] According to the above technical solution, the specific method of S200 is as follows:
[0015] The backbone guidance part based on YOLOv5 contains a replication structure, so as to realize the equivalent replication of the three imported feature layers with different depths. The replication group and the original group cross-replace the feature layers, that is, the original group becomes original layer + replication layer + original layer or replication layer + original layer + replication layer, and the replication group performs relative cross transformation.
[0016] According to the above technical solution, the specific method of S300 is:
[0017] For three feature layers of different depths, forward network convolution is performed from deep layer to shallow layer to construct different return value pooling feature layers, which are first residual feature layer 1, first residual feature layer 2, first residual feature layer 3, first residual feature layer 4 and first residual feature layer 5, respectively. Feature pyramid 1 is constructed through the above residual feature layers, and the corresponding downsampling times are 1, 2, 3, 4 and 5, respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32, respectively;
[0018] If the size of the residual feature layer 1 is too large, it will consume a lot of video memory during training and will be discarded.
[0019] According to the above technical solution, the specific method of S400 is:
[0020] Three feature layers with different depths are convolved forward from the shallow layer to the deep layer to construct different return value pooling feature layers, which are the second residual feature layer 5, the second residual feature layer 4, the second residual feature layer 3, the second residual feature layer 2 and the second residual feature layer 1. The feature pyramid 2 is constructed by the above residual feature layers. The higher the residual feature layer is, the larger the field of view is and the more feature layers are extracted. The lower the residual feature layer is, the larger the horizontal extension is to ensure the extraction of features.
[0021] According to the above technical solution, the specific method of S500 is:
[0022] The feature pyramid 1 and feature pyramid 2 obtained in S300 are matched and fused one by one from the top layer to the bottom layer of the pyramid. When a residual feature layer appears in the bottom layer, it is discarded and only the horizontal matching layer is fused. In the process of mutual fusion, the same feature points are emphasized and different feature points are spatially combined to form a complete graphic feature pool expression result.
[0023] According to the above technical solution, the replication method adopted by the replication structure to achieve equivalent replication is as follows:
[0024] The feature information of the original layer is reserved first, and then every two feature points are combined to form a fitting feature group, and then the feature information is separated by dichotomy to form an upper feature layer and a lower feature layer, and then data is copied based on the upper feature layer and the lower feature layer respectively to form an upper copy feature layer and a lower copy feature layer, and then the upper feature layer and the lower copy feature layer are superimposed, and the lower feature layer and the upper copy feature layer are superimposed, and the copied feature points are fitted correspondingly with the original feature points to achieve effective equivalent replication.
[0025] According to the above technical solution, the method for determining the size of the residual feature layer 1 is as follows:
[0026] Based on the distribution of anchors in the residual feature layer 1, they are projected into the Faceboxes model, and the size of the relative relationship between the projection and the receptive field is judged.
[0027] According to the above technical solution, the corresponding downsampling times of the second residual feature layer 1, the second residual feature layer 2, the second residual feature layer 3, the second residual feature layer 4 and the second residual feature layer 5 are 1, 2, 3, 4 and 5 respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32 respectively.
[0028] According to the above technical solution, the spatial combination method is to multiply two or more feature points to encode the nonlinear law in the feature space to form a combined feature vector, and then perform linear calculation training.
[0029] Compared with the prior art, the beneficial effects achieved by the present invention are:
[0030] By fusing the FPN neural network with the PAN neural network and processing the feature layers of the two branches synchronously, a pair of feature pyramids are generated, one of which has complete target location information and high spatial resolution, and the other has a wide receptive field. The two are then fused to strengthen each other's feature information and construct a multi-scale expression. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1It is a flowchart of the present invention. DETAILED DESCRIPTION
[0032] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0033] The present invention provides a technical solution: a feature fusion network containing FPN and PAN structures, including an FPN neural network and a PAN neural network, which contains the following steps:
[0034] S100, build a backbone network based on the YOLOv5 algorithm, and then fuse the FPN neural network and the PAN neural network based on the backbone network to form a feature fusion network with two branches;
[0035] S200, extracting three feature layers of different depths, importing them into a feature fusion network, and equivalently copying the three feature layers to form two groups;
[0036] S300, FPN neural network transfers semantic information from deep layer to shallow layer of one set of three feature layers and parses the feature layers;
[0037] S400 and PAN neural networks transfer semantic information from shallow layers to deep layers of another set of three feature layers and parse the feature layers;
[0038] S500, fusing the semantic information and feature layer analysis results obtained in S300 with the semantic information and feature layer analysis results obtained in S400 to obtain a multi-scale expression result.
[0039] Specifically, the specific methods of S100 are as follows:
[0040] Based on YOLOv5, FPN neural network and PAN neural network are added as two independently extended network branch structures, and then they are simultaneously converged to the YOLOv5 backbone network. The backbone guidance part based on YOLOv5 contains a replication structure, and the FPN neural network and PAN neural network are branches of the same level, and their information processing is carried out synchronously.
[0041] Specifically, the specific methods of S200 are as follows:
[0042] The backbone guidance part based on YOLOv5 contains a replication structure, so as to realize the equivalent replication of the three imported feature layers with different depths. The replication group and the original group cross-replace the feature layers, that is, the original group becomes original layer + replication layer + original layer or replication layer + original layer + replication layer, and the replication group performs relative cross transformation.
[0043] Specifically, the specific method of S300 is:
[0044] For three feature layers of different depths, forward network convolution is performed from deep layer to shallow layer to construct different return value pooling feature layers, which are first residual feature layer 1, first residual feature layer 2, first residual feature layer 3, first residual feature layer 4 and first residual feature layer 5, respectively. Feature pyramid 1 is constructed through the above residual feature layers, and the corresponding downsampling times are 1, 2, 3, 4 and 5, respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32, respectively;
[0045] If the size of the residual feature layer 1 is too large, it will consume a lot of video memory during training and will be discarded.
[0046] Specifically, the specific method of S400 is:
[0047] Three feature layers with different depths are convolved forward from the shallow layer to the deep layer to construct different return value pooling feature layers, which are the second residual feature layer 5, the second residual feature layer 4, the second residual feature layer 3, the second residual feature layer 2 and the second residual feature layer 1. The feature pyramid 2 is constructed by the above residual feature layers. The higher the residual feature layer is, the larger the field of view is and the more feature layers are extracted. The lower the residual feature layer is, the larger the horizontal extension is to ensure the extraction of features.
[0048] Specifically, the specific methods of S500 are:
[0049] The feature pyramid 1 and feature pyramid 2 obtained in S300 are matched and fused one by one from the top layer to the bottom layer of the pyramid. When a residual feature layer appears in the bottom layer, it is discarded and only the horizontal matching layer is fused. In the process of mutual fusion, the same feature points are emphasized and different feature points are spatially combined to form a complete graphic feature pool expression result.
[0050] Specifically, the replication method used by the replication structure to achieve equivalent replication is as follows:
[0051] The feature information of the original layer is reserved first, and then every two feature points are combined to form a fitting feature group, and then the feature information is separated by dichotomy to form an upper feature layer and a lower feature layer, and then data is copied based on the upper feature layer and the lower feature layer respectively to form an upper copy feature layer and a lower copy feature layer, and then the upper feature layer and the lower copy feature layer are superimposed, and the lower feature layer and the upper copy feature layer are superimposed, and the copied feature points are fitted correspondingly with the original feature points to achieve effective equivalent replication.
[0052] Specifically, the method for determining the size of the residual feature layer 1 is as follows:
[0053] Based on the distribution of anchors in the residual feature layer 1, they are projected into the Faceboxes model, and the size of the relative relationship between the projection and the receptive field is judged.
[0054] Specifically, the second residual feature layer 1, the second residual feature layer 2, the second residual feature layer 3, the second residual feature layer 4 and the second residual feature layer 5 have corresponding downsampling times of 1, 2, 3, 4 and 5 respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32 respectively.
[0055] Specifically, the spatial combination method is to multiply two or more feature points to encode the nonlinear law in the feature space to form a combined feature vector, and then perform linear calculation training.
[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0057] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A feature fusion network containing FPN and PAN structures, including an FPN neural network and a PAN neural network, characterized in that: It contains the following steps: S100, build a backbone network based on the YOLOv5 algorithm, and then fuse the FPN neural network and the PAN neural network based on the backbone network to form a feature fusion network with two branches; S200, extracting three feature layers of different depths, importing them into a feature fusion network, and equivalently copying the three feature layers to form two groups; S300, FPN neural network transfers semantic information from deep layer to shallow layer of one set of three feature layers and parses the feature layers; S400 and PAN neural networks transfer semantic information from shallow layers to deep layers of another set of three feature layers and parse the feature layers; S500, fusing the semantic information and feature layer analysis results obtained in S300 with the semantic information and feature layer analysis results obtained in S400 to obtain a multi-scale expression result.
2. According to claim 1, a feature fusion network containing FPN and PAN structures is characterized in that: The specific method of S100 is as follows: Based on YOLOv5, FPN neural network and PAN neural network are added as two independently extended network branch structures, and then they are simultaneously converged to the YOLOv5 backbone network. The backbone guidance part based on YOLOv5 contains a replication structure, and the FPN neural network and PAN neural network are branches of the same level, and their information processing is carried out synchronously.
3. The feature fusion network containing FPN and PAN structures according to claim 2 is characterized in that: The specific method of S200 is as follows: The backbone guidance part based on YOLOv5 contains a replication structure, so as to realize the equivalent replication of the three imported feature layers with different depths. The replication group and the original group cross-replace the feature layers, that is, the original group becomes original layer + replication layer + original layer or replication layer + original layer + replication layer, and the replication group performs relative cross transformation.
4. The feature fusion network containing FPN and PAN structures according to claim 1, characterized in that: The specific method of S300 is: For three feature layers of different depths, forward network convolution is performed from deep layer to shallow layer to construct different return value pooling feature layers, which are first residual feature layer 1, first residual feature layer 2, first residual feature layer 3, first residual feature layer 4 and first residual feature layer 5, respectively. Feature pyramid 1 is constructed through the above residual feature layers, and the corresponding downsampling times are 1, 2, 3, 4 and 5, respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32, respectively; If the size of the residual feature layer 1 is too large, it will consume a lot of video memory during training and will be discarded.
5. The feature fusion network containing FPN and PAN structures according to claim 1, characterized in that: The specific method of S400 is: Three feature layers with different depths are convolved forward from the shallow layer to the deep layer to construct different return value pooling feature layers, which are the second residual feature layer 5, the second residual feature layer 4, the second residual feature layer 3, the second residual feature layer 2 and the second residual feature layer 1. The feature pyramid 2 is constructed by the above residual feature layers. The higher the residual feature layer is, the larger the field of view is and the more feature layers are extracted. The lower the residual feature layer is, the larger the horizontal extension is to ensure the extraction of features.
6. The feature fusion network containing FPN and PAN structures according to claim 1, characterized in that: The specific method of S500 is: The feature pyramid 1 and feature pyramid 2 obtained in S300 are matched and fused one by one from the top layer to the bottom layer of the pyramid. When a residual feature layer appears in the bottom layer, it is discarded and only the horizontal matching layer is fused. In the process of mutual fusion, the same feature points are emphasized and different feature points are spatially combined to form a complete graphic feature pool expression result.
7. The feature fusion network containing FPN and PAN structures according to claim 3, characterized in that: The copy method used by the copy structure to achieve equivalent copying is as follows: The feature information of the original layer is reserved first, and then every two feature points are combined to form a fitting feature group, and then the feature information is separated by dichotomy to form an upper feature layer and a lower feature layer, and then data is copied based on the upper feature layer and the lower feature layer respectively to form an upper copy feature layer and a lower copy feature layer, and then the upper feature layer and the lower copy feature layer are superimposed, and the lower feature layer and the upper copy feature layer are superimposed, and the copied feature points are fitted correspondingly with the original feature points to achieve effective equivalent replication.
8. The feature fusion network containing FPN and PAN structures according to claim 4, characterized in that: The method for determining the size of the residual feature layer 1 is as follows: Based on the distribution of anchors in the residual feature layer 1, they are projected into the Faceboxes model, and the size of the relative relationship between the projection and the receptive field is judged.
9. The feature fusion network containing FPN and PAN structures according to claim 5, characterized in that: The corresponding downsampling times of the second residual feature layer 1, the second residual feature layer 2, the second residual feature layer 3, the second residual feature layer 4 and the second residual feature layer 5 are 1, 2, 3, 4 and 5 respectively, and the corresponding step sizes are 2, 4, 8, 16 and 32 respectively.
10. The feature fusion network containing FPN and PAN structures according to claim 6, characterized in that: The spatial combination method is to multiply two or more feature points to encode the nonlinear law in the feature space to form a combined feature vector, and then perform linear calculation training.