A multi-scale feature fusion method based on feature sharing

Abstract

The application discloses a multi-scale feature fusion method based on feature sharing, and belongs to the field of artificial intelligence.The method comprises the following steps: in the multi-scale features extracted by a backbone extraction network, all feature maps of different scales are selected as output features, wherein the selected feature maps have different spatial scales; a multi-scale feature fusion module based on a sharing mechanism is constructed for the output features, the multi-scale feature fusion module realizes strong fusion between scales through a point-by-point convolution and a deep convolution in parallel, so that each feature map contains not only the context semantic information of the original scale feature, but also the fusion features from all other scale features; and according to the fused features, a target in an image / video is segmented, classified and detected.The application can improve the utilization rate of the extracted multi-scale features to improve the model performance.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-scale feature fusion method based on feature sharing. Background Art

[0002] In recent years, with the rapid development of computer hardware, more and more image classification / segmentation algorithms based on convolutional neural networks have been proposed. Today, image classification / segmentation has been widely used in military, remote sensing, medical, social services and other fields.

[0003] Feature fusion networks, such as Inception Network (IN) and Path Aggregation Network (PANet), are an important component of mainstream multi-scale feature fusion. Their role is to fuse multi-scale features from the backbone feature extraction network, thereby obtaining features from multiple receptive fields to achieve better algorithm performance. Specifically, the features extracted by the backbone feature extraction network with a larger receptive field contain more comprehensive global features (position information), while the smaller receptive field extracts more detailed, fine-grained information. Fusion of large-scale features with small-scale features can effectively improve the performance of classification / segmentation algorithms.

[0004] However, mainstream feature fusion networks, such as IN, PANet, and the Feature Pyramid Network based on Automatic Architecture Search (NAS-FPN), primarily combine or concatenate multi-scale features extracted by feature extractors at different scales. This not only fails to consider the relationship between features at different scales but also reduces scale diversity. While other adaptive feature fusion algorithms consider proportional relationships during feature fusion, most of them simply utilize feature maps from adjacent scales or perform simple feature filtering through fully connected networks. Summary of the Invention

[0005] Purpose of the invention: In order to overcome the shortcomings of existing multi-scale feature fusion technology, a multi-scale feature fusion method based on feature sharing is provided, which can effectively solve the shortcomings of insufficient utilization of multi-scale features in the image classification / segmentation process and thus improve the algorithm performance.

[0006] Technical solution: To achieve the above objectives, the present invention provides a multi-scale feature fusion method based on feature sharing, comprising the following steps:

[0007] S1: Among the multi-scale features extracted by the backbone extraction network, all feature maps of different scales are selected as output features;

[0008] S2: Constructing a multi-scale feature fusion module based on a sharing mechanism for the output features. The multi-scale feature fusion module achieves strong fusion between scales by performing point-by-point convolution and depth-wise convolution in parallel, so that each feature map not only contains the contextual semantic information of the features at the original scale but also contains fused features from features at all other scales.

[0009] S3: Segment, classify, and detect objects in images / videos based on the fused features;

[0010] Furthermore, before selecting feature maps of different scales from the multi-scale features extracted by the backbone feature extraction network in step S1, the method further includes:

[0011] A1: For image / video classification or segmentation, a multi-scale feature extraction network is constructed as the backbone feature extraction network based on the input and output tensor sizes;

[0012] A2: Extract multi-scale features of the target to be classified or segmented in the image / video through the backbone feature extraction network.

[0013] Furthermore, in step S1, selecting feature maps of different scales from the multi-scale features extracted by the backbone feature extraction network includes:

[0014] B1: The different scale features extracted by the l scale feature extractors contained in the backbone feature extraction network are used as output features.

[0015] Furthermore, the feature maps of different scales extracted in step S1 are described by the formula:

[0016] F1=C1(input)

[0017] F2=C2(input)

[0018] …

[0019] F l =C l (input)

[0020] Among them, C i (i=1, 2, ..., l) represents the feature extractors of different scale features, the input is the image of the backbone feature extraction network, F i (i=1, 2, ..., l) represents the extracted feature maps of different scales.

[0021] Furthermore, in step S2, a multi-scale feature fusion module based on a sharing mechanism is constructed for the output features. The multi-scale feature fusion module achieves strong fusion between scales by performing point-by-point convolution and depth-wise convolution in parallel, so that each feature map not only contains the contextual semantic information of the features of the original scale but also contains other fusion features from features of all other scales, including:

[0022] C1: The feature maps F1, F2, ..., F obtained from the four claims are combined into a single image. l The multi-scale feature set F is obtained by connecting through the Concatenate function;

[0023] C2: Input the multi-scale feature set F obtained in step C1 into the pointwise convolution to obtain the multi-scale fusion feature F′;

[0024] C3: Input the multi-scale feature set F obtained in step C1 into the depthwise convolution to obtain the semantic information F″ of each scale feature;

[0025] C4: Add the multi-scale fusion feature F′ obtained in step C2 and step C3 and the semantic information F″ of each scale feature point by point to obtain the multi-scale shared fusion feature

[0026] Furthermore, the number of feature maps obtained in step C2 and step C3 should be consistent with the number of feature maps obtained in step C1. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of a method flow chart provided for an example of the present invention;

[0029] Figure 2 A schematic diagram of a multi-scale feature fusion structure based on feature sharing provided by an example of the present invention;

[0030] Figure 3 This is the Pavia University hyperspectral remote sensing image data map used in the simulation experiment of the present invention;

[0031] Figure 4This is a pseudo-color labeled image of the hyperspectral remote sensing image data of Pavia University in the simulation experiment of the present invention;

[0032] Figure 5 This is the classification result diagram of the hyperspectral remote sensing image data of Pavia University in the simulation experiment of the present invention obtained by the Inception Network with multi-scale feature fusion based on feature sharing;

[0033] Figure 6 This is the classification result diagram of the hyperspectral remote sensing image data of Pavia University obtained by the original Inception Network in the simulation experiment of the present invention; DETAILED DESCRIPTION

[0034] The present invention is further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, modifications of various equivalent forms of the present invention made by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0035] like Figure 1 As shown, the present invention provides a multi-scale feature fusion method based on feature sharing, comprising the following steps:

[0036] S1: Among the multi-scale features extracted by the backbone extraction network, all feature maps of different scales are selected as output features;

[0037] S2: Constructing a multi-scale feature fusion module based on a sharing mechanism for the output features. The multi-scale feature fusion module achieves strong fusion between scales by performing point-by-point convolution and depth-wise convolution in parallel, so that each feature map not only contains the contextual semantic information of the features at the original scale but also contains fused features from features at all other scales.

[0038] S3: Segment, classify, and detect objects in images / videos based on the fused features;

[0039] In this embodiment, before selecting feature maps of different scales from the multi-scale features extracted by the backbone feature extraction network in step S1, the method further includes:

[0040] A1: For image / video classification or segmentation, a multi-scale feature extraction network is constructed as the backbone feature extraction network based on the input and output tensor sizes;

[0041] A2: Extract multi-scale features of the objects to be classified or segmented in the image through the backbone feature extraction network.

[0042] In this embodiment, in step S1, selecting feature maps of different scales from the multi-scale features extracted by the backbone feature extraction network includes:

[0043] B1: The different scale features extracted by the l scale feature extractors contained in the backbone feature extraction network are used as output features.

[0044] The feature maps of different scales extracted in step S1 of this embodiment are described by the formula:

[0045] F1=C1(input)

[0046] F2=C2(input)

[0047] …

[0048] F l =C l (input)

[0049] Among them, C i (i=1, 2, ..., l) represents the feature extractors of different scale features, the input is the image of the backbone feature extraction network, F i (i=1, 2, ..., l) represents the extracted feature maps of different scales.

[0050] In step S2 of this embodiment, a multi-scale feature fusion module based on a sharing mechanism is constructed for the output features. The multi-scale feature fusion module achieves strong fusion between scales by performing point-by-point convolution and depth-wise convolution in parallel, so that each feature map not only contains the contextual semantic information of the features at the original scale but also contains other fusion features from features at all other scales, including:

[0051] C1: The feature maps F1, F2, ..., F obtained from the four claims are combined into a single image. l The multi-scale feature set F is obtained by connecting through the Concatenate function;

[0052] C2: Input the multi-scale feature set F obtained in step C1 into the pointwise convolution to obtain the multi-scale fusion feature F′;

[0053] C3: Input the multi-scale feature set F obtained in step C1 into the depthwise convolution to obtain the semantic information F″ of each scale feature;

[0054] C4: Add the multi-scale composite feature F′ obtained in step C2 and step C3 and the semantic information F″ of each scale feature point by point to obtain the multi-scale shared fusion feature

[0055] The number of feature maps obtained in steps C2 and C3 in the embodiment is consistent with the number of feature maps obtained in step C1.

[0056] Based on the above scheme, in order to verify the effect of the method of the present application, the Inception Network architecture is used as the backbone multi-scale feature network for feature extraction in this embodiment, and the hyperspectral image classification task is used for verification. Among them, 1% of the samples in the Pavia University hyperspectral image data set are used as the training set, and 99% of the samples are used as the test set for simulation experiment, and no training strategy is used; using the TensorFlow deep learning framework, the training parameters are set as shown in Table 1:

[0057] Table 1 Training parameter settings

[0058] Input Spatial Size 15×15 Number of Principal 10 Decay Rate 0.01 Epochs 120 Batch Size 32 Learning Rate 0.01

[0059] Implementation results

[0060] In this embodiment, the overall classification accuracy (Overall Accuracy, OA), the average classification accuracy (Average Accuracy, AA) and the Kappa coefficient are used. Table 2 shows the classification accuracy comparison between the Inception Network model without using the fusion method of the present application and the Inception Network model using the fusion method of the present application. Figure 3-4 The Pavia University hyperspectral remote sensing image data used in the experiment and the corresponding pseudo-color label map are shown in Figures 1 and 2, respectively. Figure 3 The above method is applied to the above method, and the Pavia University classification result map obtained by using the fusion method of the present application is shown in Figures 3 and 4. Figure 5 Figure 6 The classification result map of Pavia University obtained by using the fusion method of the present application is shown in Figures 3 and 4.

[0061] Table 2 Classification accuracy of each class and overall classification

[0062]

[0063] According to the data in Table 2, the classification effect of the present application is better than that of the original IN model, and the OA, average classification AA and Kappa coefficient are also higher than those of the original model. In addition, by comparing Figure 5 and Figure 6 with the classification true value map, it can be seen that Figure 6 ​The classification performance is closer to the true value, thereby verifying the effectiveness of the method of the present invention.

Claims

1. A multi-scale feature fusion method based on feature sharing, characterized in that: The steps include: S1: Among the multi-scale features extracted by the backbone feature extraction network, all feature maps with different scales are selected as output features; S2: constructing a multi-scale feature fusion module based on a sharing mechanism for the output features, wherein the multi-scale feature fusion module performs a point-by-point addition operation on the multi-scale fusion features output by the point-by-point convolution and the semantic information of each scale feature output by the depth convolution in parallel, thereby obtaining a multi-scale shared fusion feature, so that each feature map contains not only the contextual semantic information of the original scale, but also the fusion features of features of all other scales; S3: Segment, classify, and detect objects in the image based on the multi-scale shared fusion features.

2. The method according to claim 1, characterized in that Before step S1, the following steps are also included: A1: For image / video classification or segmentation, a multi-scale feature extraction network is constructed as the backbone feature extraction network based on the input and output tensor sizes; A2: Use the backbone feature extraction network to extract multi-scale features of the target to be classified or segmented in the image / video.

3. The method according to claim 1, characterized in that The step S1 of selecting feature maps of different scales includes: B1: The different scale features extracted by the l scale feature extractors included in the backbone feature extraction network are used as output features. The different scale features are described by the formula: F1=C1(input) F2=C2(input) … F l =C l (input) Among them, C i (i=1, 2, ..., l) represents the feature extractors of different scale features, the input is the image of the backbone feature extraction network, F i (i=1, 2, ..., l) represents the extracted feature maps of different scales.

4. The method according to claim 3, characterized in that The specific operations of the multi-scale feature fusion module in step S2 include: C1: The different scale feature maps F1, F2, ..., F l The multi-scale feature set F is obtained by connecting through the Concatenate function; C2: Input the multi-scale feature set into the pointwise convolution branch to obtain the multi-scale fusion feature F′; C3: Input the multi-scale feature set into the depthwise convolution branch to obtain the contextual semantic information F″ of each scale feature; C4: Add the multi-scale fusion feature F′ obtained in step C2 and step C3 and the semantic information F″ of each scale feature point by point to obtain the multi-scale shared fusion feature 5. The multi-scale feature fusion method based on feature sharing according to claim 4 is characterized in that: The number of feature maps obtained in step C2 and step C3 should be consistent with the number of feature maps obtained in step C1.

Images