Object recognition method based on neighbor-branch complementary saliency and multiple prior sparse representation

Abstract

The present invention discloses a target recognition method based on neighbor-branch complementary saliency and multi-prior sparse representation, and the method comprises the following steps: Step 1: Propose a neighbor-branch complementary saliency extraction network, mine deep-level and semantic consistency information of an image, and extract candidate salient regions of multi-scale targets; Step 2: Suppress false alarms that may be generated by the saliency extraction network by combining a sparse representation classifier with multi-prior information of the target, and realize accurate recognition of ship targets in complex scenes. The method mines the salient regional features of ship targets in images through a deep saliency feature extraction network, and is combined with a multi-prior sparse representation classification method for image data, which can not only give full play to the advantages of multi-level extraction of image features by a deep network, but also has the characteristics of robustness to complex environmental interferences such as partial occlusion of objects, complex sea clutter, port facilities, light shadows, etc., and can provide support for applications such as port rescue and maritime traffic maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and recognition, and relates to a method for identifying ships in remote sensing images, and in particular to a method for identifying ship targets based on saliency extraction of complementary adjacent feature extraction branches and multi-prior sparse representation. Background Art

[0002] Ship recognition at sea is of great significance for applications such as automatic fishery management, maritime rescue, and port traffic maintenance. Its recognition performance directly determines the accuracy and efficiency of military and civilian applications. Visible light remote sensing images have rich structural information and texture details, which conform to the visual observation habits of the human eye, and therefore have become an important data source for ship detection and recognition. However, due to the wide-area imaging characteristics of optical remote sensing, there are usually various complex environmental factors in the image scene, such as waves, light shadows, various port facilities, etc., and the dense arrangement and occlusion caused by the docking of the ship itself further increase the difficulty of ship recognition. Therefore, ship recognition in complex optical scenes is still a challenging task.

[0003] Visual saliency model is a typical method in the field of target detection and recognition. It imitates the visual attention mechanism of human eyes and can quickly locate abnormal areas or abnormal points in complex scenes. It has broad application prospects in ship recognition applications under massive data conditions. Early visual saliency models used simple regional features to highlight salient areas, such as contrast, compactness and other features. On this basis, researchers have successively proposed various strategies to achieve more accurate object contour highlighting, such as frequency domain transform extraction method, Bayesian probability calculation extraction method, etc. However, these methods are mostly designed for specific application scenarios. When directly applied to large-scale remote sensing scenes, they are prone to produce a large number of false alarms. With the development of deep learning technology, deep learning architectures represented by convolutional neural networks have provided an effective model framework for remote sensing image ship recognition with their excellent high-level semantic representation advantages. In order to make the network suitable for detecting and identifying ship targets of different scales and types, the feature pyramid module (FPN) is widely used in the feature extraction stage of the network. Although subsequent detection and recognition models have added connection channels between feature extraction branches on the basis of classic FPN to fuse and characterize multi-level target features. However, existing networks rarely consider the semantic gap between features at different levels, and the direct fusion of feature maps with large resolution differences can easily introduce unnecessary background noise, reducing the efficiency and accuracy of network learning of target features. In addition, the classification modules of existing deep detection and recognition networks are rarely designed in a targeted manner in combination with the characteristics of ship targets in remote sensing scenes, resulting in a low accuracy rate of target recognition in complex scenes with partial occlusion of objects, light shadows, and interference from various environmental factors. Therefore, it is urgent to design and optimize the network structure of the ship recognition model based on the characteristics of ships in complex optical remote sensing image scenes. Summary of the invention

[0004] Aiming at the difficulties such as the variable scales and types of ship targets in remote sensing images and the complex scene interference elements, the present invention provides a target recognition method based on neighbor-branch complementary saliency and multi-prior sparse representation.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A target recognition method based on neighbor-branch complementary saliency and multiple prior sparse representations includes the following steps:

[0007] Step 1: Propose a neighbor-branch complementary saliency extraction network to mine deep and semantically consistent information of the image and extract candidate salient regions of multi-scale objects;

[0008] Step 2: By combining the sparse representation classifier with multiple prior information of the target, the false alarms that may be generated by the saliency extraction network are suppressed, and accurate identification of ship targets in complex scenarios is achieved.

[0009] Compared with the prior art, the present invention has the following advantages:

[0010] (1) Aiming at the difficulties of variable scales and types of ship targets in remote sensing images and complex interference from false alarm sources in scenes, the present invention proposes an end-to-end ship target recognition method that combines adjacent complementary saliency extraction and multi-prior sparse representation. This method mines the semantic consistency features of ship targets in images through a deep saliency feature extraction network, and combines it with a multi-prior sparse representation classification method for image data. It can not only give full play to the advantages of multi-level extraction of image features by deep networks, but also has strong robustness to complex environmental interference such as partial occlusion of objects and complex sea clutter, port facilities, light and shadows. This method is suitable for fine-grained recognition of ship targets in various complex application scenarios such as complex sea surfaces and ports, and can provide support for applications such as port rescue and maritime traffic maintenance.

[0011] (2) In order to solve the problems of difficult multi-scale representation of ship targets and large semantic differences among different extraction branches during feature fusion, the present invention designs a deep visual saliency extraction network based on the complementarity of adjacent feature extraction branches. The network mainly includes an inter-layer feature aggregation module and an intra-layer feature self-mining module. The network enhances the edge details and semantic information of different feature extraction branches by fusing feature maps of adjacent layers, and mines the feature information of different receptive fields in the same extraction branch at the same time, so as to better capture the scale changes of the target.

[0012] (3) In order to solve the difficulties such as the complex characteristics of ship targets and the numerous false alarm sources, the present invention designs a multi-class ship classifier based on multi-prior sparse representation. By constructing low-rank and intra-class difference regularization term constraints for typical ship targets, the accuracy of ship target recognition under occlusion and shadow conditions is improved, and high-precision ship type recognition in complex sea and port scenarios is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The overall flow chart of the target recognition method based on neighbor-branch complementary saliency and multi-prior sparse representation.

[0014] Figure 2 Schematic diagram of the structure of the inter-layer feature aggregation module and the intra-layer feature self-mining module.

[0015] Figure 3 It is the network structure of the inter-layer feature aggregation module and the intra-layer feature self-mining module.

[0016] Figure 4 Singular value decomposition results of different types of ship slices and ship characteristics.

[0017] Figure 5 Typical ship target recognition results for HRSC2016 public dataset images.

[0018] Figure 6 Typical ship target recognition results of Worldview-3 satellite images. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0020] The present invention provides a target recognition method based on adjacent branch complementary saliency and multiple prior sparse representations, such as Figure 1 As shown, the method comprises the following steps:

[0021] Step 1: Considering the semantic differences between feature maps at different levels, directly fusing too many feature maps may introduce a lot of background noise and reduce the representation effect of target features in the image. Therefore, a network structure that only fuses adjacent feature extraction branches is proposed to enhance target features while ensuring the semantic consistency of feature maps as much as possible. A neighbor-branch complementary saliency extraction network is proposed to mine deep-level and semantically consistent information of the image and extract candidate salient regions of multi-scale targets. Among them, the neighbor-branch complementary saliency extraction network mainly includes two main functional modules: the inter-layer integration module (IIM) and the intra-layer self-mining module (ISM). The specific steps are as follows:

[0022] Step 1-1: For the input remote sensing image, the typical ResNet-101 network is used as the backbone feature extraction network to extract multi-level feature maps with downsampling levels of 2, 4, 8, 16, and 32 times, layer by layer, which are denoted as F1, F2, F3, F4, and F5 respectively.

[0023] Step 1-2: Use the inter-layer integration module (IIM) to fuse the feature maps of adjacent layers to enhance the edge details and semantic information of different feature extraction branches. The network diagram is shown in the figure. Figure 2 (a) shows. This module designs the connection relationship between adjacent channels and establishes cross-layer connections of residual structures to clarify the direction of feature learning and subsequent network parameter training. It innovatively combines channel pooling to generate the weights of fused features, and models and optimizes the importance of fused features, so that the module has the ability to adaptively allocate features for different application tasks. The specific structure is shown in Figure 3 As shown in (a), it mainly includes 7 3×3 convolutional layers, 6 up / down sampling processes, and 2 pooling layers. In the figure, m represents the number of channels, which is 256 by default. The specific fusion steps are:

[0024] Step 1-2-1: For the input feature map F of the adjacent feature extraction branch n-1 、F n and F n+1 , convolution, batch normalization, and ReLU activation function operations are used on the feature map to achieve channel data dimensionality reduction. The formula is written as:

[0025] F i 1 =ReLU(BN(conv(F i ))),i=n-1,n,n+1

[0026] In the formula, conv, BN and ReLU represent convolution, batch normalization and ReLU activation function operations respectively; F i 1 Represents the feature map obtained by the first processing of the i-th feature extraction branch in the IIM module.

[0027] Step 1-2-2: Upsampling and downsampling are used to keep the dimensions of feature maps at different levels consistent, and then the feature maps of adjacent branches are fused to enhance features at different levels. The formula is written as:

[0028]

[0029]

[0030]

[0031] In the formula, Up and Down represent upsampling and downsampling operations respectively; F i 2 represents the feature map obtained by the second processing of the i-th feature extraction branch. Similarly, and They represent the feature maps obtained after processing the i-1th and i+1th branches respectively.

[0032] Step 1-2-3: To improve the efficiency of network optimization, a residual structure is used to skip-connect the enhanced multi-level features and input features. The formula is written as:

[0033]

[0034] F i 4 =ReLU(BN(F i 3 )+F i 1 )

[0035] In the formula, F i 3 and F i 4 They represent the feature maps obtained by the third and fourth processing of the i-th branch respectively.

[0036] Step 1-2-4: Use the channel attention strategy to adjust the weights of the fused features to enhance the effective edge details and semantic information in the feature map. The formula is written as:

[0037] F i output =ReLU(MLP(AvgPool(F i 4))+MLP(MaxPool(F i 4 )))

[0038] Where AvgPool, MaxPool and MLP represent average pooling, maximum pooling and multi-layer perceptron processing respectively; F i output Represents the feature map of the final output of IIM.

[0039] Step 1-3: After the inter-layer feature aggregation module, in order to further aggregate the multi-scale information of the target, the intra-layer self-mining module (ISM) is used to extract the feature information of different receptive fields in the feature map of the same branch to better capture the scale change of the target. The network diagram is shown in the figure. Figure 2 (b) This module designs the information aggregation structure between different receptive fields of the same branch, and establishes a 1×1 convolution cross-layer connection to clarify the direction of feature learning and subsequent network parameter training. Considering that the contribution of each region in the feature map to the application task is different, only the task-related regions need to be paid attention to. It innovatively combines spatial pooling to generate weights of different spatial positions in the fused feature map, emphasizing the importance of significant regions in the fused feature map. The specific structure is as follows: Figure 3 As shown in (b), it mainly includes 6 3×3 convolution layers, 2 1×1 convolution layers, and 6 pooling layers. The specific steps are:

[0040] Step 1-3-1: If the input feature map is denoted as F j , dilated convolution with dilation rates n of 3, 6, and 9 are used to extract multi-scale features in the feature map respectively. The formula is written as:

[0041]

[0042] Where, dconv n represents a dilated convolution with a dilation rate of n; It represents the feature map obtained after the j-th channel in the ISM module is processed by the dilation rate n of the hole convolution.

[0043] Step 1-3-2: Use spatial attention to enhance the spatial detail area and suppress the scene false alarm interference. The formula is written as:

[0044]

[0045] In the formula, concat is the channel concatenation operation; Represents the feature map obtained by the second processing of the j-th channel.

[0046] Step 1-3-3: Use 1×1 convolution to fuse multi-scale features. At the same time, in order to retain the original resolution feature map information as much as possible, introduce a 1×1 convolution branch to emphasize the influence of the original image. The formula is written as:

[0047]

[0048] In the formula, It represents the feature map of the final output of the ISM module.

[0049] Step 1-4: Starting from the 32-fold downsampling level, the features extracted by the inter-layer feature aggregation module and the intra-layer feature self-mining module are combined in sequence, and the feature maps F5', F4', F3', F2', and F1' are obtained layer by layer through upsampling. Among them, F1' is the final output result map of the neighboring complementary saliency extraction network. This feature map contains the extracted suspected target salient area, which serves as the input of step 2.

[0050] Step 2: After obtaining the salient area slices of the suspected target, a discriminant classifier needs to be designed to effectively distinguish different types of ships. Considering that direct feature extraction of different slice images is prone to insufficient feature recognition ability, that is, the extracted features may be effective for certain specific targets, but difficult to distinguish other types of targets. Therefore, the idea of ​​direct feature extraction has certain limitations in the task of multi-class ship recognition. Aiming at the problem of multi-class ship recognition in complex port scenarios, a multi-class ship classifier design strategy based on sparse representation (SR) is adopted. The basic idea of ​​this method is to construct a mathematical representation model based on the sparse structure of the solution to achieve signal decomposition under sparse regularization constraints. This type of method is recognized to have excellent robustness in the characterization of target characteristics under complex environmental conditions (such as occlusion and low contrast). In addition, this method is easy to introduce prior information of the target, which is conducive to improving the accuracy of recognition. Based on the advantages of the above sparse representation theory, new regularization constraints are designed in combination with port application scenarios to achieve accurate recognition of multiple ship types in complex environments. By combining the sparse representation classifier with multiple prior information of the target, the false alarms that may be generated by the saliency extraction network are suppressed, and the fine-grained recognition of ship targets in complex scenes is achieved. The specific steps are as follows:

[0051] Step 2-1: First, perform singular value decomposition on the slices of the ship's salient area. Take the typical optical satellite Worldview-3 image as an example (e.g. Figure 4 (a) Figure 4(b) is the singular value decomposition result of the ship category. It can be seen that the tensor singular value decomposition curve of the ship target in the typical remote sensing image drops rapidly to 0, which indicates that there is a certain information redundancy in the ship target slice, that is, the representation vector of the ship target has a low rank. In addition, considering that the docking of the ship itself may lead to complex situations such as dense arrangement, occlusion, and shadow, it is necessary to propose an intra-class difference regularization term to describe the differences between samples of the same type.

[0052] A sparse representation model of multiple priors of the fusion target is established, and the regularization term includes the reconstruction error regularization term t err , the low-rank regularization term t of the slice features of the ship's salient regions lr , the target intra-class difference regularization term t used to characterize the characteristics of ships under conditions such as occlusion and illumination changes id , the multi-prior sparse representation model is written as:

[0053]

[0054] In the formula, x i is the i-th sample, B is the dictionary matrix, α i is the sparse representation coefficient of the i-th sample, λ1, λ2, λ3 and λ4 are weight coefficients used to adjust the influence of each part.

[0055] Among them, the first-order nuclear norm is used to constrain the low-rank nature of the ship characterization coefficients, and in order to characterize the differences between similar samples, the regularization term t id It is expressed as:

[0056]

[0057] In the formula, n represents the number of samples of the i-th type of ships, C i1 represents the representation coefficient of the first sample of the i-th category, C ix represents the representation coefficient of all samples in the i-th category except the first one, μ c shows the class center of the i-th class representation coefficient. ||·|| F represents the Frobenius norm, ||·||2 represents the 2-norm, and ε and β are regularization terms t id The balance coefficient of the two sub-items in . The distance between the class center of the sample representation coefficient and the correct class coefficient is made as small as possible, and the distance between the characteristics of different samples of the same category is made as large as possible, so that the representation model can learn the inter-class differences of the target.

[0058] Step 2-2: Based on the training set image data, the unknown variables in the multi-prior sparse representation model are solved by the alternating direction multiplier method algorithm.

[0059] Step 2-3: For the suspected target candidate region obtained in step 1, extract the image slice x through the multi-scale sliding window S = [1,2,...s]S , the final ship recognition result is obtained by calculating the minimum classification error, and the formula is recorded as:

[0060]

[0061] In the formula, is the predicted category, ψ is the established ship characterization model, e k Represents the classification error of the classification model predicted as the k-th category target.

[0062] Figure 5 and Figure 6 The recognition results of typical large and medium-sized ships in the HRSC2016 public dataset images and Worldview-3 commercial satellite images are given respectively, where the two groups of example images in the HRSC2016 dataset are images under the interference of light and shadow and complex port facilities, and the two groups of example images in Worldview-3 are optical images under the conditions of cloud occlusion and complex sea clutter interference. It can be seen from the recognition results that in complex sea and port scenes, the recognition method proposed in the present invention can correctly identify typical ship targets such as the Arleigh Burke destroyer, the Perry frigate and the Ticonderoga cruiser without generating false alarms.

Claims

1. A target recognition method based on neighbor-branch complementary saliency and multiple prior sparse representations, characterized in that The method comprises the following steps: Step 1: Propose a neighbor-branch complementary saliency extraction network to mine deep-level, semantically consistent information of the image and extract candidate salient regions of multi-scale objects. The neighbor-branch complementary saliency extraction network includes an inter-layer feature aggregation module and an intra-layer feature self-mining module. The specific steps are as follows: Step 1-1: For the input remote sensing image, the ResNet-101 network is used as the backbone feature extraction network to extract multi-level feature maps with downsampling levels of 2, 4, 8, 16, and 32 times, layer by layer, which are recorded as F1, F2, F3, F4, and F5 respectively; Step 1-2: Use the inter-layer feature aggregation module to fuse the feature maps of adjacent layers to enhance the edge details and semantic information of different feature extraction branches; Step 1-3: After the inter-layer feature aggregation module, in order to further aggregate the multi-scale information of the target, the intra-layer feature self-mining module is used to extract the feature information of different receptive fields in the feature map of the same branch to better capture the scale change of the target; Step 1-4: Starting from the 32-fold downsampling level, the features extracted by the inter-layer feature aggregation module and the intra-layer feature self-mining module are sequentially combined, and the feature maps F5', F4', F3', F2', and F1' are obtained layer by layer through upsampling. Among them, F1' is the final output result map of the neighboring complementary saliency extraction network. This feature map contains the extracted suspected target salient area, which is used as the input of step 2; Step 2: By combining the sparse representation classifier with multiple prior information of the target, the false alarms that may be generated by the saliency extraction network are suppressed to achieve accurate recognition of ship targets in complex scenes. The specific steps are as follows: Step 2-1: Establish a sparse representation model that integrates multiple priors of the target. The regularization term includes the reconstruction error regularization term t err , the low-rank regularization term t of the slice features of the ship's salient regions lr , the target intra-class difference regularization term t used to characterize the characteristics of ships under occlusion and illumination changes id , the multi-prior sparse representation model is written as: In the formula, x i is the i-th sample, B is the dictionary matrix, α i is the sparse representation coefficient of the i-th sample, λ1, λ2, λ3 and λ4 are weight coefficients used to adjust the influence of each part; Step 2-2: Based on the training set image data, the unknown variables in the multi-prior sparse representation model are solved by the alternating direction multiplier method algorithm; Step 2-3: For the suspected target candidate region obtained in step 1, extract the image slice x through the multi-scale sliding window S = [1,2,...s] S , the final ship recognition result is obtained by calculating the minimum classification error, and the formula is recorded as: In the formula, is the predicted category, ψ is the established ship characterization model, e k Represents the classification error of the classification model predicted as the k-th category target.

2. The target recognition method based on adjacent branch complementary saliency and multiple prior sparse representation according to claim 1 is characterized in that The specific steps of steps 1-2 are as follows: Step 1-2-1: For the input feature map F of the adjacent feature extraction branch n-1 、F n and F n+1 , convolution, batch normalization, and ReLU activation function operations are used on the feature map to achieve channel data dimensionality reduction. The formula is written as: F i 1 =ReLU(BN(conv(F i ))),i=n-1,n,n+1 In the formula, conv, BN and ReLU represent convolution, batch normalization and ReLU activation function operations respectively; F i 1 It represents the features obtained by the first processing of the i-th feature extraction branch in the IIM module; Step 1-2-2: Upsampling and downsampling are used to keep the dimensions of feature maps at different levels consistent, and then the feature maps of adjacent branches are fused to enhance features at different levels. The formula is written as: In the formula, Up and Down represent upsampling and downsampling operations respectively; F i 2 It represents the feature map obtained by the second processing of the i-th feature extraction branch; and Respectively represent the feature maps obtained after processing the i-1th and i+1th branches; Step 1-2-3: To improve the efficiency of network optimization, a residual structure is used to skip-connect the enhanced multi-level features and input features. The formula is written as: F i 4 =ReLU(BN(F i 3 )+F i 1 ) In the formula, F i 3 and F i 4 They represent the feature maps obtained by the third and fourth processing of the i-th branch respectively; Step 1-2-4: Use the channel attention strategy to adjust the weights of the fused features to enhance the effective edge details and semantic information in the feature map. The formula is written as: F i output =ReLU(MLP(AvgPool(F i 4 ))+MLP(MaxPool(F i 4 ))) Where AvgPool, MaxPool and MLP represent average pooling, maximum pooling and multi-layer perceptron processing respectively; F i output Represents the feature map of the final output of IIM.

3. The target recognition method based on adjacent branch complementary saliency and multiple prior sparse representations according to claim 1 is characterized in that The specific steps of steps 1-3 are as follows: Step 1-3-1: If the input feature map is denoted as F j , dilated convolution with dilation rates n of 3, 6, and 9 are used to extract multi-scale features in the feature map respectively. The formula is written as: Where, dconv n represents a dilated convolution with a dilation rate of n; It represents the feature map obtained after the j-th channel in the ISM module is processed by the dilated convolution with a dilation rate of n; Step 1-3-2: Use spatial attention to enhance the spatial detail area and suppress the scene false alarm interference. The formula is written as: In the formula, concat is the channel concatenation operation; Represents the feature map obtained by the second processing of the j-th channel; Step 1-3-3: Use 1×1 convolution to fuse multi-scale features. At the same time, in order to retain the original resolution feature map information as much as possible, introduce a 1×1 convolution branch to emphasize the influence of the original image. The formula is written as: In the formula, It represents the feature map of the final output of the ISM module.

4. The target recognition method based on adjacent branch complementary saliency and multiple prior sparse representations according to claim 1 is characterized in that The regularization term t id It is expressed as: In the formula, n represents the number of samples of the i-th type of ships, C i1 represents the representation coefficient of the first sample of the i-th category, C ix represents the representation coefficient of all samples in the i-th category except the first one, μ c represents the class center of the i-th class representation coefficient; ||·|| F represents the Frobenius norm, ||·||2 represents the 2-norm, and ε and β are regularization terms t id The balance coefficient of the two sub-items in .

Images