Target recognition method and device based on double-flow symmetric feature fusion network model

Abstract

The embodiment of the application discloses a target recognition method and device based on a double-flow symmetric feature fusion network model. The method comprises the following steps: inputting a visible light image and an infrared image into a double-flow symmetric feature fusion network with an attention mechanism to extract features, so as to obtain visible light image features and infrared image features; calling a recognition model according to the visible light image features and the infrared image features to test the features, and outputting a prediction label; using a multi-level feature fusion module in the double-flow symmetric feature fusion network with the attention mechanism to enhance key information in the visible light image features and the infrared image features, performing multi-level cascade average fusion on the enhanced feature expressions, and obtaining a fusion result; and using a recognition module to recognize the fusion result to obtain a recognition result of a target image. The application solves the problem of imaging effect defects of infrared images or visible light images in different environmental conditions in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer technology application, and particularly relates to a target recognition method and device based on a double-flow symmetric feature fusion network model. BACKGROUND

[0002] At present, countries all over the world list the ocean as a strategic development field, and the ocean target recognition technology has good application prospects in both military and civilian fields. As an important carrier in many fields such as marine transportation, fishing, ocean resource development, military reconnaissance, etc., the recognition task of the ship has important research value.

[0003] In recent years, ship target recognition has become a research hotspot in the field of pattern recognition. According to the current research status of ship target recognition at home and abroad, the ship images involved are mainly divided into synthetic aperture radar images, infrared images, visible light images, etc. according to the information source. Synthetic aperture radar is not affected by external environment such as weather and light, and can obtain all-weather long-distance sea ship images, but due to speckle noise, its recognition accuracy is poor and more false recognitions are prone to occur; the acquisition of infrared ship images also does not depend on external light and has all-weather characteristics, but the infrared ship image generally shows less detailed information of the ship, most of the infrared ship images have low resolution and cannot reflect the color information of the target; in the near sea area, the visible light ship image has high resolution and low acquisition cost, compared with infrared images and synthetic aperture radar images, it can directly show the detailed information of the ship target, such as the texture pattern, edge contour and color module of the target, so as to facilitate the direct observation and analysis of the human eye and make judgments, but the imaging effect is poor at night and in extreme weather conditions.

[0004] The problem of imaging effect defects of infrared images or visible light images in different environmental conditions in the prior art has not been effectively solved. SUMMARY

[0005] The embodiment of the present application provides a target recognition method and device based on a double-flow symmetric feature fusion network model, which at least solves the problem of imaging effect defects of infrared images or visible light images in different environmental conditions in the related art.

[0006] According to an aspect of the embodiment of the present application, a target recognition method based on a double-flow symmetric feature fusion network model is provided, which comprises: performing the same preprocessing on visible light images and infrared images of the same label to obtain preprocessed visible light images and infrared images; inputting the preprocessed visible light images and infrared images into a double-flow symmetric feature fusion network introducing an attention mechanism to extract features, to obtain visible light image features and infrared image features; calling a recognition model for feature testing according to the visible light image features and the infrared image features, and outputting the predicted label of the target image.

[0007] The dual-flow symmetric feature fusion network with attention mechanism comprises a feature extraction module, a multi-level feature fusion module, and an identification module, wherein the feature extraction module is configured to extract visible light image features of a preprocessed visible light image and infrared image features of an infrared image; the multi-level feature fusion module is configured to enhance key information in the visible light image features and the infrared image features, and perform multi-level cascaded average fusion on the enhanced feature expressions to obtain a fusion result; and the identification module is configured to identify the fusion result to obtain an identification result of the target image.

[0008] Optionally, the identification module comprises three fully connected layers and a Softmax output layer, and the identification result is obtained through the Softmax output layer.

[0009] Optionally, the method further comprises training the identification model according to the dual-flow symmetric feature fusion network and training samples, wherein the training of the identification model according to the dual-flow symmetric feature fusion network and the training samples comprises: performing feature extraction on the obtained visible light image training samples and infrared image training samples through the dual-flow symmetric feature fusion network to obtain visible light image features and infrared image features; integrating the visible light image features and the infrared image features through the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features; performing multi-level fusion on the feature map to obtain a fused image; training the fused image through a feature fusion layer and a fully connected layer using a stochastic gradient descent optimizer solver according to pre-trained weights, calculating the error between a real label and a predicted class label; and adjusting the weights and the error through a back propagation process to obtain the identification model.

[0010] Further, the feature extraction on the obtained visible light image training samples and infrared image training samples through the dual-flow symmetric feature fusion network comprises: obtaining the visible light image training samples and the infrared image training samples; and performing feature extraction on the visible light image training samples and the infrared image training samples through a feature extraction module to obtain the visible light image features and the infrared image features.

[0011] Optionally, the feature extraction channels of the visible light image training samples and the infrared image training samples, wherein each feature extraction channel comprises five convolution layers and five pooling layers.

[0012] Optionally, the visible light image features and the infrared image features are integrated by the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features, including: the spatial attention module uses a maximum pooling and a mean pooling operation to perform compression operation processing on the input visible light image features and infrared image features to obtain compressed visible light image features and infrared image features; the compressed visible light image features and infrared image features are compressed at a channel level to obtain the feature map.

[0013] Optionally, the feature map is subjected to multi-level fusion to obtain a fused image, including: the feature map is gradually reduced in resolution and increased in the number of channels of the feature map when passing through the third layer and the fifth layer and the pooling layer to obtain the fused image.

[0014] According to another aspect of the embodiment of the present application, a dual-flow symmetric feature fusion network model target recognition device is provided, including: a preprocessing module configured to perform the same preprocessing on visible light images and infrared images of the same label to obtain preprocessed visible light images and infrared images; a feature extraction module configured to input the preprocessed visible light images and infrared images into a dual-flow symmetric feature fusion network with attention mechanism to extract features to obtain visible light image features and infrared image features; and a recognition module configured to call a recognition model to perform feature testing according to the visible light image features and the infrared image features and output a predicted label of a target image; wherein the dual-flow symmetric feature fusion network with attention mechanism includes: the feature extraction module, a multi-level feature fusion module, and the recognition module, wherein the feature extraction module is configured to extract visible light image features of the preprocessed visible light images and infrared image features of the infrared images; the multi-level feature fusion module is configured to enhance key information in the visible light image features and the infrared image features and perform multi-level cascaded average fusion on the enhanced feature expressions to obtain a fusion result; and the recognition module is configured to recognize the fusion result to obtain a recognition result of the target image.

[0015] In the embodiment of the present application, the same preprocessing is performed on visible light images and infrared images of the same label to obtain preprocessed visible light images and infrared images; the preprocessed visible light images and infrared images are input into a dual-flow symmetric feature fusion network with attention mechanism to extract features to obtain visible light image features and infrared image features; and a recognition model is called to perform feature testing according to the visible light image features and the infrared image features and a predicted label of a target image is output. That is, the embodiment of the present application can solve the problem of imaging effect defects of infrared images or visible light images under different environmental conditions, thereby achieving the technical effect of effectively utilizing complementary information in dual-flow symmetric features extracted from visible light and infrared modalities, and further obtaining a more comprehensive ship feature description. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0017] Figure 1 A flowchart of a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0018] Figure 2 A schematic diagram of the overall architecture of a double-flow symmetric feature fusion network based on an attention mechanism in a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0019] Figure 3 A SAM module construction diagram in a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0020] Figure 4 A SAM module deployment diagram in a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0021] Figure 5 A working flowchart of a fusion layer Fuse_layer1 in a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0022] Figure 6 A CAM module construction diagram in a target recognition method based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure.

[0023] Figure 7 A schematic diagram of a target recognition device based on a double-flow symmetric feature fusion network model provided for an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0024] In order to enable persons skilled in the art to better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the application.

[0025] It should be noted that the terms “first”, “second”, etc. in the specification and claims and the drawings of the application are used to distinguish different objects, rather than to limit a specific order.

[0026] According to an aspect of the embodiments of the present application, a target recognition method based on a dual-flow symmetric feature fusion network model is provided, Figure 1 A flowchart of the target recognition method based on the dual-flow symmetric feature fusion network model is shown in FIG. 1. Figure 1 As shown in FIG. 1, the target recognition method based on the dual-flow symmetric feature fusion network model includes the following steps:

[0027] In step S102, the same preprocessing is performed on the visible light image and the infrared image of the same label to obtain the preprocessed visible light image and infrared image.

[0028] In step S104, the preprocessed visible light image and infrared image are input into the dual-flow symmetric feature fusion network with attention mechanism to extract features, and the visible light image features and infrared image features are obtained.

[0029] In step S106, the visible light image features and infrared image features are used to call the recognition model for feature testing, and the predicted label of the target image is output.

[0030] The dual-flow symmetric feature fusion network with attention mechanism includes a feature extraction module, a multi-level feature fusion module, and a recognition module. The feature extraction module is used to extract the visible light image features of the preprocessed visible light image and the infrared image features of the infrared image. The multi-level feature fusion module is used to enhance the key information in the visible light image features and the infrared image features, and to perform multi-level cascaded average fusion on the enhanced feature expression to obtain a fusion result. The recognition module is used to recognize the fusion result to obtain the recognition result of the target image.

[0031] Optionally, the recognition module is composed of three fully connected layers and a Softmax output layer, and the recognition result is obtained through the Softmax output layer.

[0032] Optionally, the target recognition method based on the dual-flow symmetric feature fusion network model further includes: training the recognition model according to the dual-flow symmetric feature fusion network and training samples, wherein the training of the recognition model according to the dual-flow symmetric feature fusion network and the training samples includes: performing feature extraction on the acquired visible light image training samples and infrared image training samples through the dual-flow symmetric feature fusion network to obtain visible light image features and infrared image features; integrating the visible light image features and the infrared image features through the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features; performing multi-level fusion on the feature map to obtain a fused image; training the fused image through a feature fusion layer and a fully connected layer using a stochastic gradient descent optimizer solver according to pre-trained weights, calculating errors of real labels and predicted class labels; and adjusting the weights and the errors through a back propagation process to obtain the recognition model.

[0033] Further, the feature extraction on the acquired visible light image training samples and infrared image training samples through the dual-flow symmetric feature fusion network to obtain visible light image features and infrared image features includes: acquiring the visible light image training samples and the infrared image training samples; and performing feature extraction on the visible light image training samples and the infrared image training samples through a feature extraction module to obtain the visible light image features and the infrared image features.

[0034] Optionally, the feature extraction channels of the visible light image training samples and the feature extraction channels of the infrared image training samples, wherein the feature extraction channels are composed of five convolution layers and five pooling layers.

[0035] Optionally, the integration of the visible light image features and the infrared image features through the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features includes: performing compression operation processing on the input visible light image features and infrared image features using maximum pooling and mean pooling operations through the spatial attention module to obtain compressed visible light image features and infrared image features; and performing compression operation processing on the compressed visible light image features and infrared image features at a channel level to obtain the feature map.

[0036] Optionally, the multi-level fusion of the feature map to obtain a fused image includes: gradually reducing the resolution and increasing the number of channels of the feature map when passing through the third layer and the fifth layer and the pooling layer to obtain the fused image.

[0037] In summary, Figure 2 A schematic diagram of the overall architecture of the dual-flow symmetric feature fusion network based on the attention mechanism in the target recognition method based on the dual-flow symmetric feature fusion network model is provided for the embodiments of the present application, as shown in Figure 2As shown in the figure, the ship target recognition method based on the dual-flow symmetric feature fusion network model of the application comprises the following specific steps:

[0038] Step 1, feature extraction

[0039] The registered visible light image and thermal infrared image pair is input into the feature extraction module in parallel as a training sample. The feature extraction module structure is composed of two convolution network channels for operating on the visible light and infrared thermal images respectively. Each feature extraction channel is built by five convolution layers (Conv1-Conv5) and five pooling layers (Max_pool1-Max_pool5).

[0040] Step 2, introducing an attention mechanism module

[0041] The SAM module is deployed to the feature extraction network convolution Conv3 and Conv5 output, and the obtained visible light and infrared image features are integrated.

[0042] Figure 3 A SAM module construction diagram in the target recognition method based on the dual-flow symmetric feature fusion network model provided by the embodiment of the application is shown in Figure 4 A SAM module deployment diagram in the target recognition method based on the dual-flow symmetric feature fusion network model provided by the embodiment of the application is shown in Figure 3 and Figure 4 As shown in the figure, the SAM module first uses the maximum pooling and mean pooling operations to compress the input feature map, and then performs a compression operation on the output feature map at the channel level, that is, two two-dimensional feature vectors are generated through two pooling operations, and are spliced through the Concat operation, and then the feature map with a channel number of 2 obtained by splicing is input into the NN (Network in Network) network for fusion, and the convolution is used to ensure that the output dimension is consistent with the input dimension.

[0043] Step 3, multi-level feature fusion

[0044] The original image size input into the two feature extraction channels is 224x224, which is gradually reduced in resolution and increased in the number of feature map channels after passing through the five convolution layers and the pooling layers. The feature maps extracted by different convolution layers of the VGG16 network correspond to different levels of semantic information and visual details, and the feature fusion architecture built based on the feature maps at different levels will directly affect the model recognition performance. The feature maps C1 and C2 output by the convolution layers Conv1 and Conv2 mainly capture low-level visual features such as color and line segment; the feature maps C3-C5 output by the convolution layers Conv3, Conv4 and Conv5 contain more semantic information than the shallow feature maps C1 and C2, and also retain part of the visual details.

[0045] Since the feature expressions obtained by different channels of the same volume layer are complementary to each other in describing the ship image, the feature maps of all channels output by the layer can be superimposed to obtain an overall feature map by a 1:1 ratio. Figure 5 The working flow chart of the fusion layer Fuse_layer1 in the target recognition method based on the dual-flow symmetric feature fusion network model provided by the embodiment of the application is as shown in Figure 5 Fuse_layer1 is set after the visible light feature extraction network convolution layer Conv3, feature fusion is realized for the two feature maps output by the dual-flow channel convolution layer Conv3, and the fused feature map continues to the next level of feature extraction, and Fuse_layer2 is set after the convolution layer Conv5.

[0046] Specifically, the cascaded average fusion algorithm is used to first cascade all the feature maps, and then perform 1x1 convolution operation as the fusion result. The algorithm can more reasonably retain all the elements of the input feature map, can be applied to feature maps of any dimension, and keeps the size of the fused feature map unchanged. Finally, a target recognition module is constructed by using three-level fully connected layers, and in order to prevent network overfitting, the Dropout technology is introduced in the three-level fully connected layers, and the fusion feature map Feature_fuse is output by the Softmax activation function through the fully connected layer to output the result probability of recognition. The parameters of the dual-flow symmetric feature fusion network finally constructed are as shown in Table 1.

[0047] Table 1: Overall parameter table of dual-flow symmetric feature fusion network

[0048]

[0049]

[0050] Step 4: training the network model and testing

[0051] In the training phase, the feature extraction network module uses the pre-trained weights of VGG16 to train, the feature fusion layer and the fully connected layer are trained using the stochastic gradient descent (SGD) optimization solver, and then the error of the real label and the predicted class label obtained by the Softmax function is calculated. Then, the weights and biases are adjusted through the back propagation process to minimize the error and save the optimal model.

[0052] In the test phase, the visible light image and the infrared image under the same label are also preprocessed, input into the dual-flow symmetric feature fusion network to extract features, and the optimal model is called to test the features, and the predicted label of the ship image is output.

[0053] The effects of the application are further illustrated by the following simulation experiment:

[0054] The multispectral ship data used in this experiment comes from the VAIS dataset, which is the only publicly available dataset of paired visible and long-wave infrared ship images. The visible light camera used in this dataset is ISVI IC-C25; the long-wave infrared camera used is Sofradir-EC Atom 1024, with a spectral range of 8-12 microns. The dataset includes a total of 2865 images (1623 visible light images and 1242 infrared images), including 1088 pairs of visible light and infrared images. The dataset can be divided into 6 coarse-grained categories, namely "cargo", "medium-other", "passenger", "sailing", "tug" and "small". Among them, the size of the visible light image ranges from 644 to 4478952 pixels, with an average of 181319 pixels and a median of 9983 pixels; the size of the infrared image ranges from 594 to 137240 pixels, with an average of 8544 pixels and a median of 1610 pixels. The present experiment only selects 1088 pairs of registered visible light and infrared images for experiment. The training and test samples are divided in the manner specified by the dataset author's literature, and the information of the divided training and test samples is shown in Table 2.

[0055] Table 2 Information of VAIS dataset

[0056]

[0057] The ship recognition evaluation index adopted in the present application includes recognition accuracy Acc and average model building time consumption ATPI (Average Model Building Time Consumption Per Image). Among them, Acc is defined as the ratio of correctly recognized samples to the total number of samples, the higher the ratio, the better the recognition performance, which can be expressed by the formula:

[0058]

[0059] Among them, TP, FP, TN and FN represent the number of positive samples identified as positive samples, negative samples incorrectly identified as positive samples, negative samples identified as negative samples and positive samples incorrectly identified as negative samples, respectively.

[0060] The experimental platform processor of the application is Intel(R) Core(TM) i9-10900K CPU, the graphic processor for deep CNN calculation is NVIDIA GeForce RTX 2080, and the calculation environment is Keras environment with a TensorFlow back end. The parameter settings are as follows: the model is trained in a small batch sample mode, the batch parameter is set to 16, the training iteration step number is set to 6000; the model learning rate is set to 0.0025, the SGD algorithm is used to optimize the loss function, and the impulse parameter is set to 0.9; the Dropout parameters in the full connection layer are all set to 0.5.

[0061] In addition, considering that the existing VAIS data is limited in quantity and the data quantity of each category of the original data set is unbalanced, before model training, the training data set is subjected to data enhancement in the manner of random flipping, random rotation, transformation of saturation and contrast, etc., so as to improve the model generalization capability and prevent model overfitting. The training sample information after data enhancement is shown in Table 3.

[0062] Table 3 Information of training set after VAIS data enhancement

[0063]

[0064] The verification of the experimental effect of the application is carried out from two aspects: one is the verification of the core of the model: the performance of the multi-level fusion module based on the spatial attention mechanism; and the other is the verification of the recognition accuracy of the model algorithm proposed by the application on each category of the data set. When verifying the algorithm, the performance of the model and the baseline method, other feature fusion methods and other advanced multi-source feature fusion models proposed in recent years are compared.

[0065] The core of the dual-flow symmetric feature fusion network model based on the attention mechanism is the multi-level fusion module based on the spatial attention mechanism. In order to verify the performance of the model fusion module, Table 4 compares the recognition performance of the baseline method, other feature fusion methods and the feature fusion method proposed by the application on the VAIS data set based on two evaluation indexes. Among them, the baseline method selected by the application is the VGG16 algorithm, the infrared and visible light images are input into the dual-flow network in parallel for feature extraction, TS_CFF is defined as the recognition network obtained by constructing a single-level fusion layer Fuse_layer2 after the convolution layer Conv5, TS_MCFF is defined as the recognition network obtained by constructing a multi-level fusion layer after the convolution layers Conv3 and Conv5 respectively, TS_CBAM_MCFF is defined as the recognition network obtained by introducing the CBAM attention mechanism into the multi-level feature fusion module, and TS_SAM_MCFF is defined as the recognition network obtained by introducing the SAM spatial attention mechanism into the feature fusion module, that is, the final fusion recognition network model constructed by the application.

[0066] Table 4 identification performance of baseline algorithm, other fusion algorithm and the fusion algorithm proposed in the application on the VAIS dataset

[0067]

[0068] As can be seen from Table 4, the visible light image recognition accuracy of the baseline algorithm is higher than that of the infrared image, mainly because the infrared image contains less detailed texture information; compared with using single modal image for target recognition, the recognition accuracy of all multi-modal image feature fusion algorithms in the table has been greatly improved, and the average model construction time consumption ATPI of each image has also increased within an acceptable range, so it can be confirmed that the visible light and infrared image feature level fusion can effectively improve the ship recognition model recognition accuracy.

[0069] Comparing the Acc and ATPI indicators of TS_CFF and TS_MCFF algorithms on the ship target dataset, it is obvious that under the condition of approximately the same ATPI, the recognition accuracy of TS_MCFF algorithm is significantly improved compared with TS_CFF algorithm, and it can be seen that multi-level fusion can enhance the response of the key region of the fusion feature map, so the algorithm selects the idea of retaining multi-level fusion, and introduces the attention mechanism module to further optimize the algorithm.

[0070] Comparing the recognition accuracy and ATPI consumption of TS_MCFF algorithm, TS_CBAM_MCFF algorithm and TS_SAM_MCFF algorithm on the experimental data respectively, it can be analyzed that for the double-flow symmetric feature fusion algorithm with relatively complex network architecture, introducing CBAM mechanism in the fusion module can improve the recognition accuracy, but the CAM module in CBAM mechanism almost has no help to improve the model recognition performance, and can only bring structural redundancy and increase APTI time consumption. In contrast, only introducing SAM spatial attention mechanism can improve the algorithm model target recognition accuracy while reducing ATPI, which is mainly because the SAM module can strengthen the learning of the key position of the feature map space, which will be more conducive to the model to obtain the distinguishing features of the ship image.

[0071] In order to verify the recognition performance of the TS_SAM_MCFF model algorithm proposed in the application on each category of the VAIS dataset, Table 5 compares the recognition accuracy of the baseline method, other feature fusion method and the feature fusion method proposed in the application on each category of the VAIS dataset.

[0072] As shown in Table 5, compared with other methods, the proposed TS_SAM_MCFF network model shows a more excellent recognition effect. Overall, the method has good recognition effect on "cargo", "small", "sailing", especially for "sailing" and "cargo", the recognition accuracy is more than 97%, because the difference between the two types of ships and other ship types is larger. In summary, the TS_SAM_MCFF network model proposed in the application achieves the best overall recognition accuracy on the actual VAIS dataset.

[0073] Table 5 Recognition accuracy of baseline algorithm, other fusion algorithm and the proposed fusion algorithm on each category of VAIS dataset

[0074]

[0075] Finally, the TS_SAM_MCFF model is compared with other advanced multi-source feature fusion models proposed in recent years to further verify the recognition performance of TS_SAM_MCFF. The comparison methods include four multi-source feature fusion recognition models based on VAIS dataset: 1) Gnostic field, 2) CNN, 3) Gnostic field+CNN, 4) Multimodal CNN, and three visible light image recognition models based on VAIS dataset: 1) VGG16 (baseline), 2) Gabor+CNN, 3) Gabor+MS-CLBP.

[0076] Table 6 Comparison of recognition performance of other advanced recognition algorithms and the proposed feature fusion recognition algorithm on VAIS dataset

[0077]

[0078] As can be seen from Table 6, the TS_SAM_MCFF model algorithm has a fusion recognition accuracy on the visible light and infrared ship image that is improved by 3.61% compared with the visible light ship image recognition accuracy and improved by 13.62% compared with the infrared ship image recognition accuracy, and overall, the recognition accuracy is significantly improved. In addition, compared with the three fusion recognition models of Gnostic field, CNN and MultimodalCNN, the recognition accuracy of TS_SAM_MCFF is obviously improved. Compared with the Gnostic field+CNN model, the single-mode recognition accuracy of TS_SAM_MCFF is improved, and the dual-mode image recognition accuracy is approximately flat. However, the Gnostic field+CNN model is a series combination of four classification models to realize decision-level fusion recognition (that is, the probability output of Gnostic field on infrared image, CNN on infrared image, Gnostic field on visible light image and CNN on visible light image is fused). This model needs to consume a lot of time for training and recognition, while the end-to-end TS_SAM_MCFF model greatly reduces the model construction time while approximately reaching the recognition accuracy of the model, and has better comprehensive performance. At the same time, the recognition accuracy of the TS_SAM_MCFF model on single-mode image is better than all other advanced multispectral recognition algorithms and visible light recognition algorithms.

[0079] Specifically, the application proposes a dual-flow symmetric feature fusion network model based on an attention mechanism. The network model mainly comprises a dual-flow symmetric feature extraction module, a multi-level feature fusion module with an introduced attention mechanism and a recognition module. The feature extraction module uses a dual-flow symmetric convolutional network to extract visible light image features and infrared image features after preprocessing. The fusion module based on the attention mechanism enhances the key information in the visible light and infrared image feature descriptions, and performs multi-level cascading average fusion on the enhanced feature expressions to improve the model feature expression. The recognition module comprises three fully connected layers and a Softmax output layer, and the ship recognition result is obtained through the Softmax output layer.

[0080] The dual-flow symmetric feature fusion model proposed by the application selects 13 convolutional layers of a VGG16 network as the main structure of the feature extraction module, and extracts visible light image features and infrared image features in parallel.

[0081] The fusion model feature extraction module structure provided by the application is composed of two convolution network channels respectively operating on visible light and infrared thermal images. Each feature extraction channel is built by 5 convolution layers (Conv1-Conv5) and 5 pooling layers (Max_pool1-Max_pool5). In order to make full use of the feature descriptions output by different convolution layers of the dual-source image feature extraction channel, the application designs a multi-level fusion module based on an attention mechanism: after the convolution layer Conv3, a SAM spatial attention module (Spatial Attention Module) is introduced, and then a first-level fusion layer Fuse_layer1 is constructed; after the convolution layer Conv5, a SAM module is introduced and a second-level fusion layer Fuse_layer2 is constructed. The feature map output by the fusion layer Fuse_layer1 is input into the visible light feature extraction channel for the next feature extraction step, and the feature map output by the fusion layer Fuse_layer2 is input into the maximum pooling layer Max_pool5 of the convolution kernel for feature dimension reduction, to obtain the final fusion feature map Feature_fuse. Finally, a target recognition module is constructed by using three-level fully connected layers, and in order to prevent network overfitting, the Dropout technology is introduced in the three-level fully connected layers, and the fusion feature map Feature_fuse is output by the Softmax activation function through the fully connected layer to output the recognition result probability.

[0082] The multi-level fusion module adopts a cascaded average fusion algorithm, which is as follows: the expression of feature fusion is defined as:

[0083] F=f(X,Y)

[0084] Wherein, X, Y represent the features to be fused respectively; F represents the fused feature, X, Y, F∈R HWC Wherein, H, W, C represent the height, width and channel number of the feature map respectively.

[0085] The cascaded fusion directly cascades and splices all feature maps as the fusion result, and this fusion method can be applied to feature maps of any dimension. The cascaded fusion can be represented as:

[0086] F=f(X,Y)

[0087] =[x1,x2,...,x n ,y1,y2,...,y n ]

[0088] The cascade fusion can more reasonably reserve all elements of the input feature map, and the application introduces an average module based on the cascade fusion and designs a cascade average fusion method. The cascade average fusion firstly cascades and splices all feature maps, and then performs 1x1 convolution operation to obtain the final fusion feature map. The formula of the cascade average fusion under visible light and infrared images can be expressed as:

[0089] F=f 1×1 ([x1,x2,...,x n ,y1,y2,...,y n ])

[0090] Wherein, f 1×1 is a convolution operation with a convolution kernel size of 1x1 and a number of n.

[0091] In the multi-level feature fusion module designed by the application, the sizes of the two feature maps input into the first level fusion layer are both 56x56x256, the size of the feature map after cascading splicing is 56x56x512, and the size of the output feature map after average fusion is still 56x56x256. The sizes of the two feature maps of the second level fusion layer are both 14x14x512, the size of the feature map after cascading splicing is 14x14x1024, and the size of the output feature map after average fusion is still 14x14x512.

[0092] For the attention mechanism module, the application applies the CBAM attention subnetwork module proposed by Woo et al. The CBAM attention mechanism module is composed of two independent parts in series, Figure 6 A CAM module construction diagram is provided in the target recognition method based on the dual-flow symmetric feature fusion network model of the application, as shown in Figure 6 , that is, a channel attention module CAM (Channel Attention Module) and a spatial attention module SAM (Spatial Attention Module).

[0093] The working principle of the CAM module is explained by the following formula:

[0094]

[0095] Wherein, The symbol represents element point multiplication, and M CAM represents the integration processing in the channel dimension.

[0096] The CAM module is compressed in the spatial dimension, and the idea of maximum pooling is introduced when performing the Squeeze operation, and the maximum response information is used to supplement the mean pooling information.

[0097] The SAM module first uses a maximum pooling and mean pooling operation to compress the input feature map, and then performs a compression operation on the output feature map at the channel level, that is, two two-dimensional feature vectors are generated through two pooling operations, and are spliced through a Concat operation, and then the feature map with a channel number of 2 obtained by splicing is sent into a network in network (NN) network for fusion, and the dimensions of the output are ensured to be consistent with the dimensions of the input through convolution. The specific working principle of the SAM module is as follows:

[0098]

[0099] Wherein, The symbol represents element multiplication, M SAM represents the integration processing in the spatial dimension, and the specific mathematical expression is as follows:

[0100] M SAM =σ(f 3×3 ([AvgPool(F);MaxPool(F)]))

[0101] In the formula, sigma is an activation function, and f 3×3 is a 3*3 convolution operation.

[0102] The application attempts to integrate the CBAM attention mechanism module into the double-flow symmetric feature fusion model basic network, and the recognition performance is improved, but through experiments, it is found that the CAM module has less contribution to the performance improvement of the feature fusion model, but its integration into the model will bring a certain degree of structural redundancy and increase in time consumption, therefore, the application only retains the SAM module and integrates it into the double-flow feature extraction network. The specific deployment position of the SAM module is after the output of the feature extraction network convolution Conv3 and Conv5, the integrated features of the visible light and infrared images are combined with the two-level fusion layers respectively, and the final fusion feature map output by the two times of fusion is input into the recognition network to obtain more accurate ship recognition.

[0103] In the technical field, with the rapid development of deep learning technology, many researchers have introduced deep learning technology into the field of ship target recognition, and the ship recognition technology based on deep convolutional neural network (CNN) has become a research hotspot in this field. Due to the small difference between most ship classes, it is difficult for the target recognition algorithm to distinguish them, especially the complex background of the obtained ship image will bring greater challenges to ship recognition. The visual attention mechanism developed in recent years can focus on important information in the image space rather than all image feature information when processing images by simulating the human visual system. This mechanism can guide the model to learn the image area with important information and help the model to extract key feature information with distinguishing degree in the ship image.

[0104] In summary, the ship image information acquired by the plurality of sensors has certain complementarity, the CNN plays an important role in the ship identification problem, and the application of the attention mechanism can guide the model to learn the image area with important information and help the model to extract the key feature information with distinguishing degree in the ship image. Therefore, the visible light sensor and the infrared sensor are fused to acquire images, and the attention mechanism is introduced into the CNN to identify the ship.

[0105] In the embodiment of the application, the visible light image and the infrared image of the same label are subjected to the same pretreatment to obtain the pretreated visible light image and the pretreated infrared image; the pretreated visible light image and the pretreated infrared image are input into a double-flow symmetric feature fusion network to extract features to obtain visible light image features and infrared image features; and a recognition model is called according to the visible light image features and the infrared image features to test the features and output the predicted label of the target image. That is, the embodiment of the application can solve the problem of imaging effect defects of the infrared image or the visible light image under different environmental conditions, thereby achieving the technical effect of effectively utilizing the complementary information in the double-flow symmetric features extracted from the visible light and the infrared, and further obtaining a more comprehensive ship feature description.

[0106] According to another aspect of the embodiment of the application, there is provided a double-flow symmetric feature fusion network model target identification device, Figure 7 A schematic diagram of the double-flow symmetric feature fusion network model target identification device provided by the embodiment of the application is shown in Figure 7 As shown in the drawing, the double-flow symmetric feature fusion network model target identification device provided by the embodiment of the application comprises: a pretreatment module 72, which is configured to perform the same pretreatment on the visible light image and the infrared image of the same label to obtain the pretreated visible light image and the pretreated infrared image; a feature extraction module 74, which is configured to input the pretreated visible light image and the pretreated infrared image into a double-flow symmetric feature fusion network with an attention mechanism to extract features to obtain visible light image features and infrared image features; and an identification module 76, which is configured to call a recognition model according to the visible light image features and the infrared image features to test the features and output the predicted label of the target image. The double-flow symmetric feature fusion network with the attention mechanism comprises: a feature extraction module, a multi-level feature fusion module, and an identification module. The feature extraction module is configured to extract the visible light image features of the pretreated visible light image and the infrared image features of the infrared image. The multi-level feature fusion module is configured to enhance the key information in the visible light image features and the infrared image features, and perform multi-level cascaded average fusion on the enhanced feature expression to obtain a fusion result. The identification module is configured to identify the fusion result to obtain the identification result of the target image.

[0107] The above merely describes the preferred embodiments of the present application, but is not intended to limit the protection scope of the present application.

Claims

1. A target recognition method based on a dual-stream symmetric feature fusion network model, characterized in that, include: The visible light image and the infrared image of the same label are subjected to the same preprocessing to obtain the preprocessed visible light image and the infrared image; The preprocessed visible light image and the infrared image are input into a two-stream symmetric feature fusion network with an attention mechanism to extract features, thereby obtaining visible light image features and infrared image features. Based on the visible light image features and the infrared image features, the recognition model is invoked to perform feature testing and output the predicted label of the target image; The dual-stream symmetric feature fusion network incorporating an attention mechanism includes: a feature extraction module, a multi-level feature fusion module, and a recognition module. The feature extraction module extracts visible light image features from the preprocessed visible light image and infrared image features from the infrared image. The multi-level feature fusion module enhances key information in the visible light and infrared image features and performs multi-level cascaded averaging fusion of the enhanced feature expressions to obtain a fusion result. The recognition module identifies the fusion result to obtain the recognition result of the target image. The recognition model is trained based on the dual-stream symmetric feature fusion network and training samples. The training of the recognition model includes: The feature extraction module extracts features from the acquired visible light image training samples and infrared image training samples to obtain visible light image features and infrared image features. The visible light image features and the infrared image features are integrated by the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features; The feature maps are fused at multiple levels to obtain the fused image; The fused image is trained using a stochastic gradient descent optimization solver through a feature fusion layer and a fully connected layer based on pre-trained weights, and the error between the true label and the predicted class label is calculated. The recognition model is obtained by adjusting the weights and the error through a backpropagation process. The feature extraction module consists of two convolutional network channels that operate on visible light image training samples and infrared thermal image training samples respectively. Each feature extraction channel is built with 5 convolutional layers and 5 pooling layers. The first two convolutional layers capture low-level visual features including color and line segments, while the last three convolutional layers capture more semantic information and also retain some visual details. The spatial attention module is deployed after the outputs of the third and fifth convolutional layers of the feature extraction module. The spatial attention module first uses max pooling and mean pooling operations to compress the input feature map, and then compresses the output feature map at the channel level. That is, it generates two two-dimensional feature vectors through two pooling operations and concatenates them by a Concat operation. Then, the concatenated feature map with two channels is fed into the NN network for fusion. The convolution method ensures that the dimension of the output is consistent with the dimension of the input. The step of performing multi-level fusion on the feature maps to obtain the fused image includes: The resolution of the feature map is gradually reduced and the number of channels of the feature map is increased when the feature map passes through the third and fifth layers and the pooling layer to obtain the fused image.

2. The target recognition method based on a dual-stream symmetric feature fusion network model according to claim 1, characterized in that, The recognition module consists of three fully connected layers and a Softmax output layer, and the recognition result is obtained through the Softmax output layer.

3. A target recognition device based on a dual-stream symmetric feature fusion network model, characterized in that, include: The preprocessing module is used to perform the same preprocessing on the visible light image and the infrared image of the same label to obtain the preprocessed visible light image and the infrared image; The feature extraction module is used to input the preprocessed visible light image and the infrared image into a two-stream symmetric feature fusion network with an attention mechanism to extract features, thereby obtaining visible light image features and infrared image features; The recognition module is used to call the recognition model to perform feature testing based on the visible light image features and the infrared image features, and output the predicted label of the target image; The dual-stream symmetric feature fusion network incorporating an attention mechanism includes: a feature extraction module, a multi-level feature fusion module, and a recognition module. The feature extraction module extracts visible light image features from the preprocessed visible light image and infrared image features from the infrared image. The multi-level feature fusion module enhances key information in the visible light image features and the infrared image features, and performs multi-level cascaded averaging fusion of the enhanced feature expressions to obtain a fusion result. The recognition module identifies the fusion result to obtain the recognition result of the target image. The recognition model is trained based on the dual-stream symmetric feature fusion network and training samples. The training of the recognition model includes: The feature extraction module extracts features from the acquired visible light image training samples and infrared image training samples to obtain visible light image features and infrared image features. The visible light image features and the infrared image features are integrated by the spatial attention module to obtain a feature map containing the visible light image features and the infrared image features; The feature maps are fused at multiple levels to obtain the fused image; The fused image is trained using a stochastic gradient descent optimization solver through a feature fusion layer and a fully connected layer based on pre-trained weights, and the error between the true label and the predicted class label is calculated. The recognition model is obtained by adjusting the weights and the error through a backpropagation process. The feature extraction module consists of two convolutional network channels that operate on visible light image training samples and infrared thermal image training samples respectively. Each feature extraction channel is built with 5 convolutional layers and 5 pooling layers. The first two convolutional layers capture low-level visual features including color and line segments, while the last three convolutional layers capture more semantic information and also retain some visual details. The spatial attention module is deployed after the outputs of the third and fifth convolutional layers of the feature extraction module. The spatial attention module first uses max pooling and mean pooling operations to compress the input feature map, and then compresses the output feature map at the channel level. That is, it generates two two-dimensional feature vectors through two pooling operations and concatenates them by a Concat operation. Then, the concatenated feature map with two channels is fed into the NN network for fusion. The convolution method ensures that the dimension of the output is consistent with the dimension of the input. The step of performing multi-level fusion on the feature maps to obtain the fused image includes: The resolution of the feature map is gradually reduced and the number of channels of the feature map is increased when the feature map passes through the third and fifth layers and the pooling layer to obtain the fused image.

Images