A method of target detection and recognition based on feature fusion
By introducing a multi-scale feature fusion network and attention suggestion network into the aircraft target recognition algorithm, combining global and local features, the problem of difficult to capture fine-grained features in aircraft target recognition is solved, and the recognition accuracy and automation are improved.
Patent Information
- Application Number
- CN202311351847.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2043-10-18
AI Technical Summary
Existing aircraft target recognition algorithms are difficult to identify aircraft target types in images, especially because the target occupies fewer pixels and has large size differences, resulting in less obvious feature differences and it is difficult to capture fine-grained features, affecting the recognition accuracy.
A target detection and recognition method based on feature fusion is designed. Through a multi-scale feature fusion network and attention suggestion network, combining global and local features, the different scale features of aircraft targets are gradually learned to enhance the recognition ability of aircraft types.
It improves the accuracy and automation of aircraft target recognition, reduces the pressure on identification work, and improves the efficiency and automation level of tower commanders.
Smart Images

Figure CN117197676B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of aircraft target detection and recognition, and in particular to a target detection and recognition method based on feature fusion. Background Art
[0002] Aircraft target detection and type recognition are the basis of intelligent air traffic control and tower command, and are also the key issues in air situation awareness. Rapidly and accurately identifying aircraft targets is of great significance to improving the air surveillance capabilities of aircraft and ensuring the safety of my country's airspace.
[0003] In air traffic control and situational awareness, aircraft pods equipped with radar, visible light or infrared image sensors or ground observation equipment are usually used to obtain aircraft image information. Since the background of aerial images is relatively simple, it is relatively easy to detect aircraft targets; however, the shooting distance of aircraft targets is relatively far, so the targets usually occupy fewer pixels in the image, making it relatively difficult to identify the type of aircraft targets.
[0004] For the task of aircraft target type recognition, deep learning algorithms represented by convolutional neural networks have currently achieved the best results. However, since type-related discriminability is usually concentrated in a relatively small local area in the target image, and the feature differences between different types of aircraft are relatively small, while the feature differences between the same type of aircraft are relatively large, it is difficult for existing algorithms to capture fine-grained features related to the category. On the other hand, due to changes in aircraft size and shooting distance, the collected aircraft targets often have large size differences, further increasing the difficulty of learning discriminability features. These problems restrict the performance improvement and implementation of related algorithms.
[0005] In order to solve the problem that the feature size related to the target type in natural images is usually small and the difference is not obvious, RACNN has achieved stable and good results in fine-grained classification tasks such as bird recognition by setting up three layers of sub-networks to gradually learn the type-related features of key areas. RACNN can learn the key areas related to the type recognition results in the sub-network of this layer, and then crop and enlarge the area and use it as the input of the next layer of network, so as to prompt the model to gradually focus on type-related features. However, since the input of the RACNN algorithm in the classification task of the third layer of sub-network is a local image that has been cropped twice, it lacks global information, which affects the type recognition accuracy of the algorithm.
[0006] The technical solution provided by the present invention is as follows: a target detection and recognition method based on feature fusion, characterized in that it specifically includes the following steps:
[0007] Step 1: Target detection data collection and preprocessing
[0008] Collect the target machine image, use the oblique frame to mark the target machine position, and divide it into oblique frame detection training samples and oblique frame detection test samples; for the oblique frame detection training samples, cut the pixel block where the target machine is located according to the marked target machine position, obtain the type recognition training sample x and annotate the real category label;
[0009] Step 2: Aircraft target detection
[0010] Use the oblique box detection training samples to train the object detection algorithm to learn the position of the aircraft;
[0011] Step 3: Design of the first layer sub-network of the aircraft target type recognition model
[0012] Design type recognition model I-RACNN, which contains three sub-networks. The input of the first sub-network is the original image of type recognition training sample x, and the output is the category probability prediction value and the predicted image position of the key area; the first sub-network contains a convolution Conv Layer 1, a multi-scale feature fusion network, a category probability prediction branch and an attention suggestion network branch. The design is as follows:
[0013] S3-1: Conv Layer 1 input is the original image of type recognition training sample x, and the output is a feature map of size 64×64×16;
[0014] S3-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales. Its input is the feature map output by Conv Layer 1 in S3-1, and the output length is 1024 feature FC1;
[0015] S3-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC1 learned by the multi-scale feature fusion network; its input is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and its output is the category probability prediction value;
[0016] S3-4: Attention Proposal Network Branch
[0017] The attention proposal network branch is used to predict the position of the key area image of the type recognition training sample x, and then the key area image can be obtained by cropping the original image according to the coordinates and side length; the input of the APN branch is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and the output is the predicted position of the key area image;
[0018] Step 4: Design of the second and third layer sub-networks of the aircraft target type recognition model
[0019] The input of the second layer sub-network is the original image and the key area image enlarged according to S3-4, and the output is the category probability prediction value and the key area image position predicted by the attention suggestion network branch; the second layer sub-network includes an image fusion network, a multi-scale feature fusion network, a category probability prediction branch and an attention suggestion network branch, and the design is as follows:
[0020] S41-1: The input of the image fusion network is the original image and the key area image enlarged according to S3-4, and the output is a fused feature map with a size of 64×64×16;
[0021] S41-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales. Its input is the feature map output by S41-1, and the output length is 1024 feature FC2;
[0022] S41-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC2 learned by the multi-scale feature fusion network; its input is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and its output is the category probability prediction value;
[0023] S41-4: The attention proposal network branch is used to predict the position of the key area image of the type recognition training sample x, and then the key area image can be obtained by cropping the original image according to the coordinates and side length; the input of the APN branch is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and the output is the predicted position of the key area image;
[0024] The input of the third layer sub-network includes two branches: the original image and the enlarged key area image, and the output is the category probability prediction value; the third layer sub-network includes an image fusion network, a multi-scale feature fusion network, and a category probability prediction branch, and the design is as follows:
[0025] S42-1: The input of the image fusion network is the original image and the enlarged key area image, and the output is the fused feature map with a size of 64×64×16;
[0026] S42-2: Multi-scale feature fusion network is used to learn the features of aircraft targets of different scales; its input is the feature map output by S42-1, and the output length is 1024 feature FC3;
[0027] S42-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC3 learned by the multi-scale feature fusion network; its input is the feature FC3 extracted by the multi-scale feature fusion network in S42-2, and its output is the category probability prediction value;
[0028] Step 5: Loss function design
[0029] Design loss function for the three-layer sub-network, calculate loss value and optimize the network;
[0030] Step 6: Model training
[0031] For the type recognition model I-RACNN, SDG is used as the optimizer of the network model to train the I-RACNN network;
[0032] Step 7: Model testing and result warning
[0033] First, detect and crop the aircraft target; use the algorithm trained in step 2 to detect the oblique frame detection test sample to obtain the position detection frame of the target aircraft; then cut the aircraft target image with the center of the detection frame as the center of the sample image, and resize it to obtain the type recognition test sample to be identified; input the type recognition test sample into the aircraft target type recognition model I-RACNN trained in step 6 to obtain the type recognition result of the target aircraft and the model's predicted probability value for the target aircraft; finally, display the image captured by the sensor on the display screen, mark the target aircraft with a red box according to the predicted detection frame, and display the category information and confidence predicted by the type recognition model I-RACNN.
[0034] Furthermore, in the target detection data collection and preprocessing of step 1, the collected target machine images are uniformly resized and normalized, the target machine position is marked with an oblique frame, and the oblique frame detection training samples and the oblique frame detection test samples are divided according to a ratio of 8:2 to construct a target machine oblique frame detection data set; for the oblique frame detection training samples, the pixel block where the target machine is located is cut according to the marked target machine position, the image size is adjusted, and the type recognition training sample x is obtained, and each type recognition training sample x is labeled with a true category label y j , j = 1, ..., n, where n is the number of categories, construct a target machine type recognition training sample data set, which contains the original image of the type recognition training sample x and the true category label y j .
[0035] Furthermore, in the step 2, the oblique box detection training samples and the corresponding oblique box annotations are used to train the YOLOv8 model in the aircraft target detection to obtain a trained YOLOv8 model.
[0036] Furthermore, the multi-scale feature fusion network includes a feature extraction network, a top-down fusion network, a bottom-up fusion network, and a pooling splicing network;
[0037] The feature extraction network performs four layers of convolution operations on the output feature map of size 64×64×16 in sequence, and the four layers of convolution operations output four layers of feature maps, whose height and width gradually decrease and whose depth gradually increases, and finally obtains a feature map of size 4×4×256; let the output of the four layers of convolution operations be [C1, C2, C3, C4];
[0038] The top-down fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are D1, D2, D3 and D4 respectively:
[0039]
[0040] conv(C4,1,s=1) means the result of the conv convolution operation of C4 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0041] Represents C i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as D i+1 The results of deconvolution with kernel 3×3 and step size s=2 are added element by element;
[0042] The bottom-up fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are P1, P2, P3 and P4 respectively:
[0043]
[0044] conv(D1,1,s=1) means the result of the conv convolution operation of D1 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0045] Indicates D i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as P i-1 The results of the convolution with kernel 3×3 and step size s=2 are added element by element;
[0046] The pooling concatenation network is composed of four pooling layers concatenated in parallel. Suppose the concatenated outputs are Q1, Q2, Q3, and Q4 respectively:
[0047] Q i =Reshape(conv(P i ,1,s=16 / 2 i-1 ,k=256)), i=1,2,3,4 (3)
[0048] This formula means that first a layer of convolution kernel is 1×1 and step size s=16 / 2 is passed. i-1, a convolution operation with a convolution kernel number k of 256, convolving P1-P4 into a 2×2×256 feature map, and then using Reshape to expand it to obtain the corresponding vector Q i ;
[0049] Finally, vectors Q1, Q2, Q3, and Q4 are concatenated, and a fully connected layer with 1024 nodes is added to obtain the final feature output with a length of 1024.
[0050] Furthermore, a fully connected layer with n nodes is added to the category probability prediction branch to map the features extracted by the multi-scale feature fusion network to features with a length of n to obtain a prediction output.
[0051] Furthermore, fully connected layers with lengths of 128 and 3 are added to the attention suggestion network branch, and the features extracted by the multi-scale feature fusion network are mapped into a vector with a length of 3 to achieve position prediction of the key area; the original image of the type recognition training sample x is cropped according to the horizontal and vertical coordinates and side lengths of the acquired key area image position, and enlarged to the same size as the original image through bilinear interpolation to obtain an enlarged key area image of the type recognition training sample x.
[0052] Furthermore, the loss function design in step 5 includes two loss functions: classification loss and sorting loss;
[0053] Classification loss L class For training type recognition models, the cross entropy loss function is used, and its formula can be formalized as:
[0054]
[0055] For type recognition training sample x, is the true probability that x belongs to category j, when x belongs to category j Otherwise 0. represents the probability that x predicted by the s-th layer subnetwork in the three constructed subnetworks belongs to category j; y j Represents the true category label, Y (s) Represents the label predicted by the s-th layer sub-network;
[0056] For the APN module, the ranking loss function L is used rank For training, the ranking loss function is defined as
[0057]
[0058] For the type recognition training sample x whose true category is j, Indicates the probability that the image is predicted to be of category j in the s-th layer subnetwork, It indicates the probability that the image is predicted as category j in the s+1th layer of the subnetwork; margin indicates the threshold.
[0059] Furthermore, in the step 6 model training, the I-RACNN network is trained in the following order:
[0060] ① Initialization training of subnetworks; fix the parameter weights of the two APN networks, sample m batches of type recognition training samples and corresponding type labels for training for the first layer of subnetworks constructed each time, sample m images of type recognition training samples for training for the second and third layers of subnetworks constructed, copy and form data pairs, input them into the two input branches of the above two networks, and then use the corresponding type labels to train them batch by batch; calculate the loss according to formula (4) and update the weights of the above three networks respectively, so as to complete the initialization pre-training of the type prediction part of the above three subnetworks;
[0061] ② Pre-training of the attention proposal network branch; fix the parameter weights of the type prediction part in the three-layer sub-network, use the type recognition training samples to train the APN networks in the first and second layer sub-networks, and update the weights of the above two APN networks according to formula (5) to complete the initialization pre-training of the two APN networks;
[0062] ③Fix the parameter weights of the two APN networks, and use the type recognition training samples to train the parameter weights of the type prediction part of the three-layer sub-network batch by batch for one cycle;
[0063] ④Fix the parameter weights of the type prediction part in the three-layer sub-network, and use the type recognition training samples to train two APNs batch by batch for 1 cycle;
[0064] Repeat ③ to ④, and alternately train the type prediction part and the APN network in the model; finally, the trained aircraft target type recognition model I-RACNN is obtained.
[0065] The beneficial effects of the present invention are as follows: in view of the low degree of automation of image-based aircraft target recognition in air traffic control, the present invention designs an integrated target detection-recognition-display process based on RACNN, which can realize automatic monitoring of aircraft targets. In view of the problem of global feature loss in RACNN, the present invention designs a method for fusing the original image containing global features and the key area image containing classification-related key features. In order to enhance the feature learning ability of the RACNN model for targets of different scales, a multi-scale feature fusion learning module is designed to automatically extract features of different scales. Finally, the recognition effect of the algorithm is enhanced.
[0066] The present invention aims to intelligently identify the type of aircraft by designing an aircraft detection and classification method, and realize the function of automatically detecting and identifying the type of aircraft target from aerial images such as visible light and radar taken by a pod or observation equipment, so as to reduce the work pressure of flight commanders and pilots in identifying target aircraft, improve the efficiency of tower commanders and the level of tower automation, and reduce the burden on staff.
[0067] Through step 3, the present invention designs a multi-scale feature fusion learning network. By learning the features of the four scales of the image, the features are fused in two orders, from top to bottom and from bottom to top. The abstract semantic information of the upper layer can be used to guide the shallow feature learning of the lower layer network, and the upper layer semantic information can be further optimized through the features learned by the lower layer network, so as to obtain richer type-related features of different scales, so as to better adapt to the changes in the size of the aircraft in the image, and finally improve the effect of fine-grained classification of the target aircraft.
[0068] Through step 4, the present invention designs an I-RACNN method that can fuse global and local features to more fully learn type-related features; by introducing the original image containing global features in the second and third layer sub-networks, and fusing the original image with the key area image containing local features, the algorithm can be prompted to more comprehensively learn the target machine features from two aspects: the global features contained in the original image and the local features contained in the local key area image, so as to enhance the classification effect of the algorithm.
[0069] Through steps 1 to 7, the present invention can realize automatic monitoring and identification of aircraft targets. Through the designed I-RACNN, the model can use a multi-scale feature learning network to adapt to the changes in the size differences of different types of aircraft, enhance the fusion learning ability of type-related features of different scales, and further enhance the learning ability of type features by fusing global and local image features, ultimately improving the algorithm's recognition effect on the target aircraft type. BRIEF DESCRIPTION OF THE DRAWINGS
[0070] Figure 1 It is the overall framework diagram of the present invention;
[0071] Figure 2 It is the overall framework diagram of the algorithm I-RACNN of the present invention;
[0072] Figure 3 This is a structural diagram of the multi-scale feature fusion network adopted by the I-RACNN of the present invention.
[0073] Specific implementation method: The specific implementation method of the present invention is described in detail below with reference to the accompanying drawings:
[0074] A target detection and recognition method based on feature fusion, which specifically includes the following steps:
[0075] Step 1: Target detection data collection and preprocessing
[0076] Use air detection equipment or airborne pods to collect target aircraft images, resize the collected target aircraft images to 224×224×3 and normalize them, use oblique frames (determined by center coordinates, width, height and rotation angle) to mark the position of the target aircraft, divide the oblique frame detection training samples and oblique frame detection test samples into 8:2 ratios, and construct a target aircraft oblique frame detection dataset.
[0077] For the oblique frame detection training samples, cut the pixel block where the target machine is located according to the marked target machine position, adjust the image size to 64×64×3, obtain the type recognition training samples x, and annotate the true category label y for each type recognition training sample x j , j = 1, ..., n, where n is the number of categories, construct a target machine type recognition training sample data set, which contains the original image of the type recognition training sample x and the true category label y j .
[0078] Step 2: Aircraft target detection
[0079] The target detection algorithm is trained using the oblique frame detection training sample of the target aircraft to learn the position of the aircraft. The input is the image in the oblique frame detection training sample, and the output is the center coordinates, width, height and rotation angle of the corresponding oblique frame, thereby determining the unique rectangle. Considering that the YOLOv8 algorithm proposed in the document (Real-Time Flying Object Detection with YOLOv8) has high accuracy and detection speed in target detection, the present invention adopts YOLOv8 as the target detection algorithm. Specifically, the oblique frame detection training sample and the corresponding oblique frame annotation are used to train the YOLOv8 model, the model parameters are set by default, the number of training times is 100, and the trained YOLOv8 model is obtained.
[0080] Step 3. Design of the first layer sub-network of the aircraft target type recognition model. Considering that the literature (Recurrent Attention CNN for FGVC: RACNN) has achieved stable and good results in fine-grained classification, the present invention is based on RACNN and uses the target aircraft type recognition dataset to design an improved RACNN algorithm to realize aircraft type recognition.
[0081] The designed type recognition model I-RACNN contains three layers of sub-networks, of which the first layer of sub-network is designed as follows:
[0082] The input of the first layer sub-network is the original image of the type recognition training sample x, and the output is the category probability prediction value and the predicted key area image position ax,ay,al.
[0083] Specifically, the first sub-network contains a convolution layer Conv Layer 1, a multi-scale feature fusion network, a category probability prediction branch and an attention proposal network branch. The designs of each part are as follows:
[0084] S3-1: Conv Layer 1 is a convolution layer conv(3,1,16,padding=same) with a convolution kernel size of 3×3 and 16 convolution kernels. The input is the original image of the type recognition training sample x, and the output is a feature map of size 64×64×16.
[0085] S3-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales, so as to better adapt to the changes in aircraft size. Its input is the feature map output by Conv Layer 1 in S3-1, and the output feature FC1 is 1024 in length. The multi-scale feature fusion network adds pooling splicing on the basis of the PAFPN structure to fuse all features into a feature with a length of 1024.
[0086] Its structure is as Figure 3 As shown, it includes four parts: feature extraction network, top-down fusion network, bottom-up fusion network, and pooling splicing network.
[0087] Specifically, the multi-scale feature fusion network performs four layers of convolution operations on the feature map of size 64×64×16 output by Conv Layer 1. The four layers of convolution operations output four layers of feature maps, whose height and width gradually decrease and depth gradually increase. The number of convolution kernels used in the four layers is 32, 64, 128, and 256 respectively, and finally a feature map of size 4×4×256 is obtained; let the output of the four-layer convolution operation be [C1, C2, C3, C4];
[0088] The top-down fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are D1, D2, D3 and D4 respectively:
[0089]
[0090] conv(C4,1,s=1) means the result of the conv convolution operation of C4 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0091] Represents C i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as D i+1 The results of deconvolution with kernel 3×3 and step size s=2 are added element by element;
[0092] The bottom-up fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are P1, P2, P3 and P4 respectively:
[0093]
[0094] conv(D1,1,s=1) means the result of the conv convolution operation of D1 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0095] Indicates D i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as P i-1 The results of the convolution with kernel 3×3 and step size s=2 are added element by element;
[0096] The pooling concatenation network is composed of four pooling layers concatenated in parallel. Suppose the concatenated outputs are Q1, Q2, Q3, and Q4 respectively:
[0097] Q i =Reshape(conv(P i ,1,s=16 / 2 i-1 ,k=256)), i=1,2,3,4 (3)
[0098] This formula means that first a layer of convolution kernel is 1×1 and step size s=16 / 2 is passed. i-1 , a convolution operation with a convolution kernel number k of 256, convolving P1-P4 into a 2×2×256 feature map, and then using Reshape to expand it to obtain the corresponding vector Q i .
[0099] Finally, vectors Q1, Q2, Q3, and Q4 are concatenated, and a fully connected layer with 1024 nodes is added to obtain the final feature output FC1 with a length of 1024.
[0100] S3-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC1 learned by the multi-scale feature fusion network. Its input is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and its output is the category probability prediction value.
[0101] Specifically, a fully connected layer with n nodes is added to map the feature FC1 to a feature of length n to obtain the predicted output, which is the category predicted by the first layer sub-network in the model for the original image of the type recognition training sample x, where the output value of each node represents the probability that the input original image belongs to the corresponding category among the n categories.
[0102] S3-4: Attention Proposal Network Branch
[0103] The Attention Proposal Network (APN) branch is used to predict the position (coordinates and side length) of the key area image of the type recognition training sample x, and then the key area image can be obtained by cropping the original image according to the coordinates and side length. The input of the APN branch is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and the output is the predicted position of the key area image.
[0104] Specifically, a fully connected layer with lengths of 128 and 3 is added to map the feature FC1 to a vector fp1(ax,ay,al) with a length of 3 to achieve the location prediction of the key area. Among them, ax,ay,al represent the horizontal and vertical coordinates of the upper left corner of the square area concerned by the APN and the side length of the square. The key area image corresponding to this position contains local features that play a major role in the classification effect of this layer of the network after being screened by the model.
[0105] The original image of the type recognition training sample x is cropped according to the acquired key area image position (including coordinates ax, ay and side length al), and is enlarged to the same size as the original image through bilinear interpolation to obtain the enlarged key area image of the type recognition training sample x.
[0106] Step 4: Design of the second and third layer sub-networks of the aircraft target type recognition model
[0107] The second and third sub-networks are designed based on the first sub-network of the aircraft target type recognition model designed in step 3. Since the input of the original RACNN algorithm in the third-layer sub-network classification task is a local image that has been cropped twice, there is a lack of global features about the entire aircraft target, which affects the accuracy of the algorithm. In view of the problem that the above RACNN fails to make full use of global features in feature learning, an improved recurrent attention convolutional neural network (Improved RACNN, I-RACNN) algorithm based on feature fusion is proposed. By introducing global features in the second and third sub-networks, the type-related features are learned from two perspectives: the global features of the entire aircraft target contained in the original image and the local features contained in the key area image, thereby making full use of information of different scales to improve the classification effect.
[0108] The framework of the designed second and third layer sub-networks is as follows Figure 2As shown. The input of the second layer sub-network includes two branches: the original image and the enlarged key area image, and the output is the category probability prediction value and the key area image position predicted by the attention suggestion network branch. The input of the third layer sub-network includes two branches: the original image and the enlarged key area image, and the output is the category probability prediction value. The second layer sub-network consists of an image fusion network, a multi-scale feature fusion network, a category probability prediction branch, and an attention suggestion network branch. The third layer sub-network consists of an image fusion network, a multi-scale feature fusion network, and a category probability prediction branch. Among them, the multi-scale feature fusion network, the category probability prediction branch, and the attention suggestion network branch in the second and third layer sub-networks are the same as the above networks and branches in the first layer sub-network designed in step 3.
[0109] Specifically, the second-layer subnetwork is designed as follows:
[0110] The input of the second layer sub-network is the original image and the key area image enlarged according to S3-4, and the output is the category probability prediction value and the key area image position predicted by the attention suggestion network branch. The second layer sub-network includes image fusion network, multi-scale feature fusion network, category probability prediction branch and attention suggestion network branch. The design of each part is as follows:
[0111] S41-1: The input of the image fusion network is the original image and the image of the key area enlarged according to S3-4, and the output is the fused feature map. Specifically, the original image and the enlarged image of the key area obtained by S3-4 are spliced by channel after passing through ConvLayer 2, so as to fuse the global features of the entire aircraft target contained in the original image and the local features contained in the key area image, where Conv Layer2 is set to conv(3,1,8), and by splicing the third dimension of the fused feature map, a fused feature map of size 64×64×16 is finally obtained.
[0112] S41-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales, so as to better adapt to the changes in aircraft size. Its input is the feature map output by S41-1, and the output feature FC2 with a length of 1024. The multi-scale feature fusion network adds pooling splicing on the basis of the PAFPN structure to fuse all features into a feature with a length of 1024. Its structure is as follows Figure 3 As shown, it includes four parts: feature extraction network, top-down fusion network, bottom-up fusion network, and pooling splicing network.
[0113] Specifically, the multi-scale feature fusion network performs four layers of convolution operations on the feature map of size 64×64×16 output by S41-1 in sequence. The four layers of convolution operations output four layers of feature maps, whose height and width gradually decrease and whose depth gradually increases. The number of convolution kernels used in the four layers is 32, 64, 128, and 256 respectively, and finally obtains a feature map of size 4×4×256. Suppose the output of the four-layer convolution operation is [C1, C2, C3, C4];
[0114] The top-down fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are D1, D2, D3 and D4 respectively:
[0115]
[0116] conv(C4,1,s=1) means the result of the conv convolution operation of C4 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0117] Represents C i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as D i+1 The results of deconvolution with kernel 3×3 and step size s=2 are added element by element;
[0118] The bottom-up fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are P1, P2, P3 and P4 respectively:
[0119]
[0120] conv(D1,1,s=1) means the result of the conv convolution operation of D1 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0121] Indicates D i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as P i-1 The results of the convolution with kernel 3×3 and step size s=2 are added element by element;
[0122] The pooling concatenation network is composed of four pooling layers concatenated in parallel. Suppose the concatenated outputs are Q1, Q2, Q3, and Q4 respectively:
[0123] Q i =Reshape(conv(P i ,1,s=16 / 2 i-1 ,k=256)), i=1,2,3,4 (3)
[0124] This formula means that first a layer of convolution kernel is 1×1 and step size s=16 / 2 is passed. i-1 , a convolution operation with a convolution kernel number k of 256, convolving P1-P4 into a 2×2×256 feature map, and then using Reshape to expand it to obtain the corresponding vector Q i .
[0125] Finally, vectors Q1, Q2, Q3 and Q4 are concatenated, and a fully connected layer with 1024 nodes is added to obtain the final feature output FC2 with a length of 1024.
[0126] S41-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC2 learned by the multi-scale feature fusion network. Its input is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and its output is the category probability prediction value.
[0127] Specifically, a fully connected layer with n nodes is added to map the feature FC2 to a feature of length n to obtain the predicted output, which is the category predicted by the second-layer subnetwork in the model for the original image of the type recognition training sample x, where the output value of each node represents the probability that the input original image belongs to the corresponding category among the n categories.
[0128] S41-4: The Attention Proposal Network (APN) branch is used to predict the position (coordinates and side length) of the key area image of the type recognition training sample x, and then the key area image is obtained by cropping the original image according to the coordinates and side length. The input of the APN branch is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and the output is the predicted position of the key area image.
[0129] Specifically, a fully connected layer with lengths of 128 and 3 is added to map the feature FC2 to a vector fp2(ax,ay,al) with a length of 3 to achieve the location prediction of the key area. Among them, ax,ay,al represent the horizontal and vertical coordinates of the upper left corner of the square area concerned by the APN and the side length of the square. The key area image corresponding to this position contains local features that play a major role in the classification effect of this layer of the network after being screened by the model.
[0130] The original image of the type recognition training sample x is cropped according to the acquired key area image position (including coordinates ax, ay and side length al), and is enlarged to the same size as the original image through bilinear interpolation to obtain the enlarged key area image of the type recognition training sample x.
[0131] The third layer sub-network is designed as follows: The input of the third layer sub-network includes two branches: the original image and the enlarged key area image, and the output is the category probability prediction value. Specifically, the third layer sub-network includes an image fusion network, a multi-scale feature fusion network, and a category probability prediction branch. The designs of each part are as follows:
[0132] S42-1: The input of the image fusion network is the original image and the enlarged key area image, and the output is the fused feature map. Specifically, the original image and the enlarged key area image obtained by S41-4 are spliced by channel after passing through Conv Layer3, so as to fuse the global features of the entire aircraft target contained in the original image and the local features contained in the key area image. In the present invention, Conv Layer3 is set to conv(3,1,8), and by splicing the third dimension of the fused feature map, a fused feature map with a size of 64×64×16 is finally obtained.
[0133] S42-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales, so as to better adapt to the changes in aircraft size. Its input is the feature map output by S42-1, and the output feature FC3 with a length of 1024. The multi-scale feature fusion network adds pooling splicing on the basis of the PAFPN structure to fuse all features into a feature with a length of 1024. Its structure is as follows Figure 3 As shown, it includes four parts: feature extraction network, top-down fusion network, bottom-up fusion network, and pooling splicing network.
[0134] Specifically, the multi-scale feature fusion network performs four layers of convolution operations on the feature map of size 64×64×16 output by S42-1 in sequence. The four layers of convolution operations output four layers of feature maps, whose height and width gradually decrease and whose depth gradually increases. The number of convolution kernels used in the four layers is 32, 64, 128, and 256 respectively, and finally obtains a feature map of size 4×4×256; let the output of the four-layer convolution operation be [C1, C2, C3, C4];
[0135] The top-down fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are D1, D2, D3 and D4 respectively:
[0136]
[0137] conv(C4,1,s=1) means the result of the conv convolution operation of C4 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0138] Represents C iAfter a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as D i+1 The results of deconvolution with kernel 3×3 and step size s=2 are added element by element;
[0139] The bottom-up fusion network is composed of four fusion layers connected in sequence. The outputs of the four fusion layers are P1, P2, P3 and P4 respectively:
[0140]
[0141] conv(D1,1,s=1) means the result of the conv convolution operation of D1 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1;
[0142] Indicates D i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as P i-1 The results of the convolution with kernel 3×3 and step size s=2 are added element by element;
[0143] The pooling concatenation network is composed of four pooling layers concatenated in parallel. Suppose the concatenated outputs are Q1, Q2, Q3, and Q4 respectively:
[0144] Q i =Reshape(conv(P i ,1,s=16 / 2 i-1 ,k=256)), i=1,2,3,4 (3)
[0145] This formula means that first a layer of convolution kernel is 1×1 and step size s=16 / 2 is passed. i-1 , a convolution operation with a convolution kernel number k of 256, convolving P1-P4 into a 2×2×256 feature map, and then using Reshape to expand it to obtain the corresponding vector Q i .
[0146] Finally, vectors Q1, Q2, Q3, and Q4 are concatenated, and a fully connected layer with 1024 nodes is added to obtain the final feature output FC3 with a length of 1024.
[0147] S42-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC3 learned by the multi-scale feature fusion network. Its input is the feature FC3 extracted by the multi-scale feature fusion network in S42-2, and its output is the category probability prediction value.
[0148] Specifically, a fully connected layer with n nodes is added to map the feature FC3 to a feature of length n to obtain the predicted output, which is the category predicted by the third-layer subnetwork in the model for the original image of the type recognition training sample x, where the output value of each node represents the probability that the input original image belongs to the corresponding category among the n categories.
[0149] Step 5: Loss function design
[0150] A loss function is designed for the three-layer sub-network, the loss value is calculated and the network is optimized. Specifically, the present invention adopts the same loss function as the original RACNN algorithm, including two loss functions: classification loss and sorting loss.
[0151] Classification loss L class For training type recognition models, the cross entropy loss function is used, and its formula can be formalized as:
[0152]
[0153] For type recognition training sample x, is the true probability that x belongs to category j, when x belongs to category j Otherwise 0. represents the probability that x belongs to category j predicted by the s-th layer subnetwork in the three subnetworks constructed in steps 3 and 4. j represents the true category label marked in step 1, Y (s) Represents the label predicted by the s-th layer sub-network.
[0154] For the APN module designed by S3-4, the ranking loss function L is used. rank For training, the ranking loss function is defined as
[0155]
[0156] For the type recognition training sample x whose true category is j, Indicates the probability that the image is predicted to be of category j in the s-th layer subnetwork, Indicates the probability that the image is predicted as category j in the s+1th layer of the subnetwork. Margin indicates the threshold, which is set to 0.1 here. With the above settings, the predicted probability value (i.e., confidence) of the subnetwork of this layer for the input image in the trainable model should be at least 0.1 higher than the predicted probability value of the subnetwork of the previous layer. Through this function, the prediction accuracy of the subnetwork of this layer can be made as high as possible as possible.
[0157] Step 6. Model training For the type recognition model I-RACNN, SDG is used as the optimizer of the network model, the learning rate is set to 0.0002, and the batch size is e=128. In order to enable the APN and the classification network to promote each other to improve the fine-grained classification effect, the I-RACNN network is trained in the following order.
[0158] ① Initialization training of subnetworks. Fix the parameter weights of the two APN networks, set the training batch size m = 128, and for the first layer subnetwork constructed in step 3, sample m batches from the type recognition training samples and the corresponding type labels for training each time. For the second and third layer subnetworks constructed in step 4, sample m images from the type recognition training samples in batches, copy them to form data pairs, input them to the two input branches of the above two networks, and then use the corresponding type labels to train batches. Calculate the loss according to formula (4) and update the weights of the above three networks respectively. Each subnetwork is trained 100 times to complete the initialization pre-training of the type prediction part (Conv Layer 1, 2, 3, multi-scale feature fusion network, category probability prediction branch) in the above three subnetworks.
[0159] ② Pre-training of the attention proposal network branch. Fix the parameter weights of the type prediction part in the three-layer sub-network, use the type recognition training samples to train the APN networks in the first and second sub-networks, and update the weights of the above two APN networks according to formula (5). The training number is 100 times to complete the initialization pre-training of the two APN networks.
[0160] ③ Set the training batch size m = 128, fix the parameter weights of the two APN networks, and use the type recognition training samples to train the parameter weights of the type prediction part of the three-layer sub-network batch by batch for one cycle (using all the type recognition training samples batch by batch for one time is one cycle);
[0161] ④ Set the training batch size m = 128, fix the parameter weights of the type prediction part in the three-layer subnetwork, and use the type recognition training samples to train the two APNs batch by batch for 1 cycle.
[0162] Repeat ③ to ④, and train the type prediction part and APN network in the model alternately for 100 cycles. Finally, the trained aircraft target type recognition model I-RACNN is obtained.
[0163] Step 7: Model testing and result warning
[0164] First, detect and crop the aircraft target. Use the YOLOv8 algorithm trained in step 2 to detect the oblique frame detection test sample and obtain the position detection frame of the target aircraft. Then, cut the aircraft target image with the center of the detection frame as the center of the sample image and resize it to 64×64 to obtain the type recognition test sample to be identified.
[0165] Then, the type recognition test sample is input into the aircraft target type recognition model I-RACNN trained in step 6 to obtain the output of the third layer subnetwork of the model. The output is the type recognition result of the target aircraft and the predicted probability value of the model for the target aircraft, which is also the confidence of the above algorithm in the recognition of the aircraft target type.
[0166] Finally, the image captured by the sensor is displayed on the screen. According to the detection box predicted by YOLOv8, the target machine is marked with a red box, and the category information and confidence predicted by the type recognition model I-RACNN are displayed.
[0167] It should be understood that the parts not elaborated in detail in this specification belong to the prior art. The above embodiments are only descriptions of the preferred implementation methods of the present invention, and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made by ordinary engineers and technicians in this field to the technical solution of the present invention should fall within the protection scope determined by the claims of the present invention.
Claims
1. A target detection and recognition method based on feature fusion, characterized in that The specific steps include: Step 1: Target detection data collection and preprocessing Collect the target machine image, use the oblique frame to mark the target machine position, and divide the oblique frame detection training sample and the oblique frame detection test sample; for the oblique frame detection training sample, cut the pixel block where the target machine is located according to the marked target machine position, obtain the type recognition training sample x and mark the real category label; Step 2: Aircraft target detection Use the oblique box detection training samples to train the object detection algorithm to learn the position of the aircraft; Step 3: Design of the first layer sub-network of the aircraft target type recognition model Design type recognition model I-RACNN, which contains three sub-networks. The input of the first sub-network is the original image of type recognition training sample x, and the output is the category probability prediction value and the predicted image position of the key area; the first sub-network contains a convolution Conv Layer 1, a multi-scale feature fusion network, a category probability prediction branch and an attention suggestion network branch. The design is as follows: S3-1: Conv Layer 1 input is the original image of type recognition training sample x, and the output is a feature map of size 64×64×16; S3-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales. Its input is the feature map output by ConvLayer 1 in S3-1, and the output length is 1024 feature FC1; S3-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC1 learned by the multi-scale feature fusion network; its input is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and its output is the category probability prediction value; S3-4: Attention Proposal Network Branch The attention proposal network branch is used to predict the position of the key area image of the type recognition training sample x, and then the key area image can be obtained by cropping the original image according to the coordinates and side length; the input of the APN branch is the feature FC1 extracted by the multi-scale feature fusion network in S3-2, and the output is the predicted position of the key area image; Step 4, the second and third layer sub-network design of the aircraft target type recognition model The input of the second layer sub-network is the original image and the key area image enlarged according to S3-4, and the output is the category probability prediction value and the key area image position predicted by the attention suggestion network branch; the second layer sub-network includes an image fusion network, a multi-scale feature fusion network, a category probability prediction branch and an attention suggestion network branch, and the design is as follows: S41-1: The input of the image fusion network is the original image and the key area image enlarged according to S3-4, and the output is a fused feature map with a size of 64×64×16; S41-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales. Its input is the feature map output by S41-1, and the output length is 1024 feature FC2; S41-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC2 learned by the multi-scale feature fusion network; its input is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and its output is the category probability prediction value; S41-4: The attention proposal network branch is used to predict the position of the key area image of the type recognition training sample x, and then the key area image can be obtained by cropping the original image according to the coordinates and side length; the input of the APN branch is the feature FC2 extracted by the multi-scale feature fusion network in S41-2, and the output is the predicted position of the key area image; The input of the third layer sub-network includes two branches: the original image and the enlarged key area image, and the output is the category probability prediction value; the third layer sub-network includes an image fusion network, a multi-scale feature fusion network, and a category probability prediction branch, and the design is as follows: S42-1: The input of the image fusion network is the original image and the enlarged key area image, and the output is the fused feature map with a size of 64×64×16; S42-2: The multi-scale feature fusion network is used to learn the features of aircraft targets of different scales. Its input is the feature map output by S42-1, and the output length is 1024 feature FC3; S42-3: The category probability prediction branch is used to predict the category of the input image based on the feature FC3 learned by the multi-scale feature fusion network; its input is the feature FC3 extracted by the multi-scale feature fusion network in S42-2, and its output is the category probability prediction value; Step 5: Loss function design Design loss function for the three-layer sub-network, calculate loss value and optimize the network; Step 6: Model training For the type recognition model I-RACNN, SDG is used as the optimizer of the network model to train the I-RACNN network; Step 7: Model testing and result warning First, detect and crop the aircraft target; use the algorithm trained in step 2 to detect the oblique frame detection test sample to obtain the position detection frame of the target aircraft; then cut the aircraft target image with the center of the detection frame as the center of the sample image, and resize it to obtain the type recognition test sample to be identified; input the type recognition test sample into the aircraft target type recognition model I-RACNN trained in step 6 to obtain the type recognition result of the target aircraft and the model's predicted probability value for the target aircraft; finally, display the image captured by the sensor on the display screen, mark the target aircraft with a red box according to the predicted detection frame, and display the category information and confidence predicted by the type recognition model I-RACNN.
2. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that In the step 1 target detection data collection and preprocessing, the collected target machine images are uniformly resized and normalized, the target machine position is marked with an oblique frame, and the oblique frame detection training samples and the oblique frame detection test samples are divided according to the ratio of 8:2 to construct a target machine oblique frame detection data set; for the oblique frame detection training samples, the pixel block where the target machine is located is cut according to the marked target machine position, the image size is adjusted, and the type recognition training sample x is obtained, and each type recognition training sample x is marked with a true category label y j , j = 1, ..., n, where n is the number of categories, construct a target machine type recognition training sample data set, which contains the original image of the type recognition training sample x and the true category label y j .
3. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that In the step 2, the oblique frame detection training samples and the corresponding oblique frame annotations are used to train the YOLOv8 model in the aircraft target detection to obtain a trained YOLOv8 model.
4. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that The multi-scale feature fusion network includes a feature extraction network, a top-down fusion network, a bottom-up fusion network, and a pooling splicing network; The feature extraction network sequentially performs 4 layers of convolution operations on the output feature map of size 64×64×16, and the 4 layers of convolution operations output 4 layers of feature maps, whose height and width gradually decrease and depth gradually increase, and finally obtains a feature map of size 4×4×256; the output of the 4 layers of convolution operations is assumed to be [C1, C2, C3, C4]; The top-down fusion network is composed of four fusion layers connected in sequence. Suppose the outputs of the four fusion layers are D1, D2, D3 and D4 respectively: conv(C4,1,s=1) means the result of the conv convolution operation of C4 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1; Represents C i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as D i+1 The results of deconvolution with kernel 3×3 and step size s=2 are added element by element; The bottom-up fusion network is composed of four fusion layers connected in sequence. Suppose the outputs of the four fusion layers are P1, P2, P3 and P4 respectively: conv(D1,1,s=1) represents the result of the conv convolution operation of D1 after a convolution layer with a convolution kernel of 1×1 and a step size of s=1; Indicates D i After a convolution operation with a convolution kernel of 1×1 and a step size of s=1, the result is the same as P i-1 The results of the convolution with kernel 3×3 and step size s=2 are added element by element; The pooling concatenation network is composed of four pooling layers concatenated in parallel. Suppose the concatenated outputs are Q1, Q2, Q3, and Q4 respectively: Q i =Reshape(conv(P i ,1,s=16 / 2 i-1 ,k=256)),i=1,2,3,4 (3) This formula means that first a layer of convolution kernel is 1×1 and step size s=16 / 2 is passed. i-1 , a convolution operation with a convolution kernel number k of 256, convolving P1-P4 into a 2×2×256 feature map, and then using Reshape to expand it to obtain the corresponding vector Q i ; Finally, vectors Q1, Q2, Q3, and Q4 are concatenated, and a fully connected layer with 1024 nodes is added to obtain the final feature output with a length of 1024.
5. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that A fully connected layer with n nodes is added to the category probability prediction branch to map the features extracted by the multi-scale feature fusion network to features with a length of n to obtain a prediction output.
6. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that In the attention suggestion network branch, fully connected layers with lengths of 128 and 3 are added, and the features extracted by the multi-scale feature fusion network are mapped into a vector with a length of 3 to achieve the position prediction of the key area; the original image of the type recognition training sample x is cropped according to the horizontal and vertical coordinates and side length of the acquired key area image position, and enlarged to the same size as the original image through bilinear interpolation to obtain the enlarged key area image of the type recognition training sample x.
7. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that The loss function design in step 5 includes two loss functions: classification loss and sorting loss; Classification loss L class For training type recognition models, the cross entropy loss function is used, and its formula is formalized as: For type recognition training sample x, is the true probability that x belongs to category j, when x belongs to category j =1, otherwise 0. represents the probability that x predicted by the s-th layer subnetwork in the three constructed subnetworks belongs to category j; y j Represents the true category label, Y (s) Represents the label predicted by the s-th layer sub-network; For the APN module, the ranking loss function L is used rank For training, the ranking loss function is defined as For the type recognition training sample x whose true category is j, Indicates the probability that the image is predicted to be of category j in the s-th layer subnetwork, It indicates the probability that the image is predicted as category j in the s+1th layer of the sub-network; margin indicates the threshold.
8. The target detection and recognition method based on feature fusion according to claim 1 is characterized in that In the step 6 model training, the I-RACNN network is trained in the following order: ① Initialization training of subnetworks; fix the parameter weights of the two APN networks, sample m batches of type recognition training samples and corresponding type labels for training for the first layer of subnetworks constructed each time, sample m batches of images from the type recognition training samples for the second and third layer of subnetworks constructed, copy and form data pairs, input them into the two input branches of the above two subnetworks, and then use the corresponding type labels to train them batch by batch; calculate the loss according to formula (4) and update the weights of the above three subnetworks respectively to complete the initialization pre-training of the type prediction part of the above three subnetworks; ② Pre-training of the attention proposal network branch; fix the parameter weights of the type prediction part in the three-layer sub-network, use the type recognition training samples to train the APN networks in the first and second layer sub-networks, and update the weights of the above two APN networks according to formula (5) to complete the initialization pre-training of the two APN networks; ③Fix the parameter weights of the two APN networks, and use the type recognition training samples to train the parameter weights of the type prediction part of the three-layer sub-network batch by batch for one cycle; ④Fix the parameter weights of the type prediction part in the three-layer sub-network, and use the type recognition training samples to train two APNs batch by batch for 1 cycle; Repeat ③ to ④, and train the type prediction part and the APN network in the model alternately; finally, the trained aircraft target type recognition model I-RACNN is obtained.
Citation Information
Patent Citations
Fine-grained ship image target identification method for multi-feature region
CN112668403A
Remote sensing scene classification method based on attention network scale feature fusion
CN113408594A
SAR image aircraft target detection method based on joint attention mechanism
CN115410102A
