A visible light target recognition method and device based on deep learning, an electronic device, and a storage medium

Abstract

The application provides a visible light target recognition method and device based on deep learning, electronic equipment and storage medium, and belongs to the technical field of visible light target recognition. The visible light target recognition method comprises: acquiring a visible light target recognition image; processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain multiple visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the multiple visible light target recognition feature maps; and outputting the visible light target recognition result. The small target feature decoupling extraction network in the application can better mine multi-dimensional features of different depth feature layers, thereby improving the accuracy and robustness of detection.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of visible light target recognition, and particularly relates to a visible light target recognition method and device based on deep learning, an electronic device and a storage medium. BACKGROUND

[0002] With the rapid development of remote sensing information technology, the resolution of remote sensing images is higher and higher, the information quantity is larger and larger, and the detection means is richer and richer. The importance of high-resolution remote sensing image target recognition and detection is increasingly prominent in military and civilian aspects. The emergence of artificial intelligence technology provides a new idea and method for remote sensing image target extraction and recognition. However, compared with large and medium target detection, the detection effect of small targets in actual remote sensing complex scenes is still not satisfactory.

[0003] In recent years, the remote sensing image target detection and recognition method based on deep learning has always been a research hotspot, and the effectiveness of detection and recognition lies in algorithm + data + calculation, involving the core problem of deep convolutional neural network. Especially for different target sizes and attributes under different scene conditions of aerial remote sensing, how to accurately and quickly extract small targets is still one of the difficulties to be solved. Specifically, small target detection mainly faces the following challenges: 1) in the existing target detection model, the bottom layer features of the backbone network are generally used to detect small targets, but the bottom layer features lack semantic information, which brings certain difficulties to the detection of small targets; 2) at present, the mainstream target detection algorithm widely uses small target data set with less training samples, which makes the small target not fully learned in the process of model training; 3) the backbone network of the existing target detection model is trained on the classification data set, but there is a certain difference between the scale distribution of the target in the classification data set and the scale distribution of the target in the detection data set. SUMMARY

[0004] Therefore, the embodiments of the present application propose a visible light target recognition method and device based on deep learning, an electronic device and a storage medium, which aims to improve the detection and recognition ability of small targets in complex scenes by using a small amount of data set.

[0005] To achieve the above object, embodiments of the present application provide a visible light target recognition method based on deep learning, comprising: acquiring a visible light target recognition image; processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain multiple visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the multiple visible light target recognition feature maps; and outputting the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network comprises: determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and a ResNet50-vd of a YOLO algorithm; replacing a basic network Darknet-53 in a YOLOv3 algorithm with the improved ResNet50-vd, the improved ResNet50-vd comprising first to fifth feature output layers; determining a feature pyramid network based on the respective output feature maps of the second to fifth feature output layers of the improved ResNet50-vd; determining a path aggregation network based on the feature pyramid network; determining a multi-level feature fusion layer based on the path aggregation network; and constructing the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in sequence.

[0006] Optionally, the ResNet50-vd comprises first to fifth feature output layers, each of the first to fifth feature output layers is a 3x3 convolution layer, the improved ResNet50-vd is determined based on the deformable convolution network layer, the small target detection layer and the ResNet50-vd of the YOLO algorithm, comprising: replacing the 3x3 convolution layers of the third to fifth feature output layers of the ResNet50-vd with the deformable convolution network layer, and connecting the output end of the second feature output layer of the ResNet50-vd with the small target detection layer to obtain the improved ResNet50-vd.

[0007] Optionally, the second feature output layer to the fifth feature output layer based on the improved ResNet50-vd respectively output feature map to determine the feature pyramid network, comprising: according to the number of channels of the output feature map of each deformable convolution network layer, the corresponding 1x1 convolution layer connected with each deformable convolution network layer is determined, the output end of the fifth feature output layer to the third feature output layer is connected with the input end of each corresponding 1x1 convolution layer, and the output end of the small target detection layer is connected with the input end of a 1x1 convolution layer; according to the size of the output feature map of each deformable convolution network layer, the parameters of the first upsampling layer are determined, the input end of the first upsampling layer is connected with the output end of the 1x1 convolution layer; the output end of the deformable convolution network layer above and the output end of the first upsampling layer are connected with the input end of the first fusion layer, the input end of the first 3x3 convolution layer is connected with the output end of the first fusion layer, and the first stage of the feature pyramid network is obtained; the second stage to the third stage of the feature pyramid network are determined according to the foregoing steps from top to bottom, and the feature pyramid network is obtained.

[0008] Optionally, the path aggregation network is determined based on the feature pyramid network, comprising: the output end of the third stage is connected with the input end of the 1x1 convolution layer; the output end of the first upsampling layer of the third stage and the input end of the 1x1 convolution layer are connected with the input end of the second fusion layer, the output end of the second fusion layer is connected with the second 3x3 convolution layer, and the fifth stage of the feature pyramid network is obtained; the sixth stage to the eighth stage of the feature pyramid network are constructed according to the two foregoing steps from bottom to top, and the path aggregation network is obtained.

[0009] Optionally, a multi-level feature fusion layer is constructed based on the feature pyramid network, comprising: the output ends of the sixth stage and the eighth stage are respectively connected to the input ends of the second up-sampling layer to the fourth up-sampling layer, and the output ends of the second up-sampling layer to the fourth up-sampling layer are respectively connected to the input ends of the third 3x3 convolution layer to the fifth 3x3 convolution layer; the output end of the sixth stage is connected to the input end of the down-sampling layer, and the output end of the down-sampling layer is connected to the input end of the 1x1 convolution layer; the output ends of the third 3x3 convolution layer to the fifth 3x3 convolution layer and the output end of the 1x1 convolution layer are all connected to the input end of the third fusion layer; the output end of the third fusion layer is connected to the input end of the compression layer; the output end of the compression layer is connected to the input end of the excitation layer; the output end of the excitation layer is connected to the input end of the 1x1 convolution layer; the output end of the 1x1 convolution layer is connected to the normalization layer; the output end of the normalization layer and the output end of the third fusion layer are all connected to the input end of the matrix segmentation layer; the output end of the matrix segmentation layer is connected to the input end of the fourth fusion layer; and the multi-level feature fusion layer is obtained based on the above steps.

[0010] To achieve the above object, the embodiment of the present application further provides a visible light target recognition device based on deep learning, comprising: an acquisition module, configured to acquire a visible light target recognition image; a processing module, configured to process the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network, to obtain a plurality of visible light target recognition feature maps of different scales, and to determine a visible light target recognition result based on the plurality of visible light target recognition feature maps; and an output module, configured to output the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network comprises: determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and a ResNet50-vd of a YOLO algorithm; replacing a basic network Darknet-53 in a YOLOv3 algorithm with the improved ResNet50-vd, wherein the improved ResNet50-vd comprises a first feature output layer to a fifth feature output layer; determining a feature pyramid network based on the respective output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; determining a path aggregation network based on the feature pyramid network; determining a multi-level feature fusion layer based on the path aggregation network; and constructing the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in sequence.

[0011] To achieve the above object, embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the deep learning-based visible light target recognition method provided by any one of the preceding embodiments.

[0012] To achieve the above object, embodiments of the present application further provide a computer readable storage medium storing a computer program, and the computer program is executable by a processor to implement the deep learning-based visible light target recognition method provided by any one of the preceding embodiments.

[0013] Embodiments of the present application provide a deep learning-based visible light target recognition method, device, electronic device and storage medium. The method comprises: acquiring a visible light target recognition image; processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps; and outputting the visible light target recognition result. The small target feature decoupling extraction network can better mine multi-dimensional features of different depth feature layers, thereby improving the accuracy and robustness of detection. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the deep learning-based visible light target recognition method provided in embodiments of the present application;

[0015] Figure 2 is a small target feature decoupling extraction network structure diagram of the deep learning-based visible light target recognition method provided in embodiments of the present application;

[0016] Figure 3 is a multi-level feature fusion layer structure diagram of the deep learning-based visible light target recognition method provided in embodiments of the present application;

[0017] Figure 4 is a structure block diagram of the deep learning-based visible light target recognition device provided in embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation modes of the present application. The embodiments can be combined and referenced with each other without contradiction.

[0019] Reference Figure 1 , Figure 1 The flow chart of the visible light target recognition method based on deep learning provided by the embodiments of the present application can be executed by a server or a terminal device. The visible light target recognition method comprises the following steps.

[0020] S10, acquiring a visible light target recognition image;

[0021] S20, processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps;

[0022] In the embodiments of the present application, before step S20, the process of constructing the small target feature decoupling extraction network specifically comprises the following execution steps:

[0023] S201, determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and a ResNet50-vd of YOLO algorithm;

[0024] Specifically, the ResNet50-vd comprises a first feature output layer to a fifth feature output layer, each layer of the first feature output layer to the fifth feature output layer is a 3x3 convolution layer, and the ResNet50-vd based on the deformable convolution network layer, the small target detection layer and the ResNet50-vd of YOLO algorithm determines the improved ResNet50-vd, comprising:

[0025] S202, replacing the 3x3 convolution layer of the third feature output layer to the fifth feature output layer of the ResNet50-vd with a deformable convolution network layer, and connecting the output end of the second feature output layer of the ResNet50-vd with a small target detection layer to obtain the improved ResNet50-vd.

[0026] In the specific implementation process, the visible light target detection model adopts 8 times, 16 times and 32 times down-sampling feature maps to respectively predict small, medium and large three scales of target samples. In the aerial image, the target scale is smaller, and there are many small targets. After multiple down-sampling, the feature information of the target is too little, and it is difficult to detect such small targets by using the three scale feature maps. In order to improve the feature information extraction amount of small targets, the feature map C2 is input into the neck network to perform feature fusion with the remaining three scale feature maps.

[0027] As shown in Figure 2 {C1, C2, C3, C4, C5} are five scale size feature maps, respectively corresponding to {2, 4, 8, 16, 32} times of down-sampling multiple. The traditional feature pyramid network (FPN) algorithm mainly has the following three shortcomings: 1) the semantic difference between different levels of features is not considered before feature fusion; 2) in the top-down feature fusion process, the high-level features are lost; 3) the ROI of each layer does not combine the useful information of other levels. In view of this, the feature maps C2, C3, C4 and C5 are fused by the feature pyramid, and since C2 has higher resolution and retains more feature information about small targets, it provides more sufficient target information in the fusion process of feature information transmission, thereby enhancing the learning ability of the network. An additional detection head suitable for the C2 scale size is added in this paper, which performs target prediction on the 4 times down-sampled feature map, and can realize the detection of small scale targets, thereby improving the detection ability of small targets.

[0028] In the original YOLOv3, DarkNet-53 is first used to extract feature maps of different scales. Since ResNet has been widely applied and researched more widely, there are more different variants to choose from, and it is also better optimized through a deep learning framework. Therefore, we replace the original backbone network DarkNet-53 with ResNet50-vd in the YOLO algorithm. The Resnet50 network contains 49 convolutional layers and one fully connected layer. The Resnet50 network structure can be divided into seven parts. The first part does not contain residual blocks, mainly for convolution, regularization, activation function and maximum pooling calculation on the input. The second, third, fourth and fifth parts contain residual blocks. In the Resnet50 network structure, the residual block has three convolutional layers, so the network has a total of 1+3×(3+4+6+3)=49 convolutional layers, plus the last fully connected layer, a total of 50 layers, which is the origin of the name Resnet50. The input of the network is 224×224×3, after the convolution calculation of the first five parts, the output is 7×7×2048, and the pooling layer will convert it into a feature vector. Finally, the classifier will calculate the feature vector and output the class probability.

[0029] Replacing DarkNet-53 directly with ResNet50-vd would harm the performance of YOLOv3 detector, we replace some convolutional layers in ResNet50-vd with deformable convolutional layers. A key challenge in vision tasks is how to adapt to the geometric transformations in target scale, pose, viewpoint and model geometry transformations. There are generally two approaches. The first is to build a training dataset with as much varied data as possible, which is usually achieved by augmenting existing data with affine transformations. This way can learn a robust representation, but usually the training is time-consuming and the model is complex with many parameters. The second approach is to use features and algorithms that are invariant to transformations. This includes many famous techniques such as SIFT and sliding window-based object detection algorithms. The above two methods have two disadvantages. First, the geometric transformation is assumed to be fixed and known, as prior knowledge to enhance data and design features and algorithms. This assumption prevents generalization to new tasks with unknown geometric transformations, because these geometric transformations are not properly modeled. Second, artificially designed invariant features and algorithms can be difficult and infeasible for complex transformations, especially when the transformation is unknown. In recent years, CNN has achieved great success in vision tasks. However, it still has the above two problems, and its adaptability to model geometry transformations is mostly from additional data augmentation, the ability of more complex and larger models, and some small artificially designed modules (such as max pooling). In short, CNN is still limited by large, unknown transformations of the model. This limitation comes from the fixed geometry of the CNN module: the convolution unit samples at a fixed position of the input feature map; the pooling layer reduces the spatial resolution by a fixed ratio; the RoI pooling layer generates a fixed-dimensional spatial vector for each RoI. The lack of internal mechanisms to handle geometric transformations can cause problems. For example, the receptive field of all activation units in the same CNN layer is the same. Higher-level CNNs want to encode semantic information into spatial locations, which is not desirable. Because different locations can be associated with different scales and shapes of targets, different scales or receptive field sizes need to be used. Another example is that although object detection has made great progress, recent methods all rely on the original bbox based on feature extraction, which is particularly bad for irregular targets. Therefore, deformable convolutional networks (DCN) are introduced, and the improvement of algorithm accuracy has been verified in many detection models. We apply deformable convolutional layers to all 3x3 convolutional layers of C3, C4 and C5, and remove softmax normalization in spatial aggregation. This not only enhances the dynamic characteristics of the network, but also improves its expression ability. We denote the modified backbone as ResNet50-vd-dcn, where the outputs of feature extraction networks 3, 4 and 5 are denoted as C3, C4 and C5, respectively.

[0030] S203, replacing the base network Darknet-53 in the YOLOv3 algorithm with an improved ResNet50-vd, the improved ResNet50-vd including a first feature output layer to a fifth feature output layer;

[0031] S204, determining a feature pyramid network based on the respective output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd;

[0032] Specifically, the determination of the feature pyramid network based on the respective output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd can include the following execution process:

[0033] S2041, according to the number of channels of the output feature maps of each deformable convolutional network layer, a 1x1 convolutional layer connected to each deformable convolutional network layer is determined, the output end of the fifth feature output layer to the third feature output layer is connected to the input end of each corresponding 1x1 convolutional layer, and the output end of the small target detection layer is connected to the input end of a 1x1 convolutional layer;

[0034] S2042, according to the size of the output feature maps of each deformable convolutional network layer, the parameters of a first upsampling layer are determined, the input end of the first upsampling layer is connected to the output end of the 1x1 convolutional layer;

[0035] S2043, the output end of the deformable convolutional network layer above and the output end of the first upsampling layer are connected to the input end of a first fusion layer, the output end of the first fusion layer is connected to the input end of a first 3x3 convolutional layer, and a first stage of the feature pyramid network is obtained;

[0036] S2044, the second stage to the third stage of the feature pyramid network are determined in the order from top to bottom according to the foregoing steps, and the feature pyramid network is obtained.

[0037] In the specific execution process, in the feature extraction network, the shallow feature maps generated by the shallow network have more target texture features, and therefore contain rich detail information; and the deep feature maps generated by the deep network extract more rich semantic information through a larger receptive field. The basic idea of the feature pyramid (Feature Pyramid Networks, FPN) is to train and test the model by constructing a series of images or feature maps of different scales, with the purpose of improving the robustness of the detection algorithm for detection targets of different sizes. However, if the FPN calculation is directly performed according to the original definition, a large amount of calculation overhead will be brought.

[0038] In order to reduce the amount of calculation, FPN adopts a multi-scale feature fusion method, which can significantly improve the scale robustness of feature expression without greatly increasing the amount of calculation. The FPN algorithm structure includes: a bottom-up line, a top-down line, and a horizontal connection. The bottom-up process is essentially the process of forward propagation of the convolutional network.

[0039] For example, the ResNet network contains several stride=1 and stride=2 convolutions in the forward propagation process. After the stride=1 convolution, the scale of the feature map remains unchanged, and after the stride-2 convolution, the scale of the feature map is reduced to 1 / 2 of the original. We call the continuous scale-invariant feature maps in each stage a network stage. For each stage, the last layer feature map contains the most expressive features in this stage.

[0040] When constructing the feature pyramid, FPN selects the last layer feature map of each stage to build a hierarchical structure. For the ResNet network, the feature map used to construct the feature pyramid is the last residual block of each stage. Directly selecting the channel number of each feature map is different, because the network responsible for subsequent processing needs to slide the window to extract features on different layers of feature maps, which requires all layers of feature maps to have the same depth. In order to make each feature map have the same depth, FPN adds a 1x1 convolution operation to each selected feature map to convert it to a uniform channel number.

[0041] A 1x1 convolution is used on the last residual block of the conv2, conv3, conv4, and conv5 corresponding stages to transform them into d-dimensional channels, respectively, and are labeled as C1, C2, C3, and C4. Because the feature maps between adjacent two stages have a 2-fold scale reduction, the width and height scales of C1, C2, C3, and C4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, and the depth is equal to d.

[0042] The top-down process is essentially to construct a new feature map by scaling the feature map of the upper layer, and the new feature map needs to maintain the same scale as the feature map of the lower layer, so that the feature maps can be fused together. In the long and wide directions, the up-sampling method is used to make the width and height of the upper layer feature map the same size as the lower layer feature map; in the depth direction, a 1x1 convolution is used to compress the depth of the upper layer feature map to the same depth as the lower layer feature map. By using up-sampling and 1x1 convolution operations, a new feature map is constructed from the upper layer feature map that is completely consistent in scale with the lower layer feature map.

[0043] Through the top-down process, the new feature map constructed based on the upper feature map and the original lower feature map have the same scale. The new feature map and the corresponding elements in the original lower feature map are added first, so as to realize the fusion of the upper feature and the lower feature, and then each fused feature map is output as a new feature map with a depth of d. In order to eliminate the problem of insufficient fusion that may be caused by directly adding the corresponding elements of the two feature maps, FPN uses a 3x3 convolution to smooth the feature map after fusion, so as to obtain a more fully fused feature map. At this point, the construction of the feature pyramid is completed, and each feature map of the feature pyramid fuses low-dimensional and high-dimensional features, and the length and width scales of each layer are different, but the number of channels is the same. The network of the subsequent task performs feature extraction on each feature map of the feature pyramid, so as to obtain features of multiple scales. FPN is a general method for constructing a feature pyramid using a convolutional neural network and detecting multiple scales, which can be widely applied to the screening of candidate frames, Fast R-CNN detection and other scenes.

[0044] S205, determining a path aggregation network based on the feature pyramid network;

[0045] With reference to the foregoing Figure 2 In the embodiments of the present application, step S205 can specifically include the following execution process:

[0046] S2051, connecting the output end of the third stage to the input end of the 1x1 convolution layer;

[0047] S2052, connecting the output end of the first up-sampling layer of the third stage and the input end of the 1x1 convolution layer to the input end of the second fusion layer, and connecting the output end of the second fusion layer to the second 3x3 convolution layer, to obtain the fifth stage of the feature pyramid network;

[0048] S2053, sequentially constructing the sixth stage to the eighth stage of the feature pyramid network from bottom to top according to the foregoing two steps, to obtain the path aggregation network.

[0049] In a specific embodiment, bottom-up is essentially the forward pass of the network. During the forward pass, the size of the feature maps changes after passing through some layers, but remains unchanged after passing through others. Layers whose feature map size doesn't change are grouped into one stage. Therefore, the features extracted each time are the output of the last layer of each stage, thus forming a feature pyramid. The top-down process uses upsampling, while lateral connections fuse the upsampled results with the feature maps of the same size generated from the bottom-up process. After fusion, a 3×3 convolutional kernel is used to convolve each fused result to eliminate the aliasing effect of upsampling. It is assumed that the generated feature maps are P2, P3, P4, P5, and correspond one-to-one with the original bottom-up convolutional results C2, C3, C4, C5.

[0050] However, FPN and PANet fuse feature information from different layers on an equal footing, ignoring the relationships between different feature layers. Deep network features, containing rich semantic information, are suitable for detecting large targets, while shallow network features, containing rich detail information, are better suited for detecting small targets. Therefore, the contributions of feature maps from different layers to the target objective are different. In drone aerial image target detection tasks, small target samples account for a larger proportion, and simply fusing deep feature map information weakens the shallow feature map's perception of the location information of small targets.

[0051] S206. Determine a multi-level feature fusion layer based on the path aggregation network;

[0052] In the embodiments of this application, step S206 may specifically include the following execution process:

[0053] S2061. Connect the output terminals of the sixth stage and the eighth stage to the input terminals of the second upsampling layer to the fourth upsampling layer respectively, and connect the output terminals of the second upsampling layer to the fourth upsampling layer to the input terminals of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer respectively.

[0054] S2062. Connect the output of the sixth stage to the input of the downsampling layer, and connect the output of the downsampling layer to the input of the 1×1 convolutional layer.

[0055] S2063. Connect the output terminals of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer and the output terminal of the 1×1 convolutional layer to the input terminal of the third fusion layer.

[0056] S2064. Connect the output of the third fusion layer to the input of the compression layer;

[0057] S2065. Connect the output end of the compression layer to the input end of the excitation layer;

[0058] S2066, connecting the output end of the excitation layer to the input end of the 1x1 convolution layer;

[0059] S2067, connecting the output end of the 1x1 convolution layer to the normalization layer;

[0060] S2068, connecting the output end of the normalization layer and the output end of the third fusion layer to the input end of the matrix segmentation layer;

[0061] S2069, connecting the output end of the matrix segmentation layer to the input end of the fourth fusion layer;

[0062] S20640, obtaining a multi-level feature fusion layer based on the above steps.

[0063] In the specific implementation process, this section designs a multi-level feature fusion layer (MFFL) to give different proportion weights to each scale feature layer, adaptively adjust the contribution degree of shallow and deep features, and strengthen the feature information needed for different size target detection. The shallow and deep features of the four different scale feature layers are effectively utilized, and then the feature information is enriched to guide the unmanned aerial vehicle aerial image target detection work.

[0064] As shown in Figure 3 , the MFFL fuses the four levels of feature maps to regenerate new feature maps. Here, the process of generating new feature maps L3 from P3 through multi-level feature fusion (the generation processes of L2, L4, and L5 are the same as that of L3) is described in detail. First, in order to weight and fuse the four levels of feature maps, they need to be unified to the same size and the same number of channels. P4 and P5 are upsampled to adjust the resolution, and a 3x3 convolution layer is used to adjust the number of channels; P2 is downsampled, and a 1x1 convolution layer is used to adjust the number of channels. The adjusted feature maps R1, R2, R3, and R4 are integrated in the channel dimension to obtain the fused feature map R:

[0065] R = Cat ([R1, R2, R3, R4])

[0066] Due to the different granularities between different feature maps, the fused feature map may introduce conflicting information and redundant information, reducing the expression ability of the feature map. In order to filter noise information, the channel attention of the feature map R is calculated through the squeeze and excitation module to emphasize the expression of relevant information and weaken the interference of irrelevant information. First, the global average pooling is performed on the input R H×W×C to obtain the global feature of each channel:

[0067]

[0068] where Fsq(·) denotes the compression mapping, Z c denotes the global feature space of the c-th channel of the feature map R, u C denotes a point on the c-th channel.

[0069] The nonlinear relationship between the channels is then learned by two fully connected layers W1 and W2, and the weights s of each channel are generated using a Sigmoid activation function:

[0070] s = F ex (x, W) = σ(W2δ(W1z))

[0071] where Fex(·) denotes the excitation mapping, σ(·) denotes the Sigmoid function, and δ(·) denotes the ReLU function. Multiplying the weight s with the feature map R gives the feature map U, which amplifies the response of important information and suppresses the response of irrelevant information. The excitation and compression modules capture the correlation between the channels, mine the long-distance dependency between the input and output feature channels, and reduce the interference of redundant and conflicting information.

[0072] To further capture the spatial features of the feature map, the spatial position information is extracted by compressing the channel dimension through a 1x1 convolution, and then the weight of the feature map in space is normalized using the Softmax function to obtain the weight matrix W ∈ R 4×H×W :

[0073]

[0074] Finally, the weight matrix is divided into α, β, γ, λ along the channel dimension, α, β, γ, λ ∈ R 1×H×W , to obtain the importance weight parameters of each hierarchical feature map, and the new fused feature map L3 is obtained by multiplying each hierarchical feature map with the weight parameters:

[0075] L3 = αR1 + βR2 + γR3 + λR4

[0076] The generation of weight parameters comes from the input of different hierarchical features. During the process of model training, the network continuously learns through gradient backpropagation and adaptively adjusts to the current task. The weight parameters are constantly optimized by the feature learning update process. When the input data information contains mostly small target instances, the weight parameters will give higher weights to shallow feature maps, while deep feature maps will be given lower weights according to the needs of the detection task to guide feature learning. Through the reasonable allocation of shallow and deep features, more accurate recognition of unmanned aerial vehicle image targets is achieved. MFFL fuses the input feature maps {P2, P3, P4, P5} and aggregates more context information from different receptive field feature layers. It adaptively weights and fuses each level of feature maps to get {L2, L3, L4, L5}, fully exploiting the multi-dimensional features of different depth feature layers, and better supervising the feature fusion process of the network, so that the fused features can balance strong semantic information and rich texture detail information.

[0077] The generation of weight parameters is a core step in the training process of deep learning models. During the training process, the weight parameters are constantly updated through the backpropagation algorithm, allowing the model to adaptively adjust to the current task. This feature learning update process constantly optimizes the weight parameters, enabling the model to better understand the input data and complete the task.

[0078] In the target detection task, the input data contains a large number of small target instances, so the weight parameters will give higher weights to shallow feature maps to better detect small targets. For large target instances, deep feature maps are more conducive to detection, so the weight parameters will give lower weights according to the needs of the detection task to guide feature learning.

[0079] To better exploit the multi-dimensional feature information of different hierarchical features, MFFL fuses the input feature maps. By aggregating feature layers from different receptive fields, MFFL can extract more context information and adaptively weight and fuse each level of feature maps to obtain more complete feature representations. This feature fusion process can better supervise the learning process of the network, allowing the fused features to balance strong semantic information and rich texture detail information. Through multi-level feature fusion, the detection performance of small objects can be greatly improved without increasing the amount of calculation. This is because small objects are often ignored in high-level feature maps, while they can be better represented in low-level feature maps. By fusing, we can simultaneously utilize high-level semantic information and low-level positioning information, thereby improving the detection accuracy of small objects.

[0080] In summary, the adaptive adjustment of weight parameters and the feature fusion process are very important steps in deep learning model training, which can effectively improve the performance and generalization ability of the model. In the target detection task, the feature fusion technology of MFFL can better exploit the multi-dimensional features of different depth feature layers, thereby improving the accuracy and robustness of detection.

[0081] For remote sensing target samples, shallow features focus on the details of the target texture, while deep features are more abstract and focus on the abstract information of the target and the surrounding background. Obviously, shallow features can better help locate small targets, so L3 mainly uses shallow feature information to accurately locate the target, and retains a small amount of deep features to help reduce redundant information. Overall, different sizes of targets will change the allocation of MFFL to each level of features, making it tilt in the direction that is beneficial to the current training sample. And in the process of fusing features at each level, MFFL obtains multi-scale information on different receptive fields, which can better exploit the relationship between target pixels and surrounding pixels. This object-to-object relationship, i.e. context information, is beneficial to small target detection.

[0082] In UAV aerial images, small target detection tasks occupy a dominant position. C3 represents the features extracted only through the backbone network, P3 represents the features of C3 after fusion through the feature pyramid FPN and PAN, and L3 represents the feature output of P3 after the multi-level feature fusion layer. C3 features can roughly locate the target position, but the response to the target is weak. P3, compared to C3, learns more fully about the target, making its attention to the object improved, but due to the deep semantic information of the feature pyramid fusion, P3 also adds the interference of the information around the target. L3, on the other hand, strengthens the target features while suppressing the interference of other features to some extent. Visualization analysis shows that the multi-level feature fusion layer strengthens the small target's detailed information by combining multiple levels of features, reduces interference information, and improves discriminability, thereby improving the detection rate of small targets.

[0083] In addition, the visualization of attention heat maps of different feature maps (C3, P3, L3) proves the effectiveness of MFFL in enhancing small target detection. C3 feature maps can only roughly locate weakly responsive target objects, while P3 feature maps have better attention to target objects but are more disturbed by the surrounding context. L3 feature maps not only strengthen the target features, but also suppress the interference of irrelevant features, thereby improving the discriminability and detection rate of small targets.

[0084] Overall, the MFFL module is a powerful tool in the UAV image analysis environment and can be applied to various other computer vision tasks. Its ability to integrate multiple levels of feature maps and adaptively weight their fusion has great potential in improving the performance of deep learning models.

[0085] S207, based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in turn to build the small target feature decoupling extraction network.

[0086] It should be noted that the step division of the above various methods is only for the purpose of clear description. In implementation, one step can be combined or some steps can be split and decomposed into multiple steps, as long as the same logical relationship is included, and all are within the protection scope of the present application. Adding insignificant modifications or introducing insignificant designs in the algorithm or process, but not changing the core design of the algorithm and process, are within the protection scope of the present application.

[0087] On the basis of the above embodiment, the embodiment of the present application further provides a visible light target recognition device based on deep learning. The target recognition device 300 comprises an acquisition module 3001, a processing module 3002 and an output module 3003. The acquisition module 3001 is configured to acquire a visible light target recognition image. The processing module 3002 is configured to process the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network, obtain a plurality of visible light target recognition feature maps of different scales, and determine a visible light target recognition result based on the plurality of visible light target recognition feature maps. The output module 3003 is configured to output the visible light target recognition result. The process of constructing the small target feature decoupling extraction network comprises: determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and a ResNet50-vd of a YOLO algorithm; replacing a basic network Darknet-53 in a YOLOv3 algorithm with the improved ResNet50-vd, wherein the improved ResNet50-vd comprises a first feature output layer to a fifth feature output layer; determining a feature pyramid network based on the respective output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; determining a path aggregation network based on the feature pyramid network; determining a multi-level feature fusion layer based on the path aggregation network; and building the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in turn.

[0088] It is not difficult to find that the present embodiment is a system embodiment corresponding to the above method embodiment, and the present embodiment can be implemented in cooperation with the above method embodiment. The related technical details and technical effects mentioned in the above embodiments are still effective in the present embodiment. In order to reduce repetition, they will not be described here. Accordingly, the related technical details mentioned in the present embodiment can also be applied in the above embodiments.

[0089] It is worth mentioning that each module involved in the embodiment is a logical module. In actual application, one logical unit can be one physical unit, or a part of one physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the embodiment, but this does not mean that there are no other units in the embodiment.

[0090] Another embodiment of the present application provides an electronic device, including: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the deep learning-based visible light target identification method provided by any one of the above embodiments.

[0091] The memory and the processor are connected in a bus manner, and the bus can include any number of interconnected buses and bridges, which connect one or more processors and memories of various circuits together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together, which are well known in the art, and therefore will not be further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0092] The processor is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. Magnetic memory can be used to store data used by the processor during execution.

[0093] Another embodiment of the present application relates to a computer readable storage medium storing a computer program. The computer program is executed by the processor to implement the deep learning-based visible light target identification method provided by the above embodiment.

[0094] That is, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by a program instructing relevant hardware, the program being stored in a storage medium and including a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0095] A person of ordinary skill in the art can understand that the above-mentioned embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A visible light target recognition method based on deep learning, characterized in that, The method comprises the steps of: acquiring a visible light target recognition image; processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain multiple visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the multiple visible light target recognition feature maps; outputting the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network comprises the steps of: determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and ResNet50-vd of a YOLO algorithm; replacing a basic network Darknet-53 in a YOLOv3 algorithm with the improved ResNet50-vd, wherein the improved ResNet50-vd comprises first to fifth feature output layers; determining a feature pyramid network based on output feature maps of the second to fifth feature output layers of the improved ResNet50-vd; determining a path aggregation network based on the feature pyramid network; determining a multi-level feature fusion layer based on the path aggregation network; constructing the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in sequence. 2.The deep learning-based visible light object recognition method of claim 1, wherein, The ResNet50-vd includes a first feature output layer to a fifth feature output layer, each layer being a convolutional layer of 3 3 of the ResNet50-vd based on the deformable convolutional network layer, the small target detection layer, and the YOLO algorithm determines an improved ResNet50-vd, comprising: The third feature output layer to the fifth feature output layer of the ResNet50-vd are replaced by 3 3. The convolutional layers of 3 are replaced by deformable convolutional network layers, and the output end of the second feature output layer of the ResNet50-vd is connected with a small target detection layer to obtain an improved ResNet50-vd. 3.The deep learning-based visible light object recognition method of claim 2, wherein, determining a feature pyramid network based on output feature maps of the second to fifth feature output layers of the improved ResNet50-vd, comprising the steps of: According to the number of channels of the output feature map of each deformable convolutional network layer, a first 1 1 is determined, and the output end of the fifth feature output layer to the third feature output layer is connected with the corresponding 1 1 of the convolutional layer, and the output end of the small target detection layer is connected with the input end of a 1 1 of the convolutional layer. The parameters of the first up-sampling layer are determined according to the size of the output feature map of each deformable convolutional network layer, an input end of the first up-sampling layer is connected with the output end of the convolutional layer of the first deformable convolutional network layer.

1. The output end of the deformable convolution network layer above and the output end of the first up-sampling layer are connected with the input end of a first fusion layer, and the output end of the first fusion layer is connected with a first 3 3convolution layer, to obtain a first stage of the feature pyramid network; determining the second to third stages of the feature pyramid network in a bottom-up order to obtain the feature pyramid network. 4.The deep learning-based visible light object recognition method of claim 3, wherein, determining a path aggregation network based on the feature pyramid network, comprising the steps of: connecting the output of the third stage to the input of the convolutional layer of the first stage 1; Connect the output of the first upsampling layer of the third stage to the 1 The input of the first convolutional layer is connected to the input of the second fusion layer, and the output of the second fusion layer is connected to the second fusion layer. Three convolutional layers are added to obtain the fifth stage of the feature pyramid network; constructing the sixth to eighth stages of the feature pyramid network in a bottom-up order to obtain the path aggregation network. 5.The deep learning-based visible light object recognition method of claim 4, wherein, constructing a multi-level feature fusion layer based on the feature pyramid network, comprising the steps of: The output end of each of the sixth stage and the eighth stage is respectively connected to the input end of each of the second up-sampling layer to the fourth up-sampling layer, and the output end of each of the second up-sampling layer to the fourth up-sampling layer is respectively connected to the input end of each of the third 3 3 convolutional layer to the fifth 3 3 convolutional layer. connecting an output of the sixth stage to an input of a down-sampling layer, an output of the down-sampling connected to an input of the convolutional layer of stage 1 1. The third 3 3 convolutional layers are respectively connected with the input end of the first 1 3 convolutional layers are respectively connected with the input end of the first 1 The output end of the first 1 connecting an output end of the third fusion layer to an input end of a compression layer; connecting an output end of the compression layer to an input end of an excitation layer; connecting an output of the excitation layer to an input of the convolution layer of 1 1. The output of the convolutional layer of the 1 1 is connected to a normalization layer; connecting an output end of the normalization layer and an output end of the third fusion layer to an input end of a matrix segmentation layer; connecting an output end of the matrix segmentation layer to an input end of a fourth fusion layer; obtaining the multi-level feature fusion layer based on the above steps.

6. A visible light target recognition device based on deep learning, characterized in that, The method comprises the steps of: an acquisition module for acquiring a visible light target recognition image; a processing module for processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain multiple visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the multiple visible light target recognition feature maps; an output module for outputting the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network comprises the steps of: determining an improved ResNet50-vd based on a deformable convolution network layer, a small target detection layer and ResNet50-vd of a YOLO algorithm; replace a basic network Darknet-53 in a YOLOv3 algorithm with an improved ResNet50-vd, the improved ResNet50-vd including a first feature output layer to a fifth feature output layer; determine a feature pyramid network based on each of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd outputting a feature map; determine a path aggregation network based on the feature pyramid network; determine a multi-level feature fusion layer based on the path aggregation network; construct the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer connected in sequence.

7. An electronic device, comprising: comprise: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions executed by the at least one processor to enable the at least one processor to perform the deep learning-based visible light target recognition method according to any one of claims 1 to 5.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, can implement the deep learning-based visible light target recognition method according to any one of claims 1 to 5.

Images