Visible light target recognition method and device based on deep learning, electronic equipment and storage medium

Abstract

The invention provides a visible light target recognition method and device based on deep learning, electronic equipment and a storage medium, and belongs to the technical field of visible light target recognition. Processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps; according to the invention, the small target feature decoupling extraction network can better mine the multi-dimensional features of different depth feature layers, so that the accuracy and robustness of detection are improved.

Description

Technical Field

[0001] The present application relates to the field of visible light target recognition, and in particular to a visible light target recognition method, device, electronic device and storage medium based on deep learning. Background Art

[0002] With the rapid development of remote sensing information technology, the resolution of remote sensing images is getting higher and higher, the amount of information is getting larger and larger, and the detection methods are becoming more and more abundant. The importance of high-resolution remote sensing image target recognition and detection is becoming increasingly prominent in military and civilian applications. The emergence of artificial intelligence technology has provided new ideas and methods for target extraction and recognition of remote sensing images. However, compared with large and medium target detection, there is still a certain gap in the detection effect of small targets in actual remote sensing complex scenes.

[0003] In recent years, remote sensing image target detection and recognition methods based on deep learning have become a research hotspot, and the key to the effectiveness of detection and recognition lies in algorithm + data + calculation, which involves the core issues of deep convolutional neural networks. Especially for aerial remote sensing, when the target size and attributes are different under different scene conditions, how to accurately and quickly extract small targets is still one of the difficulties that need to be solved. Specifically, small target detection mainly faces the following challenges: 1) In the existing target detection model, the underlying features of the backbone network are generally used to detect small targets, but the underlying features lack semantic information, which brings certain difficulties to the detection of small targets; 2) At present, the small target data set widely used by the mainstream target detection algorithm has few training samples, which makes it difficult to fully learn small targets during the model training process; 3) The backbone networks of existing target detection models are all trained on classification data sets, but there is a certain difference between the scale distribution of targets in classification data sets and the scale distribution of targets in detection data sets. Summary of the invention

[0004] In view of this, the embodiments of the present application propose a visible light target recognition method, device, electronic device and storage medium based on deep learning, aiming to improve the detection and recognition capabilities of small targets in complex scenarios using a small amount of data sets.

[0005] To achieve the above-mentioned purpose, an embodiment of the present application provides a visible light target recognition method based on deep learning, comprising: obtaining a visible light target recognition image; processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps; outputting the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network comprises: determining an improved ResNet50-vd based on a deformable convolutional network layer, a small target detection layer and a ResNet50-vd of a YOLO algorithm; converting Y The basic network Darknet-53 in the OLOv3 algorithm is replaced by an improved ResNet50-vd, and the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer; a feature pyramid network is determined based on the feature maps output by the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; a path aggregation network is determined based on the feature pyramid network; a multi-level feature fusion layer is determined based on the path aggregation network; and the small target feature decoupling extraction network is established based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are sequentially connected in communication.

[0006] Optionally, the ResNet50-vd includes a first feature output layer to a fifth feature output layer, each of which is a 3×3 convolution layer. The improved ResNet50-vd determined by the ResNet50-vd based on the deformable convolutional network layer, the small target detection layer and the YOLO algorithm includes: replacing the 3×3 convolutional layers of the third feature output layer to the fifth feature output layer of the ResNet50-vd with a deformable convolutional network layer, and connecting the output end of the second feature output layer of the ResNet50-vd to the small target detection layer to obtain an improved ResNet50-vd.

[0007] Optionally, the output feature maps of the second feature output layer to the fifth feature output layer based on the improved ResNet50-vd are respectively used to determine the feature pyramid network, including: according to the number of channels of the output feature maps of each deformable convolutional network layer, correspondingly determining the 1×1 convolutional layer connected to each deformable convolutional network layer, connecting the output ends of the fifth feature output layer to the third feature output layer to the input ends of the corresponding 1×1 convolutional layers, and connecting the output end of the small target detection layer to the input end of a 1×1 convolutional layer; determining the parameters of the first upsampling layer according to the size of the output feature maps of each deformable convolutional network layer, connecting the input end of the first upsampling layer to the output end of the 1×1 convolutional layer; connecting the output end of the upper deformable convolutional network layer and the output end of the first upsampling layer to the input end of the first fusion layer, connecting the output end of the first fusion layer to the input end of the first 3×3 convolutional layer, to obtain the first stage of the feature pyramid network; determining the second stage to the third stage of the feature pyramid network in order from top to bottom according to the above steps to obtain the feature pyramid network.

[0008] Optionally, determining the path aggregation network based on the feature pyramid network includes: connecting the output end of the third stage to the input end of the 1×1 convolution layer; connecting the output end of the first upsampling layer of the third stage and the input end of the 1×1 convolution layer to the input end of the second fusion layer, and connecting the output end of the second fusion layer to the second 3×3 convolution layer to obtain the fifth stage of the feature pyramid network; constructing the sixth stage to the eighth stage of the feature pyramid network sequentially from bottom to top according to the above two steps to obtain the path aggregation network.

[0009] Optionally, a multi-level feature fusion layer is constructed based on a feature pyramid network, including: the output ends of the sixth stage and the eighth stage are respectively connected to the input ends of the second upsampling layer to the fourth upsampling layer, and the output ends of the second upsampling layer to the fourth upsampling layer are respectively connected to the input ends of the third 3×3 convolution layer to the fifth 3×3 convolution layer; the output end of the sixth stage is connected to the input end of the downsampling layer, and the output end of the downsampling is connected to the input end of the 1×1 convolution layer; the output ends of the third 3×3 convolution layer to the fifth 3×3 convolution layer and the output end of the 1×1 convolution layer are both connected to the input end of the third fusion layer; the output end of the third fusion layer is connected to the input end of the compression layer; the output end of the compression layer is connected to the input end of the excitation layer; the output end of the excitation layer is connected to the input end of the 1×1 convolution layer; the output end of the 1×1 convolution layer is connected to the normalization layer; the output end of the normalization layer and the output end of the third fusion layer are both connected to the input end of the matrix segmentation layer; the output end of the matrix segmentation layer is connected to the input end of the fourth fusion layer; based on the above steps, a multi-level feature fusion layer is obtained.

[0010] To achieve the above-mentioned purpose, the embodiment of the present application also provides a visible light target recognition device based on deep learning, including: an acquisition module, used to acquire a visible light target recognition image; a processing module, used to process the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network, to obtain a plurality of visible light target recognition feature maps of different scales, and to determine a visible light target recognition result based on the plurality of visible light target recognition feature maps; an output module, used to output the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network includes: determining an improved ResNet50-vd based on a deformable convolutional network layer, a small target detection layer and a YOLO algorithm; Net50-vd; replace the basic network Darknet-53 in the YOLOv3 algorithm with an improved ResNet50-vd, wherein the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer; determine a feature pyramid network based on the feature maps output by the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; determine a path aggregation network based on the feature pyramid network; determine a multi-level feature fusion layer based on the path aggregation network; and build the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are sequentially connected in communication.

[0011] To achieve the above-mentioned objectives, an embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based visible light target recognition method provided in any of the aforementioned embodiments.

[0012] To achieve the above objectives, an embodiment of the present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the visible light target recognition method based on deep learning provided by any of the aforementioned embodiments.

[0013] The embodiments of the present application propose a deep learning-based visible light target recognition method, device, electronic device and storage medium, which obtain a visible light target recognition image; process the visible light target recognition image based on a pre-constructed small light target feature decoupling extraction network to obtain multiple visible light target recognition feature maps of different scales, and determine a visible light target recognition result based on the multiple visible light target recognition feature maps; output the visible light target recognition result. The small target feature decoupling extraction network in the present invention can better mine the multi-dimensional features of different deep feature layers, thereby improving the accuracy and robustness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of a visible light target recognition method based on deep learning provided in an embodiment of the present application;

[0015] Figure 2 It is a schematic diagram of a network structure for decoupling and extracting small target features of a visible light target recognition method based on deep learning provided in an embodiment of the present application;

[0016] Figure 3 It is a schematic diagram of the multi-level feature fusion layer structure of the visible light target recognition method based on deep learning provided in the embodiment of the present application;

[0017] Figure 4 It is a structural block diagram of a visible light target recognition device based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0019] refer to Figure 1 , Figure 1 A flow chart of a visible light target recognition method based on deep learning provided in an embodiment of the present application. The method can be executed by a server or a terminal device. The visible light target recognition method includes:

[0020] S10, obtaining a visible light target recognition image;

[0021] S20, processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps;

[0022] In an embodiment of the present application, before step S20, the process of constructing the small target feature decoupling extraction network specifically includes the following execution steps:

[0023] S201, determining an improved ResNet50-vd based on a deformable convolutional network layer, a small target detection layer, and a ResNet50-vd of a YOLO algorithm;

[0024] Specifically, the ResNet50-vd includes a first feature output layer to a fifth feature output layer, each of which is a 3×3 convolution layer, and the ResNet50-vd based on the deformable convolution network layer, the small target detection layer and the YOLO algorithm determines the improved ResNet50-vd, including:

[0025] S202. Replace the 3×3 convolutional layers from the third feature output layer to the fifth feature output layer of the ResNet50-vd with deformable convolutional network layers, and connect the output end of the second feature output layer of the ResNet50-vd to the small target detection layer to obtain an improved ResNet50-vd.

[0026] In the specific implementation process, the visible light target detection model uses feature maps that are downsampled by 8 times, 16 times, and 32 times to predict target samples of small, medium, and large scales, respectively. However, the target scale in aerial images is smaller, and there are many tiny targets. After multiple downsampling, the feature information of the target is too scarce, and the original three-scale feature maps are difficult to detect such extremely small targets. In order to increase the amount of feature information extracted from small targets, the feature map C2 is input into the neck network for feature fusion with the other three scale feature maps.

[0027] like Figure 2 As shown in the figure, {C1, C2, C3, C4, C5} are the feature maps of five scales, corresponding to the downsampling multiples of {2, 4, 8, 16, 32} respectively. The traditional feature pyramid network (FPN) algorithm has the following three main shortcomings: 1) the semantic differences between features at different levels are not considered before feature fusion; 2) high-level features are lost in the top-down feature fusion process; 3) the ROI of each layer does not combine the useful information of other levels. In this regard, the feature map C2 is fused with C3, C4, and C5 through the feature pyramid. Since C2 has a higher resolution and retains more feature information about small targets, it provides more sufficient target information in the fusion process of feature information transmission, thereby enhancing the learning ability of the network. This paper adds an additional detection head adapted to the scale of C2, which performs target prediction on the feature map downsampled 4 times, and can detect extremely small-scale targets, thereby improving the detection ability of small targets.

[0028] In the original YOLOv3, DarkNet-53 was first used to extract feature maps at different scales. As ResNet has been widely used and studied more widely, there are more different variants to choose from, and it has also been better optimized through the deep learning framework. Therefore, we replaced the original backbone network DarkNet-53 with ResNet50-vd in the YOLO algorithm. The Resnet50 network contains 49 convolutional layers and one fully connected layer. The Resnet50 network structure can be divided into seven parts. The first part does not contain residual blocks, and mainly performs convolution, regularization, activation function, and maximum pooling calculations on the input. The second, third, fourth, and fifth parts of the structure all contain residual blocks. In the Resnet50 network structure, the residual blocks have three layers of convolution, so the network has a total of 1+3×(3+4+6+3)=49 convolutional layers, plus the final fully connected layer, a total of 50 layers, which is also the origin of the name Resnet50. The input of the network is 224×224×3. After the convolution calculation of the first five parts, the output is 7×7×2048. The pooling layer will convert it into a feature vector. Finally, the classifier will calculate this feature vector and output the category probability.

[0029] Considering that directly replacing DarkNet-53 with ResNet50-vd will hurt the performance of YOLOv3 detector, we replaced some convolutional layers in ResNet50-vd with deformable convolutional layers. The key challenge in vision tasks is how to adapt to geometric transformations and model geometric transformations in object scale, pose, viewpoint and deformation. There are generally two approaches. The first is to create a training dataset with as much variation as possible, which is usually achieved by augmenting existing data, such as affine transformation. This can learn robust representations, but it is usually time-consuming to train and the model is complex and has many parameters. The second approach is to use features and algorithms that are invariant to transformations. This includes many well-known techniques, such as SIFT and sliding window-based object detection algorithms. The above methods have two disadvantages. First, it assumes that the geometric transformation is fixed and known, and uses it as prior knowledge to augment data and design features and algorithms. This assumption prevents generalization to new tasks with unknown geometric transformations because these geometric transformations are not modeled correctly. Second, manually designed invariant features and algorithms may be difficult and infeasible for complex transformations, especially when the transformation is unknown. In recent years, CNN has achieved great success in visual tasks. However, it still has the above two problems. Their adaptability to model geometric transformations mostly comes from additional data augmentation, the ability of more complex and larger models, and some small artificially designed modules (such as max pooling). In short, CNN is still limited by large, unknown transformations of the model. This limitation stems from the fixed geometric structure of CNN modules: convolution units sample at fixed positions of the input feature map; pooling layers reduce the spatial resolution at a fixed ratio; RoI pooling layers generate fixed-dimensional spatial vectors for each RoI. The lack of internal mechanisms to handle geometric transformations can cause some problems. For example, the receptive fields of all activation units in the same CNN layer are the same. This is undesirable for high-level CNNs that want to encode semantic information into spatial locations. Because different locations may be associated with objects of different scales and shapes, different scales or receptive field sizes need to be used. Another example is that despite great progress in object detection, all recent methods rely on primitive bbox-based feature extraction, which is particularly bad for irregular objects. Therefore, the deformable convolutional network (DCN) is introduced. The improvement of algorithm accuracy by the deformable convolutional network has been verified in many detection models. The deformable convolutional layer is applied to all 3x3 convolutional layers of C3, C4 and C5. In the spatial aggregation, the softmax normalization is removed. This not only enhances the dynamic characteristics of the network, but also improves its expressiveness. We denote this modified backbone as ResNet50-vd-dcn, where the outputs of the feature extraction network 3, 4 and 5 are denoted as C3, C4, C5 respectively.

[0030] S203, replacing the basic network Darknet-53 in the YOLOv3 algorithm with an improved ResNet50-vd, wherein the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer;

[0031] S204, determining a feature pyramid network based on the output feature graphs of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd;

[0032] Specifically, the determining of the feature pyramid network based on the output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd may include the following execution process:

[0033] S2041, according to the number of channels of the output feature map of each deformable convolutional network layer, correspondingly determining a 1×1 convolutional layer connected to each deformable convolutional network layer, the output end of the fifth feature output layer to the third feature output layer is connected to the input end of each corresponding 1×1 convolutional layer, and the output end of the small target detection layer is connected to the input end of one of the 1×1 convolutional layers;

[0034] S2042, determining parameters of a first upsampling layer according to the size of the output feature map of each of the deformable convolutional network layers, wherein an input end of the first upsampling layer is connected to an output end of the 1×1 convolutional layer;

[0035] S2043, connecting the output end of the upper deformable convolutional network layer and the output end of the first upsampling layer to the input end of the first fusion layer, and connecting the output end of the first fusion layer to the input end of the first 3×3 convolutional layer, to obtain the first stage of the feature pyramid network;

[0036] S2044. Determine the second stage to the third stage of the feature pyramid network in order from top to bottom according to the above steps to obtain the feature pyramid network.

[0037] In the specific execution process, in the feature extraction network, the shallow feature map generated by the shallow network has more target texture features and therefore contains rich detail information; while the deep feature map generated by the deep network extracts richer semantic information through a larger receptive field. The basic idea of ​​Feature Pyramid Networks (FPN) is to train and test the model by constructing a series of images or feature maps of different scales, in order to improve the robustness of the detection algorithm for detection targets of different sizes. However, if the FPN calculation is performed directly according to the original definition, it will bring a large amount of computational overhead.

[0038] In order to reduce the amount of calculation, FPN adopts a multi-scale feature fusion method, which can significantly improve the scale robustness of feature expression without significantly increasing the amount of calculation. The FPN algorithm structure includes: a bottom-up line, a top-down line, and horizontal connections. The bottom-up process is essentially the forward propagation process of the convolutional network.

[0039] For example, during the forward propagation of the ResNet network, there are several convolutions with stride=1 and stride=2. After the convolution with stride=1, the scale of the feature map remains unchanged, and after the convolution with stride-2, the scale of the feature map is reduced to 1 / 2 of the original. We call the successive scale-invariant feature maps a network stage. For each stage, the last layer of feature maps contains the most expressive features in this stage.

[0040] When FPN constructs the feature pyramid, it selects the last layer of feature maps in each stage to build a hierarchical structure. For the ResNet network, the feature map used to construct the feature pyramid is the last residual block in each stage. The number of channels directly selected for each feature map is different. This is because the network responsible for subsequent processing needs to slide the window on the feature maps of different layers to intercept features, which requires that the feature maps of all layers have the same depth. In order to make each feature map have the same depth, FPN adds a 1×1 convolution operation to each selected feature map to convert it into a uniform number of channels.

[0041] A 1×1 convolution is used on the last residual block of the corresponding stages of conv2, conv3, conv4, and conv5 to transform them into d-dimensional channels, and they are marked as C1, C2, C3, and C4 respectively. Because the feature maps between two adjacent stages are scaled by 2 times, the width and height scales of C1, C2, C3, and C4 are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively, and the depth is all equal to the d dimension.

[0042] The top-down process is essentially to construct a new feature map by scaling the feature map of the upper layer. The new feature map needs to maintain the same scale as the feature map of the lower layer to ensure that the feature maps can be fused together. In the length and width directions, upsampling is used to make it the same size as the width and height of the feature map of the lower layer; in the depth direction, a 1×1 convolution is used to compress the depth of the upper feature map to the same depth as the feature map of the lower layer. By combining upsampling operations and 1×1 convolution operations, a new feature map with the same scale as the feature map of the lower layer is constructed from the upper feature map.

[0043] After a top-down process, the new feature map constructed based on the upper feature map has the same scale as the original lower feature map. First, each corresponding element in the new feature map and the original lower feature map is added to achieve the fusion of the upper and lower features, and then each fused feature map is output as a new feature map with a depth of d. In order to eliminate the problem of insufficient fusion that may be caused by the direct addition of the corresponding elements of the two feature maps, FPN uses a 3×3 convolution to smooth the fused feature map, thereby obtaining a more fully fused feature map. At this point, the construction of the feature pyramid is completed. Each layer of the feature map of the feature pyramid integrates low-dimensional and high-dimensional features. The length and width scales of each layer are different, but the number of channels is the same. The network of the subsequent task performs feature interception on each feature map of the feature pyramid, and can obtain features of multiple scales. FPN is a general method for constructing a feature pyramid using a convolutional neural network for multi-scale object detection. It can be widely used in scenarios such as candidate box screening and Fast R-CNN detection.

[0044] S205, determining a path aggregation network based on the feature pyramid network;

[0045] Continue to refer Figure 2 In the embodiment of the present application, step S205 may specifically include the following execution process:

[0046] S2051, the output end of the third stage is connected to the input end of the 1×1 convolutional layer;

[0047] S2052, connecting the output end of the first upsampling layer of the third stage and the input end of the 1×1 convolutional layer to the input end of the second fusion layer, and connecting the output end of the second fusion layer to the second 3×3 convolutional layer, to obtain the fifth stage of the feature pyramid network;

[0048] S2053. Construct the sixth stage to the eighth stage of the feature pyramid network sequentially from bottom to top according to the above two steps to obtain the path aggregation network.

[0049] In a specific embodiment, bottom-up is actually the forward process of the network. In the forward process, the size of the feature map will change after passing through certain layers, but will not change when passing through other layers. The layers that do not change the size of the feature map are classified as one stage, so each extracted feature is the output of the last layer of each stage, so that a feature pyramid can be formed. The top-down process is performed by upsampling, and the lateral connection is to fuse the upsampling result with the feature map of the same size generated from the bottom up. After fusion, a 3×3 convolution kernel will be used to convolve each fusion result in order to eliminate the aliasing effect of upsampling. It is assumed that the generated feature map results are P2, P3, P4, P5, which correspond one-to-one to the original bottom-up convolution results C2, C3, C4, C5.

[0050] However, FPN and PANet fuse feature information of different levels across layers in an equal relationship, ignoring the relationship between different feature layers. The rich semantic information contained in deep network features is suitable for detecting large targets, while the rich detail information contained in shallow network features is more suitable for detecting small targets. Therefore, the contribution of feature maps of different levels to task objectives is different. In the target detection task of drone aerial images, small target samples account for a larger proportion. Simply fusing deep feature map information will weaken the perception of shallow feature maps on the location information of small targets.

[0051] S206, determining a multi-level feature fusion layer based on the path aggregation network;

[0052] In an embodiment of the present application, step S206 may specifically include the following execution process:

[0053] S2061, connecting the output ends of the sixth stage and the eighth stage to the input ends of the second upsampling layer to the fourth upsampling layer, and connecting the output ends of the second upsampling layer to the fourth upsampling layer to the input ends of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer;

[0054] S2062, connecting the output end of the sixth stage to the input end of the downsampling layer, and connecting the output end of the downsampling layer to the input end of the 1×1 convolutional layer;

[0055] S2063, connecting the output ends of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer and the output end of the 1×1 convolutional layer to the input end of the third fusion layer;

[0056] S2064, connecting the output end of the third fusion layer to the input end of the compression layer;

[0057] S2065, connecting the output end of the compression layer to the input end of the excitation layer;

[0058] S2066, connecting the output end of the excitation layer to the input end of the 1×1 convolutional layer;

[0059] S2067, connecting the output end of the 1×1 convolutional layer to a normalization layer;

[0060] S2068, connecting the output end of the normalization layer and the output end of the third fusion layer to the input end of the matrix segmentation layer;

[0061] S2069, connecting the output end of the matrix segmentation layer to the input end of the fourth fusion layer;

[0062] S20640. Based on the above steps, a multi-level feature fusion layer is obtained.

[0063] In the specific implementation process, this section designs a multi-level feature fusion layer (MFFL), assigns different proportional weights to feature layers of each scale, adaptively adjusts the contribution of shallow and deep features, strengthens the feature information required for detecting targets of different sizes, and effectively utilizes the shallow and deep features of four different scale feature layers, thereby enriching the feature information to guide the target detection work of UAV aerial images.

[0064] like Figure 3 As shown in the figure, MFFL fuses the feature maps of four levels to regenerate a new feature map. Here, the process of generating a new feature map L3 by fusing multiple levels of features from P3 is described in detail (the generation process of L2, L4, and L5 is the same as L3). First, in order to weightedly fuse the feature maps of the four levels, they need to be unified into the same size and the same number of channels. P4 and P5 are upsampled to adjust the resolution, and the number of channels is adjusted using a 3×3 convolution layer; P2 is downsampled and the number of channels is adjusted through a 1×1 convolution layer. The adjusted feature maps R1, R2, R3, and R4 are integrated in the channel dimension to obtain the fused feature map R:

[0065] R=Cat([R1,R2,R3,R4])

[0066] Due to the different granularity between different feature maps, the fused feature map may introduce conflicting and redundant information, reducing the expressiveness of the feature map. In order to filter out noise information, the channel attention of the feature map R is calculated through the Squeeze and Excitation module to emphasize the expression of relevant information and weaken the interference of irrelevant information. First, the input R∈R H×W×C Perform global average pooling to obtain the global features of each channel:

[0067]

[0068] Among them, Fsq(·) represents the compression mapping, Z c Represented as the global feature space of the cth channel of the feature map R, u C Represents a point on the cth channel.

[0069] Then, two fully connected layers W1 and W2 are used to learn the nonlinear relationship between channels, and the Sigmoid activation function is used to generate the weight s of each channel:

[0070] s=F ex (x, W) = σ(W2δ(W1z))

[0071] Among them, Fex(·) represents the excitation mapping, σ(·) represents the Sigmoid function, and δ(·) represents the ReLU function. The weight s is multiplied by the feature map R to obtain the feature map U, which amplifies the response of important information and suppresses the response of irrelevant information. The excitation and compression module captures the correlation between channels, mines the long-range dependency between input and output feature channels, and reduces the interference of redundant and conflicting information.

[0072] In order to further capture the spatial features of the feature map, the channel dimension is compressed by 1×1 convolution to extract the spatial position information, and then the Softmax function is used to normalize the weight of the feature map in space to obtain the weight matrix W∈R 4×H×W :

[0073]

[0074] Finally, the weight matrix is ​​divided into α, β, γ, λ along the channel dimension, α, β, γ, λ∈R 1×H×W , the importance weight parameters of the feature maps at each level are obtained, and the feature maps at each level are multiplied by the weight parameters to obtain the fused new feature map L3:

[0075] L3=αR1+βR2+γR3+λR4

[0076] The weight parameters are generated from the input of features at different levels. During the model training process, they are continuously learned through the network's gradient back propagation and adaptively adjusted to adapt to the current task. The weight parameters are continuously optimized by the update process of feature learning. When the input data information contains mostly small target instances, the weight parameters will give higher weights to the shallow feature maps, while the deep feature maps will give lower weights to guide feature learning according to the needs of the detection task. Through the reasonable allocation of shallow and deep features, more accurate recognition of drone aerial image targets is achieved. MFFL fuses the input feature maps {P2, P3, P4, P5}, aggregates more contextual information from feature layers of different receptive fields, and adaptively weights and fuses feature maps of each level to obtain {L2, L3, L4, L5}, fully exploiting the multi-dimensional features of different deep feature layers, which can better supervise the feature fusion process of the network, so that the fused features take into account both powerful semantic information and rich texture detail information.

[0077] The generation of weight parameters is a core step in the deep learning model training process. During the training process, the weight parameters are continuously updated through the back propagation algorithm so that the model can adaptively adjust to the current task. This feature learning update process continuously optimizes the weight parameters so that the model can better understand the input data and complete the task.

[0078] In the target detection task, the input data contains a large number of small target instances, so the weight parameter will give higher weights to the shallow feature map to better detect small targets. For large target instances, the deep feature map is more conducive to detection, so the weight parameter will give lower weights according to the needs of the detection task to guide feature learning.

[0079] In order to better mine the multi-dimensional feature information of features at different levels, MFFL fuses the input feature maps. By aggregating feature layers with different receptive fields, MFFL can extract more contextual information and adaptively weight the fusion of feature maps at each level to obtain a more adequate feature representation. This feature fusion process can better supervise the learning process of the network, so that the fused features can take into account both powerful semantic information and rich texture detail information. Through multi-level feature fusion, the detection performance of small objects can be greatly improved without increasing the amount of computation. This is because small objects are often ignored in high-level feature maps, but can be better represented in low-level feature maps. Through fusion, we can simultaneously utilize high-level semantic information and low-level positioning information to improve the detection accuracy of small objects.

[0080] In short, the adaptive adjustment of weight parameters and the feature fusion process are very important steps in deep learning model training, which can effectively improve the performance and generalization ability of the model. In the target detection task, MFFL's feature fusion technology can better mine the multi-dimensional features of different deep feature layers, thereby improving the accuracy and robustness of detection.

[0081] For remote sensing target samples, shallow features focus on the detailed texture of the target while deep features focus more generally on the abstract information of the target and the surrounding background. Obviously, shallow features can better help locate small targets. Therefore, L3 uses shallow feature information as the main factor to accurately locate the target, and retains a small amount of deep features to help reduce redundant information. In general, targets of different sizes will change MFFL's allocation of features at all levels, making it tilt in a direction that is beneficial to the current training sample. In addition, in the process of fusing features at all levels, MFFL obtains multi-scale information obtained in different receptive fields, which can better mine the connection between target pixels and surrounding pixels. This relationship between objects, namely contextual information, is conducive to the detection of small targets.

[0082] In drone aerial images, small target detection tasks occupy a major position. C3 represents the features extracted only by the backbone network, P3 represents the features of C3 after fusion of feature pyramid FPN and PAN, and L3 represents the feature output of P3 after multi-level feature fusion layer. C3 features can roughly locate the target position, but the response to the target is weak. Compared with C3, P3 learns the target more fully, which improves its attention to the object. However, due to the deep semantic information fused by the feature pyramid, P3 also adds interference from the information around the target. While strengthening the target features, L3 suppresses the interference of other features to a certain extent. From the visual analysis, it can be seen that the multi-level feature fusion layer strengthens the detailed information of small targets through multi-level features, reduces interference information, and improves discrimination, thereby improving the detection rate of small targets.

[0083] In addition, the visualization of attention heatmaps of different feature maps (C3, P3, L3) demonstrates the effectiveness of MFFL in enhancing small object detection. The C3 feature map can only roughly locate the target object with a weak response, while the P3 feature map pays better attention to the target object but is more disturbed by the surrounding context. The L3 feature map not only strengthens the target features but also suppresses the interference of irrelevant features, thereby improving the discriminability and detection rate of small objects.

[0084] Overall, the MFFL module is a powerful tool in the context of UAV image analysis and can be applied to a variety of other computer vision tasks. Its ability to integrate multi-level feature maps and adaptively weight their fusion has great potential in improving the performance of deep learning models.

[0085] S207. Building the small target feature decoupling extraction network based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are sequentially connected in communication.

[0086] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this application.

[0087] On the basis of the above embodiments, the embodiment of the present application further provides a visible light target recognition device based on deep learning, wherein the target recognition device 300 includes an acquisition module 3001, a processing module 3002 and an output module 3003, wherein the acquisition module 3001 is used to acquire a visible light target recognition image; the processing module 3002 is used to process the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determine a visible light target recognition result based on the plurality of visible light target recognition feature maps; the output module 3003 is used to output the visible light target recognition result; wherein the process of constructing the small target feature decoupling extraction network includes: based on a deformable convolutional network layer, a small target detection layer and a YO The ResNet50-vd of the LO algorithm determines the improved ResNet50-vd; the basic network Darknet-53 in the YOLOv3 algorithm is replaced by the improved ResNet50-vd, and the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer; based on the second feature output layer to the fifth feature output layer of the improved ResNet50-vd, each output feature map determines a feature pyramid network; based on the feature pyramid network, a path aggregation network is determined; based on the path aggregation network, a multi-level feature fusion layer is determined; based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are sequentially connected in communication, the small target feature decoupling extraction network is established.

[0088] It is not difficult to find that this embodiment is a system embodiment corresponding to the above method embodiment, and this embodiment can be implemented in conjunction with the above method embodiment. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiments.

[0089] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0090] Another embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the deep learning-based visible light target recognition method provided in any one of the above embodiments.

[0091] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripheral devices, voltage regulators and power management circuits together, which are all well known in the art, so this article will not describe them further. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on the wireless medium through the antenna, and further, the antenna also receives the data and transmits the data to the processor.

[0092] The processor is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interfacing, voltage regulation, power management, and other control functions. Magnetic storage may be used to store data used by the processor when performing operations.

[0093] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the method for identifying visible light targets based on deep learning provided in the above embodiment is implemented.

[0094] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM (Read-Only Memory), RAM (Random Access Memory), disk or optical disk and other media that can store program codes.

[0095] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A visible light target recognition method based on deep learning, characterized in that: include: Acquire visible light target recognition images; Processing the visible light target recognition image based on a pre-constructed small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determining a visible light target recognition result based on the plurality of visible light target recognition feature maps; Outputting the visible light target recognition result; The process of constructing the small target feature decoupling extraction network includes: Determine the improved ResNet50-vd based on the deformable convolutional network layer, small object detection layer and ResNet50-vd of the YOLO algorithm; The basic network Darknet-53 in the YOLOv3 algorithm is replaced by an improved ResNet50-vd, wherein the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer; Determine a feature pyramid network based on the output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; Determine a path aggregation network based on the feature pyramid network; Determining a multi-level feature fusion layer based on the path aggregation network; The small target feature decoupling extraction network is constructed based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are connected in sequence.

2. The visible light target recognition method based on deep learning according to claim 1, characterized in that: The ResNet50-vd includes a first feature output layer to a fifth feature output layer, each of which is a 3×3 convolution layer, and the ResNet50-vd based on the deformable convolution network layer, the small target detection layer and the YOLO algorithm determines the improved ResNet50-vd, including: The 3×3 convolutional layers from the third feature output layer to the fifth feature output layer of the ResNet50-vd are replaced with deformable convolutional network layers, and the output end of the second feature output layer of the ResNet50-vd is connected to the small object detection layer to obtain an improved ResNet50-vd.

3. The visible light target recognition method based on deep learning according to claim 2, characterized in that: The determining of the feature pyramid network based on the output feature maps of the second feature output layer to the fifth feature output layer based on the improved ResNet50-vd comprises: According to the number of channels of the output feature map of each deformable convolutional network layer, a 1×1 convolutional layer connected to each deformable convolutional network layer is correspondingly determined, the output ends of the fifth feature output layer to the third feature output layer are connected to the input ends of the corresponding 1×1 convolutional layers, and the output end of the small target detection layer is connected to the input end of one of the 1×1 convolutional layers; Determining parameters of a first upsampling layer according to the size of the output feature map of each deformable convolutional network layer, wherein an input end of the first upsampling layer is connected to an output end of the 1×1 convolutional layer; The output end of the upper deformable convolutional network layer and the output end of the first upsampling layer are connected to the input end of the first fusion layer, and the output end of the first fusion layer is connected to the input end of the first 3×3 convolutional layer to obtain the first stage of the feature pyramid network; According to the above steps, the second stage to the third stage of the feature pyramid network are determined sequentially from top to bottom to obtain the feature pyramid network.

4. The visible light target recognition method based on deep learning according to claim 3 is characterized in that: The determining of the path aggregation network based on the feature pyramid network includes: Connect the output end of the third stage to the input end of the 1×1 convolutional layer; The output end of the first upsampling layer of the third stage and the input end of the 1×1 convolutional layer are connected to the input end of the second fusion layer, and the output end of the second fusion layer is connected to the second 3×3 convolutional layer to obtain the fifth stage of the feature pyramid network; According to the above two steps, the sixth stage to the eighth stage of the feature pyramid network are sequentially constructed from bottom to top to obtain the path aggregation network.

5. The visible light target recognition method based on deep learning according to claim 4 is characterized in that: Construct a multi-level feature fusion layer based on the feature pyramid network, including: Connect the output ends of the sixth stage and the eighth stage to the input ends of the second upsampling layer to the fourth upsampling layer, respectively, and connect the output ends of the second upsampling layer to the fourth upsampling layer to the input ends of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer; Connect the output end of the sixth stage to the input end of the downsampling layer, and the output end of the downsampling is connected to the input end of the 1×1 convolutional layer; Connecting the output ends of the third 3×3 convolutional layer to the fifth 3×3 convolutional layer and the output end of the 1×1 convolutional layer to the input end of the third fusion layer; Connecting the output end of the third fusion layer to the input end of the compression layer; Connecting the output end of the compression layer to the input end of the excitation layer; Connecting the output end of the excitation layer to the input end of the 1×1 convolutional layer; Connect the output end of the 1×1 convolutional layer to a normalization layer; Connecting the output end of the normalization layer and the output end of the third fusion layer to the input end of the matrix segmentation layer; Connecting the output end of the matrix segmentation layer to the input end of the fourth fusion layer; Based on the above steps, a multi-level feature fusion layer is obtained.

6. A visible light target recognition device based on deep learning, characterized in that: include: An acquisition module, used for acquiring a visible light target recognition image; A processing module, used to process the visible light target recognition image based on a pre-built small target feature decoupling extraction network to obtain a plurality of visible light target recognition feature maps of different scales, and determine a visible light target recognition result based on the plurality of visible light target recognition feature maps; An output module, used for outputting the visible light target recognition result; The process of constructing the small target feature decoupling extraction network includes: Determine the improved ResNet50-vd based on the deformable convolutional network layer, small object detection layer and ResNet50-vd of the YOLO algorithm; The basic network Darknet-53 in the YOLOv3 algorithm is replaced by an improved ResNet50-vd, wherein the improved ResNet50-vd includes the first feature output layer to the fifth feature output layer; Determine a feature pyramid network based on the output feature maps of the second feature output layer to the fifth feature output layer of the improved ResNet50-vd; Determine a path aggregation network based on the feature pyramid network; Determining a multi-level feature fusion layer based on the path aggregation network; The small target feature decoupling extraction network is constructed based on the improved ResNet50-vd, the feature pyramid network and the multi-level feature fusion layer that are connected in sequence.

7. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the visible light target recognition method based on deep learning as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement the visible light target recognition method based on deep learning as described in any one of claims 1 to 5.

Images