Small target detection enhancement method and system based on deep learning semantic information extraction
By integrating the global acceptance domain and design semantic information extraction module in small object detection, the missed detection problem in small object detection is solved, and the detection accuracy is significantly improved.
Patent Information
- Application Number
- CN202411488309.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-03-04
AI Technical Summary
Small object detection is susceptible to interference from complex backgrounds and blurred appearance in aerial images, resulting in false positives and missed detection problems. Existing detectors still cannot effectively solve the missed detection problems after training on small object datasets.
A semantic information extraction method based on deep learning is designed, the global acceptance domain is integrated, and the semantic information extraction module is constructed. The semantic information of the image is extracted through the multi-head cross attention module and the multi-head self-attention module, reducing the loss of semantic information, thereby improving the performance of small object detection.
By reducing the loss of semantic information, the accuracy of small object detection is significantly improved, and the missed detection problem is effectively solved, so that deep learning neural networks can better learn the essential characteristics of the target and improve detection accuracy.
Smart Images
Figure CN119559370A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing enhancement, and specifically relates to a small target detection enhancement method and system based on deep learning semantic information extraction. Background Art
[0002] In the past decade, object detection has been widely used in natural scenes. With the advancement of aerial technology, object detection in aerial images has attracted widespread attention due to its important applications in traffic monitoring, surveillance, and rescue. Compared with natural images, aerial images are usually taken in a larger field of view, and objects are usually small, usually smaller than 16×16 pixels. This makes object detection in aerial images susceptible to interference from complex backgrounds and blurred appearances, leading to confusion between small targets and backgrounds. Therefore, false positives and missed detections become the main problems of small object detection.
[0003] Considering the powerful performance of popular detectors such as YOLOV5 and Faster R-CNN, the most direct way to solve small object detection is to train these existing detectors on small object datasets. However, the current mainstream detectors focus on objects of general size. Directly adopting existing popular detectors cannot produce satisfactory performance in solving false positives and missed detections.
[0004] To solve this problem, most current work improves existing detectors based on the features of tiny objects. Many methods based on YOLOV5 and Faster R-CNN focus on solving the problem of false positives in small target detection and have achieved good results.
[0005] However, missed detection is still the main problem of small object detection. The missed detection problem of small object detection is mainly caused by the lack of semantic information. In the feature extraction process, operations such as downsampling gradually increase the receptive field and improve the ability of high-level features to distinguish between objects and background when losing object resolution. However, the target information in the high-level features appears as highlights but without structural details, that is, lack of semantic information. The lack of semantic information leads to the loss of object positioning and causes missed detection. Some previous works focused on enhancing the existing semantic information in the features and failed to restore the semantic information lost by the network. Therefore, solving the problem of missed detection in small object detection requires a solution for small object features. Summary of the invention
[0006] In order to solve the problems existing in the background technology, the present invention provides a small target detection enhancement method and system based on deep learning semantic information extraction, integrates the global receptive field, and designs a semantic information extraction module to reduce the loss of semantic information, thereby solving the problem of missed detection in small target detection.
[0007] The technical solution to achieve the purpose of the present invention is:
[0008] A small target detection enhancement method based on deep learning semantic information extraction includes the following steps:
[0009] S1, integrates the global receptive field and builds a semantic information extraction module;
[0010] S2, collect a number of images to be detected and annotate the targets, build a training set, input the training set into the target detection network for training, extract the semantic information of the global receptive field of the input image through the semantic information extraction module, input the feature map output by the semantic information extraction module into the backbone network for training, and output it through the feature pyramid and the detection head in turn until the network loss function converges and the training is completed, and the trained target detection network is obtained;
[0011] S3, collect the image to be detected and input it into the trained target detection network for small target detection.
[0012] Furthermore, the extraction process of the semantic information extraction module includes: the input image is passed through a convolution layer to obtain a feature map, the feature map is input into a multi-head cross-attention module and a multi-head self-attention module respectively, the outputs of the two modules are connected through a connection layer and sequentially pass through a convolution layer, an upsampling operation, a convolution layer and an SE module to output semantic information.
[0013] Furthermore, the input image is passed through a 6×6 convolutional layer to obtain a feature map F.
[0014] Furthermore, the process of the multi-head cross attention module for processing the feature map is as follows: the feature map F is split into F by a feature splitting operation with a step size of 2. i , where i∈[1L2,3,4], through four convolutional layers, F i After processing, the feature F is processed through the multi-head cross attention mechanism i The output feature F″ is obtained by processing, namely:
[0015] F″=MHCA(MHCA(Conv(F1), Conv(F2)), MHCA(Conv(F3), Conv(F4))),
[0016] Among them, MHCA(·) represents the multi-head cross attention mechanism, and Conv(·) represents the convolutional layer.
[0017] Furthermore, the process of the multi-head self-attention module performing feature map processing is: processing the feature F through two convolutional layers and the multi-head self-attention mechanism in sequence, F′=MHSA(Conv(Conv(F))), wherein Conv(·) represents the convolutional layer, and MHSA(·) represents the multi-head self-attention mechanism.
[0018] Furthermore, the outputs of the multi-head cross-attention module and the multi-head self-attention module are connected through a connection layer and sequentially passed through a convolutional layer and an upsampling operation to obtain a feature F″′ having the same size as the feature map F, F″′=F+Upsample(Conv(Concat(F′,F″))), wherein Upsample(·) represents an upsampling operation, Conv(·) represents a convolutional layer, and Concat(·) represents a connection layer.
[0019] Furthermore, the feature F″′ passes through the convolution layer and the SE module to obtain the final output feature F″″, F″″=SE(Conv(F″′)), where Conv(·) represents the convolution layer, and SE(·) represents the compression and excitation attention module.
[0020] Furthermore, the backbone network is ResNet, VGG or CSPDarkNet53, and the feature pyramid adopts FPN, PAN or BiFPN.
[0021] Furthermore, the detection head consists of a 1×1 convolutional layer.
[0022] A small target detection enhancement system based on deep learning semantic information extraction, comprising:
[0023] The semantic information extraction module building unit integrates the global receptive field to build the semantic information extraction module;
[0024] The target detection network training unit collects a number of images to be detected and annotates the targets, constructs a training set, inputs the training set into the target detection network for training, extracts the semantic information of the global receptive field of the input image through the semantic information extraction module, and inputs the feature map output by the semantic information extraction module into the backbone network for training, and outputs it through the feature pyramid and the detection head in turn until the network loss function converges and the training is completed, thereby obtaining a trained target detection network;
[0025] The target detection unit collects the images to be detected and inputs them into the trained target detection network for small target detection.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] The method of the present invention takes into account the problem of missed detection of small target detection. Starting from reducing the loss of semantic information in the process of small target detection, the small target detection input image is integrated into the global receptive field and the semantic information is extracted to reduce the loss of semantic information. Deep learning is combined for solving and optimization to complete the detection of small targets. The application of this method can effectively improve the detection effect of existing small target detection methods based on deep learning, so that other existing small target detection methods can effectively solve the problem of missed detection of small target detection after applying this method. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 It is a structural diagram of the semantic information extraction module of the method of the present invention;
[0029] Figure 2 It is a schematic diagram of the feature segmentation operation in the semantic information extraction module of the method of the present invention.
[0030] Figure 3 2 is an enhanced effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION
[0031] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0032] The present invention takes into account the problem of missed detection in the current small target detection method based on deep learning, and integrates the global receptive field of the target image and extracts semantic information based on deep learning to reduce the loss of semantic information, so that the deep learning neural network can better learn the essential characteristics of the target through more semantic information, thereby reducing missed detection of the target and improving the detection performance of the network. The present invention proposes a small target detection enhancement method based on deep learning semantic information extraction, which specifically includes the following steps:
[0033] S1: During the network training process, a semantic information extraction module is constructed, such as Figure 1 As shown, the semantic information extraction module includes a multi-head cross attention module and a multi-head self-attention module. The multi-head cross attention module includes a sequentially connected downsampling operation, a convolutional layer, and a multi-head cross attention mechanism. The multi-head self-attention module includes a sequentially connected convolutional layer and a multi-head self-attention mechanism. The outputs of the two modules are connected through a connection layer and sequentially output through a convolutional layer, an upsampling operation, a convolutional layer, and a SE module, specifically including:
[0034] S1-1: Input a 800×800 resolution RGB three-channel small target image captured by a camera, and use a 6×6 convolution layer to initially extract features. The calculation process is as follows:
[0035] F = Conv(I),
[0036] Among them, Conv(·) represents the convolutional layer, I represents the input image, and F represents the initially extracted features. Where H and W represent the height and width of the original image respectively.
[0037] S1-2: Further process F. In the first part, feature F is split into F by a feature splitting operation with a step size of 2. i , where i∈[1, 2, 3, 4], the specific operation is as follows Figure 2 As shown in Figure 2, the feature segmentation operation can better preserve the details of small target objects in the image. Four 6×6 convolutional layers are used to segment F i After processing, the feature F is processed through the multi-head cross attention mechanism i After processing, the output feature F″ is obtained. Through the multi-head cross attention mechanism, more semantic information can be better extracted and the internal connections of the small target object structure can be better distinguished. The calculation process is as follows:
[0038] F″=MHCA(MHCA(Conv(F1), Conv(F2)), MHCA(Conv(F3), Conv(F4))),
[0039] Among them, MHCA(·) represents the multi-head cross attention mechanism, and Conv(·) represents the convolutional layer.
[0040] S1-3: Further process F. In the second part, feature F is processed by two 6×6 convolutional layers and a multi-head self-attention mechanism to obtain the output feature F′. Through two fast downsampling convolution operations, a feature size suitable for attention mechanism calculation can be formed, reducing the consumption of computing memory. Through the perception of the overall receptive field by the multi-head attention mechanism, more semantic information can be better extracted and the difference between small target objects and background can be better distinguished. The calculation process is as follows:
[0041] F′=MHSA(Conv(Conv(F))),
[0042] Among them, Conv(·) represents the convolutional layer and MHSA(·) represents the multi-head attention mechanism.
[0043] S1-4: The outputs of the two modules are connected through a connection layer and passed through a 3×3 and 1×1 convolution layer, and an upsampling operation in sequence. The size of the result is the same as the size of the feature map F. The two are added together. The calculation process is as follows:
[0044] F″′=F+Upsample(Conv(Concat(F′,F″))),
[0045] Among them, Upsample(·) represents the upsampling operation, Conv(·) represents the convolution layer, and Concat(·) represents the connection layer. By adding feature F, the problem of gradient vanishing can be effectively avoided.
[0046] S1-5: Pass the feature F″′ through a 3×3 and 1×1 convolutional layer and SE module to obtain the final output feature F″″. The calculation process is as follows:
[0047] F″″=SE(Conv(F″′)),
[0048] Among them, Conv(·) represents the convolutional layer, and SE(·) represents the compression and excitation attention module. The SE module obtains the importance of each channel and weights the features on each channel according to this importance, thereby highlighting important features and suppressing unimportant features.
[0049] S2: Input the obtained feature F″″ into the backbone network, wherein the backbone network used can be any backbone network, such as ResNet, VGG, CSPDarkNet53, etc. This embodiment uses CSPDarkNet53 as the backbone network;
[0050] The output of the backbone network is processed in sequence using a feature pyramid network and a detection head. The feature pyramid used can be any feature pyramid structure, such as FPN, PAN, BiFPN, etc. This embodiment uses FPN as the feature pyramid structure. The detection head used can be any detection head structure. This embodiment uses a 1×1 convolution to form a detection head structure and outputs the detection result.
[0051] S3: Collect the image to be detected and input it into the trained target detection network for processing. After processing, the target detection network outputs the enhanced target detection result, including the positioning and classification information of small targets, and marks the position of the target area in the image to complete the detection of small targets in the image.
[0052] A small target detection enhancement system based on deep learning semantic information extraction, comprising:
[0053] The semantic information extraction module building unit integrates the global receptive field to build the semantic information extraction module;
[0054] The target detection network training unit collects a number of images to be detected and annotates the targets, constructs a training set, inputs the training set into the target detection network for training, extracts the semantic information of the global receptive field of the input image through the semantic information extraction module, and inputs the feature map output by the semantic information extraction module into the backbone network for training, and outputs it through the feature pyramid and the detection head in turn until the network loss function converges and the training is completed, thereby obtaining a trained target detection network;
[0055] The target detection unit collects the images to be detected and inputs them into the trained target detection network for small target detection.
[0056] like Figure 3 As shown, the results of small target detection based on two small target images are performed using the method of the present invention and using only a small target detection network, and are compared with the annotated images. It can be seen that there is a missed target in the detection results when the method of the present invention is not applied, while the missed target is eliminated in the detection results when the method of the present invention is applied. Starting from reducing the loss of semantic information in the process of small target detection, the method integrates the small target detection input image into a global receptive field and extracts semantic information to reduce the loss of semantic information, so that the deep learning neural network can better learn the essential characteristics of the target through more semantic information, thereby effectively reducing the existing missed detection phenomenon of small target detection based on deep learning, and has a good small target detection effect in different data domains, effectively improving the detection accuracy of the current small target detection method based on deep learning.
[0057] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A small target detection enhancement method based on deep learning semantic information extraction, characterized in that: Includes steps: S1, integrates the global receptive field and builds a semantic information extraction module; S2, collect a number of images to be detected and annotate the targets, build a training set, input the training set into the target detection network for training, extract the semantic information of the global receptive field of the input image through the semantic information extraction module, input the feature map output by the semantic information extraction module into the backbone network for training, and output it through the feature pyramid and the detection head in turn until the network loss function converges and the training is completed, and the trained target detection network is obtained; S3, collect the image to be detected and input it into the trained target detection network for small target detection.
2. The small target detection enhancement method based on deep learning semantic information extraction according to claim 1 is characterized in that: The extraction process of the semantic information extraction module includes: the input image is passed through a convolution layer to obtain a feature map, the feature map is input into a multi-head cross attention module and a multi-head self-attention module respectively, the outputs of the two modules are connected through a connection layer and sequentially pass through a convolution layer, an upsampling operation, a convolution layer and an SE module to output semantic information.
3. The small target detection enhancement method based on deep learning semantic information extraction according to claim 1 is characterized in that: The input image is passed through a 6×6 convolutional layer to obtain a feature map F.
4. The small target detection enhancement method based on deep learning semantic information extraction according to claim 2 is characterized in that: The process of the multi-head cross attention module to process the feature map is as follows: the feature map F is split into F by a feature splitting operation with a step size of 2. i , where i∈[1,2,3,4], through four convolutional layers, F i After processing, the feature F is processed through the multi-head cross attention mechanism i Processing is performed to obtain the output feature F", namely: F″=MHCA(MHCA(Conv(F1),Conv(F2)),MHCA(Conv(F3),Conv(F4))), where MHCA(·) represents the multi-head cross attention mechanism and Conv(·) represents the convolutional layer.
5. The small target detection enhancement method based on deep learning semantic information extraction according to claim 4 is characterized in that: The process of the multi-head self-attention module performing feature map processing is: processing the feature F through two convolutional layers and the multi-head self-attention mechanism in sequence, F'=MHSA(Conv(Conv(F))), where Conv(·) represents the convolutional layer and MHSA(·) represents the multi-head self-attention mechanism.
6. The small target detection enhancement method based on deep learning semantic information extraction according to claim 5 is characterized in that: The outputs of the multi-head cross-attention module and the multi-head self-attention module are connected through a connection layer and sequentially passed through a convolutional layer and an upsampling operation to obtain a feature F"', F"' = F + Upsample (Conv (Concat (F', F"))) with the same size as the feature map F, wherein Upsample (·) represents an upsampling operation, Conv (·) represents a convolutional layer, and Concat (·) represents a connection layer.
7. The small target detection enhancement method based on deep learning semantic information extraction according to claim 6 is characterized in that: The feature F'' is passed through the convolution layer and the SE module to obtain the final output feature F'', F'''=SE(Conv(F''')), where Conv(·) represents the convolution layer and SE(·) represents the compression and excitation attention module.
8. The small target detection enhancement method based on deep learning semantic information extraction according to claim 1 is characterized in that: The backbone network is ResNet, VGG or CSPDarkNet53, and the feature pyramid uses FPN, PAN or BiFPN.
9. The small target detection enhancement method based on deep learning semantic information extraction according to claim 1 is characterized in that: The detection head consists of a 1×1 convolutional layer.
10. A small target detection enhancement system based on deep learning semantic information extraction for implementing any of the methods described in claims 1-9, characterized in that: include: The semantic information extraction module building unit integrates the global receptive field to build the semantic information extraction module; The target detection network training unit collects a number of images to be detected and annotates the targets, constructs a training set, inputs the training set into the target detection network for training, extracts the semantic information of the global receptive field of the input image through the semantic information extraction module, and inputs the feature map output by the semantic information extraction module into the backbone network for training, and outputs it through the feature pyramid and the detection head in turn until the network loss function converges and the training is completed, thereby obtaining a trained target detection network; The target detection unit collects the images to be detected and inputs them into the trained target detection network for small target detection.