Small object detection method based on semantic enhancement and Gaussian loss
Through the semantic enhancement module and the Gaussian Wasserstein distance loss function, the feature fusion and loss function are improved, and the problems of low resolution and difficulty in anchor box matching in small object detection are solved, and the detection accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202310532596.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2025-09-05
- Estimated Expiration
- 2043-05-12
AI Technical Summary
There are problems such as low resolution, loss of information, lack of semantic information, and difficulty in matching anchor boxes in small object detection, and existing methods are difficult to effectively solve.
The semantic enhancement module and the Gaussian Wasserstein distance loss function are used to improve feature fusion and loss functions, combined with deep and shallow features, and use the Gaussian distribution to measure the similarity of the box and smooth the loss calculation.
It improves the accuracy and efficiency of small object detection, solves the problem of difficulty in anchor frame matching, and enhances the fusion and prediction accuracy of feature information.
Smart Images
Figure CN116524274B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of target detection technology, and in particular relates to a small target detection method based on semantic enhancement and Gaussian loss. Background Art
[0002] Although small object detection is a sub-task of target detection, it presents distinct challenges. One of these challenges is the low resolution of small objects. As convolutional networks downsample and extract features, information is lost. In deep feature maps, small objects occupy only a single point, leaving little usable feature information. Meanwhile, the backbone network extracts hierarchical features. Low-level features have high resolution and contain more spatial and positional information, but suffer from high background noise, resulting in a lack of semantic information. High-level features, having passed through multiple convolutional layers, possess stronger semantic information, but suffer from low resolution and a lack of detail. Therefore, to eliminate background while enhancing the feature representation of the detected object, FPN is often used to combine spatial and semantic information. This method fuses feature information at different scales. It can generate feature pyramids with rich semantic information at different scales, adapting to the detection needs of objects of varying sizes and improving detection accuracy and efficiency. FPN can also address the problem of being unable to predict multiple targets when targets overlap at the same location. Another issue is the difficulty in matching anchor frames. For large and medium-sized objects, the quality of anchor frames is generally measured based on the degree of overlap between the two anchor frames (IoU). However, for small objects, IoU-based evaluation methods are not a good measure of anchor frame quality. Since small objects have fewer pixels, the number of pixel deviations accounts for a significant proportion of the total pixels, and small pixel deviations can cause drastic changes in IoU. GIoU, DIoU, and CIoU improve on IoU by adding some restrictions based on the geometric relationship of the frames on the plane to further evaluate the degree of overlap. However, these improvements still cannot eliminate the drastic changes caused by small pixel deviations between frames for small objects. RotatedObjectDetection converts the calculation of the IoU of two frames into the distance between Gaussian distributions. This method circumvents the complex calculation of the IoU between non-horizontal frames. NWD proves that for small objects, this calculation method smooths the changes caused by deviations. Summary of the Invention
[0003] The present invention aims to overcome the above shortcomings of the prior art and provides a small target detection method based on semantic enhancement and Gaussian loss. The present invention includes a semantic enhancement module and Gaussian Wasserstein distance loss to solve the problem that deep feature maps suppress the prediction of small targets in shallow feature maps and the difficulty of matching small target anchor boxes.
[0004] The present invention comprises the following steps:
[0005] A small target detection method based on semantic enhancement and Gaussian loss is proposed, which performs small target detection by improving the current feature fusion method and loss function. The method is characterized by comprising the following steps:
[0006] 1) Perform a series of image preprocessing operations on the images in the training set, including geometric scaling, region filling, affine transformation, and mosaic data enhancement;
[0007] 2) The processed image is then fed into the Darknet-53 feature extraction network. As the backbone network progresses, it outputs feature maps of three scales, 76×76, 38×38, and 19×19, denoted as C1, C2, and C3.
[0008] 3) C1, C2, and C3 then enter the improved FPN network, which is composed of the semantic enhancement fusion module proposed in this example. The feature maps output by FPN are denoted as P1, P2, and P3;
[0009] 4) The feature map after feature fusion enters the Head to directly predict the target category and prediction box position (cx, cy, w, h);
[0010] 5) Evaluate the predicted box results obtained in step 4) using the Gaussian Wasserstein distance loss function and train the model. Preferably, the scales of the three feature maps C1, C2, and C3 in step 2) are 76×76, 38×38, and 19×19 respectively;
[0011] Step 3) The semantic enhancement based on feature fusion includes semantic reorganization and factor fusion, including the following sub-steps:
[0012] (31) First, C1, C2, and C3 are semantically reorganized.
[0013] (32) During semantic reorganization, a 1×1 convolutional compression channel is first used to exchange information between channels while reducing the amount of computation.
[0014] (33) Use a 3×3 convolutional layer to encode the feature map after channel compression, so that each channel can learn the context information of the feature. The encoded feature map is denoted as W l .
[0015] (34) Use the Softmax function to calculate W l Normalize so that W l The sum of the values is 1, thus achieving soft selection of the feature area.
[0016] (35)Through W l Reorganize the local feature area. For a target position l and a square area N(X l,r), the reorganization formula is as follows:
[0017]
[0018] Where X can be any one of C1, C2, C3, and the reorganized feature map is recorded as C′1, C′2, C′3. r is W l The side length is preferably set to 3.
[0019] (36) The key parameter fusion factor α is introduced to balance the feature learning of deep and shallow layers in small object detection. C′1, C′2, and C′3 aggregate adjacent feature layers in the following way:
[0020]
[0021] in is a 1×1 convolution operation for channel matching, f upsample represents the 2× upsampling operation for resolution matching, f layer Convolution operations are usually used for feature processing, and α represents the fusion factor. α is set as a network self-learning parameter. Before the network starts training, its initial value is calculated based on the dataset labels. This parameter participates in gradient backpropagation and is controlled by the loss function. To prevent α from exploding due to network gradients, the following constraint formula is imposed:
[0022]
[0023] N up , N low are the number of scale targets corresponding to the upper and lower feature maps, respectively.
[0024] The Gauss-Wasserstein distance loss function described in step 5) is specifically:
[0025] (51) Use an ellipse to surround the target, so that the background pixels are minimized. The ellipse area surrounding the target is mapped to a 2D Gaussian distribution, and the distribution of the points in the target on the plane follows:
[0026]
[0027] (cx, cy, w, h) indicates that the target in the target detection dataset is annotated using a horizontal box.
[0028] (52) Let Gaussian distribution be N(u,∑):
[0029]
[0030] Where d = 2, indicating a two-dimensional Gaussian distribution, u is the center point of the distribution, u = [cx, cy] T, ∑ is the Covariance matrix, which reflects the shape of the distribution. The elements on the diagonal are the variances in the x and y directions, and the elements on the diagonal are the covariances of x and y, which reflect the degree of linear correlation between the points in the x and y directions. It is 0 when the distribution is horizontal:
[0031]
[0032] The WSD distance between two two-dimensional Gaussian distributions N1(u1,Σ1) and N2(u2,Σ2) can be expressed as follows:
[0033]
[0034] For any two horizontal boxes, Σ1Σ2=Σ2Σ1, and the distance will degenerate into:
[0035]
[0036] where |||| F Represents the Frobenius norm, the square root of the sum of the squares of each number in the matrix. The WSD distance between the predicted box and the true box can be converted to:
[0037]
[0038] where w gt ,h gt is the width and height of the real box, and d represents the WSD distance between the predicted box and the real box.
[0039] (53) After calculating d, it is necessary to map d to between 0 and 1. The following formula is an improved loss function that adds constraints on the width and height of the prediction box:
[0040]
[0041] Where func is the mapping function, C is a normalization parameter that is adaptively calculated during training, V is the aspect ratio, which reflects the similarity between the width and height of the predicted box and the true box, and α is calculated from V.
[0042]
[0043]
[0044] The parameter C allows the mapping function to use more general and simple functions such as e -d , On the other hand, it can better fit objects of different scales. After obtaining the IOU, the performance of the model is evaluated based on the size of the IOU, and the gradient is back-propagated to train the model.
[0045] (54) After obtaining the IOU value, the performance of the model is evaluated based on the size of the IOU, and the gradient is back-propagated to train the model.
[0046] The advantages of the present invention are:
[0047] 1) This paper proposes a semantic enhancement fusion module to improve FPN. It self-learns importance weights for fused features, imposes constraints on weight learning for deep feature maps, and uses perceptual feature reorganization to replace 1×1 convolution before feature fusion, thereby acquiring semantic information with a wider field of view. Traditional FPN suffers from poor fusion when dealing with small objects. Since small objects require sufficient feature information for network learning, they are mostly derived from high-resolution low-level feature maps, which can predict accurate bounding boxes. However, low-level feature maps lack semantic information, posing a significant challenge for subsequent classification. Furthermore, due to the fusion of adjacent layers, FPN is a multi-task learning method. More specifically, if top-down fusion is omitted in FPN, each layer only needs to focus on detecting objects of highly correlated scales. For example, shallow layers learn small-scale objects, while deep layers learn large-scale objects. However, in FPN, with indirect supervision from losses across all layers, each layer must learn objects of nearly all sizes, and even deep layers need to learn small objects. Therefore, each layer must not only focus on objects of its corresponding scale but also seek assistance from other layers to obtain more training samples. Therefore, in order to achieve the purpose of combining deep features rich in semantic information with shallow features containing detail information, a semantic enhancement fusion module is introduced here. The application scenario of this module is in FPN. This method uses feature recombination instead of the 1×1 convolution before FPN fusion. At each position to be fused, semantic recombination is used to use the underlying content information to predict the recombined kernel and recombine the features in a predetermined nearby area. Thanks to the content information, feature recombination can use adaptive and optimized recombined kernels at different positions and achieve better performance than the mainstream 1×1 convolution. In addition, the present invention adds a key parameter, the fusion factor α, to balance the weight of each layer of features participating in the fusion. The factor α is used to achieve the purpose of balancing deep and shallow feature learning in small target detection.
[0048] 2) Introducing the adaptively normalized Gaussian distribution Wasserstein distance loss, replacing the existing IoU calculation with distance adaptive normalization, makes the IoU less affected by changes in target scale, can better cope with data augmentation strategies, and also makes the loss calculation of small target boxes smoother.
[0049] Existing IoU calculation methods have very different sensitivities to objects of different scales. For small targets, a one-pixel offset in the predicted box will cause a huge error in the IoU. Therefore, IoU is not a good metric for tiny objects. This paper proposes a new metric method that uses Gaussian Wasserstein distance to measure the similarity between the predicted box and the true box, replacing the standard IoU. The main advantage of Gaussian Wasserstein distance is that it can still measure the similarity of distributions even when there is no overlap or very little overlap. In addition, Gaussian Wasserstein is insensitive to the size of the target and is therefore more suitable for measuring the similarity between tiny objects.
[0050] In summary, the feature enhancement fusion module and Gaussian distribution Wasserstein distance loss respectively improve the feature information of small targets and enhance the IoU fitting ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 Flowchart of the method of the present invention.
[0052] Figure 2 It is a schematic diagram of the model structure of the present invention. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0054] In one embodiment, Figure 1 As shown in the figure, a small target detection method based on semantic enhancement and Gaussian loss is proposed, including:
[0055] Step S1: perform data enhancement on the training set images.
[0056] The original input is a 608×608 RGB three-channel image. Images that do not meet this size are cropped. Before the image is input into the backbone network, a series of image preprocessing operations are performed, including geometric scaling, region filling, affine transformation, and mosaic data augmentation.
[0057] Step S2: Darknet-53 extracts multi-scale image feature maps.
[0058] Darknet-53, proposed by YOLOv3, draws on the residual structure of ResNet and achieves performance close to that of ResNet-152. As the backbone network deepens, it outputs feature maps at three scales, 76×76, 38×38, and 19×19, denoted as C1, C2, and C3, respectively.
[0059] Step S3: Enter the improved FPN to perform multi-scale feature fusion.
[0060] FPN is composed of the semantic enhancement fusion module proposed in this example, which includes two steps: semantic reconstruction and factor fusion. Figure 2 Specifically, for the three scale feature maps C1, C2, and C3, we first use a 1×1 convolution to compress the channel, which exchanges information between channels while reducing the amount of computation. Then, we use a 3×3 convolution layer to encode the feature map after the compression channel, so that each channel can learn the context information of the feature. The encoded feature is recorded as W l Finally, before entering the reassembly module, W l Normalize the space with Softmax function, this step makes W l The sum of the values is 1, thus achieving soft selection of the feature area. In the following reorganization part, W l The local feature area will be reorganized by the operation of the weighting function. For a square area N(X l ,r), the reorganization formula is as follows:
[0061]
[0062] Where X can be any one of C1, C2, C3, and the reorganized feature map is recorded as C′1, C′2, C′3. r is W l The side length is set to 3 in this example.
[0063] The traditional FPN fusion operation of adjacent layers makes each layer of features multi-task learning. More specifically, if the top-down fusion is omitted, each layer only needs to focus on detecting objects with highly correlated scales, for example, shallow layers learn small-scale targets and deep layers learn large-scale targets. However, in FPN, under the indirect supervision of the losses of all layers, each layer needs to learn targets of almost all sizes, and even deep layers need to learn small targets. Therefore, each layer not only needs to pay attention to its corresponding scale objects, but also needs to get help from other layers to obtain more training samples. Based on this, this example adds a key parameter, the fusion factor α, to achieve a balance between deep and shallow feature learning in small target detection. FPN aggregates adjacent feature layers in the following way:
[0064]
[0065] in is a 1×1 convolution operation for channel matching, f upsample represents the 2× upsampling operation for resolution matching, f layerConvolution operations are usually used for feature processing. Here, this module sets α as a network self-learning parameter. Before the network starts training, its initial value is calculated based on the dataset label. This parameter participates in gradient backpropagation and is controlled by the loss function. At the same time, in order to prevent α from exploding due to network gradients due to excessive size, the following constraint formula is imposed on it:
[0066]
[0067] In the above formula, N up , N low are the number of scale targets corresponding to the upper and lower feature maps, respectively.
[0068] Step S4: Head predicts the category and bounding box of the fused image
[0069] The head consists of a classification branch and a regression branch. The classification branch is used to determine the category of the object enclosed by the current bounding box. The regression branch is used to predict the center coordinates and width and height of the bounding box, recorded as (cx, cy, w, h).
[0070] Step S6: Use the Gaussian Wasserstein distance loss to evaluate the prediction results and train the model. In practice, rectangular objects are rare, which means that the corners of the annotation box are likely to be background pixels. Therefore, if computational complexity is not a concern, using an ellipse to enclose the object minimizes the introduction of background pixels. The ellipse enclosing the object is mapped to a 2D Gaussian distribution. The distribution of points in the object on the plane follows:
[0071]
[0072] Let Gaussian distribution be N(u,∑):
[0073]
[0074] Where d = 2, indicating a two-dimensional Gaussian distribution, u is the center point of the distribution, u = [cx, cy] T , ∑ is the Covariance matrix, which reflects the shape of the distribution. The elements on the diagonal are the variances in the x and y directions, and the elements on the diagonal are the covariances of x and y, which reflect the degree of linear correlation between the points in the x and y directions. It is 0 when the distribution is horizontal:
[0075]
[0076] The WSD distance between two two-dimensional Gaussian distributions N1(u1,Σ1) and N2(u2,Σ2) can be expressed as follows:
[0077]
[0078] For any two horizontal boxes, Σ1Σ2=Σ2Σ1, and the distance will degenerate into:
[0079]
[0080] where |||| F Represents the Frobenius norm, the square root of the sum of the squares of each number in the matrix. The WSD distance between the predicted box and the true box can be converted to:
[0081]
[0082] where w gt ,h gt is the width and height of the true box. After calculating the WSD distance between the predicted box and the label, the WSD value needs to be mapped to the range of 0-1. The following formula is an improved loss function that adds constraints on the width and height of the predicted box:
[0083]
[0084] Where func is the mapping function, C is a normalization parameter that is adaptively calculated during training, V is the aspect ratio, which reflects the similarity between the width and height of the predicted box and the true box, and α is calculated from V.
[0085]
[0086]
[0087] The parameter C allows the mapping function to use more general and simple functions such as e -d , On the other hand, it can better fit objects of different scales. After obtaining the IOU, the performance of the model is evaluated based on the size of the IOU, and the gradient is back-propagated to train the model.
Claims
1. A small object detection method based on semantic enhancement and Gaussian loss, comprising the following steps: 1) Perform image preprocessing operations on the images in the training set, including geometric scaling, region filling, affine transformation, and mosaic data enhancement; 2) The processed image is then input into the Darknet-53 feature extraction network. As the backbone network goes deeper, it outputs feature maps of three scales in turn, which are recorded as ; 3) Then it enters the improved FPN network for semantic enhancement fusion; the feature map output by the FPN network is recorded as The semantic enhancement fusion includes semantic reorganization and factor fusion, and the specific steps are as follows: (31) First Perform semantic reorganization; (32) When semantic reconstruction is performed, a 1×1 convolution compression channel is first used to exchange information between channels while reducing the amount of computation. (33) Use The convolutional layer of different sizes encodes the feature map after the compressed channel, allowing each channel to learn the context information of the feature; the encoded feature map is recorded as ; (34) Use Softmax function to Normalize to make The sum of the values is 1, thus achieving soft selection of the feature area; (35) Pass Reorganize the local feature area; for a target location and the square area centered on it , the reorganization formula is as follows: (1) in Can be Any one of them, the reorganized feature map is recorded as ; for The side length of (36) Introduced fusion factor Balancing deep and shallow feature learning in small object detection; Adjacent feature layers are aggregated as follows: (2) in is a 1 × 1 convolution operation for channel matching, represents a 2× upsampling operation for resolution matching, Usually it is a convolution operation used for feature processing. Represents the fusion factor; Set as the network self-learning parameter. Before the network starts training, its initial value is calculated based on the dataset label. This parameter participates in the gradient back propagation and is controlled by the loss function. At the same time, in order to prevent Too large will cause the network gradient to explode, and the following constraint formula is applied to it: (3) are the number of scale targets corresponding to the upper and lower feature maps respectively; 4) The feature map after feature fusion enters the Head to directly predict the target category and prediction box position ; 5) Evaluate the predicted box results obtained in step 4) using the Gaussian Wasserstein distance loss function and train the model.
2. The small target detection method based on semantic enhancement and Gaussian loss according to claim 1, characterized in that: Step 2) Three feature maps They are 、 and .
3. The small target detection method based on semantic enhancement and Gaussian loss according to claim 1, characterized in that: In step (35) Set to 3.
4. The small target detection method based on semantic enhancement and Gaussian loss according to claim 1, characterized in that: The Gauss-Wasserstein distance loss function described in step 5) is specifically: (51); The elliptical area surrounding the target is mapped to a 2D Gaussian distribution, and the distribution of the points in the target on the plane follows: (4) Indicates that the targets in the target detection dataset are annotated with horizontal boxes; (52) Let Gaussian distribution be : (5) Where d = 2, represents a two-dimensional Gaussian distribution, u is the center point of the distribution, , Covariancematrix, reflecting the shape of the distribution, the elements on the diagonal are Direction and The variance of the direction, the elements on the diagonal are and The covariance of The degree of linear correlation in the direction is 0 when the distribution is horizontal: (6) Two two-dimensional Gaussian distributions and The WSD distance between them can be expressed as follows: (7) For any two horizontal boxes, , the distance will degenerate into: (8) in represents the Frobenius norm, the square root of the sum of the squares of each number in the matrix; the WSD distance between the predicted box and the true box can be converted to: (9) in , is the width and height of the real box, and d represents the WSD distance between the predicted box and the real box; (53) After calculating d, we need to map d to the range of 0-1. The following formula is an improved loss function that adds constraints on the width and height of the prediction box: (10) Where func is the mapping function, C is a normalization parameter that is adaptively calculated during training, V is the aspect ratio, reflecting the similarity between the width and height of the predicted box and the true box, and α is calculated from V; (11) (12) Parameter C allows the mapping function to use more general and simple functions , , On the other hand, it can better fit targets of different scales. After obtaining the IOU, the performance of the model is evaluated according to the size of the IOU, and the gradient is back-propagated to train the model. (54) After obtaining the IOU value, the performance of the model is evaluated based on the size of the IOU, and the gradient is back-propagated to train the model.