Small target image detection method based on context semantic information aggregation

Abstract

The invention relates to a small target image detection method based on context semantic information aggregation, and belongs to the technical field of computing camera shooting. According to the method, a two-stage detection model is adopted, and a context information aggregation-based feature enhancement module and a multi-stage detection head auxiliary model are introduced, so that accurate detection of a small target is realized. According to the method, the internal association between the similar small target features can be quickly learned in the complex natural image features, and the learned internal association between the similar small target features and the context information are aggregated with the small target feature information extracted by the network in the reasoning process of target detection. And the feature representation of the small target is effectively enhanced. According to the method, a good detection and recognition effect can be achieved without relying on specific data set training.

Description

Technical Field

[0001] The invention relates to a small target image detection method based on context semantic information aggregation, and belongs to the technical field of computational photography. Background Art

[0002] Object detection is a core task in the field of computer vision, which aims to accurately identify and locate objects from images or videos. In object detection, image denoising technology is crucial to improving detection performance and accuracy. In object detection, single-stage detection, two-stage detection, and detection methods for small targets are important research directions for technicians.

[0003] Single-stage detectors are a type of detection model that directly extracts the location and category of a target from an image. They usually have a simple network structure and fast inference speed, and are widely used in real-time applications and resource-constrained environments. Representative algorithms of single-stage detectors include the YOLO (You Only Look Once) series and SSD (Single Shot MultiBox Detector). These methods achieve target detection by sampling dense candidate boxes on an image and directly classifying and regressing their locations. However, single-stage detectors are relatively weak in handling small and dense targets, and their performance is limited in practical applications.

[0004] In contrast to the single-stage detector, the two-stage detector usually uses a two-stage process to complete the target detection task. First, the candidate boxes are generated using the Region Proposal Network (RPN) or other methods; then, these candidate boxes are further processed by the classification and position regression network. Representative two-stage detectors include Faster R-CNN, Mask R-CNN, etc. This type of method usually has better detection performance, especially for small and dense targets.

[0005] Detection of small targets is an important challenge in the field of target detection. Because small targets are often small in size and have less information in the image, they are easily affected by noise and background interference. In order to solve this problem, researchers have proposed many detection methods for small targets. These methods usually include strategies such as enhancing the receptive field of the model, introducing multi-scale feature fusion mechanisms, and improving loss functions. In addition, there are some methods based on feature enhancement and transfer learning, which learn the relationship between small target and large target features, fit small target features into the large feature space, or use the knowledge of pre-trained large target detection models to improve the performance of small target detection. However, most of these methods focus on the association between small target and large target features, and introduce complex prior knowledge to enhance the representation of small targets, which often ignores the unique features of small targets themselves and the potential association between small target features. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings and defects of the prior art and creatively propose a small target image detection method based on contextual semantic information aggregation, which can quickly learn the intrinsic associations between similar small target features in complex natural image features, and aggregate the learned intrinsic associations and contextual information between similar small target features with the small target feature information extracted by the network during the reasoning process of target detection, so as to effectively enhance the feature representation of small targets.

[0007] The innovative features of the present invention include:

[0008] First, a simple yet effective feature memory module is introduced, which aims to mine the potential association information between small objects of the same category and save it as common small object feature information.

[0009] Second, a context-based feature aggregation mechanism is proposed, which uses the category information learned from small objects to enhance the feature representation of small objects, further enhancing the feature representation capability.

[0010] Third, the results are gradually improved through multiple stages of regression heads, achieving more accurate predictions. This approach of combining contextual information and feature memory makes small object detection more robust in complex scenarios.

[0011] The present invention is implemented by adopting the following technical solutions.

[0012] A small target image detection method based on contextual semantic information aggregation adopts a two-stage detection model and introduces a feature enhancement module based on contextual information aggregation and a multi-stage detection head auxiliary model to achieve accurate detection of small targets.

[0013] Specifically, the steps include:

[0014] First, a small object feature extraction module based on the ResNet backbone network is constructed. The present invention uses the ResNet50 model pre-trained on ImageNet to initialize the weights of the backbone network, and inputs natural images into the backbone network for image feature extraction.

[0015] After that, a feature pyramid is designed, and a feature pyramid with the same number of channels is obtained by convolution processing on the image features extracted by the backbone network. As a result, each layer of features contains feature information of different depths, which helps to characterize the image more comprehensively.

[0016] Then, the first stage of regression and classification is performed using the Region Proposal Network to obtain the regions of interest of the detection model. Through the above processing, the present invention can effectively extract potential small target regions from the original image.

[0017] Finally, the second stage regression and classification networks are constructed, including multi-level refined detection heads, which can gradually refine the regions of interest obtained in the first stage and use different thresholds at different stages to obtain more accurate detection results.

[0018] Beneficial Effects

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] 1. The present invention not only does not require complex model structures and prior knowledge, but also learns and constructs a feature memory module suitable for small target features by exploring the potential correlation information between and within small target features. This feature memory module can save and utilize feature information related to small targets, enhance the representation of small target features during the detection process, and thus achieve more accurate small target detection results.

[0021] 2. The present invention is an accurate general small target detection method that does not rely on a specific data set. The general small target features learned by the feature memory module can effectively deal with the problem of small target detection in images with different environmental complexity, and has good robustness and generalization. It can be applied to practical application fields including autonomous driving and drone photography, and can achieve good detection and recognition effects without relying on specific data set training, which is of great significance to the development of scientific research and industrial detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is the overall flow chart of the method of the present invention;

[0023] Figure 2 It is a characteristic memory module structure diagram of the method of the present invention;

[0024] Figure 3 It is the overall flow chart of small target detection based on contextual semantic information aggregation of the method of the present invention; DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0026] Example

[0027] like Figure 1 As shown, a small target image detection method based on contextual semantic information aggregation includes the following steps:

[0028] Step 1: In order to address the challenges of scarcity of small object samples and difficulty in fully learning small object features due to their insignificance, the present invention proposes a context-aware feature enhancement mechanism.

[0029] First, given an input image Represents a matrix, W represents the width of the input image, and H represents the height of the input image. Through the backbone network B and the region proposal network RPN, the feature R of the region of interest is extracted:

[0030] R= RPN(B(I)) (1) The size of the matrix R is preferably N×256×7×7, which represents the features of the region of interest extracted by the backbone network in the image I, where there are N regions.

[0031] Step 2: If Figure 3 As shown, after two stages, feature map context semantic information aggregation is performed before prediction in the third stage.

[0032] In order to make the semantic information learned by the model more consistent with the target detection task model, the region of interest features (such as 256×7×7) extracted by the backbone network in image I are first expanded into a 1-dimensional vector, and a linear layer is passed to obtain features with a channel number of C, maintaining the feature integrity of each region of interest to explore the contextual information F of the input feature. intra :

[0033] F intra =A intra (Linear(R)) (2)

[0034] Among them, A intra Represents the context module inside the feature, Linear represents the linear layer used to aggregate feature information, and the matrix F intra The dimension is N×(N C +1)×C, represents the classification representation of each pixel in the proposed feature, N CIndicates the number of categories that the model can detect.

[0035] Step 3: Use the contextual semantic information learned between small objects to enhance the semantic representation of small object features:

[0036]

[0037] in, A represents the representation of the general category feature information of the small object category learned from the model stored in the feature memory module, inter is the context aggregation scheme between features, F inter It is a matrix of size B×N×C that incorporates the semantic context information between features.

[0038] Step 4: Combine the contextual semantic information between the proposed features and the classification representation of each pixel within the proposed feature to enhance the original proposed feature representation R Enh :

[0039] R Enh =A Enh (R,F inter ) (4)

[0040] Among them, A Enh It is a fusion module that aims to aggregate the original features R and the contextual information F between features. inter .

[0041] Step 5: Represent the enhanced features as R Enh Sent to the regression head of the detector for object classification and regression.

[0042] Furthermore, in order to help the model better store the general small target feature information, the present invention uses a simple feature memory module to store the general small target features. Figure 2 As shown, the present invention introduces a feature storage module M, whose size is (N C +1)×C′, which is used to store the general distribution information of small object features of each category, where C′ represents the number of hyperparameters used to represent the feature category distribution, which is the same as the value of the feature channel number C mentioned above.

[0043] During training, the same random seed is used to initialize each category in the feature storage module, representing it as a normal distribution. Then, after each training iteration, the information in M ​​is updated using a moving average:

[0044]

[0045] Among them, i represents the current iteration number, m is the momentum, It represents the weighted average of the true labeled features in each iteration, and the weight of each true labeled feature is its cosine similarity with the feature memory module.

[0046] Since the area predicted by the detector is not completely accurate, there are many wrong predictions, which may interfere with the feature storage module learning the correct feature category representation. In order to help the model learn the correct feature representation, the present invention introduces an additional background class in the feature memory module, and each time selects the prediction with the smallest coincidence with the real mark in the image to update the background class feature, which helps the model better distinguish the foreground and background.

[0047] Then, for each category c included in the set of small object features j , extract the corresponding pixel-level feature representation

[0048]

[0049] in, represents the true annotation of the kth small target feature, c j represents the jth category; Indicates that it belongs to category c j The feature corresponding to the kth true annotation; Represents the number of detectable target categories; then, the weighted mean of the cosine similarity of all features representing the jth category is calculated.

[0050] Then, the feature representation of the jth class extracted by Formula 5 is used to update the feature representation in the feature memory module.

[0051] In order to obtain higher quality small object features, the present invention designs a feature enhancement mechanism based on contextual semantic information aggregation and a simple feature memory module. This allows small object features that may be masked or difficult to identify by the information of larger objects to be enhanced through the contextual information between small target features learned by the feature memory module.

[0052] In order to constrain the feature storage module to learn the common features of small targets, the present invention introduces a simple storage loss function to enhance the learning ability of the feature storage module, symbolized by p i Represents the prediction result corresponding to category i in the feature storage module:

[0053]

[0054] in, represents feature memory loss, N c Indicates the number of categories that the model can detect.

[0055] Overall loss function The expression is as follows:

[0056]

[0057] in, and They represent the original loss for detecting the head, namely the cross entropy loss and the smooth L1 loss respectively.

[0058] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A small object image detection method based on contextual semantic information aggregation, characterized in that: The following steps are involved: Step 1: Establish a context-aware feature enhancement mechanism; First, given an input image Represents a matrix, W represents the width of the input image, and H represents the height of the input image; Through the backbone network B and the region proposal network RPN, the feature R of the region of interest is extracted, and R has N regions; Step 2: After two stages, feature map context semantic information aggregation is performed before the third stage prediction; First, the features of the region of interest extracted by the backbone network in image I are expanded into a 1-dimensional vector, and a linear layer is passed to obtain features with a channel number of C, maintaining the feature integrity of each region of interest to explore the contextual information F of the input features. intra ; Step 3: Use the contextual semantic information learned between small objects to enhance the semantic representation of small object features F inter ; Step 4: Combine the contextual semantic information between the proposed features and the classification representation of each pixel within the proposed feature to enhance the original proposed feature representation; Step 5: The enhanced feature representation is sent to the regression head of the detector for object classification and regression.

2. A small object image detection method based on contextual semantic information aggregation as claimed in claim 1, characterized in that: In step 1, R = RPN (B (I)), where R represents the region of interest features extracted by the backbone network in image I; In step 2, F intra =A intra (Linear(R)), where A intra Represents the context module inside the feature, Linear represents the linear layer used to aggregate feature information, and the matrix F intra The dimension is N×(N C +1)×C, represents the classification representation of each pixel in the proposed feature, N C Indicates the number of categories that the model can detect.

3. The small object image detection method based on contextual semantic information aggregation as claimed in claim 2, characterized in that: The size of R is N×256×7×7.

4. The small object image detection method based on contextual semantic information aggregation as claimed in claim 1, characterized in that: In step 3, in, A represents the representation of the general category feature information of the small object category learned from the model stored in the feature memory module, inter is the context aggregation scheme between features, F inter It is a matrix of size B×N×C that incorporates the semantic context information between features.

5. The small object image detection method based on contextual semantic information aggregation as claimed in claim 1, characterized in that: In step 4, R Enh =A Enh (R,F inter ), where A Enh It is a fusion module that aims to aggregate the original features R and the contextual information F between features. inter .

6. The small object image detection method based on contextual semantic information aggregation as claimed in claim 1, characterized in that: A feature memory module is used to store the general small target features. The feature storage module M is introduced, and its size is (N C +1)×C′, used to store the general distribution information of small object features of each category, where C′ represents the number of hyperparameters used to represent the feature category distribution; During training, the same random seed is used to initialize each category in the feature storage module, representing it as a normal distribution, and then, after each training iteration, the information in M ​​is updated using a moving average: Among them, i represents the current iteration number, m is the momentum, represents the weighted average of the true labeled features in each iteration, and the weight of each true labeled feature is its cosine similarity with the feature memory module; Introduce an additional background class in the feature memory module, and select the prediction with the smallest coincidence with the true label in the image to update the background class feature each time; Then, for each category c included in the set of small object features j , extract the corresponding pixel-level feature representation in, represents the true annotation of the kth small target feature, c j represents the jth category; Indicates that it belongs to category c j The feature corresponding to the k-th true annotation; Represents the number of detectable target categories; then, the weighted mean of the cosine similarity of all features representing the jth category is calculated; Then, the feature representation of the jth class is used to update the feature representation in the feature memory module.

7. The small object image detection method based on contextual semantic information aggregation as claimed in claim 6, characterized in that: Using the storage loss function, we can enhance the learning ability of the feature storage module, symbolized by p i Represents the prediction result corresponding to category i in the feature storage module: in, Indicates feature memory loss; N c Indicates the number of categories that the model can detect; Overall loss function The expression is as follows: in, and They represent the original loss of the detection head respectively.

Images