A small target detection method with fuzzy residual image

Abstract

The application discloses a small target detection method with fuzzy residual image, comprising the following steps: constructing a generative adversarial network, inputting a small target image with fuzzy residual image into the generative adversarial network, removing the fuzzy residual image, and generating a deblurring image; constructing a Yolov7 feature extraction network, inputting the deblurring image into the Yolov7 feature extraction network to generate a feature pyramid, introducing a cascade query structure into the feature pyramid, and performing small target feature detection. The application effectively solves the problem of poor detection effect of small targets with fuzzy residual image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a small target detection method with fuzzy residual image. BACKGROUND

[0002] Small target detection technology is different from the task of detecting targets of regular size. Small targets cannot provide much color, shape, texture or any other type of visually recognizable information due to their small size in the image, making the accuracy of small target detection not high.

[0003] The definition of small targets in the professional field mainly involves relative scale and absolute scale. Among them, the relative scale small target definition method takes the relative proportion of the target and the image as the judgment basis: 1) the median of the boundary box area and the image area ratio is 0.08%-0.58%; 2) the width-height ratio of the boundary box and the image is less than 0.1, and the area ratio is less than 0.03. The definition based on absolute scale generally uses the judgment basis of MS COCO dataset, that is, the target with a pixel range less than 32x32 is defined as a small target. In this application, the absolute definition is used to define the small target in the application, that is, the target with a pixel range less than 32x32 is defined as a small target.

[0004] Most current target detection technologies, especially small target detection technologies, have certain challenges in processing targets with fuzzy residual images. Because they are usually trained based on clear images in the training set. For targets with fuzzy residual images, due to the lack of clear contours and feature information, the model may not be able to accurately identify and locate the target.

[0005] Therefore, how to provide a method for detecting small targets with fuzzy residual images is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0006] The present application provides a small target detection method with fuzzy residual image, which improves the detection accuracy of small targets by introducing a deblurring network based on the improved yolov7.

[0007] The small target detection method with fuzzy residual image provided by the present application has the characteristics that it comprises the following steps:

[0008] S1: Construct a generative adversarial network, input the small target image with fuzzy residual image into the generative adversarial network, remove the fuzzy residual image, and generate a deblurring image;

[0009] S2: Constructing a Yolov7 feature extraction network, inputting the deblurred image into the Yolov7 feature extraction network to generate a feature pyramid, introducing a cascade query structure into the feature pyramid for small target feature detection.

[0010] Preferably, the S1 includes a GAN generator network construction step:

[0011] The GAN generator network includes: two convolution blocks with a stride of 1 / 2 arranged in sequence according to the image processing order, nine residual modules and two transpose convolution blocks, a dropout layer is added after the first convolution layer of each residual block, and the input small target image feature with blur residual and the output image feature are fused through a global residual connection.

[0012] Preferably, the small target image with blur residual is expressed as:

[0013] I B =k(M)*I S +N

[0014] Where I S is the original clear image, which becomes a small target image I B with blur residual after adding relevant noise, k(M) is a blur kernel determined by a motion field, N is additive noise, and * represents convolution.

[0015] The content loss function of the generator is:

[0016]

[0017] Where I S represents the original clear image, I B represents a blurred image synthesized by adding noise to the original clear image, represents the clear image generated by the generator, the original clear image I S and the clear image generated by the generator are input into a pre-trained VGG19 network for feature extraction, and the extracted features are input into a discriminator network for comparison. Among them, is the feature map obtained by the jth convolution before the ith maxpooling layer in the VGG19 network; W i,j and H i,j are the dimensions of the feature map width and height.

[0018] Preferably, the S1 includes a GAN discriminator network construction step:

[0019] The GAN discriminator network is used for refining the texture features of the generated picture, and includes six convolutional layer groups, and Instancenorm and Leakyrelu layers are added behind the first five convolutional layers.

[0020] Preferably, the adversarial loss function of the GAN discriminator network is:

[0021]

[0022] wherein I B is a small target image with blurred residual images, is a generator, is a discriminator.

[0023] Preferably, the input of the deblurred image to the Yolov7 feature extraction network in S2 to generate a feature pyramid includes:

[0024] The Yolov7 feature extraction network includes a feature extraction network and a feature fusion network.

[0025] The deblurred image generates feature maps of different scales after passing through the feature extraction network and the feature fusion network; and the feature maps are grouped into a feature pyramid according to the scale.

[0026] Preferably, the cascaded query structure in S2 includes:

[0027] S21: Starting from the smallest layer of the feature pyramid, the minimum distance between each feature point and each target center point on the feature map is calculated;

[0028] S22: A threshold is set according to the minimum anchor frame on the feature map, and a feature point value is obtained according to the relationship between the minimum distance and the threshold, thereby generating an attention map of the corresponding feature pyramid layer;

[0029] S23: The attention map of the current feature pyramid layer is mapped to the feature map of the next layer of the feature pyramid layer, so as to guide the feature map of the next layer of the feature pyramid layer to perform target detection within the range guided by the attention map;

[0030] Meanwhile, the feature map of the next layer of the feature pyramid layer generates an attention map of the current layer and executes S23 again, until all the feature pyramid layer levels are traversed.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The application effectively solves the problem of poor detection effect of small targets with fuzzy residual images. The application combines an image deblurring network and a target detection network to realize the detection of small targets with fuzzy residual images. By introducing a generative adversarial network, the generator network is used for the reconstruction of the fuzzy image content, and the discriminator network is used for the refinement of the target boundary texture. The target detection network receives the generated clear image, and realizes the detection of small targets by introducing a cascade query structure in the feature pyramid. The practicability of the network structure can be applied in the fields of deep learning, machine vision, image deblurring, target detection, etc. Compared with the prior art, the detection ability of small targets with fuzzy residual images is stronger. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0034] Figure 1 is a flowchart of the small target detection method with fuzzy residual images provided by the embodiment of the present application;

[0035] Figure 2 is a generator network structure diagram of the generative adversarial network provided by the embodiment of the present application;

[0036] Figure 3 is a discriminator network structure diagram of the generative adversarial network provided by the embodiment of the present application;

[0037] Figure 4 is an image feature pyramid structure diagram provided by the embodiment of the present application;

[0038] Figure 5 is a cascade query mechanism principle diagram provided by the embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] The present application provides a small target detection method with fuzzy residual images, which is a brand new network designed based on yolov7.

[0041] Yolov7 is a network with very superior detection speed and detection accuracy at present, which proposes a high aggregation network structure E-ELAN to enhance the feature learning ability of the network. Secondly, the structure reparameterization is introduced, and the multi-branch network structure is used during training to make the model obtain better feature expression. During test reasoning, the branches are fused in series to reduce the calculation parameter amount and improve the detection speed.

[0042] The application introduces a deblurring network on the basis of yolov7. The network uses a generative adversarial technology to process small targets with blur residual, generates clear target features for the subsequent detection process. In order to improve the detection accuracy of small targets, a cascade query mechanism is introduced in the detection head part of yolov7 in the detection process, so as to improve the detection accuracy of small targets by yolov7.

[0043] As shown in Figure 1 The method comprises the following steps:

[0044] S1: constructing a generative adversarial network, inputting small target images with blur residual into the generative adversarial network, removing the blur residual, and generating deblurring images;

[0045] S2: constructing a yolov7 feature extraction network, inputting the deblurring images into the yolov7 feature extraction network to generate a feature pyramid, introducing a cascade query structure in the feature pyramid, and detecting small target features.

[0046] The object of the embodiment of the application for detecting small targets with blur residual includes small targets with a pixel range less than 32*32.

[0047] In one embodiment, the generative adversarial network comprises a GAN generator network, and the generator network is used to remove the blur residual and generate the content of the residual part.

[0048] The GAN generator network comprises two convolution blocks with a stride of 1 / 2, nine residual modules and two transpose convolution blocks arranged in sequence according to the image processing order, a dropout layer is added after the first convolution layer of each residual block, and the input image features and the output image features are fused through a global residual connection.

[0049] As shown in Figure 2As shown, the generator network model based on the generative adversarial network of the application is shown, the blurred image first passes through a network module composed of a 7x7 convolution layer, an Instancenorm normalization layer and a Relu activation layer. Then pass through two network modules composed of 3x3 convolution, Instancenorm normalization layer and Relu, and then pass through nine residual modules, the composition of the residual module is 3x3 convolution, Instancenorm normalization, Relu activation layer, 3x3 convolution, Instancenorm. Then pass through two modules composed of ConvTranspose transposed convolution, Instancenorm, Relu, and a 7x7 convolution layer and a tanh activation layer. In addition, a global residual connection is added to fuse the input image features and the output image features. The blurred image is removed by the generator network to generate an image with clear features.

[0050] In one embodiment, the small target image expression with blur residual is:

[0051] I B = k (M) * I S + N (1)

[0052] Where, I S is the original clear image, which becomes a small target image I B with blur residual after adding relevant noise, k (M) is the blur kernel determined by the motion field, N is the additive noise, and * represents convolution; I B The image is input to the generator network of the generative adversarial network, which generates an image with clear target features according to the image with blur residual.

[0053] The content loss function of the generator is:

[0054]

[0055] Where, I S represents the original clear image, I B represents the blurred image synthesized by adding noise to the original clear image, and the clear image generated by inputting the blurred image into the generator network, represents the clear image generated by the generator. The original clear image I S and the clear image generated by the generator are sent to the pre-trained VGG19 network for feature extraction. The extracted features are sent to the discriminator network for comparison. is the feature map obtained by the jth convolution before the ith maxpooling layer in the VGG19 network; W i,j and H i,jis the dimension of the width and height of the feature map, which is defined based on the difference between the feature map generated by the generator and the target image CNN feature map.

[0056] In one embodiment, the generative adversarial network comprises a GAN discriminator network, and the discriminator network is used to refine the generated picture and depict the texture details to obtain an image with clear target features.

[0057] As shown in Figure 3 , the discriminator network model built based on the generative adversarial network of the application, the GAN discriminator network is used to refine the texture features of the generated picture, which comprises six convolutional layer groups, and Instancenorm and Leakyrelu layers are added behind the first five convolutional layers. It adopts a local-to-local discrimination method, and the input image is composed of two parts of the original clear image and the clear image generated by the generator after feature extraction by VGG19. The adversarial network divides the input image feature map into 64x64 small blocks, and sequentially operates on each part. After the last convolutional layer, the sigmod function is used to map the pixel value to 0-1, and a sequence is output, each pixel point representing the probability that the corresponding local area in the image is a real sample.

[0058] In one embodiment, the adversarial loss function of the GAN discriminator network is:

[0059]

[0060] Where, I B is a small target image with blurred residual shadow, is a generator, is a discriminator. The Wasserstein distance is used as the adversarial loss function, and a gradient penalty term is introduced to enhance the training stability and avoid the problem of gradient disappearance.

[0061] The total loss function of the GAN network is:

[0062] L=L GAN +λL X (4)

[0063] Where L is the total loss of the GAN network, L GAN is the adversarial loss, L X is the content loss, and λ is a balance factor used to adjust the proportion of adversarial loss and content loss in the total loss. The value can be 100 in experiments.

[0064] In one embodiment, inputting the deblurring image to the Yolov7 feature extraction network to generate a feature pyramid in S2 comprises:

[0065] The Yolov7 feature extraction network includes a feature extraction network and a feature fusion network.

[0066] The deblurred image is input into the feature extraction network and the feature fusion network to generate feature maps with different scales. The feature maps are grouped into a feature pyramid according to the scale.

[0067] In one embodiment, as shown in FIG. 2, the S2 cascade query structure includes: Figure 4

[0068] S21: Starting from the smallest layer of the feature pyramid, the minimum distance between each feature point and each target center point on the feature map is calculated.

[0069] S22: A threshold is set according to the minimum anchor box on the feature map, and a feature point value is obtained according to the relationship between the minimum distance and the threshold, thereby generating an attention map for the corresponding feature pyramid layer.

[0070] S23: The attention map of the current feature pyramid layer is mapped to the feature map of the next feature pyramid layer, guiding the feature map of the next feature pyramid layer to perform target detection within the range guided by the attention map.

[0071] Meanwhile, the feature map of the next feature pyramid layer generates an attention map for the current layer and executes S23 again until all feature pyramid layers are traversed.

[0072] The specific execution steps are as follows:

[0073] S21: Starting from the layer with the smallest number of pixels in the feature pyramid, the minimum distance between each feature position on the layer and each target center is calculated.

[0074]

[0075] X, Y are feature points on the feature map, is the center point position of the target on the feature map.

[0076] S22: Starting from the layer with the smallest number of pixels in the feature pyramid, CSQ is applied to extract query features. First, sparse convolution is added to generate an attention map for the query. According to the minimum anchor box on the feature map, a threshold s l is set. l When the distance between D l [x][y] and s is less than s

[0077] , the value of this feature point is 1, otherwise it is 0, thereby generating an attention map for the layer. The generation method is as follows:

[0078] where V l *[x][y] is the generated attention map representation, D l [x][y] is the minimum distance from each feature point of the feature map to each target center, s l is a threshold value, which is the minimum distance of the anchor box corresponding to the feature point on the feature map.

[0079] S23: As shown in Figure 5 , the generated attention map is mapped to the next layer, and the feature points contained in the attention map are mapped to the next layer feature Figure Four The nearest key position. The specific mapping method is:

[0080]

[0081] Wherein is the key position of the new attention map generated by mapping, i.e. the key feature point, is the key position of the original attention map.

[0082] The new feature layer performs target detection on the position corresponding to the feature map of the layer according to the key position of the attention map generated by mapping.

[0083] At the same time, the new feature layer generates the attention map of the layer according to the method of S23, and maps the attention map to the next layer. The above steps are iterated for all layers.

[0084] The greatest advantage of the present application is that it not only improves the precision of small target detection, but also extends the small target detection technology of clear features to small target detection of blurred features. At the same time, the present application improves the precision of small target detection without affecting the detection effect of medium and large targets.

[0085] The above describes in detail the small target detection method with blurred residual image provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

[0086] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional elements of the same name in the process, method, article, or apparatus.

Claims

1. A small target detection method with fuzzy residual image, characterized in that: The method comprises the following steps: S1: constructing a generative adversarial network, inputting a small target image with blur residual into the generative adversarial network, removing the blur residual, and generating a deblurring image; the construction of the generative adversarial network comprises the following steps of constructing a GAN generator network: The GAN generator network comprises two convolution blocks with a stride of 1 / 2, nine residual modules and two transpose convolution blocks arranged in sequence according to an image processing sequence, a dropout layer is added after a first convolution layer of each residual block, and the features of the input small target image with blur residual are fused with the output image features through a global residual connection; The expression of the small target image with blur residual is: I B = k(M) * I S + N where I S is the original sharp image, plus correlated noise, resulting in a small target image I B with blur and residual artifacts, k(M) is the blur kernel determined from the motion field, N is additive noise, and * denotes convolution. The content loss function of the generator is: wherein, I S represents the original clear image, I B represents a blurred image synthesized by adding noise to the original clear image, represents a clear image generated by the generator, and the original clear image and the clear image generated by the generator are input into a pre-trained VGG19 network for feature extraction, is a feature map obtained by the jth convolution before the ith maxpooling layer in the VGG19 network; W i,j and H i,j are the dimensions of the width and height of the feature map; The construction of the generative adversarial network comprises the following steps of constructing a GAN discriminator network: The GAN discriminator network is used for refining the texture features of the generated image, comprises six convolution layer groups, and Instancenorm and Leakyrelu layers are added after the first five convolution layers; the adversarial loss function of the GAN discriminator network is: wherein I B is a small target image with fuzzy residual, is a generator, is a discriminator; S2: constructing a Yolov7 feature extraction network, inputting the deblurring image into the Yolov7 feature extraction network to generate a feature pyramid, introducing a cascade query structure into the feature pyramid, and performing small target feature detection; the cascade query structure comprises: S21: starting from the smallest layer of the feature pyramid, calculating the minimum distance between each feature point and each target center point on the feature map; S22: setting a threshold according to the minimum anchor frame on the feature map, obtaining a feature point value according to the relationship between the minimum distance and the threshold, and generating an attention map of the corresponding feature pyramid layer according to the feature point value; S23: mapping the attention map of the current feature pyramid layer to the feature map of the next layer of the feature pyramid layer, guiding the feature map of the next layer of the feature pyramid layer to perform target detection within the range guided by the attention map; Meanwhile, the feature map of the next layer of the feature pyramid layer generates an attention map of the current layer and executes S23 again until all the feature pyramid layer levels are traversed.

2. The small object detection method with fuzzy ghosting according to claim 1, wherein, The inputting of the deblurring image into the Yolov7 feature extraction network to generate the feature pyramid in S2 comprises: The Yolov7 feature extraction network comprises a feature extraction network and a feature fusion network; The deblurring image generates feature maps with different scales after passing through the feature extraction network and the feature fusion network; and the feature maps are grouped into a feature pyramid according to the scale.

Images