Small target detection method in spatial domain based on deep learning semantic feature separation

Abstract

The present invention discloses a method for detecting small targets in a spatial domain based on deep learning semantic feature separation. The method comprises: preprocessing the input image to eliminate noise interference and improve the learning ability of small targets; using a neural network to process the target image and extracting the semantic information and texture information of the neural network through deconvolution operation; using a detector to generate a probability map of the target area, using probability map filtering to process the target semantic and texture features, eliminating background feature interference, retaining target foreground features, and sparsely representing the target feature representation space; using a Transformer structure network to query and process target semantic information and texture feature information, extracting the correlation between target features, and generating multiple target prediction box sets; using the Hungarian algorithm to dynamically match the predicted box information with the real box information, and realizing the optimal dynamic matching mechanism between the predicted box and the real box by minimizing the loss function, thereby realizing target prediction without relying on prior information.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for detecting small targets in spatial domain based on deep learning semantic feature separation. Background Art

[0002] In recent years, with the continuous development of drone technology, its application has been increasingly widespread in various sectors, including transportation, aerial photography, electricity, and security, promoting the prosperity of related industries and businesses. At the same time, while drone technology has brought convenience, its illegal use has also increased, posing a significant threat to public safety and personal privacy. Compared to traditional aircraft, drones are smaller, more maneuverable, more concealed, and more readily available. Therefore, the "illegal" and "indiscriminate" use of drones has become a serious security issue that threatens personal and public safety. How to effectively detect, track, and counter illegal drones has become an increasingly important area of ​​technical security research. Effective drone detection is a prerequisite for tracking and countering drones and is therefore particularly important. Due to their small size, high altitude, and long range, drones often appear as small targets when imaged: small pixels and limited feature information, making accurate detection difficult. Current small target detection in airspace presents numerous challenges. Existing neural network target detection methods cannot directly and accurately detect targets. There is an urgent need to develop an accurate, reliable, and low-missing small target detection method in airspace.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] In view of this, the present invention provides a spatial small target detection method based on deep learning semantic feature separation. The input image is first preprocessed by Gaussian filtering, size change, and small target expansion. Then, a neural network is used to extract semantic and texture features in the image, deconvolution is performed for upsampling, and channel point multiplication is performed for background filtering. Finally, on the basis of suppressing background noise and extracting suspected target features, global features are extracted through Transformer to improve the perception ability of the neural network detector for small targets, so as to achieve high-accuracy and high-recall spatial small target detection.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0006] According to the first aspect of the present disclosure, a method for detecting small targets in the spatial domain based on deep learning semantic feature separation is provided, which includes five steps: image preprocessing, semantic and texture feature extraction, probabilistic filtering, global prediction box generation, and dynamic prediction box matching.

[0007] Step 1: Preprocess the input spatial image. First, use Gaussian filtering to eliminate noise interference and highlight the image edges. Then, use size transformation and small object expansion methods to improve the learning ability of small objects in the spatial domain.

[0008] Step 2: Use Resnet18 and Resnet34 to extract semantic information and texture information from the target image processed in step 1, and then upsample them through deconvolution operations to improve feature resolution and information richness;

[0009] Step 3: First, use the detector to generate a probability map representing the presence of the target in the space. Then, convert and splice the semantic features and texture features of the target. Use probability map filtering to process the spliced ​​features to eliminate background feature interference and retain the target foreground features. By reducing the digital feature values ​​in the background area, the target feature representation space is sparse.

[0010] Step 4: Using the Transformer structure network, the encoder and decoder are used to encode and decode the feature space respectively, extract the correlation between target features, suppress background noise through the correlation, and enhance the regional features related to the target. Finally, a linear network is used to generate multiple target prediction box sets.

[0011] Step 5: Using the Hungarian algorithm, the prediction problem is treated as a linear programming problem. Without relying on prior information, the optimal dynamic matching mechanism between the predicted box information and the true box information is achieved by minimizing the category loss, the predicted box regression loss, and the intersection-over-union loss. The network-related parameters are then adjusted.

[0012] Furthermore, in the image preprocessing, a convolution operation is performed on the image using a Gaussian kernel with a fixed value and odd length and width to eliminate noise interference and highlight the target features; then, image normalization is used to transform the pixel values ​​into a Gaussian distribution and convert them into values ​​between [0, 1], eliminating errors caused by numerical distribution and facilitating input into a deep neural network;

[0013] In the Gaussian filtering algorithm, the Gaussian kernel size is U×U. During the convolution operation, the Gaussian kernel is slid horizontally, and the value of the filtered image at the center of the Gaussian kernel is calculated. The calculation process is shown in formula (1):

[0014]

[0015] where Y ij is the value of the output image at ij, α is a hyperparameter, α hw is the value of the Gaussian kernel at hw, y (i-h)(j-w) is the value of the input image at (ih)(jw), and

[0016] During the image normalization process, the three RGB channels are normalized separately. First, each pixel is divided by 255 to limit the value to [0, 1]. Then, the value of each channel is Gaussian normalized. In the high-dimensional space, the mathematical distribution of all pixels is converted into a Gaussian distribution with a mean of 0 and a variance of 1, eliminating the error caused by uneven pixel distribution. The normalization formula is shown in formula (2):

[0017]

[0018] Where c represents the number of channels, c∈{R,G,B}, Y cij is the normalized output value at channel c ij, y cij is the input value before normalization at channel c ij, μ c is the mean of the c channel, σ c is the mean square error of the c channel.

[0019] Furthermore, in the image preprocessing, the small target expansion method randomly distributes the pixels of the small target area in the image and copies them in the image, thereby enhancing the deep neural network's attention and learning ability for small targets during the deep neural network training process; the small target expansion formula is shown in formula (3):

[0020]

[0021] in express The pixel value of the area, ch, cw is the center point of the small target area, a, b are the height and width of the small target area, ch′, cw′ is the center point of the expanded area generated by the random algorithm, and s is the size of the input image.

[0022] Furthermore, in the step 2, Resnet18 and Resnet34 are used to extract semantic information and texture information of the target image processed in the step 1 respectively;

[0023] The input of Resnet18 and Resnet34 networks are both images of size s×s and channels 3. First, the input image is converted into a basic layer BasicLayer consisting of a convolution layer, a batch normalization layer, and an activation layer. Feature map with 64 channels;

[0024] The Resnet18 and Resnet34 networks are mainly composed of the residual module BasicBlock, which consists of two parts. One part is the backbone network, which directly inputs the input image to the output layer without any processing. The other part extracts the target features of the input image through multiple layers of BasicLayer as a small amount, and finally adds the output values ​​of the two parts. The formula of BasicBlock is shown in formula (4):

[0025] y=x+BasicLayer(x)*N and BasicLayer(x)=Relu(BachNorm(Conv(x))) (4)

[0026] Where y is the output of BasicBlock, x is the input of BasicBlock, N represents the number of layers containing BasicLayer in BasicBlock, Conv is the convolution function, BachNorm is the batch normalization function, and Relu is the Relu activation function;

[0027] Formula (4) ensures that the output of the branch BasicLayer(x)*N is small compared to x, and ensures that the sizes of y and x remain relatively stable. During the network training process, the output y of BasicBlock is guaranteed to maintain a close data distribution with the input x, making the training of deep neural networks possible.

[0028] Furthermore, in the step 2, the features extracted by Resnet are deconvolved to expand the size of the convolution extracted features and reduce the number of channels for extracting the features; the deconvolution operation can be regarded as an upsampling operation on the features to expand the spatial size of the features.

[0029] Furthermore, in step three, a detector is used to generate a probability map representing the existence of the target in the space, and then the semantic features and texture features of the target are converted and spliced, including:

[0030] To better detect small objects, after Resnet18 feature extraction and upsampling, the network treats an 8×8 area of ​​the image as a detection unit. The detector uses the value of the corresponding point in the probability map to represent the probability of whether an object exists in the corresponding area.

[0031] Set the loss function L on the probability graph to represent the prediction effect k , as shown in formula (5):

[0032]

[0033] in is the probability prediction of the target center at xy in the probability graph, α, β are hyperparameter values, and Yxyc The true value of the preset probability map; the predicted value The closer to the preset value Y xyc , the smaller the loss of the corresponding position;

[0034] Default value Y xyc Indicates whether there is a target in the corresponding area; when there is no target in the area, the preset value Y xyc is 0; when there is a target center in the corresponding area, the preset value Y xyc =1; when there is no target center in the corresponding area but the area is within the target range, Gaussian distribution is used to represent the probability of the target existing in the area and the distance from the area to the target center; Y xyc As shown in formula (6):

[0035]

[0036] where p x , p y are the x and y coordinates of the target center point; σ is the variance value related to the target object size. Specifically, σ is related to the size and aspect ratio of the target detection box, as shown in formula (7):

[0037]

[0038] Where γ is a hyperparameter, h b , w b is the height and width of the target detection box;

[0039] After generating the probability map, the semantic information and texture information are channel-wise concatenated, as shown in formula (8):

[0040]

[0041] Among them, F1 and F2 are semantic features and texture features extracted by two channels, c1 and c2 are the number of channels of the two features, and Y c is the feature after splicing, and the number of channels after splicing is c=c1+c2.

[0042] Furthermore, in step 3, the spliced ​​features are processed using probability map filtering to eliminate background feature interference, retain target foreground features, and sparse the target feature representation space by reducing the digital feature value of the background area, including:

[0043] The scene semantic information and texture information extracted by the Resnet18 and Resnet34 networks contain background information and target information. Therefore, by suppressing the background, the deep neural network's attention to the target can be enhanced, eliminating the influence of the background. The probability map generated in step 3 correctly reflects the distribution of the target. The probability map value is larger in the area where the target exists, close to 1, while the probability map value in the background area is close to 0. After performing a dot product operation on the probability map and the spliced ​​features, the background area is closer to 0, the feature tensor is sparsed, the background information is filtered out, and the target area information is retained, achieving the purpose of suppressing the background and enhancing the foreground.

[0044] The probability map is regarded as a filter. The dot product operation can eliminate the influence of background information and highlight the target information area, as shown in formula (9):

[0045] Y cij =P ij ·X cij (9)

[0046] Among them, P ij is the value of the probability map generated in step 3 at ij, X cij is the vector of c channels of the feature after the semantic information and texture information are spliced ​​at ij, Y cij is the vector of c channels at position ij after the filtering operation.

[0047] Furthermore, the step 4 includes:

[0048] The multi-layer Encoder and Decoder are both composed of the Transformer structure, which mainly consists of a linear layer and a multi-head attention mechanism module MultiHead;

[0049] First, the concatenation of the sparse semantic information and texture information obtained in step 3 is downsampled using a convolutional layer to reduce the channel dimension of the feature and the input dimension of the subsequent Transformer structure.

[0050] For each input Transformer data, three data K (Key), Q (Query), and V (Value) are first generated through the linear layer of MultiHead. K, Q, and V are one-dimensional vector data with a dimension of d. MultiHead divides each K, Q, and V into multiple small-scale vectors, each with a dimension of d. h =d / n h , d h is the number of Heads contained in MultiHead; then the correlation matrix between each data is calculated by multiplying K and Q, and then the output is generated by multiplying the correlation matrix with V;

[0051] The output Attention (Q, K, V) of each Head in MultiHead is calculated as shown in formula (10):

[0052]

[0053] Then, the output results of all heads are connected. The specific process is shown in formula (11):

[0054]

[0055] in n h The output of the attention head, W O is the transformation matrix used to project the concatenated multi-head attention results into the feature space;

[0056] Since the position of the value in the feature space is related to the overall semantics, position coding is added to represent the position in the feature space; the position coding at (i, 2j), (i, 2j+1) in the feature space is shown in formula (12):

[0057]

[0058]

[0059] Through multiple encoders and decoders, the features obtained in step 4 are converted into feature vectors containing global information and correlation information, achieving the effect of suppressing background noise and strengthening regional features related to the target;

[0060] The obtained feature vector is detected by three detectors to generate NB prediction boxes; the three detectors respectively generate the target type, target center point, and prediction box size. By integrating the output information of the three detectors, NB prediction boxes can be obtained.

[0061] Furthermore, the step five includes:

[0062] During the training phase, the Hungarian algorithm is used for bipartite graph matching. The predicted box set is matched with the real box set. The predicted box with the highest matching degree is selected as the best predicted box to minimize the overall loss. The loss is then reduced through the gradient descent algorithm.

[0063] In the prediction stage, the prediction box whose prediction category confidence is higher than the threshold is directly selected as the final prediction box.

[0064] According to a second aspect of an embodiment of the present disclosure, a device for detecting small targets in a spatial domain based on deep learning semantic feature separation is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the method for detecting small targets in a spatial domain based on deep learning semantic feature separation as described in the first aspect.

[0065] The beneficial effects of the present invention are:

[0066] 1. To address the poor performance of neural network detectors in detecting small targets in the spatial domain, a network that separates semantic and texture feature extraction is designed to separate the semantic recognition and box coordinate regression tasks, avoiding structural coupling caused by the tasks.

[0067] 2. The detector processes the semantic information to obtain a probability map reflecting the existence of the target and background. The probability map is used to perform point multiplication filtering on the features of the semantic information and texture information to sparse the feature space and suppress background noise.

[0068] 3. Design a Transformer-based network to process features to obtain global correlation features. The detector then predicts a certain number of predicted boxes. During the training phase, the Hungarian algorithm dynamically matches the predicted boxes with the ground-truth boxes, and the network is trained using backpropagation. During the prediction phase, the predicted boxes whose predicted category confidence exceeds a threshold are directly selected as the final predicted boxes.

[0069] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0071] Figure 1 A method for detecting small targets in spatial domain based on deep learning semantic feature separation is presented;

[0072] Figure 2 The Resnet network structure for image feature extraction is shown;

[0073] Figure 3 Shows the Transformer-based Encoder and Decoder;

[0074] Figure 4 The results show the detection effect of the network on small targets in airspace;

[0075] Figure 5 The figure shows the detection performance of the network under different IOU standards. DETAILED DESCRIPTION

[0076] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0077] Figure 1 A flow chart of a method for detecting small targets in a spatial domain based on deep learning semantic feature separation is shown. According to one aspect of an embodiment of the present disclosure, a method for detecting small targets in a spatial domain based on deep learning semantic feature separation is provided, comprising the following five steps:

[0078] Step 1: Preprocess the input spatial image. First, use Gaussian filtering to eliminate noise interference and highlight the image edges. Then, use size transformation and small object expansion methods to improve the learning ability of small objects in the spatial domain.

[0079] Step 2: Use Resnet18 and Resnet34 to extract semantic information and texture information from the target image processed in step 1, and then upsample them through deconvolution operations to improve feature resolution and information richness;

[0080] Step 3: First, use the detector to generate a probability map representing the presence of the target in the space. Then, convert and splice the semantic features and texture features of the target. Use probability map filtering to process the spliced ​​features to eliminate background feature interference and retain the target foreground features. By reducing the digital feature values ​​in the background area, the target feature representation space is sparse.

[0081] Step 4: Using the Transformer structure network, the encoder and decoder are used to encode and decode the feature space respectively, extract the correlation between target features, suppress background noise through the correlation, and enhance the regional features related to the target. Finally, a linear network is used to generate multiple target prediction box sets.

[0082] Step 5: Using the Hungarian algorithm, the prediction problem is treated as a linear programming problem. Without relying on prior information, the optimal dynamic matching mechanism between the predicted box information and the true box information is achieved by minimizing the category loss, the predicted box regression loss, and the intersection-over-union loss. The network-related parameters are then adjusted.

[0083] For the input image in step 1, we first perform a convolution operation on the image using a Gaussian kernel with a fixed value and odd length and width to eliminate noise interference and highlight the target features; then we use image normalization to convert the pixel values ​​into a Gaussian distribution and convert them into values ​​between [0,1] to eliminate the error caused by the numerical distribution and facilitate input into the deep neural network; finally, we use the small target expansion method to expand the number of small targets in the image, thereby improving the neural network's attention to small targets and feature extraction capabilities.

[0084] In the Gaussian filtering algorithm, the Gaussian kernel size is U×U. During the convolution operation, the Gaussian kernel is slid horizontally, and the value of the filtered image at the center of the Gaussian kernel is calculated. The calculation process is shown in formula (1):

[0085]

[0086] where Y ij is the value of the output image at ij, α is a hyperparameter, α hw is the value of the Gaussian kernel at hw, y (i-h)(j-w) is the value of the input image at (ih)(jw), and

[0087] During the image normalization process, the three RGB channels are normalized separately. First, each pixel is divided by 255 to limit the value to [0, 1]. Then, the value of each channel is Gaussian normalized. In the high-dimensional space, the mathematical distribution of all pixels is converted into a Gaussian distribution with a mean of 0 and a variance of 1, eliminating the error caused by uneven pixel distribution. The normalization formula is shown in formula (2):

[0088]

[0089] Where c represents the number of channels, c∈{R,G,B}, Y cih is the normalized output value at channel c ih, y cij is the input value before normalization at channel c ij, μ c is the mean of the c channel, σ c is the mean square error of channel c;

[0090] The small target expansion method randomly distributes the pixels of the small target area in the image and copies them in the image. During the deep neural network training process, the deep neural network's attention to and learning ability of small targets are enhanced. The small target expansion formula is shown in formula (3):

[0091]

[0092] in express The pixel value of the area, ch, cw is the center point of the small target area, a, b are the height and width of the small target area, ch′, cw′ is the center point of the expanded area generated by the random algorithm, s is the size of the input image, and in this embodiment, s is 416.

[0093] Figure 2 The framework of the ResNet network used to extract semantic and texture features from an image is shown. In step 2, ResNet18 and ResNet34 are used to extract semantic and texture information from the target image processed in step 1, respectively. Deconvolution is then performed on the features extracted by the ResNet network, increasing the size of the convolutional features and reducing the number of channels used to extract them. The deconvolution operation can be considered an upsampling operation, increasing the spatial size of the features.

[0094] The input of Resnet18 and Resnet34 networks are both 416×416 images with 3 channels. First, the input image is converted into a feature map of 208×208 with 64 channels through the basic layer consisting of convolution layer, batch normalization layer, and activation layer.

[0095] The Resnet18 and Resnet34 networks are mainly composed of the residual module BasicBlock, which consists of two parts. One part is the backbone network, which directly inputs the input image to the output layer without any processing. The other part extracts the target features of the input image through multiple layers of BasicLayer as a small amount, and finally adds the output values ​​of the two parts. The formula of BasicBlock is shown in formula (4):

[0096] y=x+BasicLayer(x)*N and BasicLayer(x)=Relu(BachNorm(Conv(x))) (4)

[0097] Where y is the output of BasicBlock, x is the input of BasicBlock, N represents the number of layers containing BasicLayer in BasicBlock, Conv is the convolution function, BachNorm is the batch normalization function, and Relu is the Relu activation function;

[0098] Formula (4) ensures that the output of the branch BasicLayer(x)*N is small compared to x, and ensures that the sizes of y and x remain relatively stable. During the network training process, the output y of BasicBlock is guaranteed to maintain a close data distribution with the input x, making the training of deep neural networks possible.

[0099] The convolution function Conv is composed of the convolution kernel size, and a convolution kernel of size 1×1 or 3×3 is usually used. The formula for the convolution operation is shown in the following formula (5):

[0100]

[0101] Where y is the output, h and w are the height and width of the convolution kernel respectively, and α ij is the coefficient of the convolution kernel ij, x ij is the pixel value at the image ij. Formula (5) has a similar mathematical processing process to the convolution process of the signal in the information processing process. By adjusting α ij Effectively extract regional data features.

[0102] The batch normalization function BachNorm performs normalization on each channel, as shown in formulas (6), (7), (8), and (9):

[0103]

[0104]

[0105]

[0106]

[0107] The input data is β=x1,x2……,x m There are m vectors in total, and the output is y i =BN(x). The BatchNorm steps are as follows:

[0108] Find the input data x1, x2, ..., x m The mean μ β ; Find the variance of this batch Normalize this batch of data by x i Obtain normalized data Scaling and translation variables γ and β are introduced to calculate the normalized values. γ and β are called translation parameters and scaling parameters respectively. This ensures that the learned features are retained after each data normalization, while completing the normalization operation and accelerating training.

[0109] The formula of the Relu activation function is shown in formula (10):

[0110]

[0111] The Relu activation function outputs 0 when the input is less than or equal to 0, and directly outputs the input when the input is greater than 0. The Relu function is nonlinear, and its derivatives are different in the two regions of x≤0 and x>0, ensuring that the neural network has high-dimensional nonlinearity and avoiding overfitting of the neural network.

[0112] The deconvolution operation can be regarded as an upsampling operation on the features to expand the spatial size of the features.

[0113] After the convolution operation is stretched into a matrix form, it is as shown in formula (11):

[0114] y=CX (11) Where X is the column vector after the input data is stretched, X=[x1,x2,……,x N ], N = H × W, where H and W are the height and width of the input data, respectively. The size of the C matrix is ​​M × N, where M is the number of features output by the convolution, which is also the product of the height and width of the output feature. Each row of data is a sparse row vector padded with zeros after the convolution kernel is stretched. y is the column vector output after the convolution operation, with size M × 1, and each element represents the relative position of the output after the convolution operation.

[0115] The deconvolution operation upsamples the output y of the convolution operation, as shown in formula (12):

[0116] Y=C T y=C T Cx 12

[0117] Where Y is the output of the deconvolution operation, y is the output of the convolution operation in formula (11), and C T is the transpose of the convolution matrix C, with a size of N×M, through C T The convolution output y is restored to its original size N by matrix multiplication, and the output of the convolution operation is upsampled. It should be noted that the newly generated deconvolution output Y is not x, but only has the same size features as x. Therefore, the deconvolution operation is not the inverse function of the convolution operation, but only an upsampling operation on the features.

[0118] To better detect small objects, after ResNet18 feature extraction and upsampling, the network treats each 8×8 region in the image as a detection region. The detector uses the value of the corresponding point in the probability map to represent the probability of whether an object exists in the corresponding region. If the object exists in the corresponding region, the predicted value of the corresponding point in the probability map is as close to 1 as possible. If the object does not exist in the corresponding region, the predicted value of the corresponding point in the probability map is as close to 0 as possible.

[0119] Set the loss function L on the probability graph to represent the prediction effect k , as shown in formula (13):

[0120]

[0121] in is the probability prediction of the target center at xy in the probability graph, α, β are hyperparameter values, and Y xyc The true value of the preset probability map; the predicted value The closer to the preset value Y xyc , the smaller the loss of the corresponding position;

[0122] Default value Y xyc Indicates whether there is a target in the corresponding area; when there is no target in the area, the preset value Y xyc is 0; when there is a target center in the corresponding area, the preset value Y xyc When the target center does not exist in the corresponding area but the area is within the target range, the Gaussian distribution is used to represent the probability of the target existing in the area and the distance from the area to the target center. xyc As shown in formula (14):

[0123]

[0124] where p x , p y are the x and y coordinates of the target center point; σ is the variance value related to the target object size. Specifically, σ is related to the size and aspect ratio of the target detection box, as shown in formula (15):

[0125]

[0126] Where γ is a hyperparameter, h b , w b is the height and width of the target detection box;

[0127] After generating the probability map, the semantic information and texture information are concatenated channel by channel, as shown in formula (16):

[0128]

[0129] Among them, F1 and F2 are semantic features and texture features extracted by two channels, c1 and c2 are the number of channels of the two features, and Y c is the feature after splicing, and the number of channels after splicing is c=c1+c2.

[0130] The scene semantic information and texture information extracted by the Resnet18 and Resnet34 networks contain background information and target information. Therefore, by suppressing the background, the deep neural network's attention to the target can be enhanced, eliminating the influence of the background. The probability map generated in step three correctly reflects the distribution of the target. The probability map value is larger in the area where the target exists, close to 1, while the probability map value in the background area is close to 0. After performing a dot product operation on the probability map and the spliced ​​features, the background area is closer to 0, the feature tensor is sparsed, the background information is filtered out, and the target area information is retained, achieving the purpose of suppressing the background and enhancing the foreground.

[0131] The probability map is regarded as a filter. The dot product operation can eliminate the influence of background information and highlight the target information area, as shown in formula (17):

[0132] Y cij =P ij ·X cij (17)

[0133] Among them, P ij is the value of the probability map generated in step 3 at ij, X cij is the vector of c channels of the feature after the semantic information and texture information are spliced ​​at ij, Y cij is the vector of c channels at position ij after the filtering operation.

[0134] Figure 3 The figure shows the encoder and decoder built on the Transformer. The encoder and decoder respectively encode and decode the feature space, extracting the correlation between target features, suppressing background noise and enhancing target-related regional features through correlation, and finally generating multiple sets of target prediction boxes through a linear network.

[0135] The multi-layer Encoder and Decoder are both composed of the Transformer structure, which mainly consists of a linear layer and a multi-head attention mechanism module MultiHead;

[0136] First, the concatenation of the sparse semantic information and texture information obtained in step 3 is downsampled using a convolutional layer to reduce the channel dimension of the feature and the input dimension of the subsequent Transformer structure.

[0137] For each input Transformer data, three data K (Key), Q (Query), and V (Value) are first generated through the linear layer of MultiHead. K, Q, and V are one-dimensional vector data with a dimension of d. MultiHead divides each K, Q, and V into multiple small-scale vectors, each with a dimension of d. h =d / n h , n h is the number of Heads contained in MultiHead; then the correlation matrix between each data is calculated by multiplying K and Q, and the output is generated by multiplying the correlation matrix with V;

[0138] The output Attention (Q, K, V) of each Head in MultiHead is calculated as shown in formula (18):

[0139]

[0140] Then, the output results of all heads are connected. The specific process is shown in formula (19):

[0141]

[0142] in n h The output of the attention head, W O is the transformation matrix used to project the concatenated multi-head attention results into the feature space;

[0143] Since the position of the value in the feature space is related to the overall semantics, position coding is added to represent the position in the feature space; the position coding at (i, 2j), (i, 2j+1) in the feature space is shown in formula (20):

[0144]

[0145]

[0146] Through multiple encoders and decoders, the features obtained in step 4 are converted into feature vectors containing global information and correlation information, achieving the effect of suppressing background noise and strengthening regional features related to the target;

[0147] The obtained feature vector is detected by three detectors to generate NB prediction boxes; the three detectors respectively generate the target type, target center point, and prediction box size. By integrating the output information of the three detectors, NB prediction boxes can be obtained.

[0148] After obtaining NB prediction boxes, the Hungarian algorithm is used to treat the prediction problem as a linear programming problem. Without relying on prior information, the optimal dynamic matching mechanism between the predicted box information and the real box information is achieved by minimizing the category loss, the predicted box regression loss and the intersection-over-union loss, and adjusting the network related parameters.

[0149] During the training phase, the Hungarian algorithm is used for bipartite graph matching. The predicted box set is matched with the real box set. The predicted box with the highest matching degree is selected as the best predicted box to minimize the overall loss. The loss is then reduced through the gradient descent algorithm.

[0150] In the prediction stage, the prediction box whose prediction category confidence is higher than the threshold is directly selected as the final prediction box.

[0151] Figure 4 The left side shows the original image, and the right side shows the network detection result.

[0152] Figure 5 This is the curve of network detection accuracy change under different confidence scores.

[0153] According to the second aspect of an embodiment of the present disclosure, a device for detecting small targets in the spatial domain based on deep learning semantic feature separation is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the above-mentioned method for detecting small targets in the spatial domain based on deep learning semantic feature separation.

[0154] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0155] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for detecting small targets in spatial domain based on deep learning semantic feature separation, characterized in that: It includes five steps: image preprocessing, semantic texture feature extraction, probability filtering, global prediction box generation, and dynamic prediction box matching: Step 1: Preprocess the input spatial image. First, use Gaussian filtering to eliminate noise interference and highlight the image edges. Then, use size transformation and small object expansion methods to improve the learning ability of small objects in the spatial domain. Step 2: Use Resnet18 and Resnet34 to extract semantic information and texture information from the target image processed in step 1, and then upsample them through deconvolution operations to improve feature resolution and information richness; Step 3: First, use the detector to generate a probability map representing the presence of the target in the space. Then, convert and splice the semantic features and texture features of the target. Use probability map filtering to process the spliced ​​features to eliminate background feature interference and retain the target foreground features. By reducing the digital feature values ​​in the background area, the target feature representation space is sparse. Step 4: Using the Transformer structure network, the encoder and decoder are used to encode and decode the feature space respectively, extract the correlation between target features, suppress background noise through the correlation, and enhance the regional features related to the target. Finally, a linear network is used to generate multiple target prediction box sets. Step 5: Using the Hungarian algorithm, the prediction problem is treated as a linear programming problem. Without relying on prior information, the optimal dynamic matching mechanism between the predicted box information and the true box information is achieved by minimizing the category loss, the predicted box regression loss, and the intersection-over-union loss. The network-related parameters are then adjusted.

2. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In the image preprocessing, a convolution operation is performed on the image using a Gaussian kernel with a fixed value and odd length and width to eliminate noise interference and highlight the target features; Then, the pixel values ​​are normalized to transform into Gaussian distribution and converted into values ​​between [0, 1], eliminating the error caused by the numerical distribution and facilitating input into the deep neural network. In the Gaussian filtering algorithm, the Gaussian kernel size is U×U. During the convolution operation, the Gaussian kernel is slid horizontally, and the value of the filtered image at the center of the Gaussian kernel is calculated. The calculation process is shown in formula (1): where Y ij is the value of the output image at ij, α is a hyperparameter, α hw is the value of the Gaussian kernel at hw, y (i-h)(j-w) is the value of the input image at (ih)(jw), and During the image normalization process, the three RGB channels are normalized separately. First, each pixel is divided by 255 to limit the value to [0, 1]. Then, the value of each channel is Gaussian normalized. In the high-dimensional space, the mathematical distribution of all pixels is converted into a Gaussian distribution with a mean of 0 and a variance of 1, eliminating the error caused by uneven pixel distribution. The normalization formula is shown in formula (2): Where c represents the number of channels, c∈{R,G,B}, Y cij is the normalized output value at channel c ij, y cij is the input value before normalization at channel c ij, μ c is the mean of the c channel, σ c is the mean square error of the c channel.

3. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In the image preprocessing, the small target expansion method randomly distributes the pixels of the small target area in the image and copies them in the image, thereby enhancing the deep neural network's attention and learning ability for small targets during the deep neural network training process; the small target expansion formula is shown in formula (3): in express The pixel value of the area, ch, cw is the center point of the small target area, a, b are the height and width of the small target area, ch′, cw′ is the center point of the expanded area generated by the random algorithm, and s is the size of the input image.

4. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In the step 2, Resnet18 and Resnet34 are used to extract semantic information and texture information of the target image processed in the step 1 respectively; The input of Resnet18 and Resnet34 networks are both images of size s×s and channels 3. First, the input image is converted into a basic layer BasicLayer consisting of a convolution layer, a batch normalization layer, and an activation layer. Feature map with 64 channels; The Resnet18 and Resnet34 networks are mainly composed of the residual module BasicBlock, which consists of two parts. One part is the backbone network, which directly inputs the input image to the output layer without any processing. The other part extracts the target features of the input image through multiple layers of BasicLayer as a small amount, and finally adds the output values ​​of the two parts. The formula of BasicBlock is shown in formula (4): y = x + BasicLayer(x) * N and BasicLayer(x) = Relu(BachNorm(Conv(x))) (4) where y is the output of BasicBlock, x is the input of BasicBlock, N represents the number of layers containing BasicLayer in BasicBlock, Conv is the convolution function, BachNorm is the batch normalization function, and Relu is the Relu activation function; Formula (4) ensures that the output of the branch BasicLayer(x)*N is small compared to x, and ensures that the sizes of y and x remain relatively stable. During the network training process, the output y of BasicBlock is guaranteed to maintain a close data distribution with the input x, making the training of deep neural networks possible.

5. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In the second step, the features extracted by Resnet are deconvolved to expand the size of the convolution extracted features and reduce the number of channels for extracting features. The deconvolution operation can be regarded as an upsampling operation on the features. Expand the spatial size of the feature.

6. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In step 3, a detector is used to generate a probability map representing the presence of the target in space, and then the semantic features and texture features of the target are converted and spliced, including: After Resnet18 feature extraction and upsampling, the network regards an 8×8 area of ​​the image as a detection unit. The detector uses the value of the corresponding point in the probability map to represent the probability of whether the target exists in the corresponding area. Set the loss function L on the probability graph to represent the prediction effect k , as shown in formula (5): in is the probability prediction of the target center at xy in the probability graph, α, β are hyperparameter values, and Y xyc The true value of the preset probability map; the predicted value The closer to the preset value Y xyc , the smaller the loss of the corresponding position; Default value Y xyc Indicates whether there is a target in the corresponding area; when there is no target in the area, the preset value Y xyc is 0; when there is a target center in the corresponding area, the preset value Y xyc =1; when there is no target center in the corresponding area but the area is within the target range, Gaussian distribution is used to represent the probability of the target existing in the area and the distance from the area to the target center; Y xyc As shown in formula (6): where p x , p y are the x and y coordinates of the target center point; σ is the variance value related to the target object size. Specifically, σ is related to the size and aspect ratio of the target detection box, as shown in formula (7): Where γ is a hyperparameter, h b , w b is the height and width of the target detection box; After generating the probability map, the semantic information and texture information are channel-wise concatenated, as shown in formula (8): Among them, F1 and F2 are semantic features and texture features extracted by two channels, c1 and c2 are the number of channels of the two features, and Y c is the feature after splicing, and the number of channels after splicing is c=c1+c2.

7. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: In the step 3, the spliced ​​features are processed using probability map filtering to eliminate background feature interference, retain target foreground features, and sparse the target feature representation space by reducing the digital feature value of the background area, including: The probability map generated in step 3 correctly reflects the distribution of the target. The probability map value is larger in the area where the target exists, close to 1, while the probability map value in the background area is close to 0. After performing a dot product operation on the probability map and the spliced ​​features, the background area is closer to 0, the feature tensor is sparsed, the background information is filtered out, and the target area information is retained, achieving the purpose of suppressing the background and enhancing the foreground. The probability map is regarded as a filter. The dot product operation can eliminate the influence of background information and highlight the target information area, as shown in formula (9): Y cij =P ij ·X cij (9) Among them, P ij is the value of the probability map generated in step 3 at ij, X cij is the vector of c channels of the feature after the semantic information and texture information are spliced ​​at ij, Y cij is the vector of c channels at position ij after the filtering operation.

8. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: The fourth step includes: The multi-layer Encoder and Decoder are both composed of the Transformer structure, which mainly consists of a linear layer and a multi-head attention mechanism module MultiHead; First, the concatenation of the sparse semantic information and texture information obtained in step 3 is downsampled using a convolutional layer to reduce the channel dimension of the feature. For each input Transformer data, three data K (Key), Q (Query), and V (Value) are first generated through the linear layer of MultiHead. K, Q, and V are one-dimensional vector data with a dimension of d. MultiHead divides each K, Q, and V into multiple small-scale vectors, each with a dimension of d. h =d / n h , n h is the number of Heads contained in MultiHead; then the correlation matrix between each data is calculated by multiplying K and Q, and the output is generated by multiplying the correlation matrix with V; The output Attention (Q, K, V) of each Head in MultiHead is calculated as shown in formula (10): Then, the output results of all heads are connected. The specific process is shown in formula (11): in n h The output of the attention head, W O is the transformation matrix used to project the concatenated multi-head attention results into the feature space; Through multiple encoders and decoders, the features obtained in step 4 are converted into feature vectors containing global information and correlation information, achieving the effect of suppressing background noise and strengthening regional features related to the target; The obtained feature vector is detected by three detectors to generate NB prediction boxes; the three detectors respectively generate the target type, target center point, and prediction box size. By integrating the output information of the three detectors, NB prediction boxes can be obtained.

9. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 8, characterized in that: In step 4, since the position of the value in the feature space is related to the overall semantics, a position code is added to represent the position in the feature space; the position code at (i, 2j), (i, 2j+1) in the feature space is shown in formula (12):

10. The method for detecting small targets in spatial domain based on deep learning semantic feature separation according to claim 1, characterized in that: The step five includes: During the training phase, the Hungarian algorithm is used for bipartite graph matching. The predicted box set is matched with the real box set. The predicted box with the highest matching degree is selected as the best predicted box to minimize the overall loss. The loss is then reduced through the gradient descent algorithm. In the prediction stage, the prediction box whose prediction category confidence is higher than the threshold is directly selected as the final prediction box.

Images