High-precision target detection method for small remote sensing targets

Abstract

The present invention belongs to the field of computer vision technology, and in particular, relates to a high-precision target detection method for remote sensing small targets, comprising: S1: inputting a remote sensing image containing a target into an improved Transformer network for feature extraction, obtaining local features and global features, and inputting both local features and global features into a feature pyramid network for cross-scale feature fusion to obtain fusion features; S2: presetting no less than 10,000 anchor frames in a region proposal network, and allocating positive and negative samples to each target based on a scale-balanced sample allocation strategy; S3: generating a region of interest based on the allocation result of step S2, and inputting the region of interest into a detection head network for prediction to obtain classification and positioning results. The present invention can focus on the local data around the small target, and explore the global data with high correlation with the small target from a long distance, and extract local features and global features for the small target.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to a high-precision target detection method for remote sensing small targets. Background Art

[0002] As an important means of obtaining information on the earth's surface at a distance, remote sensing technology is widely used in environmental monitoring, disaster assessment, agricultural management, urban planning and other fields. With the continuous advancement of remote sensing technology, remote sensing platforms such as satellites and drones can provide higher resolution image data, and obtaining detailed information from massive data is becoming more and more critical. In remote sensing images, small target detection (such as pedestrians, vehicles, ships, etc.) has received widespread attention due to its importance in military and civilian applications. However, compared with other target detection tasks, remote sensing small target detection faces greater challenges. These small targets are widely distributed in remote sensing images and have the characteristics of small size, low signal-to-noise ratio and poor separability. These characteristics make it difficult to achieve ideal results in the process of processing remote sensing small targets using traditional target detection algorithms. In practical applications, small targets usually contain more key information, and once missed detection or false detection occurs, it may lead to greater losses. In addition, various interdisciplinary fields, such as maritime rescue, marine protection, precision agricultural observation and wildlife detection, require accurate detection of small targets. Therefore, the problem of small target detection in remote sensing images needs to be solved urgently. Summary of the invention

[0003] In view of this, the invention aims to provide a high-precision target detection method for small remote sensing targets to solve the problem that the existing technology cannot accurately detect small targets. The present invention provides an improved Transformer network structure guided from local to global, which can focus on the local data around the small target and explore the global data with high correlation with the small target from a long distance, so as to extract key local features and global features for the small target. At the same time, the present invention provides a scale-balanced sample allocation strategy, which starts with uniform allocation, combines the scale of the small target and the certainty of the network, dynamically determines the positive samples of the small target, and can allocate more positive samples to small targets with low signal-to-noise ratios. In the training stage, the network structure is guided to transfer the learning focus to the small target, thereby improving the detection effect of the small target. In addition, the allocation strategy gradually adjusts the sample size according to the training process, thereby improving the robustness of the network structure.

[0004] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0005] A high-precision target detection method for remote sensing small targets specifically includes the following steps:

[0006] S1: Input the remote sensing image containing the target into the improved Transformer network for feature extraction to obtain local features and global features, and input both the local features and the global features into the feature pyramid network for cross-scale feature fusion to obtain fusion features;

[0007] S2: Preset no less than 10,000 anchor boxes in the region proposal network, and assign positive and negative samples to each target based on the scale-balanced sample allocation strategy;

[0008] S3: Generate a region of interest based on the allocation result of step S2, and input the region of interest into the detection head network for prediction to obtain classification and positioning results.

[0009] Furthermore, in step S1, before the remote sensing image containing the small target is input into the improved Transformer network, the remote sensing image containing the small target is segmented using a 3×3 convolutional layer to obtain an image subset to be processed, and the image subset to be processed includes n sub-images to be processed.

[0010] Furthermore, the improved Transformer network includes a first-stage network module, a second-stage network module, a third-stage network module, and a fourth-stage network module that are connected in sequence;

[0011] The Local Transformer Block is the LTB module. The first-stage network module includes at least one LTB module and a linear mapping layer. The features input to the first-stage network module are mapped by the linear mapping layer and then extracted by the sequentially connected LTB modules.

[0012] The second-stage network module includes at least one LTB module and a linear mapping layer. The features input to the second-stage network module are mapped by the linear mapping layer and then local features are extracted by the sequentially connected LTB modules.

[0013] The Global Transformer Block is the GTB module. The third-stage network module includes at least one GTB module and a linear mapping layer. The features input to the third-stage network module are mapped by the linear mapping layer and then extracted by the GTB modules connected in sequence for global features.

[0014] The fourth-stage network module includes at least one GTB module and a linear mapping layer. The features input to the fourth-stage network module are mapped by the linear mapping layer and then globally extracted by the sequentially connected GTB modules.

[0015] Furthermore, step S1 specifically includes the following steps:

[0016] S101: Input each sub-image to be processed into the first-stage network module in sequence, and each sub-image to be processed is processed by the linear mapping layer in sequence, and then mapped by three parallel linear mapping layers of the LTB module to obtain the Q matrix, K matrix and V matrix of each sub-image to be processed;

[0017] S102: Taking the elements of Q matrix As the center, set the local window of size w×w, and in the current local window, use the element Select the locally relevant token feature matrices in the K matrix and the V matrix respectively:

[0018] ;

[0019] ;

[0020] ;

[0021] in, is the position in the Q matrix Elements , for In the local window neighboring token feature matrix, for In the local window neighboring token feature matrix, , , are all mapping matrices, and Use The selected locally relevant token feature matrix, The position coordinates are The pixel value of the pixel;

[0022] S103: Elements based on local window attention mechanism Perform sparse local attention calculation:

[0023]

[0024] ;

[0025] in, is the matrix transpose, is the softmax function, is the scaling factor, To use The token feature matrix performs the calculation result of the local window attention. For the position coordinates , take k locally related token matrices for self-attention calculation, Attention is self-attention calculation, and Use Selected local relevant token feature matrix;

[0026] S104: Use the next element of the Q matrix Replace the current element , repeat step S102 to step S103 until the calculation of all elements of the Q matrix is ​​completed;

[0027] S105: Divide all token feature matrices selected in step S104 into n parts on the C channel, perform local window attention calculation on the n token feature matrices, and output the cascaded n calculation results through a feedforward neural network to obtain local features. :

[0028] ;

[0029] ;

[0030] ;

[0031] ;

[0032] in, is the input token feature matrix, For cascade operation, is the local feature, LN is the normalization layer, MLP is the linear mapping layer, FFN is the feedforward neural network, is the nth attention head, MultiHead is the multi-head self-attention calculation, is the input of the feedforward neural network, is the token feature matrix in the Q matrix on the nth attention head, is the token feature matrix in the K matrix on the nth attention head, is the token feature matrix in the V matrix on the nth attention head;

[0033] S106: inputting the local features output by the first-stage network module into the second-stage network module, repeating steps S101 to S105, and obtaining the local features output by the second-stage network module;

[0034] S107: The local features output by the second-stage network module are input to the third-stage network module. After being processed by the three parallel linear mapping layers of the GTB module, the Q matrix, K matrix and V matrix are output. The offset network of the GTB module samples the surrounding positions based on the Q matrix. The coordinates of {(-1, -1), (-1, 0), ..., (0, 0), ..., (1, 0), (1, 1)} predict the offset matrix ,in, are the length and width of the current image to be processed, respectively. is the predicted value at each pixel position of the current image to be processed The x- and y-direction offset values ​​of the relevant token feature matrix;

[0035] S108: Introducing the void coefficient , void coefficient The basic sampling area is {(-r, -r), (-r, 0), ..., (0, 0), ..., (r, 0), (r, r)}, and the following formula is used to predict the element in the Q matrix The location of the relevant token feature matrix:

[0036] ;

[0037] in, is the ceiling function, The void coefficient For position The token feature matrix of the predicted position is obtained by performing relevant token feature matrix calculation on the token feature matrix of the predicted position, where x(i, j) is the pixel value at the position coordinate (i, j). (i, j) is the position offset matrix of 9 relevant positions calculated with x(i, j) as the center;

[0038] S109: Based on the calculation result of step S108, the element The global correlation matrix with the token feature matrix of the predicted position is:

[0039] ;

[0040] ;

[0041] ;

[0042] in, All in the element The global correlation matrix established by introducing the void coefficient r;

[0043] S110: Based on the calculation result of step S109, Perform global deformable attention calculation with the global correlation matrix:

[0044]

[0045] ;

[0046] in, For The calculation result of global deformable attention performed on the token feature matrix;

[0047] S111: Use the next element of the Q matrix Replace the current element , repeat step S108 to step S110 until the calculation of all elements of the Q matrix is ​​completed;

[0048] S112: Divide all token feature matrices obtained in step S111 into n parts on the C channel, perform independent global attention calculation on each channel for the token feature matrices divided into n parts, and cascade the global attention calculation results to output the global features;

[0049] S113: inputting the global features output by the third-stage network module into the fourth-stage network module, repeating steps S107 to S112, and obtaining the global features output by the fourth-stage network module;

[0050] S114: Input the local features extracted by the first-stage network module and the second-stage network module, and the global features extracted by the third-stage network module and the fourth-stage network module into the feature pyramid network for multi-scale feature fusion to obtain fused features.

[0051] Furthermore, in step S112, the GTB module consists of two branches, the first branch performs a global deformable attention operation, and the second branch performs a local window attention operation:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] in, is the calculation result of the nth attention head of the first branch, is the calculation result of the nth attention head of the second branch, is the global deformable attention operation, is a local window attention operation, Concat is a cascade operation, is the calculation result of the multi-head self-attention output of the first branch, The calculation result of multi-head self-attention output by the second branch;

[0057] Before the extracted global features are input into the feedforward neural network, the calculation results of the two branches are added and fused to achieve contextual integration of local information and global information:

[0058] ;

[0059] ;

[0060] in, It is the global feature output by the third-stage network module or the fourth-stage network module.

[0061] Furthermore, step S2 specifically includes the following steps:

[0062] S21: Region Proposal Network according to the preset training process Determine the total number of samples of positive samples :

[0063] ;

[0064] ;

[0065] in, is a time-dependent variable, and are the maximum epoch and the minimum epoch respectively, and All are control variables is a hyperparameter of the scope of action, N is the total number of samples of remote sensing images containing the target, is the rounding function;

[0066] S22: Calculate the average number of samples of each target in the remote sensing image containing the target:

[0067] ;

[0068] in, is the average number of samples of each target, G is the number of targets in the remote sensing image containing the target;

[0069] S23: The region proposal network uses the anchor box to measure the prediction accuracy of the current target, and uses the classification score and IoU score predicted by the region proposal network to evaluate the quality of the anchor box:

[0070] ;

[0071] in, Anchor box The corresponding current target Quality score, yes The category label of Anchor box Belongs to category label The predicted classification score of Anchor box The coordinates of For the goal The coordinates of is the importance coefficient of the classification score and IoU score controlled by the hyperparameters, is a matching indicator, and the anchor box On Target Candidate set middle;

[0072] S24: The corresponding front of each target The average quality score of the anchor boxes is used to evaluate the prediction certainty of the region proposal network for the current target. :

[0073] ;

[0074] in, For quantity, are the top k anchor boxes with the highest quality scores;

[0075] S25: Calculate current target The number of positive samples assigned :

[0076] ;

[0077] in, , There are two hyperparameters responsible for controlling the ability of the mapping function. is the target j, w and h are the targets The width and height of

[0078] S26: According to the IoU score, the target’s front anchor frames as positive samples, and according to the proportion of positive samples in the remote sensing image containing the target, negative samples of equal proportion are allocated to the remote sensing image containing the target;

[0079] S27: Replace the current target with the next target and repeat steps S23 to S26 to achieve the allocation of positive and negative samples for each target.

[0080] Furthermore, in step S3, the positive and negative samples and the fused features are input into the ROI alignment layer to generate a region of interest, and the region of interest is input into the detection head network for detection to obtain classification and positioning results.

[0081] Compared with the prior art, the invention can achieve the following beneficial effects:

[0082] The present invention creates the high-precision target detection method for small remote sensing targets, and provides a new improved Transformer network from local to global to solve the problem of small target detection in remote sensing images. In order to extract local information around small targets, an LTB module (Local Transformer Block) is provided, which can limit the self-attention calculation to a local window, and perform attention interaction through locally related data to extract local features. In order to explore the global features associated with small targets, a GTB module (Global Transformer Block) is designed, which can search for data associated with small targets in the distance to supplement global context features. In order to allocate learning resources to small targets, the present invention provides a scale-balanced sample allocation strategy based on uniform distribution, which can allocate sufficient positive samples to small targets, thereby generating more training content. The small target detection method provided by the present invention provides more learning resources for small targets from the two aspects of feature extraction and sample allocation. A large number of experiments were carried out on three large-scale remote sensing datasets, DIOR, AI-TOD, and NWPU VHR-10, using the small target detection method proposed in the present invention. The experimental results show that the target detection method proposed in the present invention can effectively identify small targets in remote sensing images and achieve high-precision detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The drawings constituting part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments and descriptions of the present invention are used to explain the present invention and do not constitute an improper limitation on the present invention. In the drawings:

[0084] Figure 1 A flowchart of a high-precision target detection method for small remote sensing targets according to an embodiment of the present invention;

[0085] Figure 2 A schematic diagram of the structure of high-precision target detection for small remote sensing targets according to an embodiment of the present invention;

[0086] FIG3 (a) is a schematic diagram of the structure of the local window attention mechanism described in an embodiment of the present invention;

[0087] FIG3( b ) is a schematic diagram of the structure of the global deformable attention mechanism according to an embodiment of the present invention;

[0088] Figure 4 A schematic diagram of the structure of the global deformable attention calculation related data described in the embodiment of the present invention;

[0089] FIG5 (a) is a schematic diagram of the structure of the LTB module according to an embodiment of the present invention;

[0090] FIG5( b ) is a schematic diagram of the structure of the GTB module according to an embodiment of the present invention;

[0091] Figure 6 A schematic diagram of the results of small target detection in remote sensing images according to an embodiment of the present invention;

[0092] Figure 7 A schematic diagram of the results of small target detection in a large-scale remote sensing image as described in an embodiment of the present invention. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solution and advantages of the invention more clear, the invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the invention and do not constitute a limitation of the invention.

[0094] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0095] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0096] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0097] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0098] like Figure 1 As shown, the present invention proposes a high-precision target detection method for small remote sensing targets, which specifically includes the following steps:

[0099] S1: The remote sensing image containing small targets is input into the improved Transformer network for feature extraction to obtain local features and global features. Both local features and global features are input into the feature pyramid network for cross-scale feature fusion to obtain fused features.

[0100] In step S1, before the remote sensing image containing the small target is input into the improved Transformer network, the remote sensing image containing the small target is divided into images using a 3×3 convolutional layer to obtain an image subset to be processed, where the image subset to be processed includes n sub-images to be processed.

[0101] S101: Input each sub-image to be processed into the first-stage network module in sequence. After each sub-image to be processed is processed by the linear mapping layer in sequence, it is mapped by the three parallel linear mapping layers of the LTB module to obtain the Q matrix, K matrix and V matrix of each sub-image to be processed. Each element of the Q matrix, K matrix and V matrix is ​​referred to as element q, element k and element v respectively. That is, the size is The remote sensing image is linearly mapped into a dimension of The token feature matrix of , where the size of each token feature matrix is , The number of channels for each token feature matrix.

[0102] S102: Taking the elements of Q matrix As the center, set the local window of size w×w, and in the current local window, use the element Select the locally relevant token feature matrices in the K matrix and the V matrix respectively:

[0103] ;

[0104] ;

[0105] ;

[0106] in, is the position in the Q matrix Elements , for In the local window neighboring token feature matrix, for In the local window neighboring token feature matrix, , , are all mapping matrices, and Use The selected locally relevant token feature matrix, The position coordinates are The pixel value of the pixel, and In space Close, able to provide local information.

[0107] S103: Elements based on local window attention mechanism Perform sparse local attention calculation:

[0108]

[0109] ;

[0110] in, is the matrix transpose, is the softmax function, is the scaling factor, To use The token feature matrix performs the calculation result of the local window attention. For the position coordinates , take k locally related token matrices for self-attention calculation, Attention is self-attention calculation, and Use Selected local relevant token feature matrix.

[0111] S104: Use the next element of the Q matrix Replace the current element , repeat step S102 to step S103 until the calculation of all elements of the Q matrix is ​​completed. At this time, each position on the Q matrix can interact with the local data. For the data at the edge of the Q matrix, zero padding in the convolution is used to ensure the consistency of the dimension. The size of the local window strictly controls the size of the K matrix and the V matrix, that is, the operation space of the local window attention. The local window attention can capture local patterns in the features and is suitable for small targets with local dependencies. At the same time, the interaction only occurs on the token feature matrix within the local window, which reduces the computational complexity.

[0112] S105: In order to further save computing resources,

[0113] Divide all token feature matrices selected in step S104 into n parts on the C channel, perform local window attention calculation on the n token feature matrices, and output the cascaded n calculation results through a feedforward neural network to obtain local features. :

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] in, is the input token feature matrix, For cascade operation, is the local feature, LN is the normalization layer, MLP is the linear mapping layer, FFN is the feedforward neural network, is the nth attention head, MultiHead is the multi-head self-attention calculation, is the input of the feedforward neural network, is the token feature matrix in the Q matrix on the nth attention head, is the token feature matrix in the K matrix on the nth attention head, is the token feature matrix in the V matrix on the nth attention head. The FFN consists of two linear layers and a GELU activation layer, and Dropout random inactivation is introduced.

[0119] S106: Input the local features output by the first-stage network module to the second-stage network module, and repeat steps S101 to S105 to obtain the local features output by the second-stage network module.

[0120] After the local features are extracted through the above processing, the local features are sent to two consecutive GTB modules for global feature extraction. The GTB module is used for global deformable attention to establish long-range dependencies (searching for data related to the current target in the entire image, and the relevant data plays a positive role in the decision-making of the current target. For example, when you want to detect a small boat on the sea, you cannot determine what it is based on its own information, but you can determine that the current target is also a boat based on the boats in the distance, although they are far away. This situation is represented by long-range dependencies).

[0121] S107: The local features output by the second-stage network module are input to the third-stage network module. After being processed by the three parallel linear mapping layers of the GTB module, the Q matrix, K matrix and V matrix are output. The offset network of the GTB module samples the surrounding positions based on the Q matrix. The coordinates of {(-1, -1), (-1, 0), ..., (0, 0), ..., (1, 0), (1, 1)} predict the offset matrix ,in, are the length and width of the current image to be processed, respectively. is the predicted value at each pixel position of the current image to be processed The x- and y-direction offset values ​​of the related token feature matrix.

[0122] S108: In order to prevent the prediction offset from being too large and interfering with the training process, a hole coefficient is introduced here to control this, specifically based on the sampling area The hole coefficient is set to expand the extraction area of ​​the relevant token feature matrix. Based on the expanded sampling area, the predicted offset value can be further reduced, and long-distance modeling can also be achieved.

[0123] Introducing the void coefficient , void coefficient The basic sampling area is {(-r, -r), (-r, 0), ..., (0, 0), ..., (r, 0), (r, r)}, and the following formula is used to predict the element in the Q matrix The location of the relevant token feature matrix:

[0124] ;

[0125] in, is the ceiling function, The void coefficient For position The token feature matrix of the predicted position is obtained by performing relevant token feature matrix calculation on the token feature matrix of the predicted position, where x(i, j) is the pixel value at the position coordinate (i, j). (i, j) is the position offset matrix of 9 related positions calculated with x(i, j) as the center.

[0126] After obtaining the position relative to element q, Matrix and The matrix is ​​indexed by position, the corresponding global correlation matrix is ​​obtained, and the global deformable attention is calculated. Like the local window attention, the global deformable attention is also a sparse attention representation.

[0127] S109: Based on the calculation result of step S108, the element The global correlation matrix with the token feature matrix of the predicted position is:

[0128] ;

[0129] ;

[0130] ;

[0131] in, is in the element The global correlation matrix established by introducing the void coefficient r;

[0132] S110: Based on the calculation result of step S109, Perform global deformable attention calculation with the global correlation matrix:

[0133]

[0134] ;

[0135] in, For The result of the global deformable attention calculation performed on the token feature matrix.

[0136] S111: Use the next element of the Q matrix Replace the current element , repeat step S108 to step S110 until the calculation of all elements of the Q matrix is ​​completed.

[0137] S112: Divide all token feature matrices obtained in step S111 into n parts on the C channel, and perform independent global attention calculation on each channel for the token feature matrices divided into n parts, and cascade the global attention calculation results to output the global features.

[0138] In step S112, the GTB module consists of two branches, the first branch performs a global deformable attention operation, and the second branch performs a local window attention operation:

[0139] ;

[0140] ;

[0141] ;

[0142] ;

[0143] in, is the calculation result of the nth attention head of the first branch, is the calculation result of the nth attention head of the second branch, is the global deformable attention operation, is a local window attention operation, Concat is a cascade operation, is the calculation result of the multi-head self-attention output of the first branch, The calculation result of multi-head self-attention output by the second branch;

[0144] Before the extracted global features are input into the feedforward neural network, the calculation results of the two branches are added and fused to achieve contextual integration of local information and global information:

[0145] ;

[0146] ;

[0147] in, It is the global feature output by the third-stage network module or the fourth-stage network module.

[0148] S113: input the global features output by the third-stage network module to the fourth-stage network module, and repeat steps S107 to S112 to obtain the global features output by the fourth-stage network module.

[0149] S114: Input the local features extracted by the first-stage network module and the second-stage network module, and the global features extracted by the third-stage network module and the fourth-stage network module into the feature pyramid network for multi-scale feature fusion to obtain fused features.

[0150] S2: Preset no less than 10,000 anchor boxes in the region proposal network (the same as the network structure of the region proposal network RPN of the Faster R-CNN network), and assign positive and negative samples to each small target based on the scale-balanced sample allocation strategy.

[0151] After the features are input into the region proposal network, the region proposal network predicts the probability and position offset of the foreground background for the preset anchor box based on the input features. According to the prediction, the region proposal network assigns samples as network calculation loss items to generate learning content.

[0152] S21: Region Proposal Network according to the preset training process Determine the total number of samples of positive samples :

[0153] ;

[0154] ;

[0155] in, is a time-dependent variable, and are the maximum epoch and the minimum epoch respectively, and All are control variables is a hyperparameter of the scope of action, N is the total number of samples of remote sensing images containing the target, is the rounding function.

[0156] S22: Calculate the average number of samples of each target in the remote sensing image containing small targets:

[0157] ;

[0158] in, is the average number of samples of each target, and G is the number of targets in the remote sensing image containing small targets.

[0159] As training progresses, the total number of samples gradually decreases. The region proposal network can dynamically change the difficulty of training. In the early stage, the region proposal network uses a large number of samples to cover a variety of situations. In the later stage, the number of samples is reduced to focus on high-quality samples and improve the overall performance of the small object detection model. At the same time, reducing the number of samples can speed up training and save computing resources.

[0160] In order to make the allocation rule directional, the sample size of small targets should be dynamically adjusted according to the difficulty of identifying small targets. Here, the difficulty of identifying small targets is considered from two aspects, namely, their own characteristics and prediction complexity. For their own characteristics, scale changes will seriously affect the judgment of the network structure, and small-scale targets will tend to be detected incorrectly.

[0161] S23: The region proposal network uses the anchor box to measure the prediction accuracy of the current small target, and uses the classification score and IoU score (the intersection and union score of the predicted box (anchor box) and the target's labeled box) predicted by the region proposal network to evaluate the quality of the anchor box:

[0162] ;

[0163] in, Anchor box The corresponding current target Quality score, yes The category label of Anchor box Belongs to category label The predicted classification score of Anchor box The coordinates of For the goal The coordinates of is the importance coefficient of the classification score and IoU score controlled by the hyperparameters, is a matching indicator, and the anchor box On Target Candidate set middle.

[0164] Low certainty reflects that the network structure has not fully learned the small target. The present invention combines the target scale and the calculated prediction certainty to characterize the identification difficulty of the small target. According to the identification difficulty, the network is guided to focus on the small target with high difficulty, providing an optimization direction for the training process. Specifically, the calculated identification difficulty is mapped into a regularized adjustment factor through a quadratic function, and the adjustment factor is related to Multiplication realizes dynamic allocation of samples.

[0165] Next, the quality score of each anchor box for each object is calculated, which represents the quality of the anchor box matching the small object.

[0166] S24: The corresponding front of each target The average quality score of the anchor boxes is used to evaluate the prediction certainty of the region proposal network for the current target. :

[0167] ;

[0168] in, For quantity, are the top k anchor boxes with the highest quality scores;

[0169] S25: Calculate current target The number of positive samples assigned :

[0170] ;

[0171] in, , There are two hyperparameters responsible for controlling the ability of the mapping function. is the target j, w and h are the targets width and height.

[0172] S26: According to the IoU score, the target’s front Anchor boxes are used as positive samples, and negative samples of equal proportion are allocated to the remote sensing images containing the target according to the proportion of positive samples occupying the remote sensing images containing the target.

[0173] S27: Replace the current target with the next target and repeat steps S23 to S26 to achieve the allocation of positive and negative samples for each target.

[0174] S3: Generate a region of interest based on the allocation result of step S2, and input the region of interest into the detection head network (with the same network structure as the region proposal network RPN of the Faster R-CNN network) for prediction to obtain classification and positioning results.

[0175] In step S3, the positive and negative samples and the fused features are input into the ROI alignment layer (the network structure is the same as the ROI alignment layer of the Faster R-CNN network) to generate the region of interest, and the region of interest is input into the detection head network for detection to obtain the classification and positioning results.

[0176] like Figure 2 As shown, the present invention proposes a high-precision target detection method for small remote sensing targets. The target detection method sends the remote sensing image containing the small target into the improved Transformer network guided from local to global to extract local features and global features, and sends the extracted features into the feature pyramid network for cross-scale feature fusion. The fused features are then sent to the region proposal network to generate a large number of proposals, and positive and negative samples are allocated to each small target through the proposed scale-balanced sample allocation strategy. A region of interest (RoI) is generated based on the allocated samples, and the RoI is sent to the detection head network for prediction to obtain classification and positioning results.

[0177] The improved Transformer network guided from local to global of the present invention comprises a first-stage network module, a second-stage network module, a third-stage network module and a fourth-stage network module connected in sequence;

[0178] The Local Transformer Block is the LTB module. The first-stage network module includes at least one LTB module and a linear mapping layer. The features input to the first-stage network module are mapped by the linear mapping layer and then extracted by the sequentially connected LTB modules.

[0179] The second-stage network module includes at least one LTB module and a linear mapping layer. The features input to the second-stage network module are mapped by the linear mapping layer and then local features are extracted by the sequentially connected LTB modules.

[0180] The Global Transformer Block is the GTB module. The third-stage network module includes at least one GTB module and a linear mapping layer. The features input to the third-stage network module are mapped by the linear mapping layer and then extracted by the GTB modules connected in sequence for global features.

[0181] The fourth-stage network module includes at least one GTB module and a linear mapping layer. The features input to the fourth-stage network module are mapped by the linear mapping layer and then globally extracted by the sequentially connected GTB modules.

[0182] Remote sensing images with a resolution of 800×800×3 , each stage module is set to [2, 2, 6, 2] as an example. First, the remote sensing image When input into the network structure, it will be divided into a subset of images to be processed with a dimension of 200×200×48, and the dimension will be mapped to 200×200×72 through linear mapping, and sent to the LTB module of the first two stage network modules for local feature extraction. Each stage network module consists of 2 consecutive LTB modules, and outputs local features with dimensions of 200×200×72 and 100×100×144 respectively. The features with dimensions stretched to 100×100×144 are then fed into the GTB modules of the next two stages of network modules for global feature extraction. The third stage network module contains 6 consecutive GTB modules, and the fourth stage network module contains 2 consecutive GTB modules, and outputs global features with dimensions of 50×50×288 and 25×25×576 respectively. , and then the multi-scale features and Send it into the feature pyramid network for multi-scale feature fusion.

[0183] As shown in Figure 3 (a), a single element in the Q matrix Take the example to demonstrate the calculation of local window attention. First, the input features are passed through three parallel linear mapping layers to output Q matrix, K matrix and V matrix. For an element in Q matrix , select the element Centered at The local window of , the 9 token feature matrices in the local window are indexed in the K matrix and the V matrix, and then the single element Multiply the 9 token feature matrices selected from the K matrix and output the attention weights through softmax. Next, multiply the attention weights with the 9 token feature matrices selected from the V matrix to output local features.

[0184] As shown in Figure 3(b), a single element in the Q matrix Take the calculation of global deformable attention as an example. First, the input features are passed through three parallel linear mapping layers to output Q matrix, K matrix and V matrix. For an element in Q matrix , the element Input to the offset network, output 9 AND elements The position index of the relevant token feature matrix is ​​calculated, and the 9 token feature matrices are selected in the K matrix and the V matrix using the calculated position index. Secondly, the single element Multiply the 9 token feature matrices selected from the K matrix and output the attention weights through softmax. Next, multiply the attention weights with the 9 token feature matrices selected from the V matrix and output the global features.

[0185] The scale-balanced sample allocation strategy of the present invention is introduced below. In the embodiment of the present invention, it is assumed that the proposed target detection method is trained for a total of 12 epochs, and the total number of samples of a single image is specified to be , the hyperparameters are set to , , to control the sample size, the number of real targets in the current image , , control the ratio of classification score and IoU score, and select the first 9 anchor boxes for each small target ( ) is used to evaluate the degree of certainty, making the hyperparameter , To control the ability of the mapping function, the training process The total number of positive samples that should be sampled for:

[0186] ;

[0187] ;

[0188] Secondly, calculate the average number of samples per target in the current image and the current process:

[0189] ;

[0190] Next, the quality score of each anchor box matching the true target is calculated as:

[0191] ;

[0192] Then, the certainty of each small target is evaluated based on the average quality score of the first 9 anchor boxes of each small target. :

[0193] ;

[0194] Then, the number of positive samples that should be assigned to each small target is:

[0195] ;

[0196] Finally, negative samples are randomly sampled in equal proportion according to the total number of these 10 target positive samples to complete the scale-balanced sample allocation.

[0197] like Figure 4 As shown, the global deformable attention needs to first find the position of the element related to the current position globally, and calculate the multi-head self-attention according to the relevant element index. For the input feature x, the offset prediction network first predicts the offset of the element related to each position of the input feature x in the x direction and the y direction. The offset prediction network consists of a separable convolution layer, an LN layer, a point convolution layer and an LN layer. The present invention obtains the coordinate matrix of the relevant token feature matrix by adding the predicted offset to the position of the corresponding element on the input feature x and the hole position, and then after the position rounding operation, the features in the K matrix and the V matrix are indexed to obtain the relevant token feature matrix corresponding to the position index, and the current position feature and the calculated related feature are subjected to multi-head self-attention processing after linear mapping, and then the global feature is obtained after being processed by the feedforward neural network.

[0198] The global deformable attention calculation is introduced by taking 3 attention calculation heads with hole coefficients of (1, 2, 3) as an example. Assuming that the input feature is , use each element q in the Q matrix to calculate 9 token feature matrices related to it, for the position in the Q matrix The element q of , the position of the relevant token feature matrix predicted by each of the three attention calculation heads is:

[0199] ;

[0200] ;

[0201] ;

[0202] in, , , Represents the elements calculated with the void coefficients (1, 2, 3) The relevant position of the K matrix and the V matrix is ​​indexed according to the position to obtain the element The matrix with the elements k and v at the respective deformation positions in the three attention calculation heads:

[0203] ;

[0204] ;

[0205] ;

[0206] ;

[0207] ;

[0208] ;

[0209] in, , , , , , They are the matrices of the related elements k and v calculated using the void coefficients (1, 2, 3).

[0210] According to the calculation results of the global deformable attention, the three-head attention calculation of the local deformable attention calculation branch in the GTB module with a hole coefficient of (1, 2, 3) is expressed as:

[0211] ;

[0212] ;

[0213] ;

[0214] ;

[0215] Calculated Add it to the output of LWA from the other branch and output the final global feature.

[0216] The present invention provides an LTB (Local Transformer Block) module and a GTB (Global Transformer Block) module, and the structure thereof is explained as follows. As shown in FIG5 (a), the LTB module is a single-branch network, which is composed of an LN layer, a linear mapping layer, a local window attention mechanism, an LN layer, and a feedforward neural network. The detailed processing flow is as follows: first, the input features are regularized by the LN layer, and linearly mapped into a Q matrix, a K matrix, and a V matrix, and the Q matrix, the K matrix, and the V matrix are sent to the local window attention mechanism for local attention calculation, and the output features are added to the input features to realize feature fusion, and then the fused features are sent to the LN layer for regularization again, and finally, the regularized features are sent to the feedforward neural network for feature conversion, and the extracted local features are output. As shown in Figure 5 (b), the GTB module is a two-branch network. The first branch includes an LN layer, a linear mapping layer, and a local window attention mechanism. The second branch includes an LN layer, a linear mapping layer, and a global deformable attention mechanism. After the two branches merge, they pass through the LN layer and the feedforward neural network in sequence. The detailed processing flow is as follows: first, the input features are simultaneously input into the LN layers of the two branches for regularization processing, and each is linearly mapped into a Q matrix, a K matrix, and a V matrix. For the first branch, the Q matrix, the K matrix, and the V matrix are sent to the local window attention mechanism for local attention calculation. For the second branch, the Q matrix, the K matrix, and the V matrix are sent to the global deformable attention mechanism for global attention calculation. The features output by the two branches are added to the input features to achieve feature fusion. Subsequently, the fused features are sent to the LN layer for regularization again. Finally, the regularized features are sent to the feedforward neural network for feature transformation, and the extracted global features are output.

[0217] like Figure 6 As shown, the benchmark target detection method has a serious phenomenon of missing detection of vehicles in the parking lot, while the target detection method proposed in the present invention can accurately locate these densely distributed vehicles and ensure a high recall rate. Similarly, the benchmark target detection method cannot effectively identify ships of extremely small scale. For such small targets that may only occupy dozens of pixels, the proposed target detection method can still maintain high accuracy. For targets such as camouflaged wind power stations, the target detection method proposed in the present invention can still accurately identify such weak targets from backgrounds of similar colors. This fully demonstrates the detection ability of the method of the present invention for small targets.

[0218] like Figure 7As shown in FIG. 1 , compared with the benchmark detection method, the target detection method provided by the present invention correctly predicts more targets. Figure 7 In the enlarged area, the target detection method proposed in the present invention can detect ships at sea and vehicles on the shore. However, the benchmark method seriously misses these small targets in the marked area. This shows that the target detection method proposed in the present invention can target small targets densely distributed in complex backgrounds, which are of great value in practical decision-making.

[0219] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0220] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-precision target detection method for small remote sensing targets, characterized by: The specific steps include: S1: Input the remote sensing image containing the target into the improved Transformer network for feature extraction to obtain local features and global features, and input both the local features and the global features into the feature pyramid network for cross-scale feature fusion to obtain fusion features; S2: Preset no less than 10,000 anchor boxes in the region proposal network, and assign positive and negative samples to each target based on the scale-balanced sample allocation strategy; S21: Region Proposal Network according to the preset training process Determine the total number of samples of positive samples : ； ； in, is a time-dependent variable, and are the maximum epoch and the minimum epoch respectively, and All are control variables is a hyperparameter of the scope of action, N is the total number of samples of remote sensing images containing the target, is the rounding function; S22: Calculate the average number of samples of each target in the remote sensing image containing the target: ； in, is the average number of samples of each target, G is the number of targets in the remote sensing image containing the target; S23: The region proposal network uses the anchor box to measure the prediction accuracy of the current target, and uses the classification score and IoU score predicted by the region proposal network to evaluate the quality of the anchor box: ； in, Anchor box The corresponding current target Quality score, yes The category label of Anchor box Belongs to category label The predicted classification score of Anchor box The coordinates of For the goal The coordinates of is the importance coefficient of the classification score and IoU score controlled by the hyperparameters, is a matching indicator, and the anchor box On Target Candidate set middle; S24: The corresponding front of each target The average quality score of the anchor boxes is used to evaluate the prediction certainty of the region proposal network for the current target. : ； in, For quantity, are the top k anchor boxes with the highest quality scores; S25: Calculate current target The number of positive samples assigned : ； in, , There are two hyperparameters responsible for controlling the ability of the mapping function. is the target j, w and h are the targets Width and height; S26: According to the IoU score, the target’s front anchor frames as positive samples, and according to the proportion of positive samples in the remote sensing image containing the target, negative samples of equal proportion are allocated to the remote sensing image containing the target; S27: Replace the current target with the next target, and repeat steps S23 to S26 to achieve the allocation of positive and negative samples for each target; S3: Generate a region of interest based on the allocation result of step S2, and input the region of interest into the detection head network for prediction to obtain classification and positioning results.

2. The high-precision target detection method for remote sensing small targets according to claim 1, characterized in that: In step S1, before the remote sensing image containing the small target is input into the improved Transformer network, the remote sensing image containing the small target is segmented using a 3×3 convolutional layer to obtain an image subset to be processed, wherein the image subset to be processed includes n sub-images to be processed.

3. The high-precision target detection method for remote sensing small targets according to claim 2 is characterized in that: The improved Transformer network includes a first-stage network module, a second-stage network module, a third-stage network module and a fourth-stage network module which are connected in sequence; The Local Transformer Block is an LTB module. The first-stage network module includes at least one LTB module and a linear mapping layer. The features input to the first-stage network module are mapped by the linear mapping layer and then local features are extracted by the sequentially connected LTB modules. The second-stage network module includes at least one LTB module and a linear mapping layer, and the features input to the second-stage network module are mapped by the linear mapping layer and then local features are extracted by the sequentially connected LTB modules; The Global Transformer Block is a GTB module. The third-stage network module includes at least one GTB module and a linear mapping layer. The features input to the third-stage network module are mapped by the linear mapping layer and then extracted by the sequentially connected GTB modules for global features. The fourth-stage network module includes at least one GTB module and a linear mapping layer. The features input to the fourth-stage network module are mapped by the linear mapping layer and then global features are extracted by the sequentially connected GTB modules.

4. The high-precision target detection method for remote sensing small targets according to claim 3 is characterized in that: Step S1 specifically includes the following steps: S101: Input each sub-image to be processed into the first-stage network module in sequence, and each sub-image to be processed is processed by the linear mapping layer in sequence, and then mapped by three parallel linear mapping layers of the LTB module to obtain the Q matrix, K matrix and V matrix of each sub-image to be processed; S102: Taking the elements of Q matrix As the center, set the local window of size w×w, and in the current local window, use the element Select the locally relevant token feature matrices in the K matrix and the V matrix respectively: ； ； ； in, is the position in the Q matrix Elements , for In the local window neighboring token feature matrix, for In the local window neighboring token feature matrix, , , are all mapping matrices, and Use The selected locally relevant token feature matrix, The position coordinates are The pixel value of the pixel; S103: Elements based on local window attention mechanism Perform sparse local attention calculation: ； in, is the matrix transpose, is the softmax function, is the scaling factor, To use The token feature matrix performs the calculation result of the local window attention. For the position coordinates , take k locally related token matrices for self-attention calculation, Attention is self-attention calculation, and Use Selected local relevant token feature matrix; S104: Use the next element of the Q matrix Replace the current element , repeat step S102 to step S103 until the calculation of all elements of the Q matrix is ​​completed; S105: Divide all token feature matrices selected in step S104 into n parts on the C channel, perform local window attention calculation on the n token feature matrices, and output the cascaded n calculation results through a feedforward neural network to obtain local features. : ； ； ； ； in, is the input token feature matrix, For cascade operation, is the local feature, LN is the normalization layer, MLP is the linear mapping layer, FFN is the feedforward neural network, is the nth attention head, MultiHead is the multi-head self-attention calculation, is the input of the feedforward neural network, is the token feature matrix in the Q matrix on the nth attention head, is the token feature matrix in the K matrix on the nth attention head, is the token feature matrix in the V matrix on the nth attention head; S106: inputting the local features output by the first-stage network module into the second-stage network module, repeating steps S101 to S105, and obtaining the local features output by the second-stage network module; S107: The local features output by the second-stage network module are input to the third-stage network module. After being processed by the three parallel linear mapping layers of the GTB module, the Q matrix, K matrix and V matrix are output. The offset network of the GTB module samples the surrounding positions based on the Q matrix. The coordinates of {(-1, -1), (-1, 0), ..., (0, 0), ..., (1, 0), (1, 1)} predict the offset matrix ,in, are the length and width of the current image to be processed, respectively. is the predicted value at each pixel position of the current image to be processed The x- and y-direction offset values ​​of the relevant token feature matrix; S108: Introducing the void coefficient , void coefficient The basic sampling area is {(-r, -r), (-r, 0), ..., (0, 0), ..., (r, 0), (r, r)}, and the following formula is used to predict the element in the Q matrix The location of the relevant token feature matrix: ； in, is the ceiling function, The void coefficient For position The token feature matrix of the predicted position is obtained by performing relevant token feature matrix calculation on the token feature matrix of the predicted position, where x(i, j) is the pixel value at the position coordinate (i, j). (i, j) is the position offset matrix of 9 relevant positions calculated with x(i, j) as the center; S109: Based on the calculation result of step S108, the element The global correlation matrix with the token feature matrix of the predicted position is: ； ； ； in, All in the element The global correlation matrix established by introducing the void coefficient r; S110: Based on the calculation result of step S109, Perform global deformable attention calculation with the global correlation matrix: ； in, For The calculation result of global deformable attention performed on the token feature matrix; S111: Use the next element of the Q matrix Replace the current element , repeat step S108 to step S110 until the calculation of all elements of the Q matrix is ​​completed; S112: Divide all token feature matrices obtained in step S111 into n parts on the C channel, perform independent global attention calculation on each channel for the token feature matrices divided into n parts, and cascade the global attention calculation results to output the global features; S113: inputting the global features output by the third-stage network module into the fourth-stage network module, repeating steps S107 to S112, and obtaining the global features output by the fourth-stage network module; S114: Input the local features extracted by the first-stage network module and the second-stage network module, and the global features extracted by the third-stage network module and the fourth-stage network module into the feature pyramid network for multi-scale feature fusion to obtain fused features.

5. The high-precision target detection method for remote sensing small targets according to claim 4 is characterized in that: In step S112, the GTB module consists of two branches, the first branch performs a global deformable attention operation, and the second branch performs a local window attention operation: ； ； ； ； in, is the calculation result of the nth attention head of the first branch, is the calculation result of the nth attention head of the second branch, is the global deformable attention operation, is a local window attention operation, Concat is a cascade operation, is the calculation result of the multi-head self-attention output of the first branch, The calculation result of multi-head self-attention output by the second branch; Before the extracted global features are input into the feedforward neural network, the calculation results of the two branches are added and fused to achieve contextual integration of local information and global information: ； ； in, It is the global feature output by the third-stage network module or the fourth-stage network module.

6. The high-precision target detection method for remote sensing small targets according to claim 1, characterized in that: In step S3, the positive and negative samples and the fused features are input into the ROI alignment layer to generate a region of interest, and the region of interest is input into the detection head network for detection to obtain classification and positioning results.

Images