A method for detecting a target

Abstract

The application discloses a kind of directional target detection methods, the method includes the following steps: input target image;Image is converted sequence data in Stem network, i.e.image Token, image Token is sparsified sampling, and the sparse Token obtained by sampling is sent into stem network;Image information is extracted in stem network Abstract feature;From the feature output from stem network, in Head head network, category and bounding box prediction are judged;Output detection result.The application is the first end-to-end standard encoding and decoding structure Transformer network carries out target detection;The network proposed in the application is asymmetric form whether from network structure or from data input, effectively alleviate the high computing overhead problem of Transformer network;The application efficiently extracts features from sparse image Token, largely eliminates the information redundancy in image, reduces the interference between information, thereby reduces the network training convergence period, improves network detection effect.

Description

Technical Field

[0001] This invention relates to the fields of target detection and remote sensing image processing, and specifically to a method for directional target detection. Background Technology

[0002] With the continuous application and innovation of self-attention Transformer technology in computer vision, significant results have been achieved. DETR, first applied by Carion et al. to object detection, uses a Transformer structure as the head network after extracting features through a convolutional neural backbone network to encode and decode the features, ultimately outputting the detection results in parallel. Its performance on the COCO dataset is comparable to Faster R-CNN, and it outperforms Faster R-CNN on large targets. Deformable DETR, an improvement on DETR by Jifeng Dai's team at SenseTime, addresses the slow convergence speed and limited feature space resolution of DETR. Its attention module focuses only on a small set of key sampling points, achieving better performance than DETR with a 10-fold reduction in training time. It particularly alleviates the unacceptable computational complexity of DETR caused by high-resolution feature maps, making it especially beneficial for object detection in high-resolution remote sensing images. Swin Transformer borrows the design principles and prior knowledge of convolutional neural networks to propose a hierarchical structure to model information of targets at various scales. It uses a self-attention network model within a local window, which can replace convolutional networks as a general backbone network to extract image features. Combined with existing head networks, it can be used for tasks such as image classification, object detection, and dense prediction.

[0003] As can be seen from the above, the application of Transformer technology in the field of object detection is currently as a component of the network, or as the head network, or as a general backbone network. The overall network is a hybrid of convolutional neural networks, and the network structure is relatively complex.

[0004] Currently, mainstream convolutional detection networks offer limited performance improvements in remote sensing target detection. Therefore, it is worthwhile to investigate whether a Transformer network with a simpler structure and more powerful pure encoder-decoder architecture can be designed. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention aims to provide a method for directional target detection. In general, the present invention uses a Transformer-based backbone network as an encoder, adopts a decoder with a structure similar to the original Transformer, and finally directly outputs the detection results.

[0006] Specifically:

[0007] 1. This invention performs abstract feature extraction on the image token after sparse sampling of the image, which serves as the guiding information query. For the decoder, the input is the full image token, but the decoder itself is lightweight. The full image information is first encoded and embedded in the decoder through a self-attention module, and finally, under the guidance of the encoder output query, feature alignment is performed to extract the target encoded embedding information.

[0008] 2. Constructing a simpler network structure. The network of this invention has only two main components, the encoder and decoder, with no other components, making the network simpler and more efficient.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A method for detecting directional targets, the method comprising the following steps:

[0011] S1 is the input target image;

[0012] S2 transforms images into sequence data, i.e., image tokens, in the Stem network, performs sparse sampling on the image tokens, and sends the sampled sparse tokens into the backbone network.

[0013] S3 extracts abstract features of image information from the backbone network;

[0014] S4 features output from the backbone network are used to determine the category and predict bounding boxes in the Head network;

[0015] S5 outputs the detection results.

[0016] It should be noted that each patch obtains a sequence data token by concatenating its internal raw pixels, and the patch size is P×P (in this invention, P is set to 4); secondly, the dimension is mapped to C through a linear embedding layer, resulting in the 3D image being transformed into 1D sequence data X. T ∈R N×C N = HW / P 2 This indicates the number of tokens; finally, the obtained full tokens are sent to the decoder after position encoding, and all tokens are uniformly and randomly sampled. Unsampled tokens are discarded directly, and the sampled sparse tokens are sent to the encoder after position encoding. The sampling rate is set to r.

[0017] It should be noted that the patch is derived from image segmentation, that is, by segmenting the RGB image X∈R. H× W ×3Divide into non-overlapping patches

[0018] It should be noted that the backbone network consists of encoders and decoders.

[0019] It should be noted that the encoder includes a standard VIT network or a hierarchical Swing Transformer; wherein:

[0020] In the VIT network, the sampled sparse tokens are fed into a series of Transformer blocks, and a pooling operation is performed at the end of the encoder to further aggregate the number of tokens and adjust the dimension to 256.

[0021] Using the aforementioned hierarchical structure of the Swin Transformer, the Token data is first transformed into... The image information features are extracted by performing self-attention calculation within a window using the Swing Transformer network, and then compressed to 256 channels using a linear layer. The final output feature X is... output The number of tokens in the input is reduced to X. input 1 / 64 of.

[0022] It should be noted that the input to the decoder comes partly from the output of the encoder and partly from all the tokens in the entire image converted into sequence data, and then positionally encoded.

[0023] It should be noted that in step S4, the Head network includes two feedforward neural networks that predict the bounding box and category of the target instance based on the features output by the backbone network; these can be represented as follows:

[0024] b=sigmoid(FFN(f)), c=FFN(f);

[0025] In the above formula, f is the extracted feature, and b is a five-dimensional vector [b cx b cy b w b h b θ The first four dimensions represent the center point coordinates, length, and width of the bounding box, respectively, and the last dimension represents the angle of the bounding box; the sigmoid function is used to process the predicted vector into the interval [0,1], which represents the position of the image relative to the length and width of the image, and c represents the predicted class.

[0026] The beneficial effects of this invention are as follows:

[0027] 1. This invention is the first end-to-end standard codec structure Transformer network for target detection;

[0028] 2. The network proposed in this invention is asymmetric in both its network structure and data input, which effectively alleviates the high computational overhead problem of Transformer networks;

[0029] 3. This invention efficiently extracts features from sparse image tokens, which largely eliminates information redundancy in the image and reduces interference between information, thereby reducing the network training convergence cycle and improving the network detection effect.

[0030] 4. Based on the characteristics of the rotating frame representation, this invention improves the loss function, eliminating abrupt loss changes caused by angle periodicity and the interchange of width and height, resulting in more stable training. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the process of the present invention;

[0032] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0033] Figure 3 This is a schematic diagram of the encoder with the VIT structure of the present invention;

[0034] Figure 4 This is a schematic diagram of the encoder in the Swing Transformer structure of the present invention. Detailed Implementation

[0035] The present invention will be further described below with reference to the accompanying drawings. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0036] like Figure 1 As shown, the present invention is a method for directional target detection, the method comprising the following steps:

[0037] S1 is the input target image;

[0038] S2 transforms images into sequence data, i.e., image tokens, in the Stem network, performs sparse sampling on the image tokens, and sends the sampled sparse tokens into the backbone network.

[0039] S3 extracts abstract features of image information from the backbone network;

[0040] S4 features output from the backbone network are used to determine the category and predict bounding boxes in the Head network;

[0041] S5 outputs the detection results.

[0042] It should be noted that each patch obtains a sequence data token by concatenating its internal original pixels, and the patch size is P×P (in this invention, P is set to 4); secondly, the dimension is mapped to C through a linear embedding layer, resulting in the 3D image being transformed into 1D sequence data X. T ∈R N×C N = HW / P 2 This indicates the number of tokens; finally, the obtained full tokens are sent to the decoder after position encoding, and all tokens are uniformly and randomly sampled. Unsampled tokens are discarded directly, and the sampled sparse tokens are sent to the encoder after position encoding. The sampling rate is set to r.

[0043] It should be noted that the patch is derived from image segmentation, that is, by segmenting the RGB image X∈R. H×W×3 Divide into non-overlapping patches

[0044] It should be noted that the backbone network consists of encoders and decoders.

[0045] It should be noted that the encoder includes a standard VIT network or a hierarchical Swing Transformer; wherein:

[0046] In the VIT network, the sampled sparse tokens are fed into a series of Transformer blocks, and a pooling operation is performed at the end of the encoder to further aggregate the number of tokens and adjust the dimension to 256.

[0047] Using the aforementioned hierarchical structure of the Swin Transformer, the Token data is first transformed into... The image information features are extracted by performing self-attention calculation within a window using the Swing Transformer network, and then compressed to 256 channels using a linear layer. The final output feature X is... output The number of tokens in the input is reduced to X. input 1 / 64 of.

[0048] It should be noted that the input to the decoder comes partly from the output of the encoder and partly from all the tokens in the entire image converted into sequence data, and then positionally encoded.

[0049] It should be noted that in step S4, the Head network includes two feedforward neural networks that predict the bounding box and category of the target instance based on the features output by the backbone network; these can be represented as follows:

[0050] b=sigmoid(FFN(f)), c=FFN(f);

[0051] In the above formula, f is the extracted feature, and b is a five-dimensional vector [b cx b cy b w b h b θ The first four dimensions represent the center point coordinates, length, and width of the bounding box, respectively, and the last dimension represents the angle of the bounding box; the sigmoid function is used to process the predicted vector into the interval [0,1], which represents the position of the image relative to the length and width of the image, and c represents the predicted class.

[0052] Example

[0053] In the Stem network, this invention first follows the same operation as the VIT network, converting the RGB image X∈R... H×W×3 Divide into non-overlapping patches Each patch generates a sequence data token by concatenating its internal raw pixels. The patch size is P×P (in this invention, P is set to 4). Next, a linear embedding layer maps the dimensions to C, transforming the 3D image into 1D sequence data X. T ∈R N×C N = HW / P 2 This represents the number of tokens. The process is divided into two steps: first, all obtained tokens are position-encoded and fed into the decoder; second, all tokens are uniformly and randomly sampled, with unsampled tokens discarded. The sampled sparse tokens are then position-encoded and fed into the encoder, with the sampling rate set to r.

[0054] encoder

[0055] The encoder in this invention is a backbone network that encodes the input sparse tokens to obtain abstract features. There are two different options, including the standard VIT network or the hierarchical Swing Transformer.

[0056] 1. VIT network

[0057] like Figure 3As shown, a standard VIT network is used. The sampled sparse tokens are fed into a series of Transformer blocks, and a pooling operation is performed at the end of the encoder to further aggregate the token count and adjust the dimension to 256.

[0058] 2. Swing Transformer Network

[0059] like Figure 4 As shown, the Swin Transformer employs a hierarchical structure. First, the Token data is transformed using a Reshape operation. The image information features are extracted by performing self-attention calculation within a window using the Swing Transformer network, and then compressed to 256 channels using a linear layer. The final output feature X is... output The number of tokens in the input is reduced to X. input 1 / 64 of.

[0060] In a standard Transformer encoder-decoder architecture, the encoder's output is typically provided as both a key and a value to the decoder as input. However, in this invention, the encoder's output is used as a query value, guiding the network to query the key value constructed from the decoder and extract target features layer by layer—a process known as feature alignment. This invention envisions an inductive bias, allowing the encoder to efficiently learn visual features from sparse tokens obtained through sampling, encoding semantic concepts into these visual tokens. These few tokens are sufficient to represent all visual features, eliminating the need to model all concepts in the image. In the current field of computer vision, the "less is more" principle has been proven in numerous studies, significantly saving memory and computation, which is particularly important for self-attention models where computational complexity increases quadratically. Therefore, although the encoder uses a heavily loaded VIT or Swing Transformer as its backbone network, the computational overhead is not significantly increased due to the sparse data input.

[0061] object Query

[0062] Similarly, this invention sets up a learnable tensor, Object Query, as the location query encoder. Unlike the original DETR network, this invention uses the encoder output Query as the content information query and the Object Query as the spatial location information query. The two are concatenated, and the content information and spatial information are matched separately in the decoder. This can be described as follows:

[0063]

[0064] The first item is content similarity, and the second item is spatial similarity.

[0065] decoder

[0066] This invention adopts the basic architecture of the decoder in DETR. The decoder's input comes from two parts: the encoder's output and all tokens from the entire image converted into sequential data, with positional encoding performed. This differs from the original DETR decoder, where the input is a learnable tensor object query. Although the input here is all data tokens, the decoder is lightweight. Experimental results show that a decoder composed of 5 Transformer Blocks can achieve good results. Operating in the decoder is similar to the human visual system: first, lock onto the approximate region of interest, then gradually focus until the target is clearly seen.

[0067] Head Network

[0068] The Head network consists of two feedforward neural networks that predict the bounding box and category of the target instance based on the features extracted by the Query from the decoder. They can be represented as follows:

[0069] b = sigmoid(FFN(f)) (2)

[0070] c = FFN(f) (3)

[0071] f is the extracted feature, b is a five-dimensional vector [b cx b cy b w b h b θ The first four dimensions represent the coordinates of the center point of the bounding box, as well as its length and width, respectively, and the last dimension represents the angle of the bounding box. The sigmoid function is used to process the predicted vector into the interval [0,1], representing the position of this image relative to the image's length and width. c represents the predicted class.

[0072] Loss function design

[0073] This invention employs a Hungarian loss function similar to that in DETR, namely:

[0074]

[0075] L is the index from the true value i to the predicted value obtained by the Hungarian bilateral matching method. box Specifically

[0076]

[0077] Since the detection results of this invention are represented by rotated boxes with an angle, and the inherent periodicity of the angle represented by the rotated box, as well as sudden changes in the related width and height, can lead to abrupt changes in loss, causing training instability and performance degradation, some improvements have been made to the original design. The calculation of the difference between the predicted box and the ground truth box has been improved as follows:

[0078]

[0079] in This indicates the loss at the center point.

[0080] This makes the loss function continuous, which can eliminate the periodicity of the angle and the abrupt loss caused by the exchange of width and height.

[0081] Based on this, the innovative aspects of this invention are as follows:

[0082] 1. This invention directly uses a Transformer-based backbone network as the encoder to perform image feature encoding and embedding. The decoder adopts the same structure as the decoder in DETR, thus forming an asymmetric model with structural decoupling between the encoder and decoder.

[0083] 2. The input of the encoder and decoder is different from that of the original Transformer. The input of the encoder in this invention is the image token after sparse sampling, and the input of the decoder is the full token after transformation. Therefore, the input information is asymmetric.

[0084] 3. In this invention, the encoder output is used as the Query value as guidance information to find matching target features, while in the original Transformer structure, the encoder output is used as the Value and Key values.

[0085] 4. This invention uses the encoder output as the content information query and the learnable tensor object query as the spatial information query. The two are concatted and the encoded information is queried independently. Compared with the original design, the training converges faster and the training cycle can be reduced.

[0086] 5. The encoder trained with sparse image information after sampling can extract key image features more efficiently, further improving the detection effect.

[0087] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.

Claims

1. A method for detecting directional targets, characterized in that, The method includes the following steps: S1 Input target image; S2 transforms the image into sequence data, i.e., image tokens, in the Stem network, performs sparse sampling on the image tokens, and sends the sampled sparse tokens into the backbone network. S3 extracts abstract features of image information from the backbone network; S4 features output from the backbone network are used to determine the category and predict bounding boxes in the Head network; S5 outputs the detection results; In step S2, a sequence data token is obtained by concatenating the original pixels within each patch. That is, each patch is mapped to dimension C through a linear embedding layer, thus converting the three-dimensional image into one-dimensional sequence data. ,in, This indicates the number of tokens; the first step is to encode all the tokens and send them to the decoder; the second step is to uniformly and randomly sample all the tokens, discarding any unsampled tokens, and then encode the sampled sparse tokens and send them to the encoder. The backbone network consists of encoders and decoders; a portion of the decoder's input comes from the encoder's output. In step S2, the patch is derived from image segmentation, that is, by segmenting the RGB image. Divide into multiple non-overlapping patches The patch size is P×P.

2. The directional target detection method according to claim 1, characterized in that, The encoder includes a standard VIT network or a hierarchical Swin Transformer; wherein: In the VIT network, the sampled sparse tokens are fed into a series of Transformer blocks, and a pooling operation is performed at the end of the encoder to further aggregate the number of tokens and adjust the dimension to 256. Using the aforementioned hierarchical structure, the Swing Transformer first transforms the Token data into... The image information features are extracted by performing self-attention calculation within a window using the Swing Transformer network, and then compressed to 256 channels using a linear layer; the final output features are then processed. The number of tokens in the input is reduced. 1 / 64 of.

3. The directional target detection method according to claim 1, characterized in that, In step S4, the Head network includes two feedforward neural networks that predict the bounding box and category of the target instance based on the features output by the backbone network; these can be represented as follows: 、 ； In the above formula, f is the extracted feature, and b is a five-dimensional vector. The first four dimensions represent the center point coordinates, length, and width of the bounding box, respectively, and the last dimension represents the angle of the bounding box; the sigmoid function is used to process the predicted vector into the interval [0,1], which represents the position of the image relative to the length and width of the image, and c represents the predicted class.

Images