Transformer Target Detection Method and Apparatus Incorporating Early Detectors

Abstract

This invention discloses a Transformer object detection method incorporating an early detector, comprising: inputting an image into a visual Transformer to extract semantically rich feature maps; applying an early detector to the feature maps to output coarsely estimated targets; extracting the top N estimated targets by confidence, using the four coordinates of the target's detection box as reference points for the decoder in the detection Transformer, and selecting feature points on the feature maps predicting these selected estimated targets as target identifiers for the decoder in the detection Transformer; wherein N is a preset value; the decoder of the detection Transformer uses the reference points and target identifiers provided in step three to interact with the feature maps extracted in step one using a cross-attention mechanism, thereby continuously refining the coordinates of the target identifiers and reference points, and finally outputting accurate target detection boxes and target categories. This invention also provides a corresponding Transformer object detection device incorporating an early detector.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and computer vision technology, and more specifically, relates to a Transformer object detection method and apparatus that incorporates an early detector. Background Technology

[0002] Object detection is one of the fundamental tasks in computer vision, requiring the localization and classification of objects in images. Achieving this often necessitates complex designs, such as anchor boxes and post-processing. The Detection Transformer addresses this issue by introducing object identifiers to represent objects and incorporating bipartite graph matching in sample allocation, eliminating the need for manually designed anchor boxes and post-processing. The Detection Transformer is concise and elegant, and boasts good performance. However, its speed, accuracy, and parameter count still lag significantly behind industry-leading traditional detection models based on convolutional neural networks. Summary of the Invention

[0003] This invention addresses the problem of traditional Transformer detection methods lacking good initialization and features by proposing a target detection scheme based on the Transformer detection framework. This scheme aims to make the speed, accuracy, and parameter count of Transformer detection approach those of industry-leading traditional detection models based on convolutional neural networks.

[0004] This invention presents a detection Transformer framework based on early detectors, which provides the detection Transformer decoder with semantically rich, spatially aligned deep feature maps and target identifiers associated with the input. This framework further integrates industry-leading visual Transformers as the backbone network, demonstrating state-of-the-art performance across various model scales.

[0005] To achieve the above objectives, according to one aspect of the present invention, a Transformer target detection method incorporating an early detector is provided, comprising the following steps:

[0006] Step 1: Input the image into a visual Transformer to extract semantically rich feature maps;

[0007] Step 2: Use an early detector on this feature map to output a coarse estimate of the target;

[0008] Step 3: Extract the top N estimated targets with the highest confidence scores, use the four coordinates of the detection box of the target as reference points for the decoder in the detection Transformer, and select the feature points on the feature maps of these selected estimated targets as target identifiers for the decoder in the detection Transformer; N is a preset value.

[0009] Step 4: The Transformer decoder uses the reference points and target identifiers provided in Step 3 to interact with the feature maps extracted in Step 1 using a cross-attention mechanism, thereby continuously refining the coordinates of the target identifiers and reference points, and finally outputting accurate target detection boxes and target categories.

[0010] In one embodiment of the present invention, the Transformer detection framework includes an early detector and a detection Transformer, wherein the early detector consists of a visual Transformer and a detection head; and the detection Transformer is a stack of two lightweight decoders.

[0011] In one embodiment of the present invention, the method further includes: employing local bipartite graph matching to stabilize sample allocation, and an auxiliary loss from attention map to feature map to further optimize the representation of the feature map to optimize the early detector.

[0012] In one embodiment of the present invention, a constant loss is added to the loss matrix, and positive and negative samples are assigned to the loss matrices of the predicted samples and the actual target, so that the actual target will only be assigned positive samples to the predicted samples near its center region.

[0013] In one embodiment of the present invention, a constant loss is added to the loss matrix, specifically:

[0014] The predicted target is obtained by combining the predicted target with the distance loss and class loss of the actual target, based on the output of the early detector.

[0015] Based on the central region range of the actual target, a large constant loss is added to the loss matrix for the samples predicted outside the range, and the constant loss is greater than a preset threshold.

[0016] The loss matrix obtained in the previous step is used to perform bipartite graph matching to obtain the positive samples corresponding to the actual target.

[0017] In one embodiment of the present invention, rich instance visual signals in the cross-attention map of the decoder in the Transformer are used to supervise the features extracted by the visual Transformer.

[0018] In one embodiment of the present invention, rich instance visual signals in the cross-attention map of the decoder in the Transformer are used to supervise the features extracted by the visual Transformer, specifically including:

[0019] For each layer of the Transformer decoder, aggregate all target identifiers and feature maps into a cross-attention map;

[0020] The cross-attention maps obtained from each layer are superimposed to obtain the final cross-attention map;

[0021] A linear layer is added to the early detector to predict the cross-attention map obtained in the previous step. Between the predicted mask and the binarized cross-attention map, a loss function is used to calculate the loss to assist in the training of the model.

[0022] In one embodiment of the present invention, the loss function is the binary cross-entropy (BCE) loss function.

[0023] In one embodiment of the present invention, the value of N is 300.

[0024] According to another aspect of the present invention, a Transformer target detection apparatus incorporating an early detector is also provided, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions executable by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the above-described Transformer target detection method incorporating an early detector.

[0025] In summary, compared with existing technologies, the technical solutions conceived in this invention have the following beneficial effects: Addressing the problem of traditional detection Transformer methods lacking good initialization and features, this invention proposes a target detection scheme based on a detection Transformer framework. This detection Transformer framework, based on an early detector, can provide the detection Transformer decoder with semantically rich, spatially aligned deep feature maps and target identifiers associated with the input. Furthermore, this framework integrates the industry-leading visual Transformer as the backbone network, demonstrating a leading level of performance across various model scales. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a Transformer target detection method incorporating an early detector, provided by the present invention.

[0027] Figure 2 This is a schematic diagram of a full Transformer-based detection framework provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0029] like Figure 1 As shown, this invention provides a Transformer target detection method incorporating an early detector, comprising:

[0030] Step 1: Input the image into a visual Transformer to extract semantically rich feature maps;

[0031] Step 2: Use an early detector on this feature map to output a coarse estimate of the target;

[0032] Step 3: Extract the top N estimated targets with the highest confidence scores, use the four coordinates of the detection box of the target as reference points for the decoder in the detection Transformer, and select the feature points on the feature maps of these selected estimated targets as target identifiers for the decoder in the detection Transformer; N is a preset value.

[0033] Step 4: The Transformer decoder uses the reference points and target identifiers provided in Step 3 to interact with the feature maps extracted in Step 1 using a cross-attention mechanism, thereby continuously refining the coordinates of the target identifiers and reference points, and finally outputting accurate target detection boxes and target categories.

[0034] Furthermore, within the aforementioned framework, this invention proposes local bipartite graph matching to stabilize sample allocation and an auxiliary loss from attention map to feature map to further optimize feature map representation and improve early detectors.

[0035] During training, positive and negative samples are assigned based on the loss matrices of predicted samples and actual targets. Traditional methods directly use bipartite graph matching, which causes actual targets to assign corresponding positive samples to predicted samples that are far away from them, thus impairing model optimization. To address this problem, this invention adds a constant loss to the loss matrix, ensuring that actual targets only assign positive samples to predicted samples near their center region. The specific steps are as follows:

[0036] Step 1: Based on the output of the early detector, the predicted target is used as the prediction sample, and the loss matrix is ​​obtained by combining it with the distance loss and class loss of the actual target;

[0037] Step 2: Based on the actual target's central region range, add a large constant loss to the loss matrix for the samples predicted outside the range, where the constant loss is greater than a preset threshold.

[0038] Step 3: Perform bipartite graph matching on the loss matrix obtained in the previous step to obtain the positive samples corresponding to the actual target.

[0039] During training, the features output by the backbone network of the visual Transformer do not consider the feature regions that the target identifier in the decoder of the detection Transformer focuses on through the cross-attention mechanism. This results in a misalignment between the feature responses output by the visual Transformer and the feature responses that the detection Transformer truly focuses on. To address this issue, this invention proposes an auxiliary loss from attention map to feature map. The features extracted by the visual Transformer are supervised by using rich instance visual signals from the cross-attention map of the decoder in the detection Transformer. The specific steps are as follows:

[0040] Step 1: For each layer of the Transformer decoder, aggregate all cross-attention maps between target identifiers and feature maps;

[0041] Step 2: Stack the cross-attention maps obtained from each layer to obtain the final cross-attention map.

[0042] Step 3: Add a linear layer to the early detector to predict the cross-attention map obtained in Step 2. Between the predicted mask and the binarized cross-attention map, calculate a loss function to aid model training; for example, use the Binary Cross Entropy (BCE) loss function.

[0043] like Figure 2 The diagram shows the detection framework based on the full Transformer proposed in this invention. This framework mainly consists of two parts: an early detector and a detection Transformer. The early detector comprises a visual Transformer and a small detection head; the detection Transformer is a lightweight decoder consisting of two stacked layers. To further improve the framework's performance, this invention also introduces local bipartite graph matching for positive and negative sample allocation and attention map-to-feature map auxiliary loss to further refine the feature maps output by the early detector during training. Here, H and W represent the size of the feature maps, C represents the number of classes, K represents the number of actual targets, and N represents the number of target identifiers introduced in this invention; in this framework, N = 300 by default.

[0044] The detection framework based on the full Transformer architecture of this invention outperforms previous methods in speed, accuracy, and parameter count, achieving industry-leading performance. On the COCO validation set, the small model achieves an accuracy of 43.4 mAP with 12M parameters and a speed of 27.2 FPS; the large model achieves an accuracy of 50.2 mAP with 94M parameters and a speed of 10 FPS. It maintains its advantage over previous detection models across all model scales. The detector of this invention achieves an optimal balance between speed and accuracy, as well as between accuracy and parameter count. It can run on devices with smaller models, faster speed, and higher accuracy.

[0045] The following specific embodiment illustrates the Transformer target detection method of the present invention that incorporates an early detector, including:

[0046] Step 1: Input the 800x1300 image into the visual Transformer Swin Transformer to extract the semantically rich feature map HxWxC.

[0047]

[0048] Step 2: On the feature map of HxWxC, use an early detector composed of linear layers to output dense detection results in a sliding window manner as a rough estimate of the target.

[0049]

[0050] Step 3: Extract the top 300 estimated targets with the highest confidence scores, use the four coordinates of the detection box of the target as the reference points of the decoder in the detection Transformer, and select the feature points on the feature map of the selected estimated targets as the target identifiers of the decoder in the detection Transformer.

[0051] topN_indexes = TopN(obj_cls),

[0052]

[0053] Step 4: The Transformer decoder uses the reference points and target identifiers provided in Step 3 to interact with the feature maps extracted in Step 1 using a cross-attention mechanism, thereby continuously refining the coordinates of the target identifiers and reference points, and finally outputting accurate target detection boxes and target categories.

[0054]

[0055] Furthermore, the present invention also provides a Transformer target detection device incorporating an early detector, comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions executable by the at least one processor, wherein the instructions, after being executed by the processor, are used to complete the aforementioned Transformer target detection method incorporating an early detector.

[0056] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A Transformer target detection method incorporating an early detector, characterized in that, Includes the following steps: The Transformer detection framework consists of an early detector and a detection Transformer. The early detector is composed of a visual Transformer and a detection head. The detection Transformer is a stack of two lightweight decoders. The image is input into a visual Transformer to extract semantically rich feature maps; Using an early detector on this feature map, a coarse estimate of the target is output; Extract the top N estimated targets by confidence, use the four coordinates of the detection box of each target as reference points for the decoder in the detection Transformer, and select the feature points on the feature maps of these selected estimated targets as target identifiers for the decoder in the detection Transformer; N is a preset value; use the rich instance visual signals in the cross-attention map of the decoder in the detection Transformer to supervise the features extracted by the visual Transformer; specifically, this includes: for each layer of the detection Transformer decoder, aggregating all the cross-attention maps of target identifiers and feature maps; superimposing the cross-attention maps obtained from each layer to obtain the final cross-attention map; adding a linear layer in the early detector to predict the cross-attention map obtained in the previous step, and calculating the loss between the predicted mask and the binarized cross-attention map using a loss function to assist in model training; The decoder that detects the Transformer uses the reference point and the target identifier to interact with the feature map using a cross-attention mechanism, thereby continuously refining the coordinates of the target identifier and the reference point, and finally outputting an accurate target detection box and target category.

2. The Transformer target detection method incorporating an early detector as described in claim 1, characterized in that, Also includes: Local bipartite graph matching is used to stabilize sample allocation, and an auxiliary loss from attention map to feature map is used to further optimize the representation of feature map to improve the early detector.

3. The Transformer target detection method incorporating an early detector as described in claim 1, characterized in that, Add a constant loss to the loss matrix and assign positive and negative samples to the loss matrices of the predicted samples and the actual target, so that the actual target will only be assigned positive samples to the predicted samples near its center region.

4. The Transformer target detection method incorporating an early detector as described in claim 3, characterized in that, Add a constant loss to the loss matrix, specifically: The estimated target output by the early detector is used as the prediction sample, and the loss matrix is ​​obtained by combining it with the distance loss and class loss of the actual target. Based on the central area range of the actual target, a constant loss is added to the loss matrix for the samples predicted outside the range, and the constant loss is greater than a preset threshold. The loss matrix obtained in the previous step is used to perform bipartite graph matching to obtain the positive samples corresponding to the actual target.

5. The Transformer target detection method incorporating an early detector as described in claim 1, characterized in that, The loss function is the binary cross-entropy (BCE) loss function.

6. The Transformer target detection method incorporating an early detector as described in claim 1, characterized in that, The value of N is 300.

7. A Transformer target detection device incorporating an early detector, characterized in that: It includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the Transformer target detection method with an early detector as described in any one of claims 1-6.

Images