A Lightweight Transformer-Based Object Detection Method

Abstract

This invention relates to a lightweight Transformer-based object detection method, comprising the following steps: Utilizing an improved EfficientViT backbone network based on a multi-scale linear attention mechanism and a lightweight convolutional structure, features are extracted from the input image to obtain multiple feature maps with different spatial scales; based on a preset target spatial scale and the size relationship between the spatial scales of the feature maps, the spatial scales of the multiple feature maps are unified to the target spatial scale through upsampling, downsampling, and direct pass-through; a fused feature is obtained through Hadamard product; based on the fused feature, the predicted bounding box information of the target is obtained; a loss function value considering the distance between the center points of the predicted bounding box and the ground truth bounding box, as well as the aspect ratio difference, is calculated to train the lightweight Transformer model. Compared with existing technologies, this invention enables object detection in complex scenes, particularly improving the accuracy and real-time performance of small object detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to a target detection method based on a lightweight Transformer. Background Technology

[0002] Object detection technology has made significant progress in recent years, driven by deep learning. Traditional object detection methods such as Faster-RCNN and YOLO, while effectively improving detection accuracy and speed, fall short when handling complex backgrounds, severe occlusion, or objects with significant shape variations. Furthermore, with the advent of Transformer models, detection methods based on self-attention mechanisms, such as DETR, have shown excellent performance in global feature extraction and multi-scale object detection. However, the Transformer architecture suffers from high computational complexity and slow inference speed, limiting its application in real-time detection tasks.

[0003] Therefore, developing a lightweight, efficient, and accurate Transformer object detection method has become a research hotspot in the field. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a target detection method based on a lightweight Transformer to achieve accurate real-time detection of small targets in complex scenes.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] One aspect of the present invention provides a target detection method based on a lightweight Transformer, which uses the image to be detected as input to a trained lightweight Transformer model to obtain the predicted bounding box information of the target, thereby achieving target detection. The training process of the lightweight Transformer model includes the following steps:

[0007] Step S1: Using the improved EfficientViT backbone network based on multi-scale linear attention mechanism and lightweight convolutional structure, features in the input image are extracted to obtain feature maps with different spatial scales.

[0008] Step S2: Based on the preset target spatial scale and the size relationship between the spatial scales corresponding to the feature maps, the spatial scales of multiple feature maps are unified to the target spatial scale through upsampling, downsampling and direct transfer, and the fused features are obtained through Hadamard product.

[0009] Step S3: Based on the fusion features, obtain the predicted bounding box information of the target, calculate the loss function value that takes into account the distance between the center point of the predicted bounding box and the ground truth bounding box and the difference in aspect ratio, and realize the training of the lightweight Transformer model.

[0010] As a preferred technical solution, in step S1, the backbone network is constructed based on the EfficientViT architecture, which includes four stages of gradually reducing the size of the feature map space. In the second and third stages, EfficientViTBlock is used to dynamically aggregate global context information to model long-distance dependencies between different locations.

[0011] As a preferred technical solution, in the backbone network, the fourth stage uses EfficientViT Block to replace the traditional Mobile inverted bottleneck convolution (MBConv), thereby reducing the amount of computation by reducing convolution operations.

[0012] As a preferred technical solution, in step S2, when the spatial scale of the feature map is smaller than the target scale, the feature map is upsampled to match the spatial scale of the feature map with the target scale; when the spatial scale of the feature map is larger than the target scale, the feature map is pooled to match the spatial scale of the feature map with the target scale; when the spatial scale of the feature map is consistent with the target scale, the feature map is transferred.

[0013] As a preferred technical solution, step S2, after unifying the spatial scales of multiple feature maps to the target spatial scale, further includes:

[0014] Lightweight convolution operations are performed on each feature map separately.

[0015] As a preferred technical solution, the lightweight convolution operation is GSConv.

[0016] As a preferred technical solution, in step S3, the loss function value is calculated based on the intersection-union ratio of the predicted box and the ground truth box, the distance between the center points of the predicted box and the ground truth box, and the difference in aspect ratio between the predicted box and the ground truth box.

[0017] As a preferred technical solution, in step S3, the loss function is:

[0018]

[0019] Where EIoU loss is the loss function value, IoU represents the intersection-union ratio of the predicted box and the ground truth box, d(center) is the Euclidean distance between the center points of the predicted box and the ground truth box, c is the diagonal length of the smallest bounding rectangle enclosing the predicted box and the ground truth box, used to normalize the center point distance, λ is a weighting factor for aspect ratio differences, used to balance the influence of width and height terms on the loss, and w pred and h pred w represents the width and height of the prediction bounding box, respectively. gt and h gt These represent the width and height of the actual bounding box, respectively.

[0020] In another aspect, an electronic device is provided, comprising: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the aforementioned target detection method based on lightweight Transformer.

[0021] In another aspect, the present invention provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing a target detection method based on a lightweight Transformer as described above.

[0022] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0023] (1) Strong feature representation capability and low computational burden of the backbone network: This invention utilizes an improved EfficientViT backbone network based on a multi-scale linear attention mechanism and a lightweight convolutional structure to extract features from the input image, obtaining multiple feature maps with different spatial scales. By introducing EfficientViT Block to dynamically aggregate global context information, it effectively models long-distance dependencies between different locations. The introduction of a lightweight convolutional structure reduces the computational burden, making the entire model more efficient. While reducing computation and model parameters, it maintains strong feature representation capability, making it particularly suitable for tasks requiring real-time detection.

[0024] (2) Achieving efficient and near-lossless feature fusion: Based on the relationship between the scale of the input feature map and the target scale, this invention uses upsampling, downsampling, and direct transfer methods to align the feature maps. By utilizing the Hadamard product to flexibly model the relationship between different feature maps and effectively preserve the local information of the input features, more efficient and lossless feature fusion is achieved.

[0025] (3) Ensure that the predicted box is more accurately aligned with the target box: For complex scenes and multi-scale target detection, this invention further considers the distance between the center points of the predicted box and the ground box and the difference in aspect ratio when calculating the loss function, so that the model can optimize the position and shape of the box more accurately outside the overlapping area. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the training process of the lightweight Transformer model in this embodiment.

[0027] Figure 2 This is a schematic diagram of the lightweight Transformer model structure in the embodiment;

[0028] Figure 3 This is a schematic diagram of the EfficientViT backbone network in the embodiment;

[0029] Figure 4 This is a schematic diagram of the cascaded detail fusion module in the embodiment. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] Example 1

[0032] To address the problems existing in the aforementioned technologies, this embodiment provides a lightweight Transformer-based object detection method. This method includes a lightweight Transformer model that combines an improved EfficientViT backbone network, a Cascade Detail Infusion (CDI) module, and an EIoU (Extended IoU)-based loss function to achieve object detection in complex scenes, particularly improving the accuracy and real-time performance of small object detection. Figure 2 As shown in the figure, this method achieves high-precision, low-cost target detection by reducing computational complexity, improving feature extraction capabilities, and optimizing loss function design, making it particularly suitable for real-time scenarios.

[0033] like Figure 1 As shown, the training process of a lightweight Transformer model includes the following steps:

[0034] Step S1: Using a backbone network based on a multi-scale linear attention mechanism and a lightweight convolutional structure, features in the input image are extracted to obtain feature maps with different spatial scales.

[0035] This step uses an improved EfficientViT as the backbone network of the detection model, such as... Figure 3 As shown, EfficientViT is the backbone network of the model. Through a combination of self-attention and convolutional networks, it significantly reduces computational costs while retaining powerful feature extraction capabilities. The EfficientViT architecture consists of four stages, each gradually decreasing the feature map size while increasing the number of channels. Compared to traditional convolutional neural networks (CNNs), EfficientViT further enhances feature representation and reduces redundant computation in convolutional operations by introducing deep convolutions and feedforward neural networks.

[0036] Specifically, the EfficientViT Block is introduced in the second and third stages. This method dynamically aggregates global contextual information and effectively models long-distance dependencies between different locations. In the fourth stage, the EfficientViT Block replaces the traditional MBConv. This stage reduces the computational burden by removing convolutional operations, making the entire model more efficient. The EfficientViT Block includes a lightweight multi-scale linear attention (MSA) mechanism, a depthwise convolution (DWConv), and a feedforward neural network (FNN). The MSA module utilizes the self-attention mechanism of the Transformer architecture to effectively simulate long-term dependencies between different locations in the feature map. By calculating the attention weights between locations, MSA dynamically aggregates global contextual information, enabling the model to better understand the overall structure and semantics of the image. The combination of FNN and DWConv is responsible for capturing local details. FNN enhances feature representation through pointwise nonlinear transformations, while depthwise separable convolution reduces the number of parameters and computational load, making the model both lightweight and efficient.

[0037] Step S2: Based on the preset target spatial scale and the size relationship between the spatial scales corresponding to the feature maps, the spatial scales of multiple feature maps are unified to the target spatial scale through upsampling, downsampling and direct transfer, and the fused features are obtained through Hadamard product.

[0038] To address the computational cost and information loss issues in multi-scale feature fusion, this step designs a Cascade Detail Infusion (CDI) module, such as... Figure 4 As shown, traditional feature fusion methods, such as upsampling or downsampling operations, while able to align the scale of feature maps, introduce significant computational overhead and information loss. The CDI module provided in this step addresses these issues, achieving more efficient and lossless feature fusion.

[0039] The core idea of ​​the CDI module is to use different operations (upsampling, downsampling, and direct pass) to process feature maps of different scales and align them to the target scale. The specific operations are as follows:

[0040] Upsampling: When the scale of the input feature map is smaller than the target scale, a convolutional upsampling operation is applied to match the target scale;

[0041] Downsampling: When the scale of the input feature map is larger than the target scale, pooling operations are used to reduce the resolution of the feature map;

[0042] Direct transfer: When the scale of the input feature map is consistent with the target scale, the feature map is directly transferred without adjustment.

[0043] The aligned feature maps are further enhanced with lightweight convolutional operations (such as GSConv) to improve their feature representation. Finally, the feature maps are fused using the Hadamard Product. Compared to traditional concatenation or addition operations, the Hadamard Product can more flexibly model the relationships between different feature maps and effectively preserve the local information of the input features. The Hadamard Product efficiently fuses multiple feature maps through element-wise multiplication, improving the model's ability to detect small objects.

[0044] This modular design reduces unnecessary computational overhead while maintaining the integrity of multi-scale features, thereby improving the model's target detection accuracy in complex scenarios.

[0045] Step S3: Based on the fusion features, obtain the predicted bounding box information of the target, calculate the loss function value that takes into account the distance between the center point of the predicted bounding box and the ground truth bounding box and the difference in aspect ratio, and realize the training of the lightweight Transformer model.

[0046] This step introduces an EIoU (Extended IoU) loss function to optimize the bounding box regression problem in object detection tasks. Traditional IoU loss functions only focus on the overlap between the predicted and ground truth boxes, while EIoU further considers the distance between the center points of the predicted and ground truth boxes and the aspect ratio difference, enabling the model to more accurately optimize the position and shape of the boxes outside the overlapping area.

[0047] The EIoU loss function is defined as follows:

[0048]

[0049] Where IoU represents the Intersection over Union (IoU) ratio between the predicted and ground truth boxes, it measures the ratio of the overlapping area to the union area of ​​the two boxes; the closer it is to 1, the better the overlap between the two boxes; d(center) is the Euclidean distance between the center points of the predicted and ground truth boxes; c is the diagonal length of the smallest bounding rectangle enclosing the predicted and ground truth boxes, used to normalize the center point distance; λ is a weighting factor for aspect ratio differences, used to balance the impact of width and height terms on the loss; w pred and h pred These are the width and height of the prediction box, respectively; w gt and h gt These represent the width and height of the actual bounding box, respectively.

[0050] The EIoU loss function not only optimizes for overlapping regions but also improves the distance between the center points of the predicted and ground truth boxes and their aspect ratios, enabling the model to more accurately align the target boxes. Especially in complex scenes and multi-scale object detection, EIoU significantly improves the fitting accuracy of bounding boxes.

[0051] To verify the effectiveness of this method, experiments were conducted on a custom object detection dataset. The experimental platform used the PyTorch deep learning framework, and the hardware environment included an NVIDIA RTX 3090 graphics card, a 13th generation Intel i7-13700KF processor, and 64GB of RAM. All input images were adjusted to a resolution of 640×640, the batch size was set to 24, the optimizer was AdamW, the initial learning rate was 0.01, and the training iterations were 200.

[0052] The experiment used a custom object detection dataset containing 5000 labeled images, showcasing target objects of varying shapes, sizes, and complex backgrounds. The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. Due to the large number of small targets in the dataset and the fact that target objects often resemble the background, the detection model was required to possess strong detail capture capabilities.

[0053] To evaluate the performance of this method, the proposed model was compared with existing models such as RT-DETR, Faster-RCNN, and SwinTransformer, as shown in Table 1. Experimental results show that the lightweight Transformer model in this embodiment achieves an accuracy (mAP@0.5) of 91.5%, a frame processing speed of 43 frames per second (FPS), and a computational complexity of only 8.4 GFLOPs. Compared to RT-DETR, the proposed model improves accuracy by 1.9% while reducing computational complexity by 32.8%.

[0054] Table 1 Comparison of different models

[0055]

[0056] To further verify the effectiveness of each module in this method, we conducted ablation experiments, as shown in Table 2. Experimental results show that when using the improved EfficientViT backbone network alone, the model's average accuracy is 86.7%. After introducing the Cascaded Fusion of Details (CDI) module, the accuracy improves to 89.0%, and the computational complexity is significantly reduced. After introducing an EIoU-based loss function into the complete model, the final model's average accuracy reaches 91.5%, while maintaining a low computational cost (GFLOPs of 8.4). This indicates that each module plays a crucial role in improving the overall performance of the model.

[0057] Table 2 Ablation experiments of different modules

[0058]

[0059] This paper proposes a lightweight Transformer-based object detection method. By combining an improved EfficientViT backbone network, a cascaded fusion of details (CDI) module, and an EIoU-based loss function, the model significantly reduces computational complexity and the number of parameters while maintaining high detection accuracy. Experimental results demonstrate that this method performs well in detecting small targets, complex backgrounds, and targets of different scales, and possesses strong real-time detection capabilities. It is suitable for applications with high real-time requirements, such as UAV inspection and autonomous driving.

[0060] Example 2

[0061] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the target detection method based on lightweight Transformer as described in Embodiment 1.

[0062] Example 3

[0063] This embodiment provides a computer-readable storage medium including one or more programs executable by one or more processors of an electronic device, the one or more programs including instructions for performing the target detection method based on the lightweight Transformer as described in Embodiment 1.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for object detection based on lightweight Transformer, characterized in that, The trained lightweight Transformer model is input with a picture to be detected to obtain prediction box information of a target, and target detection is realized, wherein the training process of the lightweight Transformer model comprises the following steps: In step S1, an improved EfficientViT backbone network based on a multi-scale linear attention mechanism and a lightweight convolution structure is used to extract features in the input picture to obtain a plurality of feature maps with different spatial scales; In step S2, the spatial scales of the plurality of feature maps are unified to a target spatial scale by upsampling, downsampling and direct transmission according to a preset size relationship between the target spatial scale and the spatial scales corresponding to the feature maps, and a fused feature is obtained by Hadamard product; In step S3, the prediction box information of the target is obtained based on the fused feature, a loss function value considering the center point distance and the width-height ratio difference between the prediction box and the real box is calculated, and the training of the lightweight Transformer model is realized. In step S1, the backbone network is constructed based on the EfficientViT architecture, including four stages of gradually reducing the spatial size of the feature map, and in the second and third stages, the EfficientViT Block is used to dynamically aggregate global context information to model the long-distance dependency relationship between different positions. In the backbone network, the fourth stage replaces the traditional mobile flip bottleneck convolution with the EfficientViT Block to reduce the computational complexity by reducing the convolution operation, In step S2, after the spatial scales of the plurality of feature maps are unified to the target spatial scale, it further comprises: A lightweight convolution operation is performed on each feature map, and the lightweight convolution operation is GSConv. In step S3, the loss function is: wherein, is a loss function value, denotes the intersection over union of the predicted and the ground truth box, is the Euclidean distance between the centers of the predicted and the ground truth box, is the length of the diagonal of the smallest enclosing rectangle surrounding the predicted and the ground truth box, used to normalize the center distance, is a weighting factor for the width-height ratio difference, used to balance the influence of the width and height terms on the loss, and are the width and height of the predicted box, respectively, and are the width and height of the ground truth box, respectively.

2. The lightweight Transformer-based object detection method of claim 1, wherein, In step S2, when the spatial scale of the feature map is smaller than the target scale, the spatial scale of the feature map is matched with the target scale by upsampling the feature map; when the spatial scale of the feature map is greater than the target scale, the spatial scale of the feature map is matched with the target scale by pooling the feature map; and when the spatial scale of the feature map is consistent with the target scale, the feature map is transmitted.

3. The lightweight Transformer-based object detection method of claim 1, wherein, In step S3, the loss function value is calculated based on the intersection over union of the prediction box and the real box, the center point distance between the prediction box and the real box, and the width-height ratio difference between the prediction box and the real box.

4. An electronic device, comprising: It comprises one or more processors and a memory, the memory stores one or more programs, the one or more programs include instructions for executing the lightweight Transformer-based target detection method according to any one of claims 1-3. It comprises one or more programs for one or more processors of an electronic device to execute, the one or more programs include instructions for executing the lightweight Transformer-based target detection method according to any one of claims 1-3.

5. A computer readable storage medium, characterized in that, ​

Images