Target detection method based on lightweight Transform

Abstract

The invention relates to a target detection method based on a lightweight Transform, and the method comprises the following steps: extracting features in an input picture through employing an improved OfficientViT backbone network based on a multi-scale linear attention mechanism and a lightweight convolution structure, and obtaining a plurality of feature maps with different spatial scales; according to a size relationship between a preset target spatial scale and the spatial scales corresponding to the feature maps, unifying the spatial scales of the plurality of feature maps into a target spatial scale through up-sampling, down-sampling and direct transmission, and obtaining a fusion feature through a Hadamard product; and obtaining prediction frame information of the target based on the fused features, calculating a loss function value considering a central point distance between a prediction frame and a real frame and an aspect ratio difference, and realizing training of a lightweight Transform model. Compared with the prior art, target detection in a complex scene is realized, and particularly, the precision and the real-time performance of small target detection are improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a target detection method based on a lightweight Transformer. Background Art

[0002] In recent years, object detection technology has made significant progress driven by deep learning. Although traditional object detection methods such as Faster-RCNN and YOLO can effectively improve detection accuracy and speed, they are not good enough when dealing with complex backgrounds, severe occlusions, or large differences in target shapes. In addition, with the emergence of the Transformer model, detection methods based on the self-attention mechanism such as DETR have performed well in global feature extraction and multi-scale object detection. However, the Transformer architecture has the problems of high computational complexity and slow inference speed, which limits its application in real-time detection tasks.

[0003] Therefore, developing a lightweight, efficient and accurate Transformer target detection method has become a research hotspot in the current field. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a target detection method based on lightweight Transformer to achieve accurate real-time detection of small targets in complex scenes.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] In one aspect of the present invention, a method for object detection based on a lightweight Transformer is provided, wherein a picture to be detected is used as an input of a trained lightweight Transformer model to obtain prediction box information of the object and implement object detection, wherein the training process of the lightweight Transformer model comprises the following steps:

[0007] Step S1, using the improved EfficientViT backbone network based on multi-scale linear attention mechanism and lightweight convolution structure to extract features from the input image and obtain feature maps of multiple spatial scales;

[0008] Step S2, according to the preset target spatial scale and the size relationship between the spatial scales corresponding to the feature maps, the spatial scales of multiple feature maps are unified into the target spatial scale through upsampling, downsampling and direct transfer, and the fusion feature is obtained through the Hadamard product;

[0009] Step S3, obtaining the predicted box information of the target based on the fusion features, calculating the loss function value considering the center point distance between the predicted box and the real box and the aspect ratio difference, and realizing the training of the lightweight Transformer model.

[0010] As a preferred technical solution, in step S1, the backbone network is constructed based on the EfficientViT architecture, including four stages of gradually reducing the spatial size of the feature map. In the second and third stages, EfficientViTBlock is used to dynamically aggregate global context information to achieve modeling of long-distance dependencies between different positions.

[0011] As a preferred technical solution, in the backbone network, the fourth stage uses EfficientViT Block to replace the traditional mobile inverted bottleneck convolution (MBConv), reducing the amount of computation by reducing convolution operations.

[0012] As a preferred technical solution, in step S2, when the spatial scale of the feature map is smaller than the target scale, the spatial scale of the feature map is matched with the target scale by performing an upsampling operation on the feature map; when the spatial scale of the feature map is larger than the target scale, the spatial scale of the feature map is matched with the target scale by performing a pooling operation on the feature map; when the spatial scale of the feature map is consistent with the target scale, the feature map is transferred.

[0013] As a preferred technical solution, in the step S2, after unifying the spatial scales of the multiple feature maps into the target spatial scale, the step further includes:

[0014] A lightweight convolution operation is performed on each feature map.

[0015] As a preferred technical solution, the lightweight convolution operation is GSConv.

[0016] As a preferred technical solution, in step S3, the loss function value is calculated based on the intersection-over-union ratio of the predicted box and the real box, the center point distance between the predicted box and the real box, and the aspect ratio difference between the predicted box and the real box.

[0017] As a preferred technical solution, in step S3, the loss function is:

[0018]

[0019] Among them, EIoU loss is the loss function value, IoU represents the intersection-over-union ratio of the predicted box and the real box, d(center) is the Euclidean distance between the center points of the predicted box and the real box, c is the diagonal length of the minimum circumscribed rectangle surrounding the predicted box and the real box, which is used to normalize the center point distance, λ is the weighting factor of the aspect ratio difference, which is used to balance the impact of width and height terms on the loss, and w pred and h pred are the width and height of the prediction box, w gt and h gt are the width and height of the real box respectively.

[0020] Another aspect of the present invention provides an electronic device, comprising: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the aforementioned lightweight Transformer-based target detection method.

[0021] Another aspect of the present invention provides a computer-readable storage medium, comprising one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the aforementioned lightweight Transformer-based target detection method.

[0022] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0023] (1) The backbone network has strong feature representation capabilities and low computational burden: The present invention uses an improved EfficientViT backbone network based on a multi-scale linear attention mechanism and a lightweight convolutional structure to extract features from the input image and obtain feature maps of multiple spatial scales. By introducing the EfficientViT Block to dynamically aggregate global context information, effectively modeling the long-distance dependencies between different positions, and introducing a lightweight convolutional structure to reduce the computational burden, the entire model is more efficient, while reducing the amount of computation and model parameters while maintaining a strong feature expression capability, which is particularly suitable for tasks that require real-time detection.

[0024] (2) Achieve efficient and nearly lossless feature fusion: According to the relationship between the scale of the input feature map and the target scale, the present invention uses upsampling, downsampling and direct transfer methods to achieve feature map alignment. The Hadamard product is used to flexibly model the relationship between different feature maps and effectively maintain the local information of the input features, achieving more efficient and lossless feature fusion.

[0025] (3) Ensure that the predicted box is more accurately aligned with the target box: For complex scenes and multi-scale target detection, the present invention further considers the center point distance and aspect ratio difference between the predicted box and the real box when calculating the loss function, so that the model can more accurately optimize the position and shape of the box outside the overlapping area. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a training flowchart of the lightweight Transformer model in the embodiment;

[0027] Figure 2 This is a schematic diagram of the structure of a lightweight Transformer model in the embodiment;

[0028] Figure 3 Schematic diagram of the EfficientViT backbone network in the embodiment;

[0029] Figure 4 Schematic diagram of a cascade detail fusion module in an embodiment. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] Example 1

[0032] In view of the problems existing in the above-mentioned prior art, this embodiment provides a target detection method based on a lightweight Transformer, which includes a lightweight Transformer model. The model combines an improved EfficientViT backbone network, a cascade detail fusion module (Cascade Detail Infusion, CDI) and a loss function based on EIoU (Extended IoU) to achieve target detection in complex scenes, especially to improve the accuracy and real-time performance of small target detection, such as Figure 2 This method achieves high-precision and low-cost target detection by reducing computational complexity, improving feature extraction capability, and optimizing loss function design, which is particularly suitable for real-time scenarios.

[0033] like Figure 1 As shown in the figure, the training process of the lightweight Transformer model includes the following steps:

[0034] Step S1, using a backbone network based on a multi-scale linear attention mechanism and a lightweight convolutional structure to extract features from the input image and obtain feature maps of multiple spatial scales.

[0035] This step uses the improved EfficientViT as the backbone network of the detection model, such as Figure 3 As shown in the figure. EfficientViT is the backbone network of the model. It significantly reduces the computational cost while retaining strong feature extraction capabilities by combining the self-Attention mechanism and the convolutional network. The EfficientViT architecture is divided into four stages, each of which gradually reduces the size of the feature map while increasing the number of channels. Compared with traditional convolutional neural networks (CNNs), EfficientViT further enhances feature representation and reduces redundant calculations of convolution operations by introducing deep convolution and feedforward neural networks.

[0036] Specifically, the EfficientViT Block is introduced in the second and third stages, which can dynamically aggregate global context information and effectively model long-distance dependencies between different positions. In the fourth stage, the traditional MBConv is replaced by the EfficientViT Block, which reduces the computational burden by removing the convolution operation, making the entire model more efficient. The EfficientViT Block includes a lightweight multi-scale linear attention (MSA) mechanism, a depthwise convolution (DWConv), and a feedforward neural network (FNN). The MSA module uses the self-attention mechanism of the Transformer architecture to effectively model the long-term dependencies between different positions in the feature map. By calculating the attention weights between positions, MSA dynamically aggregates global context information, allowing the model to better understand the overall structure and semantics of the image. The combination of FNN and DWConv is responsible for capturing local details. FNN enhances feature representation through point-by-point nonlinear transformations, while deep separable convolution reduces the number of parameters and computational load, making the model both lightweight and efficient.

[0037] Step S2, according to the preset target spatial scale and the size relationship between the spatial scales corresponding to the feature maps, the spatial scales of multiple feature maps are unified into the target spatial scale through upsampling, downsampling and direct transfer, and the fusion feature is obtained through the Hadamard product.

[0038] In order to solve the computational cost and information loss problems in multi-scale feature fusion, a cascade detail fusion module (CDI) is designed in this step. Figure 4 As shown in Figure 2. Traditional feature fusion methods, such as upsampling or downsampling operations, can align the scale of feature maps, but introduce a lot of computational overhead and information loss. The CDI module provided in this step solves these problems and achieves more efficient and lossless feature fusion.

[0039] The core idea of ​​the CDI module is to use different operations (upsampling, downsampling, and direct transfer) to process feature maps of different scales and align them to the target scale. The specific operations are as follows:

[0040] Upsampling: When the scale of the input feature map is smaller than the target scale, a convolution upsampling operation is applied to match the target scale;

[0041] Downsampling: When the scale of the input feature map is larger than the target scale, a pooling operation is used to reduce the resolution of the feature map;

[0042] Direct transfer: When the scale of the input feature map is consistent with the target scale, the feature map is directly transferred without adjustment.

[0043] The aligned feature maps are further enhanced through lightweight convolution operations (such as GSConv). Finally, the feature maps are fused through the Hadamard Product. Compared with traditional concatenation or addition operations, the Hadamard Product can more flexibly model the relationship between different feature maps and effectively maintain the local information of the input features. The Hadamard Product efficiently fuses multiple feature maps by element-by-element multiplication, improving the model's ability to detect small targets.

[0044] This modular design not only reduces unnecessary computational overhead, but also maintains the integrity of multi-scale features, thereby improving the model's target detection accuracy in complex scenarios.

[0045] In step S3, the predicted box information of the target is obtained based on the fusion features, and the loss function value considering the center point distance between the predicted box and the real box and the aspect ratio difference is calculated to realize the training of the lightweight Transformer model.

[0046] This step introduces an EIoU (Extended IoU)-based loss function to optimize the bounding box regression problem in the object detection task. The traditional IoU loss function only focuses on the overlap between the predicted box and the true box, while EIoU further considers the center point distance and aspect ratio difference between the predicted box and the true box, allowing the model to more accurately optimize the position and shape of the box outside the overlapping area.

[0047] The EIoU loss function is defined as follows:

[0048]

[0049] Among them, IoU represents the intersection over union of the predicted box and the true box, which measures the ratio of the overlapping area of ​​the two boxes to the union area. The closer it is to 1, the better the overlap of the two boxes. d(center) is the Euclidean distance between the center points of the predicted box and the true box. c is the diagonal length of the minimum circumscribed rectangle enclosing the predicted box and the true box, which is used to normalize the center point distance. λ is a weighting factor for the difference in aspect ratio, which is used to balance the impact of width and height terms on the loss. w pred and h pred are the width and height of the prediction box respectively; w gt and h gt are the width and height of the real box respectively.

[0050] The EIoU loss function not only optimizes the overlapping area, but also optimizes the center point distance and aspect ratio of the predicted box and the real box, so that the model can align the target box more accurately. Especially in complex scenes and multi-scale object detection, EIoU significantly improves the fitting accuracy of the bounding box.

[0051] To verify the effectiveness of this method, experiments were conducted on a custom target detection dataset. The experimental platform used the PyTorch deep learning framework, and the hardware environment included an NVIDIA RTX 3090 graphics card, a 13th-generation Intel i7-13700KF processor, and 64G memory. The resolution of all input images was adjusted to 640×640, the batch size was set to 24, the optimizer was AdamW, the initial learning rate was 0.01, and the number of training iterations was 200.

[0052] The experiment used a custom object detection dataset containing 5,000 annotated images, which shows objects of different shapes, sizes, and complex backgrounds. The dataset is divided into training set, validation set, and test set in a ratio of 7:2:1. Since there are many small objects in the dataset and the target objects are often similar to the background, the detection model is required to have strong detail capture capabilities.

[0053] In order to evaluate the performance of this method, the proposed model is compared with the existing RT-DETR, Faster-RCNN and SwinTransformer models, as shown in Table 1. The experimental results show that the lightweight Transformer model of this embodiment achieves 91.5% accuracy (mAP@0.5), a frame processing speed of 43 frames per second (FPS), and a computational complexity of only 8.4GFLOPs. Compared with RT-DETR, the proposed model improves the accuracy by 1.9% and reduces the computational complexity by 32.8%.

[0054] Table 1 Comparative experiments of different models

[0055]

[0056] To further verify the effectiveness of each module in this method, we conducted an ablation experiment, as shown in Table 2. The experimental results show that when the improved EfficientViT backbone network is used alone, the average accuracy of the model is 86.7%. After the introduction of the cascaded detail fusion module (CDI), the accuracy is improved to 89.0%, and the computational complexity is significantly reduced. After introducing the EIoU-based loss function in the complete model, the average accuracy of the final model reaches 91.5%, and the computational cost remains at a low level (GFLOPs is 8.4). This shows that each module plays a key role in improving the overall performance of the model.

[0057] Table 2 Ablation experiments of different modules

[0058]

[0059] This method proposes a target detection method based on lightweight Transformer, combined with the improved EfficientViT backbone network, cascade detail fusion module (CDI) and EIoU-based loss function, so that the model can significantly reduce the computational complexity and parameter quantity while maintaining high detection accuracy. Through experimental verification, this method performs well in detecting small targets, complex backgrounds and target objects of different scales, and has strong real-time detection capabilities. It is suitable for application scenarios with high real-time requirements such as drone inspection and autonomous driving.

[0060] Example 2

[0061] This embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the target detection method based on the lightweight Transformer as described in Example 1.

[0062] Example 3

[0063] This embodiment provides a computer-readable storage medium, including one or more programs for execution by one or more processors of an electronic device, wherein the one or more programs include instructions for executing the lightweight Transformer-based target detection method as described in Example 1.

[0064] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A target detection method based on lightweight Transformer, characterized in that: The image to be detected is used as the input of the trained lightweight Transformer model to obtain the predicted box information of the target and realize target detection, wherein the training process of the lightweight Transformer model includes the following steps: step S1, using the improved EfficientViT backbone network based on the multi-scale linear attention mechanism and the lightweight convolution structure to extract the features in the input image and obtain multiple feature maps with different spatial scales; step S2, according to the preset target spatial scale and the size relationship between the spatial scales corresponding to the feature map, the spatial scales of multiple feature maps are unified into the target spatial scale through upsampling, downsampling and direct transfer, and the fusion feature is obtained through the Hadamard product; step S3, based on the fusion feature, the predicted box information of the target is obtained, and the loss function value considering the center point distance between the predicted box and the real box and the aspect ratio difference is calculated to realize the training of the lightweight Transformer model.

2. According to claim 1, a target detection method based on lightweight Transformer is characterized in that: In the step S1, the backbone network is constructed based on the EfficientViT architecture, including four stages of gradually reducing the spatial size of the feature map. In the second and third stages, the EfficientViT Block is used to dynamically aggregate global context information to achieve modeling of long-distance dependencies between different positions.

3. According to claim 2, a target detection method based on lightweight Transformer is characterized in that: In the backbone network, the fourth stage uses EfficientViT Block to replace the traditional mobile flip bottleneck convolution, reducing the amount of computation by reducing convolution operations.

4. The object detection method based on lightweight Transformer according to claim 1, characterized in that: In the step S2, when the spatial scale of the feature map is smaller than the target scale, the spatial scale of the feature map is matched with the target scale by performing an upsampling operation on the feature map; when the spatial scale of the feature map is larger than the target scale, the spatial scale of the feature map is matched with the target scale by performing a pooling operation on the feature map; when the spatial scale of the feature map is consistent with the target scale, the feature map is transferred.

5. The object detection method based on lightweight Transformer according to claim 1, characterized in that: In the step S2, after unifying the spatial scales of the plurality of feature maps into the target spatial scale, the step further includes: A lightweight convolution operation is performed on each feature map.

6. The object detection method based on lightweight Transformer according to claim 5, characterized in that: The lightweight convolution operation is GSConv.

7. The object detection method based on lightweight Transformer according to claim 1, characterized in that: In the step S3, the loss function value is calculated based on the intersection-over-union ratio of the predicted box and the real box, the center point distance between the predicted box and the real box, and the aspect ratio difference between the predicted box and the real box.

8. The object detection method based on lightweight Transformer according to claim 1, characterized in that: In step S3, the loss function is: Among them, EIoU loss is the loss function value, IoU represents the intersection-over-union ratio of the predicted box and the real box, d(center) is the Euclidean distance between the center points of the predicted box and the real box, c is the diagonal length of the minimum circumscribed rectangle surrounding the predicted box and the real box, which is used to normalize the center point distance, λ is the weighting factor of the aspect ratio difference, which is used to balance the impact of width and height terms on the loss, and w pred and h pred are the width and height of the prediction box, w gt and h gt are the width and height of the real box respectively.

9. An electronic device, characterized in that: include: One or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the lightweight Transformer-based target detection method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: It includes one or more programs for execution by one or more processors of an electronic device, and the one or more programs include instructions for executing the lightweight Transformer-based target detection method as described in any one of claims 1-8.

Images