A lightweight target detection method and system based on context-aware local enhancement

Abstract

The application discloses a lightweight target detection method and system based on context-aware local enhancement, and the method comprises the following steps: (1) constructing a data set for target detection, and dividing the data set into a training set and a validation set; (2) improving a backbone based on YOLOv8, and establishing an improved model; (3) pruning the improved YOLOv8 target detection model; (4) inputting the training set into the improved YOLOv8 network model to obtain the best weight, and finally inputting the validation set into the network to output a detection result; the application adopts a lightweight feature extraction network and a multi-feature fusion mode to greatly reduce the parameter quantity and the calculation cost of the target detection model and improve the detection speed of target detection in the case that the computing capacity of low-end equipment is limited.

Description

A lightweight object detection method and system based on context-aware local enhancement Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a lightweight object detection method and system based on context-aware local enhancement. Background Technology

[0002] In recent years, Vision Transformers (ViTs) have been applied to various vision tasks such as visual recognition, object detection, and semantic segmentation, demonstrating superior performance. However, due to the large number of parameters and high floating-point computations in ViTs architecture models, adapting them for mobile devices leads to a significant performance degradation. Therefore, developing a ViT suitable for mobile devices has become an important research area.

[0003] Object detection aims to identify specific object categories and their precise locations within a given image. Currently, object detection algorithms primarily fall into two categories: first-stage algorithms like YOLO and SSD offer strong real-time performance and fast inference speeds, but their accuracy is insufficient; second-stage algorithms, such as the R-CNN series, offer high accuracy but suffer from slow training speeds and difficulties in real-time deployment. Applying Vitality Scales (ViTs) to the state-of-the-art YOLOv8 object detection visual task, while meeting real-time requirements, leverages the superior performance of ViTs, making it a popular research trend in object detection. However, most methods aim to reduce model parameters or FLOPS, failing to reflect the actual inference throughput of the model and hindering real-time deployment. Furthermore, existing methods often reduce memory access costs by using sparse attention, sacrificing accuracy and limited acceleration. Specifically, methods like EdgeViT and MobileViT simply use raw convolutions to extract local representations, processing only high-frequency local information using globally shared weights within the convolutions. Other methods, such as LVT, first expand the tokens into a window and then use in-window attention to obtain high-frequency information. These methods use only context-aware weights specific to each token for local perception. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a lightweight object detection method and system based on context-aware local enhancement to solve the problems of computational redundancy between attention heads and inefficient model parameter allocation in existing VITs technology.

[0005] Technical solution: The present invention provides a lightweight target detection method based on context-aware local enhancement, which includes the following steps:

[0006] (1) Construct a dataset for object detection and divide the dataset into a training set and a validation set;

[0007] (2) Improve the YOLOv8-based backbone and build an improved model;

[0008] (3) Prune the improved YOLOv8 object detection model;

[0009] (4) Input the training set into the improved YOLOv8 network model to obtain the optimal weights, and finally input the validation set into the network to output the detection results.

[0010] Further, step (1) specifically involves selecting the VisDrone2019 dataset, which contains ten classes: 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', and 'motor'.

[0011] Further, step (2) specifically involves the following: the backbone is divided into 5 layers in total. The first layer is a Conv stem, which consists of four 3×3 convolutions with strides of 2, 2, 1, 1 and padding of 1. The second to fourth layers are Ultra-Effificient Blocks, which are sandwish layouts. The last layer includes a global average pooling layer and a fully connected layer.

[0012] Furthermore, the feature is that the Sandwish layout is divided into three layers: a ConvFFN layer, a CAConv layer, and a ConvFFN layer; specifically, it includes the following steps:

[0013] (21) The specific operation of the first ConvFFN layer is as follows: First, the input image is processed by a convolutionstem to extract features, resulting in feature map F1 corresponding to the image to be processed; then, feature processing is performed, F1 is passed through the BN layer, and then enters the first FC layer and ReLU; next, a DWConv operation with a stride of 1 is performed; then, the second FC layer is entered to obtain F2, and then an add operation is performed with F1 to obtain F3, which increases the information content of the feature map and improves the model's resolution.

[0014] (22) The specific operation of the second CAConv layer is as follows: CAConv adopts a dual-branch structure. The first branch extracts local representations; the second branch uses context-aware weights to enhance local features; as follows:

[0015] F3, through the FC layer, undergoes a linear transformation to obtain three vectors: Query, Key, and Value. The first branch processes the Query and Key, which are then fed into the DWConv layer and multiplied by a convolution operator (Mul) before entering the second FC layer. Next, they pass through a Swish activation function to the third FC layer, followed by a Tanh activation function to generate context-aware weights in the range [-1, 1]. The second branch first processes the Value using depthwise separable convolution (DWConv) to extract local representations. DWConv shares weights. The result is then combined with the result from the first branch using a convolution operator (Mul), and the resulting value is then added to F3 using global average pooling (GAP) to obtain F4, which is then fed into the next ConvFFN module.

[0016] (23) The specific operation of the third ConvFFN layer is as follows: feature extraction is further performed on feature map F4. First, feature selection is performed through a BN layer, then through an FC layer, a ReLU activation function, and then a depthwise separable convolution (DWConv) operation with a stride of 2. Then, through an FC layer, F5 is obtained. In the branch, a depthwise separable convolution operation with a stride of 2 is performed on feature map F4 to collect local spatial information features and extend them to deep features. Then, the BN layer is used to accelerate network convergence and control overfitting. Finally, through an FC layer, F6 is obtained. An add operation is performed with F5 to obtain feature map F7. The combined features with low discriminative power are removed to achieve dimensionality reduction.

[0017] Furthermore, step (3) is as follows: the improved YOLOv8 object detection model is pruned using the Dependcy Graph method.

[0018] The lightweight target detection system based on context-aware local enhancement described in this invention includes the following modules:

[0019] Dataset building module: Used to build the object detection dataset and divide the dataset into training set and validation set;

[0020] Improvement and Model Building Module: Used to improve the YOLOv8-based backbone and build improved models;

[0021] Pruning module: Used to prune the improved YOLOv8 object detection model;

[0022] The testing module is used to input the training set into the improved YOLOv8 network model to obtain the optimal weights, and finally input the validation set into the network to output the detection results.

[0023] Furthermore, in the dataset construction module, the VisDrone2019 dataset was selected, which contains ten classes: 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', and 'motor'.

[0024] Furthermore, in the improved model building module, the backbone is divided into 5 layers in total. The first layer is a Conversion, which consists of four 3×3 convolutions with strides of 2, 2, 1, 1 and padding of 1. The second to fourth layers are Ultra-Effificient Blocks, which are Sandwish layouts. The last layer includes a global average pooling and a fully connected layer.

[0025] Furthermore, in the improvement and model building module, the Sandwish layout is divided into three layers: ConvFFN layer, CAConv layer, and ConvFFN layer; specifically, it includes the following steps:

[0026] (S1) The specific operation of the first ConvFFN layer is as follows: First, the input image is processed by a convolutionstem to extract features, resulting in feature map F1 corresponding to the image to be processed; further feature processing is performed, F1 is passed through a BN layer, and then enters the first FC layer and ReLU; next, a DWConv operation with a stride of 1 is performed; then it enters the second FC layer to obtain F2, and then an add operation is performed with F1 to obtain F3, which increases the information content of the feature map and improves the model's resolution;

[0027] (S2) The specific operations for the second CAConv layer are as follows: CAConv adopts a dual-branch structure. The first branch extracts local representations; the second branch uses context-aware weights to enhance local features; specifically as follows:

[0028] F3, through the FC layer, undergoes a linear transformation to obtain three vectors: Query, Key, and Value. The first branch processes the Query and Key, which are then fed into the DWConv layer and multiplied by a convolution operator (Mul) before entering the second FC layer. Next, they pass through a Swish activation function to the third FC layer, followed by a Tanh activation function to generate context-aware weights in the range [-1, 1]. The second branch first processes the Value using depthwise separable convolution (DWConv) to extract local representations. DWConv shares weights. The result is then combined with the result from the first branch using a convolution operator (Mul), and the resulting value is then added to F3 using global average pooling (GAP) to obtain F4, which is then fed into the next ConvFFN module.

[0029] (S3) The specific operation of the third ConvFFN layer is as follows: feature extraction is further performed on feature map F4. First, feature selection is performed through a BN layer, followed by an FC layer, a ReLU activation function, and then a depthwise separable convolution (DWConv) operation with a stride of 2. After that, an FC layer is used to obtain F5. In the branch, a depthwise separable convolution operation with a stride of 2 is performed on feature map F4 to collect local spatial information features and extend them to deep features. Then, a BN layer is used to accelerate network convergence and control overfitting. Finally, an FC layer is used to obtain F6, and an add operation is performed with F5 to obtain feature map F7. The combined features with low discriminative power are removed to achieve dimensionality reduction.

[0030] Furthermore, in the pruning module, the improved YOLOv8 object detection model is pruned using the Dependcy Graph method.

[0031] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: In view of the limited computing power of low-end devices, a lightweight feature extraction network and a multi-feature fusion method are adopted to significantly reduce the number of parameters and computing costs of the target detection model and improve the detection speed of the target detection. Attached Figure Description

[0032] Figure 1 is a structural diagram of the Ultra-Effificient Block module of the present invention;

[0033] Figure 2 is a structural diagram of the Convolution Feed Forward Network of the present invention;

[0034] Figure 3 is a schematic diagram of the Context-aware Attention Convolution principle of the present invention;

[0035] Figure 4 is a flowchart of the Backbone of the present invention;

[0036] Figure 5 shows the detection results of this invention on the VisDrone2019 dataset. Detailed Implementation

[0037] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0038] This invention provides a lightweight target detection method based on context-aware local enhancement, which includes the following steps:

[0039] (1) Construct a dataset for object detection and divide it into a training set and a validation set. Specifically, the VisDrone2019 dataset is used, which contains ten classes: 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', and 'motor'. The test set contains 6471 images, and the validation set contains 548 images. Some regions in the original images contain a large number of dense small objects that cannot be labeled. These regions are occluded and converted to YOLO format to improve the detection accuracy of these regions.

[0040] As shown in Figure 4, (2) the YOLOv8-based backbone is improved and an improved model is established. The backbone consists of 5 layers. The first layer is a Conv stem, which is composed of four 3×3 convolutions with strides of 2, 2, 1, 1 and padding of 1. The second to fourth layers are Ultra-Effificient Blocks. The Ultra-Effificient Blocks are sandwish layouts. The last layer includes a global average pooling layer and a fully connected layer. The sandwish layout uses fewer memory-constrained self-attention layers and more memory-efficient FFN layers for channel communication. Specifically, as shown in Figure 1, it uses a single self-attention layer for spatial mixing, which is sandwiched between FFNs. This design reduces the memory time consumption caused by self-attention layers in the model and applies more FFN layers, allowing for effective communication between different feature channels.

[0041] The Sandwish layout consists of three layers: ConvFFN layer, CAConv layer, and ConvFFN layer; specifically, it includes the following steps:

[0042] As shown in Figure 2, (21) the specific operation of the first ConvFFN layer is as follows: First, the input image is processed by a convolution stem to extract features and obtain the feature map F1 corresponding to the image to be processed; then the feature is further processed, F1 passes through the BN layer, and then enters the first FC layer and ReLU; then, a DWConv operation with a stride of 1 is performed; then the second FC layer is entered to obtain F2, and then an add operation is performed with F1 to obtain F3, which increases the information content of the feature map and improves the model's resolution.

[0043] As shown in Figure 3, (22) the specific operation of the second CAConv layer is as follows: CAConv adopts a dual-branch structure. The first branch extracts local representations; the second branch uses context-aware weights to enhance local features. Specifically, F3 passes through the FC layer and undergoes a linear transformation to obtain three sets of vectors: Query, Key, and Value. The first branch processes Query and Key. Query and Key enter the DWConv layer respectively, and then are multiplied by a convolution operator (Mul). Then, they enter the second FC layer, and then enter the third FC layer through a Swish activation function. After passing through a Tanh activation function, context-aware weights in the range of [-1,1] are generated. The second branch first uses depthwise separable convolution (DWConv) to process Value and extract local representations. DWConv has shared weights. The result obtained is calculated by passing the result obtained from the first branch through a convolution operator (Mul). The generated result is then added to F3 through a global average pooling (GAP) to obtain F4, and then sent to the next ConvFFN module.

[0044] (23) The specific operation of the third ConvFFN layer is as follows: feature extraction is further performed on feature map F4. First, feature selection is performed through a BN layer, then through an FC layer, a ReLU activation function, and then a depthwise separable convolution (DWConv) operation with a stride of 2. Then, through an FC layer, F5 is obtained. In the branch, a depthwise separable convolution operation with a stride of 2 is performed on feature map F4 to collect local spatial information features and extend them to deep features. Then, the BN layer is used to accelerate network convergence and control overfitting. Finally, through an FC layer, F6 is obtained. An add operation is performed with F5 to obtain feature map F7. The combined features with low discriminative power are removed to achieve dimensionality reduction.

[0045] (3) Prune the improved YOLOv8 object detection model; perform pruning operations on the improved YOLOv8 object detection model using the Dependcy Graph method.

[0046] DepGraph first groups the layers based on their interdependencies. A dependency graph is generated based on these dependencies, and the network is then decomposed into basic layers, including each parameterized layer (e.g., convolution) or non-parameterized layer (e.g., ReLU). The focus here is not on modeling hierarchical relationships, but rather on fine-grained relationships between layer inputs and outputs. The dependency graph learns consistent sparsity across all grouped layers while zeroing out certain dimensions. DepGraph flattens and merges the grouped parameters into a large parameter matrix, retrieving all parameters belonging to the k-th adjustable dimension, much like the k-th channel of a CNN block. Consistent sparsity can be further enhanced with a simple weighted shrinkage. Finally, this dependency graph is used for channel pruning, compressing the object detection algorithm proposed in this embodiment, making it more suitable for mobile deployment.

[0047] (4) Input the training set into the improved YOLOv8 network model to obtain the optimal weights, and finally input the validation set into the network to output the detection results.

[0048] This invention uses a Windows operating system, an Nvidia GeForce RTX 3060 GPU, and Python version 3.7. The batch size is set to 8, the maximum number of iterations is 100, the optimizer uses SGD to optimize the network, the weight decay is 0.0005, the learning rate (lr) is adjusted using cosine annealing algorithm, and the learning rate is 0.01. VFL Loss + CIOU Loss are used as the classification loss.

[0049] This invention was tested on the VisDrone2019 dataset. To intuitively compare these methods, this invention uses FLOPs and Params as evaluation metrics for model performance. Table 1 shows the detection results of various advanced YOLOv8 models used as the backbone. In this invention, 100 iterations were performed on a mobile device, and compared with other advanced models. This invention achieves a larger FLOPs with fewer parameters, greatly reducing the complexity of the model and making it applicable even on low-end devices, thus expanding the model's applicability. The detection results are shown in Figure 5. The YOLOv8 object detection algorithm using this invention as the backbone significantly improves the false detection rate for small objects, which is more beneficial for practical applications.

[0050] Table 1: Comparison of Experimental Results for Different Models

[0051]

Claims

1. A lightweight target detection method based on context-aware local enhancement, characterized in that, Includes the following steps: (1) Construct a dataset for object detection and divide the dataset into a training set and a validation set; (2) Improve the YOLOv8-based backbone and build an improved model; Specifically: the backbone is divided into 5 layers. The first layer is a Conv stem, which consists of four 3×3 convolutions with a stride of 2, 2, 1, 1 and a padding of 1; the second to fourth layers are Ultra-Effificient Blocks. Ultra-EffificientBlock is a sandwish layout; the last layer includes a global average pooling and a fully connected layer; the sandwish layout is divided into three layers: ConvFFN layer, CAConv layer, and ConvFFN layer; specifically, the following steps are included: (21) The specific operation of the first ConvFFN layer is as follows: First, the input image is processed by a convolutionstem to extract features and obtain the feature map F1 corresponding to the image to be processed; further feature processing is performed, F1 passes through the BN layer, and then enters the first FC layer and ReLU; secondly, a DWConv operation with a stride of 1 is performed; then it enters the second FC layer to obtain F2, and then with F 1. Perform the add operation to obtain F3, improve the information content of the feature map and improve the model's discrimination ability; (22) The specific operation of the second CAConv layer is as follows: CAConv adopts a dual-branch structure. The first branch extracts local representations; the second branch uses context-aware weights to enhance local features; specifically as follows: F3 passes through the FC layer and obtains three sets of vectors: Query, Key, and Value after linear transformation. The first branch processes Query and Key. Query and Key enter the DWConv layer respectively, and then pass through a convolution operator Mul multiplication, and then enter the second FC layer. Then, it passes through a Swish activation function to enter the third FC layer, and then passes through a Tanh activation function to generate context-aware weights in the range [-1,1]. The second branch first uses depthwise separable convolution DWConv to process Value and extract local representations, where DWConv With shared weights, the result obtained is calculated by a convolution operator Mul and the result obtained from the first branch. The generated result is then added to F3 by a global average pooling GAP to obtain F4, and then sent to the next ConvFFN module. (23) The specific operation of the third ConvFFN layer is as follows: the feature map F4 is further extracted. First, a BN layer is used for feature selection, then an FC layer, a ReLU activation function, and then a depthwise separable convolution (DWConv) operation with a stride of 2 is performed. F5 is obtained through the FC layer. In the branch, the feature map F4 is subjected to a depthwise separable convolution operation with stride=2 to collect local spatial information features and extend them to deep features. Then, the network convergence is accelerated through the BN layer to control overfitting. Finally, F6 is obtained through an FC layer. The feature map F7 is obtained by adding F5 and removing the combined features with low discriminative power to achieve dimensionality reduction. (3) The improved YOLOv8 object detection model is pruned. (4) The training set is input into the improved YOLOv8 network model to obtain the best weights. Finally, the validation set is input into the network to output the detection results.

2. The lightweight target detection method based on context-aware local enhancement according to claim 1, characterized in that, The specific steps (1) are as follows: Select the VisDrone2019 dataset, which contains ten classes: 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', and 'motor'.

3. The lightweight target detection method based on context-aware local enhancement according to claim 1, characterized in that, The specific steps (3) are as follows: the improved YOLOv8 object detection model is pruned using the Dependcy Graph method.

4. A lightweight target detection system based on context-aware local enhancement, characterized in that, The method described in any one of claims 1-3 includes: a dataset construction module for constructing a dataset for object detection and dividing the dataset into a training set and a validation set; an improvement and model construction module for improving the YOLOv8-based backbone and building an improved model; a pruning module for pruning the improved YOLOv8 object detection model; and a testing module for inputting the training set into the improved YOLOv8 network model to obtain the optimal weights, and finally inputting the validation set into the network to output the detection results.

5. A lightweight target detection system based on context-aware local enhancement according to claim 4, characterized in that, In the dataset construction module, the VisDrone2019 dataset was selected, which contains ten classes: 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', and 'motor'.

6. A lightweight target detection system based on context-aware local enhancement according to claim 4, characterized in that, In the improvement and model building module, the backbone is divided into 5 layers in total. The first layer is a Conv stem, which consists of four 3×3 convolutions with strides of 2, 2, 1, 1 and padding of 1. The second to fourth layers are Ultra-Effificient Blocks. The Ultra-Effificient Block uses a sandwish layout; the last layer includes a global average pooling layer and a fully connected layer.

7. A lightweight target detection system based on context-aware local enhancement according to claim 6, characterized in that, In the improvement and model building module, the Sandwish layout is divided into three layers: ConvFFN layer, CAConv layer, and ConvFFN layer; specifically, it includes the following steps: (S1) For the first ConvFFN layer, the specific operation is as follows: First, the input image is processed by a convolutionstem to extract features, resulting in a feature map F1 corresponding to the image to be processed; further feature processing is performed, F1 passes through a BN layer, then enters the first FC layer and ReLU; secondly, a DWConv operation with a stride of 1 is performed; then it enters the second FC layer to obtain F2, and then performs an add operation with F1 to obtain F3, increasing the information content of the feature map and improving... High model resolution; (S2) The specific operation of the second CAConv layer is as follows: CAConv adopts a dual-branch structure. The first branch extracts local representations; the second branch uses context-aware weights to enhance local features; specifically as follows: F3 passes through the FC layer and undergoes linear transformation to obtain three sets of vectors: Query, Key, and Value. The first branch processes the Query and Key, which then enter the DWConv layer and are multiplied by a convolution operator Mul. They then enter the second FC layer, pass through a Swish activation function, and enter the third FC layer. Finally, they pass through a Tanh activation function, generating a range in the range [-1,1]. The second branch first processes Value using a depthwise separable convolution (DWConv) to extract local representations. DWConv has shared weights. The result is then processed with the result from the first branch using a convolution operator Mul. The generated result is then added to F3 using a global average pooling (GAP) to obtain F4, which is then fed into the next ConvFFN module. (S3) The specific operation of the third ConvFFN layer is as follows: Feature map F4 is further extracted. First, a BN layer is used for feature selection. Then, an FC layer and a ReLU activation function are used. A depthwise separable convolution (DWConv) with a stride of 2 are then performed. After that, an FC layer is used to obtain F5. In the branch, a depthwise separable convolution with a stride of 2 is performed on feature map F4 to collect local spatial information features and extend them to deep features. Then, a BN layer is used to accelerate network convergence and control overfitting. Finally, an FC layer is used to obtain F6. An add operation is performed with F5 to obtain feature map F7. The combined features with low discriminative power are removed to achieve dimensionality reduction.

8. A lightweight target detection system based on context-aware local enhancement according to claim 4, characterized in that, In the pruning module, the improved YOLOv8 object detection model is pruned using the Dependcy Graph method.

Images