A YOLOv8 Partial Convolutional Network Object Detection Method
By introducing a sparse attention mechanism and a partially convolutional network FasterNet into the YOLOv8 model, combined with the BiFormer attention module and the Wise-IoU loss function, the problem of high computational cost of the YOLO model on low- and mid-range devices is solved, and efficient object detection is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HUAIYIN INSTITUTE OF TECHNOLOGY
- Filing Date
- 2023-08-31
- Publication Date
- 2026-05-26
AI Technical Summary
In pursuing target detection accuracy, existing YOLO models increase model depth and the number of parameters, resulting in high computational costs and large memory consumption, making them difficult to apply effectively on low- and mid-range devices and neglecting real-time performance.
We employ a sparse attention mechanism based on dynamic query awareness, combined with FasterNet and BiFormer attention modules from some convolutional networks. By using sparse attention and token-to-token attention, we reduce the number of model parameters and computational complexity, while using the Wise-IoU loss function to optimize detection accuracy.
While reducing the number of model parameters and computational costs, it improves the target detection speed and maintains the detection accuracy, making it suitable for low- to mid-range devices.
Smart Images
Figure CN117173395B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a YOLOv8 partially convolutional network object detection method, and more particularly to a YOLOv8 partially convolutional network object detection method based on a dynamic query-aware sparse attention mechanism. Background Technology
[0002] Neural networks have experienced rapid development in various computer vision tasks such as image classification, detection, and segmentation. While their impressive performance has powered many applications, a significant trend is the pursuit of fast neural networks with low latency and high throughput for reasons such as good user experience, instant response, and security. Networks like MobileNet, ShuffleNet, and GhostNet utilize depthwise convolutions (DWConv) and / or group convolutions (GConv) to extract spatial features. However, in reducing FLOPs, operators are often affected by the side effect of increased memory access. MicroNet further decomposes and sparses the network, pushing its FLOPs to extremely low levels. While this approach improves FLOPs, its fragmented computation efficiency is low. Furthermore, the aforementioned networks are often accompanied by additional data operations such as concatenation, shuffle, and pooling, the runtime of which is often critical for small models.
[0003] In recent years, the YOLO model has become increasingly mature, with numerous improvement methods. However, many methods fail to consider practical application issues, and the high computational cost significantly reduces its real-time performance, making them unfriendly to certain devices. In October 2018, Google published the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," releasing the BERT model. Later, in 2022, Beijing University of Aeronautics and Astronautics published the paper "TPH-YOLOv5: Improved YOLOv5 Based on Transformer Prediction Head for Object Detection on Drone-captured Scenarios" at ECCV, introducing a YOLO model that incorporates transformers. However, existing technologies often increase model depth and stack parameters in pursuit of object detection accuracy, neglecting floating-point calculations. This results in redundant detection parameters, high memory consumption, and high computational costs, making them unsuitable for low- to mid-range devices and ignoring real-world application scenarios. Summary of the Invention
[0004] Purpose of the invention: The purpose of this invention is to provide a YOLOv8 partially convolutional network object detection method based on a dynamic query-aware sparse attention mechanism that reduces the number of parameters and floating-point numbers in the object detection model while improving the object detection speed and ensuring accuracy.
[0005] Technical solution: The present invention provides a YOLOv8 partially convolutional network object detection method, comprising the following steps:
[0006] S1: Select the MS COCO 2017 dataset;
[0007] S2: Construct a feature extraction network based on YOLOv8, replace the backbone network of the YOLOv8 model with a partially convolutional network FasterNet, and extract features from the initial target image.
[0008] S3: Add a BiFormer attention module to the neck of the c2f module outside the backbone of the YOLOv8 model to purify the extracted features.
[0009] S4: Determine the participating region and apply token-to-token attention to capture key information in the input tensor;
[0010] S5: Replace the CIoU loss function with the Wise-IoU loss function to complete the construction of the initial object detection model;
[0011] S6: Train the initial object detection model using the new dataset to obtain the final object detection model;
[0012] S7: Use the final object detection model to perform object detection on the image to be detected.
[0013] Furthermore, in step S1, the dataset includes categories of pedestrians, cars, and bicycles.
[0014] Further, in step S2, the partially convolutional network FasterNet is constructed through the following steps:
[0015] S21: Set up a PConv layer, and then set up two PWConv layers or Conv1×1 layers after the PConv layer;
[0016] S22: Place the normalization layer and activation layer between two PWConv layers or Conv1×1 layers for feature transformation and classification, thus completing the construction of a FasterNetBlock block;
[0017] S23: Set up a Conv4×4 Embedding layer with a stride of 4, and then set up 4 stages in sequence after the Embedding layer. Each stage is composed of several FasterNetBlock blocks stacked together. Set up a Conv2×2 Merging layer with a stride of 2 between each stage for spatial downsampling and expanding the number of channels, thus completing the construction of the overall architecture of the partially convolutional network FasterNet.
[0018] Furthermore, in step S22, each FasterNetBlock block consists of a PConv layer and two PWConv layers or a Conv1×1 layer, which together form an inverted residual block. The number of intermediate layer channels is not limited and can be further expanded, and a Shortcut is placed to reuse the input features.
[0019] Furthermore, in step S23, the number of FasterNetBlock blocks in the latter two stages is greater than that in the former two stages.
[0020] Furthermore, in step S2, feature extraction of the initial target image is achieved by first subjecting the target image to spatial downsampling and channel expansion in the embedding layer, and then inputting it into the first layer FasterNetBlock block, thereby enabling the FasterNetBlock block to extract features from the initial target image.
[0021] Furthermore, in step S3, an image is divided into n blocks, where n is a positive integer, and each block serves as a routing area.
[0022] Furthermore, in step S3, the BiFormer attention module constructs a region-level association graph and prunes it so that each node retains only the first k connections, where k is a positive integer less than or equal to n, and each region only focuses on the first k routing regions.
[0023] Furthermore, the BIFormer attention module sequentially includes a multilayer perceptual MLP layer, a laterally normalized LN layer, a two-layer routing BRA layer, a laterally normalized LN layer, and a depthwise separable convolutional DWConv layer.
[0024] Furthermore, in step S4, the method for applying the token-to-token attention is to filter out the least relevant key-value pairs at the coarse-grained region level, construct and prune the region-level directed graph, and then apply fine-grained token-to-token attention in the union of routing regions.
[0025] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: by adopting a lightweight feature extraction network and semantic feature extraction method, the number of parameters and floating-point numbers of the target detection model are reduced, thereby improving the target detection speed while ensuring accuracy. Attached Figure Description
[0026] Figure 1 This is a schematic diagram illustrating the mechanism and principle of FasterNet in this invention;
[0027] Figure 2 This is a schematic diagram of the pointwise convolution PWConv structure of the present invention;
[0028] Figure 3 This is a structural diagram of the BIFormer attention module according to an embodiment of the present invention;
[0029] Figure 4 This is a diagram of the two-layer routing attention structure according to an embodiment of the present invention;
[0030] Figure 5 This is a flowchart of the BIFormer attention module added to the c2f layer in this invention. Detailed Implementation
[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0032] As shown in the attached figure, the YOLOv8 partially convolutional network object detection method of this embodiment includes the following steps:
[0033] S1: Select the MS COCO 2017 dataset, including categories such as pedestrians, cars, and bicycles;
[0034] S2: Construct a feature extraction network based on YOLOv8, and replace the backbone network of the YOLOv8 model with a partially convolutional network FasterNet for feature extraction of the initial target image;
[0035] The original YOLOv8 backbone was replaced with a lightweight feature extraction network, FasterNetBlock. Each FasterNetBlock block has a PConv layer followed by two PWConv (or Conv1×1) layers, which together are displayed as an inverted residual block. The intermediate layers have expanded channel counts and incorporate shortcuts to reuse input features. Normalization and activation layers are placed after each intermediate PWConv layer to maintain feature diversity and achieve low latency. Batch normalization (BN) layers, global average pooling, and fully connected 1×1 convolutional layers are used for feature transformation and classification. Finally, a Conv4×4 embedding layer with a stride of 4 is added, followed by four stages, each consisting of stacked FasterNetBlock blocks. A Conv2×2 merging layer with a stride of 2 is placed between each stage for spatial downsampling and expanding channel counts, completing the overall architecture of the partially convolutional network FasterNet. The latter two stages contain more FasterNetBlock blocks than the first two.
[0036] Specifically, this is achieved as follows: First, an input image is fed into the embedding layer for spatial downsampling and channel expansion. Then, it is fed into the first FasterNetBlock module, specifically a FasterNetBlock unit that extracts features from the initial image. The PConv layer extracts spatial features using a portion of the input channels. The subsequent pointwise convolution (PWConv) is a T-shaped conv structure, which focuses more on feature extraction at the center compared to the original approach. This significantly reduces FLOPs. The combined throughput of PConv and PWConv is:
[0037] h×w×(k 2 ×c p ×c+c×(cc p ))
[0038] h is the height of the target image, w is the width of the target image, c is the number of channels in the target image, and k is the size of the convolution kernel. p For the first or last consecutive channel, each subsequent FasterNetBlock module operates on the same principle as described above. Each group of convolutional kernels is convolved with its corresponding input data to obtain different output data, which is then reduced in dimensionality using global pooling. The input is then fed into fully connected (FC) layers to integrate the features (highly refining them), making them easier to pass to the final classifier or regression and reducing the dimensionality of the output image.
[0039] S3: Add a BiFormer attention module to the neck of the c2f module outside the backbone of the YOLOv8 model to purify the extracted features.
[0040] The improved YOLOv8 lightweight feature extraction network extracts features from images of different sizes, resulting in three effective feature layers, which are then input into the Neck layer. To enhance the expressive power of the lightweight network's features, this embodiment introduces a BiFormer attention module in the Neck layer. An image is divided into several blocks, each block serving as a routing region. The BiFormer attention module constructs a region-level association graph and prunes it, ensuring that each node retains only the first k connections, where k is a positive integer less than or equal to n. Therefore, each region only needs to focus on the first k routing regions. The specific operation is structured as follows: Figure 3 As shown, a BIFormer attention module contains an MLP layer (multilayer perceptron), an LN layer (lateral normalization), a BRA layer (two-layer routing), and a DWConv layer (depth-separable convolution).
[0041] Specifically, after the features enter the C2F module, they undergo a series of convolutions and then enter concat for feature fusion. Following this, depthwise separable convolutions are performed to reduce the number of parameters and computational cost. Then, after standardization through an LN layer, the features enter the BRA two-layer routing module. This module collects key-value pairs from the first k relevant windows and uses sparsity operations to skip the computation of the least relevant regions, saving parameters and computational cost. Finally, the features enter the MLP layer. Because the MLP layer is fully connected, it can highly refine the features. The final feature output is then given.
[0042] S4: Determine the participating region and apply token-to-token attention to capture key information in the input tensor;
[0043] To ensure that each query processes the most semantically relevant key-value pairs, one approach is to select key-value pairs for each query one by one. This requires calculating the relevance between all queries and all keys, resulting in the same complexity as the original self-attention mechanism. Another approach is to predict the attention offset based on the local context of each query. While this reduces computational complexity, it affects the modeling of long-distance dependencies. To address this issue, a two-layer routing attention mechanism is proposed. The core idea of the BRA module is to filter out the least relevant key-value pairs at the coarse-grained region level. This is achieved by first constructing and pruning a region-level directed graph, and then applying fine-grained token-to-token attention in the union of routing regions.
[0044] The specific implementation method is as follows: Input an image. Where H represents the width of the image, W represents the height of the image, and C represents the number of channels in the image. H×W is the resolution of the image, which is also the number of pixels. Each pixel represents a color, described by a C-dimensional vector. X represents this image, which is first divided into S×S regions, where each region contains... eigenvectors, that is, transforming X into... Then, through linear mapping, we obtain: in These are the projected weights of the query, key, and value, respectively; by constructing a directed graph to find the regions that each given region should participate in, we obtain: Then, the adjacency matrix of inter-region correlation is calculated: A r =Q r (K r ) T Only the first k connections of each region are retained to prune the correlation map; resulting in: I r =topkIndex(A r ), where the area routing index matrix For each query token in region i, it will focus on all key-value pairs in the union of k routing regions, first aggregating the key and value tensors, i.e.:
[0045] K g =gather(K, I) r V g =gather(V, I) r ), where K g and V g It is a tensor of aggregated key-value pairs, and then attention operations are applied to the aggregated KV pairs:
[0046] O = Attention(Q, K) g V g )+LCE(V), where a local context enhancement term LCE(V) is introduced. The function LCE() is parameterized with depthwise separable convolution, and the kernel size is set to 5.
[0047] S5: Replace the CIoU loss function with the Wise-IoU loss function to complete the construction of the initial object detection model;
[0048] S6: Train the initial object detection model using the new dataset to obtain the final object detection model;
[0049] S7: Use the final object detection model to perform object detection on the image to be detected.
Claims
1. A target detection method using a YOLOv8 partially convolutional network, characterized in that, Includes the following steps: S1: Select the MS COCO 2017 dataset; S2: Based on the dataset, construct a YOLOv8-based feature extraction network, replace the YOLOv8 model backbone network with a partially convolutional network FasterNet, and extract features from the initial target image. The FasterNet partial convolutional network is constructed through the following steps: S21: Set up a PConv layer, and then set up two PWConv layers or Conv1×1 layers after the PConv layer; S22: Place the normalization layer and activation layer between two PWConv layers or Conv1×1 layers for feature transformation and classification, thus completing the construction of a FasterNetBlock block; S23: Set up a Conv4×4 Embedding layer with a stride of 4, and then set up 4 stages in sequence after the Embedding layer. Each stage is composed of several FasterNetBlock blocks stacked together. Set up a Conv2×2 Merging layer with a stride of 2 between each stage for spatial downsampling and expanding the number of channels, thus completing the construction of the overall architecture of the partially convolutional network FasterNet. The latter two stages have more FasterNetBlock blocks than the former two. The feature extraction of the initial target image is achieved by first having the target image enter the embedding layer for spatial downsampling and channel expansion, and then inputting it into the first layer FasterNetBlock block, so that the FasterNetBlock block can extract features from the initial target image. S3: Add a BiFormer attention module to the neck of the c2f module outside the backbone of the YOLOv8 model to purify the extracted features. S4: Determine the participating region and apply token-to-token attention to capture key information in the input tensor; S5: Replace the CIoU loss function with the Wise-IoU loss function to complete the construction of the initial object detection model; S6: Train the initial object detection model using the new dataset to obtain the final object detection model; S7: Use the final object detection model to perform object detection on the image to be detected.
2. The YOLOv8 partially convolutional network target detection method according to claim 1, characterized in that, In step S1, the dataset includes categories of pedestrians, cars, and bicycles.
3. The YOLOv8 partially convolutional network target detection method according to claim 1, characterized in that, In step S22, each FasterNetBlock block consists of a PConv layer and two PWConv layers or a Conv1×1 layer, which together form an inverted residual block. The number of intermediate layer channels is not limited and can be further expanded, and a Shortcut is placed to reuse the input features.
4. The YOLOv8 partially convolutional network target detection method according to claim 1, characterized in that, In step S3, an image is divided into n blocks, where n is a positive integer, and each block is a routing area.
5. The YOLOv8 partially convolutional network target detection method according to claim 4, characterized in that, In step S3, the BiFormer attention module constructs a region-level association graph and prunes it so that each node retains only the first k connections, where k is a positive integer less than or equal to n, and each region only focuses on the first k routing regions.
6. The YOLOv8 partially convolutional network target detection method according to claim 5, characterized in that, The BIFormer attention module sequentially includes a multilayer perceptual (MLP) layer, a laterally normalized (LN) layer, a two-layer routing (BRA) layer, another laterally normalized (LN) layer, and a depthwise separable convolutional (DWConv) layer.
7. The YOLOv8 partially convolutional network target detection method according to claim 1, characterized in that, In step S4, the token-to-token attention is applied by filtering out the least relevant key-value pairs at the coarse-grained region level, constructing and pruning a region-level directed graph, and then applying fine-grained token-to-token attention in the union of routing regions.