Real-time target detection methods, terminal devices and storage media for mobile devices

Abstract

This invention discloses a real-time target detection method, terminal device, and storage medium for mobile devices. The target detection method includes extracting features from an input feature map by channel and then concatenating them to obtain a feature map after information interaction; dividing the feature map into a first channel and a second channel; extracting local features from the first channel using a PGConv module, and extracting local and global features from the second channel using both the PGConv module and a ShiftMLP module; and concatenating the local features from the first channel with the local and global features from the second channel to obtain the target features. This real-time target detection method for mobile devices solves the problem of channels being unable to communicate with each other, while significantly reducing computational load and parameter count, and enabling accurate target detection results to be obtained on mobile devices.

Description

Technical Field

[0001] This invention belongs to the field of real-time target detection technology, specifically relating to a mobile real-time target detection method, terminal device, and computer-readable storage medium for lightweight, multi-scale objects. Background Technology

[0002] Object detection is a crucial task in computer vision, primarily aimed at locating and identifying specific objects in images. Over the past few decades, researchers have proposed numerous object detection methods, including those based on traditional machine learning algorithms and deep learning. While these methods have achieved some success, they still suffer from drawbacks such as high computational complexity, low accuracy, and poor adaptability. Particularly in multi-scale object detection, the locality of convolution and the effects of downsampling operations can lead to inaccurate detection due to changes in object scale. Meanwhile, with the continuous development of computers, improvements in computing power, the development of neural network models, and convenient data acquisition methods, object detection technology has made breakthroughs in fields such as autonomous driving, security monitoring, and smart homes. However, deep neural networks are placing increasingly higher demands on hardware platforms. Object detection is no longer content with simply performing complex data computation and inference in the cloud; it is gradually migrating to mobile devices with real-time detection needs, such as smartphones, in-vehicle computers, and smart sensors.

[0003] To address these issues, researchers have proposed various lightweight networks to improve detection speed and reduce model parameters and computational complexity. In 2015, researchers proposed a lightweight version, Tiny-YOLO v1, alongside the original YOLO v1 algorithm. This algorithm simplifies the original 24-layer convolutional structure of YOLO v1 to 9 layers, achieving a 3.4x speedup on the VOC2007 dataset at the cost of a 10.7% reduction in mAP compared to YOLO v1. However, because the feature extraction network uses a fully convolutional network, the input image size must be fixed. This results in a decrease in detection accuracy when the input image size varies significantly, impacting the network's ability to detect targets. Furthermore, YOLO v1 uses a single-scale prediction bounding box, which cannot adapt to the detection needs of targets at different scales; for smaller targets, the detection performance is relatively poor. These issues limit the performance of YOLO v1 in practical applications. In 2016, Iandola et al. proposed the lightweight SqueezeNet network, which borrowed design ideas from the Inception network. Built using convolutional layers, fire modules, and pooling layers, the SqueezeNet model was only 4.8MB in size, yet achieved a TOP-1 classification accuracy of 57.5% on ImageNet, slightly better than AlexNet. However, due to the use of fewer convolutional layers and kernels, its feature extraction capability was relatively weak, failing to extract complex image features. In 2017, Howard et al. proposed the lightweight MobileNetv1 network, which used a depthwise separable convolution (DWConv) structure instead of the traditional convolutional structure. This method reduced the computational cost of convolution operations to 1 / 8 to 1 / 9 of the original. Meanwhile, by introducing a time factor α and a resolution factor ρ, the number of model parameters was further reduced, resulting in MobileNetv1 with approximately 4.2M parameters. It achieved a top-1 classification accuracy of 70.6% on the ImageNet dataset. While MobileNetv1 used separable convolutions to reduce computational complexity, this led to a loss of accuracy and insufficient feature representation capabilities. In the same year, Zhang et al. proposed the lightweight ShuffleNetv1 network. This model improved upon the conventional residual structure by replacing the first convolutional layer connected to the input feature map with grouped convolutions. It also utilized a Channel shuffle operation to exchange information between the outputs of each group of convolutions, thus maintaining network performance while reducing computational cost. ShuffleNetv1 ultimately achieved a top-1 classification accuracy of 73.7% on the ImageNet dataset.In 2018, Ma et al. proposed the ShuffleNetv2 lightweight network based on ShuffleNetv1, such as... Figure 1 As shown, Figure 1 In the ShuffleNetv2 lightweight network, Channel Split is the channel separation module, 1×1Conv is the 1×1 convolution module, 3×3DWConv is the 3×3 depthwise separable convolution module, Concat is the concatenation module, and ChannelShuffle is the channel shuffling module. When processing the input feature map, ShuffleNetv2 first performs channel splitting. One branch does not perform any operation, while the other branch undergoes convolution processing, is concatenated with the first branch, and then a Channel Shuffle operation is performed to achieve information exchange. Ultimately, ShuffleNetv2 achieved a TOP-1 classification accuracy of 74.9% on the ImageNet dataset. ShuffleNet uses convolution and depthwise separable convolution to reduce computation, but the locality of convolutional networks still cannot handle global feature processing.

[0004] Due to the locality of information, convolutional networks struggle to capture global features. Models based on self-attention mechanisms allow neural networks to focus more on important information and adaptively attend to information at different locations, enabling them to process global features. In 2021, Dosovitskiy et al. proposed the ViT model, an alternative to convolutional neural networks that effectively improves network performance. However, this increases computational cost and the number of parameters, making it unsuitable for inference on mobile devices, primarily due to their low computing power. To reduce network parameters, Touvron et al. proposed the DelT model in the same year, which has approximately 5-6 million parameters but achieves 3% lower accuracy than MobileNetv3. In 2022, the MobileViT series of models were developed to focus on lightweight ViT models. Since MobileViTv1 used the traditional ViT model, the network model was relatively heavy. Therefore, MobileViTv2 and MobileViTv3 adopted a self-attention mechanism with linear complexity to improve the network inference time. While stacking a large number of ViT models can improve accuracy, it still cannot avoid the problem of a large number of parameters, making it difficult to apply to mobile devices. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that existing target detection methods have a large number of parameters and low accuracy when applied to mobile devices. Therefore, the present invention provides a real-time target detection method, terminal device and computer-readable storage medium for lightweight, multi-scale objects on mobile devices.

[0006] The first aspect of this invention discloses a real-time target detection method for lightweight, multi-scale objects on a mobile device, comprising:

[0007] After extracting features from the input feature map by channel, the features are concatenated to obtain the feature map after information interaction.

[0008] The feature map after information interaction is divided into two channels: the first channel and the second channel.

[0009] The PGConv module is used to extract local features in the first channel, and the PGConv module and the ShiftMLP module are used to extract local features and global features in the second channel, respectively.

[0010] By splicing the local features in the first channel with the local and global features in the second channel, the features of the target are obtained;

[0011] The PGConv module includes a first convolutional module and a group convolutional module arranged sequentially.

[0012] The ShiftMLP module includes a Shift operation module and an MLP network module, which are configured sequentially.

[0013] Preferably, the input feature map is processed by extracting features from each channel and then concatenating them to obtain the feature map after information interaction, specifically including:

[0014] The input feature map is divided into two channels: the third channel and the fourth channel.

[0015] The fourth channel includes a 1×1 convolution module and a ShiftMLP module arranged sequentially.

[0016] The features extracted from the fourth channel are concatenated with the features from the third channel to obtain a feature map after information interaction.

[0017] Preferably, the PGConv module is used to extract local features in the first channel, specifically including:

[0018] Local features in the first channel are extracted sequentially using a 1×1 convolution module, a 3×3 PGConv module, and a 1×1 convolution module.

[0019] Preferably, the PGConv module and the ShiftMLP module are used to extract local and global features from the second channel, respectively, specifically including:

[0020] Local features in the second channel are extracted using a 3×3PGConv module;

[0021] Global features in the second channel are extracted using the ShiftMLP module connected to the 3×3PGConv module;

[0022] The ShiftMLP module is also connected to a 1×1 convolution module.

[0023] Preferably, the first convolution module includes a first channel division module, a 1×1 convolution module, and a first stitching module;

[0024] The first channel division module is used to divide the feature map after information interaction into two channels: the fifth channel and the sixth channel.

[0025] The sixth channel includes a 1×1 convolution module;

[0026] The first stitching module is used to stitch the features extracted from the sixth channel with the features from the fifth channel.

[0027] Preferably, the group convolution module includes a second channel partitioning module, a 1×1 convolution module, a second splicing module, and a 3×3 depth-separable convolution module;

[0028] The second channel division module is used to divide the features output by the first splicing module into two channels: the seventh channel and the eighth channel.

[0029] The eighth channel includes a 1×1 convolution module;

[0030] The second splicing module is used to splice the features extracted from the eighth channel with the features from the seventh channel;

[0031] The 3×3 depth-separable convolution module is connected to the second splicing module.

[0032] Preferably, the ShiftMLP module further includes a layer normalization module and a regularization module;

[0033] The ShiftMLP module specifically includes, in sequence, a Shift operation module, a normalization module, an MLP network module, and a regularization module.

[0034] Preferably, the MLP network module includes a 1×1 convolution module, a Dropout module, another 1×1 convolution module, and a Dropout module arranged sequentially.

[0035] A second aspect of the present invention discloses a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0036] A third aspect of the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0037] The beneficial effects of this invention are:

[0038] Convolutional Neural Networks (CNNs) are locally connected and weight-sharing neural networks whose spatial inductive bias allows them to learn representations with relatively few parameters. However, due to the local nature of convolutional operations, CNNs struggle to capture global information features. To learn global features, Self-Attention-based ViT models can be used. While ViT offers stronger representation capabilities, it often has a large number of parameters. Therefore, this invention proposes a lightweight real-time detection model for multi-scale objects that can run on mobile devices: Channel Shuffling Faster Networks (SFasterNet). SFasterNet is an efficient and lightweight network designed based on the advantages of both ViT and CNNs. Since ShuffleNet uses CNNs as its backbone for mobile object detection, it suffers from weak global representation capabilities and limited feature extraction. The ShiftMLP module proposed in this invention has fewer parameters to effectively improve global representation capabilities. Meanwhile, using a multi-scale single-channel convolutional network structure as the backbone and fusing features extracted from different scales using PAN-FPN feature pyramids results in high FLOPS and a large amount of redundant computation. To address this, a module PGConv was designed to replace ordinary convolution. This module effectively reduces redundant computation and memory access and can extract spatial features more efficiently. On the Pascal VOC2007 object detection task, SFasterNet achieved 42% mAP. Under the same network structure, ShuffleNetv2, using conventional convolution, showed a 1% improvement; MobileViTv3, using the ViT structure, showed a 7.4% improvement. Furthermore, SFasterNet improved performance across various metrics by 0.3%–4.5%, while possessing only 1.4G of FLOPs and 1.3M of parameters. Attached Figure Description

[0039] Figure 1 The following are building blocks for ShuffleNetV2 in the prior art, where (a) is the basic unit of ShuffleNet; and (b) is the unit of ShuffleNet downsampling (2×).

[0040] Figure 2 This is a construction module for a mobile real-time target detection method (SFasterNet) for lightweight, multi-scale objects in this embodiment of the invention, wherein (a) is a lightweight layer basic unit of SFasterNet; and (b) is a core unit of SFasterNet.

[0041] Figure 3 This is a schematic diagram of the standard ViT model.

[0042] Figure 4 The diagrams illustrate the convolution process, where (a) is a schematic diagram of a conventional convolution process; (b) is a schematic diagram of a depthwise separable convolution process; (c) is a schematic diagram of a local convolution process; and (d) is a schematic diagram of the PGConv convolution process of this invention.

[0043] Figure 5 The feature maps are visualized, where (a) is the feature map of the intermediate layer of SFasterNet; and (b) is the feature map of the intermediate layer of ShuffleNetv2, which is the feature map of Stage3.

[0044] Figure 6 This is a diagram illustrating the Shift operation.

[0045] Figure 7 The images show a comparison of features at each stage, where (a) is the feature map of each stage of SFasterNet; and (b) is the feature map of each stage of ShuffleNetv2.

[0046] Figure 8 Here are the feature maps corresponding to Stage 4 in ShuffleNetv2, where (a) is the feature map of Stage 4 obtained by the PGConv version; and (b) is the feature map of Stage 4 obtained by the DWConv version.

[0047] Figure 9 The following are feature maps for each stage of MobileViTv3, where (a) is the feature map for each stage of MobileViTv3 using the ShiftMLP module; and (b) is the feature map for each stage of MobileViTv3 using the Linear Transformer module.

[0048] Figure 10 This is a flowchart of a mobile real-time target detection method for lightweight, multi-scale objects in an embodiment of the present invention. Detailed Implementation

[0049] With the development of deep learning technology, object detection algorithms have been greatly improved. Among them, multi-scale processing technology has become an important development direction in object detection algorithms. This is because in natural scenes, the size and position of objects change with the distance of the camera and the scene, and different objects also have different sizes. Therefore, multi-scale processing technology has become a necessary step to improve the robustness of object detection algorithms.

[0050] Furthermore, with the widespread adoption of mobile devices, research on lightweight models has received considerable attention. Due to the limited computing power and storage resources of mobile devices, it is necessary to design lightweight models to meet their requirements. One common solution is to reduce the model size and computational load through network compression and quantization techniques; another solution is to design a specific backbone network for mobile devices.

[0051] Considering the needs for multi-scale processing and lightweight models, several successful solutions have emerged. For example, lightweight object detection models like YOLOv3-Tiny (based on the MobileNetv2 backbone) and SSDMobileNetv2 (based on the ShuffleNet backbone) employ multi-scale processing techniques, adapting to targets of varying sizes and running efficiently on mobile devices. MobileNet uses depthwise separable convolutions to construct a lightweight neural network and introduces two simple global hyperparameters to effectively balance latency and accuracy. MobileNet processes images through depthwise separable convolutional layers, activation functions, and pooling layers to obtain the final output. Simultaneously, global hyperparameters are used during training to control model size and latency, and width and resolution multipliers are used to further optimize model efficiency. ShuffleNet is a convolutional neural network architecture with two key techniques: channel-wise group convolution kernel channel rearrangement. Channel-wise group convolution divides the input tensor into multiple smaller tensors and then performs convolution operations on each smaller tensor. This method reduces computational complexity and memory consumption. Channel rearrangement techniques can rearrange the channels in the input tensor to minimize computational complexity before channel-wise group convolution.

[0052] The application of self-attention mechanisms in object detection is receiving increasing attention. In multi-scale object detection, self-attention mechanisms can adaptively learn the correlations between features at different scales, thereby better capturing the semantic information of the target. Furthermore, self-attention mechanisms can improve performance in lightweight models. Currently, several multi-scale object detection models based on self-attention mechanisms have been proposed, such as SAPD and SOGNet. These models use self-attention mechanisms to fuse features at different scales, thereby better capturing the detailed and contextual information of the target. In addition, self-attention mechanisms are also widely used in lightweight models. For example, the EfficientDet-Lite model uses a lightweight backbone network based on depthwise separable convolutions and self-attention mechanisms to improve model performance. Similarly, MobileViT combines the advantages of CNNs and ViTs to build a lightweight network model. It utilizes Transformers to encode global information and learn the information between patches. MobileViT first uses n×n convolutional kernels for local modeling of the image, and then uses Transformers for global modeling, thereby achieving better performance and generalization ability.

[0053] Based on this, some models still have problems. For example, ShuffleNet uses channel-wise group convolution, which reduces computational complexity and memory consumption, but channels cannot communicate with each other. To solve this problem, channel rearrangement is used to restore communication as much as possible. However, this problem is still not perfectly solved. To solve the problem of inter-channel communication, MobileViT introduced a Transformer structure to achieve global communication. Due to the large number of parameters in the Transformer structure, the inference speed is slow for mobile devices, making it impractical for real-world applications. MobileViTv2 replaced the original standard Transformer structure with a linear time complexity Transformer structure, effectively reducing the number of parameters. Based on this, MobileViTv3 found that the linear Transformer structure used for global modeling in MobileViTv2 was not as effective as the standard Transformer structure, so a residual structure was used for feature fusion to improve accuracy.

[0054] Analysis of the above problems reveals that group convolution can reduce computational complexity and memory consumption. Therefore, this invention designs PGConv to replace depthwise separable convolution, effectively solving the problem of communication barriers between channels. This is achieved by dividing the feature map channels into two parts according to a certain ratio; one part is convolved to extract features, and the other part is concatenated with the residual. Since this module cannot fully achieve global modeling, this invention, through analysis of the MobileViT model, finds that the MLP module is a core component of the ViT model, capable of extracting useful features without the need for convolution. Therefore, this invention designs the ShiftMLP module to replace the Transformer module, reducing parameters and computational cost while effectively improving accuracy. As shown in Table 1, the parameter and computational cost of each module are compared at the same scale. This invention sacrifices a suitable number of FLOPs to improve detection accuracy. The overall scheme of this invention is as follows: Figure 2 and Figure 10 As shown, Figure 2 In the middle, Channel Split is the channel separation module, 1×1Conv is the 1×1 convolution module, ShiftMLP is a module composed of Shift operation and MLP module, Concat is the concatenation module, PGConv is a module composed of PConv and GConv, 3×3DWConv is the 3×3 depthwise separable convolution module, Channel Shuffle is the channel shuffling module, BatchNorm is the layer normalization module, MLP is a multilayer perceptron, and DropPath is the regularization module.

[0055] Table 1. Comparison of Parameters for Each Module

[0056]

[0057] A standard ViT model, such as Figure 3 As shown, input The image is divided into equal-sized patches, and each patch is resized to a fixed size. Then, each patch is flattened into a vector. It is then mapped to a d-dimensional embedding space through a learnable linear transformation. In computer vision, the computational cost of self-attention in a Transformer is O(N). 2d). Here, C, H, and W represent the channels, height, and width of the tensor, respectively; P = wh is the number of pixels in a patch of height h and width w; and N is the number of patches. Because these models ignore the inherent spatial inductive bias of CNNs, they require more parameters to learn visual representations. For example, compared to the CNN-based DeepLabv3, the ViT-based DPT learned up to 6 times more parameters to provide similar segmentation performance (DPT vs. DeepLabv3: 345M vs. 59M). Furthermore, these models exhibit non-standard optimizability compared to CNNs. These models are sensitive to L2 regularization and require significant data augmentation to prevent overfitting.

[0058] This invention proposes a lightweight model, SFasterNet. The core idea is to use PGConv for local feature extraction from images, and then employ the ShiftMLP module (composed of MLP components and Shift operations in ViT) for global feature extraction. This enables the invention to extract more valuable features.

[0059] The present invention provides a real-time mobile target detection method for lightweight, multi-scale objects, such as... Figure 2 As shown, it includes:

[0060] Step 1: Extract features from the input feature map by channel and then concatenate them to obtain the feature map after information interaction, specifically including:

[0061] The input feature map is divided into two channels: a third channel and a fourth channel. The third channel is... Figure 2 (a) The channel on the left, from Channel Split to Concat, with the fourth channel being... Figure 2 (a) The right channel, which includes a 1×1 convolution module and a ShiftMLP module arranged in sequence; wherein the ShiftMLP module includes a Shift operation module and an MLP network module arranged in sequence.

[0062] Then, the features extracted from the fourth channel are concatenated with the features from the third channel to obtain the feature map after information interaction.

[0063] Step 2: Divide the feature map after information interaction into two channels: the first channel and the second channel. Figure 2 (b) The left-hand passage is the first passage, and the right-hand passage is the second passage;

[0064] Step 3: Use the PGConv module to extract local features in the first channel, and use the PGConv module and the ShiftMLP module to extract local and global features in the second channel, respectively.

[0065] Step 4: Combine the local features in the first channel with the local and global features in the second channel to obtain the target's features.

[0066] In step 3 of this embodiment, the local features in the first channel are extracted using the PGConv module. Specifically, the local features in the first channel are extracted sequentially using a 1×1 convolution module, a 3×3 PGConv module, and a 1×1 convolution module.

[0067] Step 3 of this embodiment of the invention, which uses the PGConv module and the ShiftMLP module to extract local and global features from the second channel respectively, specifically includes:

[0068] Local features in the second channel were extracted using a 3×3PGConv module.

[0069] Global features in the second channel were extracted using the ShiftMLP module connected to the 3×3PGConv module.

[0070] The ShiftMLP module is also connected to a 1×1 convolution module.

[0071] The PGConv module includes a first convolutional module and a group convolutional module set sequentially.

[0072] Furthermore, the first convolution module includes a first channel partitioning module, a 1×1 convolution module, and a first stitching module;

[0073] The first channel segmentation module is used to divide the feature map after information interaction into two channels: the fifth channel and the sixth channel.

[0074] The sixth channel includes a 1×1 convolution module;

[0075] The first stitching module is used to stitch together the features extracted from the sixth channel with the features from the fifth channel.

[0076] The group convolution module includes a second channel partitioning module, a 1×1 convolution module, a second stitching module, and a 3×3 depthwise separable convolution module (3×3DWConv);

[0077] The second channel division module is used to divide the features output by the first splicing module into two channels: the seventh channel and the eighth channel.

[0078] The eighth channel includes a 1×1 convolution module;

[0079] The second stitching module is used to stitch the features extracted from the eighth channel with the features from the seventh channel;

[0080] A 3×3 depth-separable convolutional module is connected to a second stitching module.

[0081] Depthwise Separable Convolution (DWConv) is a popular variant of Conv and has been widely used as a key module in many neural networks. For input DWConv uses c filters To calculate the output like Figure 4 As shown in (b), each filter slides spatially over one input channel and contributes to one output channel. This is similar to having h×w×k. 2 ×c 2 Compared to conventional Conv, this depth-based computation reduces the FLOPs of DWConv to h×w×k. 2 ×c. While DWConv can effectively reduce FLOPs, it cannot simply replace regular Conv because it leads to a significant drop in precision. Therefore, in practice, the number of channels c in DWConv is increased to c′ (c′ > c) to compensate for the precision loss; for example, the width of DWConv is increased by 6 times in inverted residual blocks. However, this results in higher memory accesses, leading to non-negligible latency and slowing down overall computation, especially for I / O-bound devices. In particular, the number of memory accesses is now upgraded to...

[0082] h×w×2c′+k 2 ≈h×w×2c′ (1)

[0083] This is higher than the regular Conv, that is,

[0084] h×w×2c+k 2 ×c 2 ≈h×w×2c (2)

[0085] Note that memory access to h×w×2c′ is consumed by I / O operations.

[0086] Since directly replacing conventional Conv with DWConv leads to a decrease in accuracy, Chen et al. proposed PConv, such as... Figure 4 As shown in (c), PConv will input The convolution is divided into two parts. One part of the channels is used for spatial feature extraction using a regular Convolutional method, and the result is concatenated with the other part of the channels to obtain the final result. PConv has lower computational cost than regular Convolutional, but higher than DWConv and group convolutions. Its purpose is to reduce computational redundancy and memory access. The FLOPs of PConv convolution are only:

[0087]

[0088] Where cp It refers to the number of channels in the filter, representing a ratio for a specific portion. PConv's FLOPs are only 1 / 16 of those of a regular Conv. Furthermore, PConv has lower memory access requirements.

[0089]

[0090] Therefore, this invention combines the advantages of all three and designs PGConv to input... The data is divided into two parts. One part of the channels is used for feature extraction with regular Conv, and then concatenated with the remaining channels. Finally, the concatenated result is used as input, and group convolution is used to reduce FLOPs. The size of the group is the same as the size of the previous filter, i.e., c. p Since the computational cost of PConv varies depending on the feature map scale, with larger scale feature maps requiring more computation than smaller scale ones, using PConv instead of conventional Conv can effectively reduce computation. However, the computational cost is still higher than that of networks built using DWConv and group convolutions. Therefore, this invention uses PGConv to process the feature map at the beginning of each stage, effectively reducing the computational cost of feature maps at different scales. Using PConv in intermediate stages improves accuracy. Figure 5 As shown, Figure 5 In the diagram, (a) represents the feature maps of the intermediate layers of SFasterNet. The top-left image of (a) serves as the input, and the entire image of (a) is the feature map of Stage 2. The row below the input image is the feature map of Stage 3, and the row below Stage 3 is the feature map of Stage 4. Figure 5 (b) in the diagram represents the feature map of an intermediate layer in ShuffleNetv2, which is a feature map of Stage 3. A comparison of the effects of SFasterNet using PGConv and ShuffleNetv2 using DWConv shows that DWConv disrupts communication between channels, preventing the extraction of finer features. Therefore, the two flexible transformations of PGConv can not only be used for channel alignment but also as a form of depthwise convolution. More importantly, they enable communication between channels, minimizing channel separation as much as possible. Without loss of generality, the input and output feature maps of this module have the same number of channels.

[0091] While replacing the conventional Conv with PConv ratio can effectively reduce FLOPs, our experimental analysis on the Pascal VOC 2007 object detection dataset revealed that the improvement in detection performance was not significant. Therefore, this invention redesigned the ratio factor value method, stipulating that the ratio factor should differ for different channels. Through ablation experiments, this invention found that when the input channel is c, and g... i When i∈[1, c] and i∈[1, n] are divisible by c, consider g i Let g be a valid ratio factor, where n represents the number of valid ratio factors. This invention demonstrates through experiments that the valid ratio factor should be g. (n+1) / 2-1 This achieves optimal performance, significantly improves detection efficiency, and substantially reduces FLOPs. Therefore, the FLOPs of the improved PConv in this invention are:

[0092]

[0093] in Therefore, the FLOPs of PGConv are:

[0094]

[0095] The detailed architecture of the ShiftMLP module of this invention is as follows: Figure 2 As shown, specifically, this module consists of four sequentially stacked components: a Shift operation module, a layer normalization module, an MLP network module, and a regularization module (DropPath module). The MLP network module includes a 1×1 convolution module, a Dropout module, another 1×1 convolution module, and a Dropout module arranged in sequence.

[0096] The shift operation has been well studied in CNNs. It can be chosen from many design options, such as active shifting and sparse shifting. In this work, the invention follows a partial shift operation from TSM, such as... Figure 6 As shown, given an input tensor, a small subset of channels will shift along four spatial directions: left, right, up, and down, while the remaining channels remain unchanged. Pixels outside the shifted range are simply discarded, and empty pixels are zero-filled. Therefore, the Shift operation does not involve any parameterization or arithmetic calculations. The only implementation is memory copying. Thus, the Shift operation is efficient, very easy to implement, and compared to self-attention mechanisms, it is cleaner and more compatible with deep learning inference libraries such as TensorRT. Furthermore, this invention sets the shift step size to 1 pixel.

[0097] Typically, we assume input features Where C is the number of channels, and H and W are the spatial height and width, respectively. Output features It has the same shape as the input. It can be written as:

[0098]

[0099]

[0100]

[0101]

[0102]

[0103] Here, γ is a ratio factor used to control how much of the channel will be shifted. In most experiments, the value of γ is set to...

[0104] The ShiftMLP of this invention relates to recent variants of multilayer perceptrons (MLPs). Specifically, these MLP variants propose pure MLP-like architectures for extracting image features. They also move beyond the attention-based framework in ViT. For example, instead of using a self-attention matrix, MLP-Mixer introduces a token-mixing MLP to directly connect all spatial locations. This eliminates the dynamic nature of ViT without sacrificing accuracy. Subsequent work has investigated further MLP designs, such as spatially gated units or cyclic connections.

[0105] The ShiftMLP module of this invention is a simple variant of the MLP, which replaces the linear layers with 1×1 convolutions, sacrificing a small number of FLOPs to improve accuracy. Compared with existing MLP work, the ShiftMLP of this invention is simpler and more efficient. Furthermore, due to the fixed linear weights, ordinary MLP variants cannot handle variable input sizes; the ShiftMLP of this invention overcomes this obstacle, thus making the model suitable for lightweight network structures.

[0106] This invention designs SFasterNetBlock using Channel Shuffle operations. Figure 1 (a) is based on the design principle. It is a residual block. In its residual branch, we replace depthwise convolution and regular convolution with the ShiftMLP module, and remove the final Channel Shuffle operation. The purpose of doing this is to improve accuracy to a certain extent, such as... Figure 2 As shown in (a). For Figure 1(b) In this invention, the depthwise convolution in the left branch is replaced with regular convolution and the PGConv module, and the depthwise convolution and regular convolution in the right branch are replaced with the PGConv and ShiftMLP modules. Finally, the results are concatenated, and a Channel Shuffle operation is performed to ultimately construct an SFasterBlock, as shown below. Figure 2 As shown in (b).

[0107] Based on SFasterBlock units, the overall SFasterNet architecture is presented in Table 2. The proposed network mainly consists of a set of SFasterBlock units divided into three stages. The first building block of each stage uses stride=2. Other hyperparameters within a stage remain unchanged, while the output channels of the next stage are doubled. This invention sets the number of bottleneck channels to 1 / 4 of the output channels of each SFasterBlock unit. The rest is similar to the ShuffleNetv2 network structure, where the number of channels in each block is scaled to generate networks of different complexities, denoted as...

[0108] 0.5x, 1.0x, etc., are shown in Table 2. The purpose of this invention is to provide a reference design that is as simple as possible, although we have found that further tuning of the hyperparameters may produce better results.

[0109] Table 2. Overall Structure of SFasterNet

[0110]

[0111] A second aspect of the present invention discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0112] A third aspect of the present invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0113] To address the issue of channels being unable to communicate with each other in depthwise separable convolution, this invention proposes PGConv, which can solve the problem of channels being unable to communicate with each other while significantly reducing the computational load.

[0114] Self-attention mechanisms can handle global features well, but they involve a large amount of computation and parameters, making it impossible for mobile devices to perform inference. To address this, this invention proposes ShiftMLP, which can replace the self-attention module and complete global feature processing with extremely low computation and parameter requirements.

[0115] SFasterNet employs a dual-branch structure, allowing for flexible and effective selection of different structural units, making it better suited for more complex environments.

[0116] This invention improves network performance with fewer parameters and fewer FLOPS. Furthermore, it redesigns the lightweight ViT model, SFasterViT, which learns global features with minimal parameters. Previous networks heavily relied on DWConv to reduce network parameters and FLOPS, resulting in extremely low accuracy. To address this issue, this invention designs SFasterConv, which effectively reduces network parameters and FLOPS without compromising performance, thereby improving accuracy.

[0117] To verify the effectiveness of the method of this invention, the experimental results of SFasterNet on different datasets will be described in detail below. More detailed ablation experiments were conducted on the Pascal VOC 2007 dataset, mainly focusing on the ablation experiments of the PGConv and ShiftMLP modules, and the parameter selection schemes of each module were analyzed through experiments. Simultaneously, we conducted detailed experimental comparisons with the latest methods on this dataset, analyzing the effectiveness of the model through different evaluation metrics. To further demonstrate the superiority of the model of this invention, we compared and analyzed the feature map information corresponding to each stage of SFasterNet with ShuffleNet. Application was performed on the Microsoft COCO dataset. We analyzed the current mainstream lightweight network models and used different evaluation metrics to judge the model's superiority. The training environment for this experiment was a 15vCPU Intel(R)Xeon(R)Platinum 8350C CPU@2.60GHz, NVIDIA RTX4090 (24G), 30GRAM, Python interpreter version 3.9, trained using CUDA 10.2 acceleration in PyTorch version 1.9.1. Meanwhile, we follow most of the training settings and hyperparameters used in "Aggregated residual transformations for deep neural networks".

[0118] The main evaluation indicators are as follows:

[0119] (1) mAP: The average value of various detection accuracies.

[0120] (2) AP 50 and AP 75 : represent IoU≥0.5 and IoU≥0.75 respectively, where IoU represents the ratio of the intersection and union between the ground truth bounding box and the predicted bounding box.

[0121] (3) AP s : Indicates that the pixel area is less than 32 2 The AP measurement value of the target box.

[0122] (4) AP m : Indicates a pixel area of ​​32 2 ~96 2 AP measurement values ​​of the target boxes between them.

[0123] (5) AP1: indicates that the pixel area is greater than 96 2 The AP measurement value of the target box.

[0124] (6) FLOPs: refers to floating-point operations, i.e. computational complexity, used to measure the complexity of an algorithm or model.

[0125] (7) Params: The number of parameters, which is related to the size of the model.

[0126] 1. Experimental results using the Pascal VOC 2007 dataset

[0127] This summary primarily evaluates the model of this invention on the Pascal VOC 2007 object detection dataset. This dataset contains approximately 5k training images and 5k test images for over 20 object categories. The SFasterNet network was trained for 600 epochs on an NVIDIA GPU using PyTorch, and experiments were conducted using image sizes of 416×416 and 320×320. To ensure fairness in the comparative experiments, the various network models were tested using the NanaoDet framework. The hyperparameters and learning rate of the AdamW optimizer for SFasterNet were set according to the specifications in "Shufflenet v2: Practical guidelines for efficient CNN architecture design". Our primary evaluation metric is the mean accuracy (mAP), as this is a practical indicator for object detection. Table 3 shows the detection results of SFasterNet for each category on the Pascal VOC 2007 dataset.

[0128] Table 3. Performance of SFasterNet on various categories of Pascal VOC 2007

[0129]

[0130] 1.1 Comparison with state-of-the-art methods

[0131] This invention will be compared with the following models:

[0132] (1)ShuffleNetv2: We will use the parameters set in the original paper and compare them with a model size of 1.5x.

[0133] (2) Mobilevitv3: We will use the parameters set in the original paper. Since the number of repetitions in each stage of ShuffleNetv2 is 4, 8, and 4, the FLOPs are much larger than we expected. Therefore, we set them to 2, 2, and 2 so that the FLOPs meet our expectations.

[0134] (3) GhostNet: We will use the parameters set in the original paper and compare them with a model of size midth_mult=1.0.

[0135] (4) Efficientnet: We will use the parameters set in the original paper and compare them with the efficientnet_lite4 size model.

[0136] Table 4 shows all the comparison results, and we analyze the results from different aspects.

[0137] Table 4. Comparison of different networks

[0138]

[0139] While EfficientNet boasts higher accuracy than other models, its FLOPs reach nearly 9G, which is unacceptable for mobile devices. GhostNet, with only 0.9G FLOPs, suffers from poor detection performance. MobileViTv3's accuracy is reduced due to its 2-2-2 repetition setting in each stage, yet even with this, its FLOPs still reach nearly 13G, and its parameter count reaches 22.8M, leading to slow inference when applied to mobile object detection. Considering these factors, ShuffleNetv2 extensively utilizes Conv and DWConv to reduce its FLOPs to 1.8G. Addressing these shortcomings, SFasterNet, as presented in this invention, excels in all aspects. By using PGConv instead of traditional Conv and DWConv, SFasterNet reduces both FLOPs and parameter count. Therefore, SFasterNet not only surpasses traditional models in computational cost and parameter count but also improves accuracy. This means sacrificing a significant number of FLOPs to improve accuracy, resulting in SFasterNet achieving improvements of over 1% across various metrics. When taking... At that time, the SFasterNet model reached its optimal performance, improving across all metrics by 0.3% to 4.5%, with only 1.4G of FLOPs and 1.3M of parameters. It also improved small-scale detection performance by 4.5%.

[0140] 1.2 Feature Map Analysis at Each Stage

[0141] SFasterNet improves performance by 4.5% on small targets, mainly due to the information in the feature maps extracted at different stages.

[0142] like Figure 7 As shown in (a), the feature map information extracted at each stage is displayed. It can be seen from the figure that the use of PGConv before the start of each stage has little impact on the subsequent stages and the middle part of each stage. Figure 7 As shown in (b), the feature maps are extracted at each stage of ShuffleNetv2. Because ShuffleNet uses a large amount of DWConv, communication between its channels is impossible, and targets with a pixel area smaller than 322 cannot extract more feature information in later stages. Therefore, later layers have a greater impact, for example... Figure 7 In stage 4 of (b), there is a lot of noise and the extractable features are rather blurry.

[0143] 1.3 PGConv Ablation Experiment

[0144] To demonstrate the effectiveness of the proposed PGConv, we will completely replace DWConv with PGConv in the ShuffleNetv2 network model to conduct ablation experiments on this module. Simultaneously, we will use a scaling factor g... (n+1) / 2-1 To conduct training. From formula (6), we know that when g... (n+1) / 2-1 When the value is too large, the number of FLOPs will also increase. (n+1) / 2-1 If the input size is too small, the PConv increases accordingly, as shown in Table 5, which illustrates the changes in parameters and computational complexity for different scaling factors with an input size of 416×416 and 100 channels. It can be seen that FLOPs are minimized when the scaling factor is taken as the median divisible by the number of channels.

[0145] Table 5. Situations corresponding to different proportional coefficients

[0146]

[0147] The experimental results are shown in Table 6. Replacing DWConv directly with PGConv only slightly increases FLOPs and the number of parameters, but all parameters increase directly, with mAP increasing from 40.9% to 41.2%. Figure 8 As shown in the image, a comparison of the effects of using DWConv and PGConv reveals that PGConv produces better results.

[0148] Table 6. Correspondence between DWConv and PGConv in ShuffleNetv2

[0149]

[0150] 1.4 ShiftMLP Ablation Experiment

[0151] Ablation experiments of the ShiftMLP module were conducted using the MobileViTv3 backbone network on NanoDet for comparative analysis. MobileViTv3 employs a linear complexity self-attention mechanism for global feature processing, namely the LinearTransformer. We replaced the LinearTransformer module with ShiftMLP, using the same parameters and repeating Stages 2, 2, and 2 (Stage 2 to Stage 3 were each repeated twice). As shown in Table 7, the computational complexity of the MobileViTv3 model using ShiftMLP instead of the LinearTransformer structure is only 3.5G, and the number of parameters is only 4.6M.

[0152] Table 7. Ablation experiments using Linear Transformer and ShiftMLP

[0153]

[0154] like Figure 9 The figures show a comparison of feature maps for ShiftMLP and Linear Transformer at various stages in MobileViTv3. When ShiftMLP is used instead of Linear Transformer, the feature information presented at each stage is roughly the same. However, ShiftMLP performs slightly worse than Linear Transformer in processing global feature information. But ShiftMLP has significantly fewer FLOPs and parameters than Linear Transformer, making it more suitable for lightweight network models.

[0155] 1.2 Experimental Results of COCO Dataset

[0156] We conducted experiments on the Microsoft COCO object detection dataset, which covers 80 object categories. We used 80k images as the training set, 40k images as the validation set, and 20k images as the test set. Similarly, we used mAP (IoU ∈ [0.5:0.95]) as the model evaluation metric. To better port the model to mobile platforms, we combined SFasterNet with NanoDet. The SFasterNet network was trained on an NVIDIA GPU using PyTorch for 300 epochs, and the experimental results are shown in Table 8.

[0157] We will compare it with the following model:

[0158] (1) YOLOv3-Tiny: Based on the experimental results in the paper, its backbone network uses a small DarkNet.

[0159] (2) YOLOv4-Tiny: The experimental results in the paper are adopted, and its backbone network uses CSPDarkNet.

[0160] (3) YOLOX-Nano: The experimental results in the paper are adopted, and its backbone network uses GhostNet.

[0161] (4) YOLOv5-n: The experimental results in the paper are adopted, and the backbone network is CSPDarkNet53.

[0162] (5) MobileDet: The experimental results in the paper are used, and its backbone network uses MobileNetv3.

[0163] (6)SSDLite: It adopts the experimental results in the MobileViTv1 paper, and its backbone network uses the MobileNet series.

[0164] (7) MobileViT-S: SSDLite uses MobileViT as the backbone network, and the experimental results are those in the paper.

[0165] (8) NanoDet: SFasterNet was used as the backbone network for training, and experiments were conducted using cp = c / 2 and cp = c / g(n+1) / 2-1 respectively.

[0166] Table 8. Comparison of each model on the COCO dataset

[0167]

[0168] By applying it to the COCO dataset, SFasterNet achieves real-time detection performance on NanoDet. Furthermore, when using SFasterNet as the backbone network, the model not only has lower parameter count and FLOPs, but also higher detection accuracy than other models. Therefore, SFasterNet can serve as a general and efficient backbone network.

[0169] This invention experimentally demonstrates that SFasterNet can improve accuracy by sacrificing a small number of FLOPs. Simultaneously, the proposed PGConv can effectively replace DWConv, enhancing communication between channels. Directly applying the ViT structure from large models to lightweight network structures not only results in FLOPs reaching levels unacceptable for mobile devices but also introduces a massive number of parameters that are inefficient for mobile device computing power. Therefore, the proposed ShiftMLP replaces the ViT model in lightweight network structures, effectively improving accuracy while reducing FLOPs and the number of parameters, making it perfectly suitable for mobile devices. SFasterNet of this invention can not only be widely applied in lightweight networks but also enable future mobile small target detection tasks.

Claims

1. A mobile terminal real-time target detection method, characterized in that, The method comprises the following steps: splitting the input feature map into two channels after extracting features from the input feature map, and obtaining a feature map after information interaction; the feature map after information interaction is divided into a first channel and a second channel, a total of two channels; local features in the first channel are extracted by using a PGConv module, and local features and global features in the second channel are extracted by using a PGConv module and a ShiftMLP module respectively; the local features in the first channel and the local features and global features in the second channel are spliced to obtain the features of the target; the PGConv module comprises a first convolution module and a group convolution module arranged in sequence; the ShiftMLP module comprises a Shift operation module and an MLP network module arranged in sequence; the first convolution module comprises a first channel division module, a 1x1 convolution module and a first splicing module; the first channel division module is used to divide the feature map after information interaction into a fifth channel and a sixth channel, a total of two channels; the sixth channel comprises a 1x1 convolution module; the first splicing module is used to splice the features extracted by the sixth channel with the features of the fifth channel; the group convolution module comprises a second channel division module, a 1x1 convolution module, a second splicing module and a 3x3 depth separable convolution module; the second channel division module is used to divide the features output by the first splicing module into a seventh channel and an eighth channel, a total of two channels; the eighth channel comprises a 1x1 convolution module; the second splicing module is used to splice the features extracted by the eighth channel with the features of the seventh channel; the 3x3 depth separable convolution module is connected with the second splicing module.

2. The method of claim 1 wherein, splitting the input feature map into two channels after extracting features from the input feature map, and obtaining a feature map after information interaction, specifically comprising: the input feature map is divided into a third channel and a fourth channel, a total of two channels; the fourth channel comprises a 1x1 convolution module and a ShiftMLP module arranged in sequence; the features extracted by the fourth channel are spliced with the features of the third channel to obtain the feature map after information interaction.

3. The method of claim 1 wherein, extracting local features in the first channel by using a PGConv module, specifically comprising: local features in the first channel are extracted by using a 1x1 convolution module, a 3x3 PGConv module and a 1x1 convolution module in sequence.

4. The method of claim 1 wherein, local features and global features in the second channel are extracted by using a PGConv module and a ShiftMLP module respectively, specifically comprising: local features and global features in the second channel are extracted by using a 3x3 PGConv module, a ShiftMLP module and a 1x1 convolution module in sequence respectively.

5. The method of claim 1 wherein, The ShiftMLP module further comprises a layer normalization module and a regularization module; the ShiftMLP module specifically comprises a Shift operation module, a normalization module, an MLP network module and a regularization module arranged in sequence.

6. The method of claim 5 wherein, the MLP network module comprises a 1x1 convolution module, a Dropout module, a 1x1 convolution module and a Dropout module arranged in sequence.

7. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 6.

Images