Lightweight service robot target detection method and device

Abstract

The invention relates to the technical field of robot target detection, in particular to a lightweight service robot target detection method and device.The lightweight service robot target detection method comprises the steps that a lightweight service robot detection model is constructed, and the lightweight service robot detection model is constructed on the basis of YOLOv5n; a MobileViT module, a C3DSC module and a convolution triple attention module are introduced into a backbone network, the MobileViT module, the C3DSC module and a deformable convolution module are introduced into a neck network, and then the lightweight service robot detection model is used for carrying out target detection on collected images of a service robot. The lightweight service robot detection model constructed by the invention effectively improves the detection performance of a multi-scale target, especially a deformed object, has relatively high detection precision while ensuring high real-time performance of the model, and is suitable for target detection requirements of the service robot.

Description

Technical Field

[0001] The present invention relates to the technical field of robot target detection, and in particular to a lightweight service robot target detection method and device. Background Art

[0002] In recent years, deep learning-based object detection methods have gradually become the main solution in the field of robotics and are also the basic perception capabilities required for robot intelligence. However, since robots use edge devices with limited resources, they need to reasonably utilize computing resources while pursuing high accuracy. In order to apply high-performance object detection algorithms to service robots and expand the application scope of deep neural network models, designing efficient and lightweight network models has become the focus of joint research between academia and industry.

[0003] At present, the research on lightweight networks is mainly based on the design of convolutional neural networks. YOLOv5 is a typical end-to-end network model, which has five versions, namely YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x. The overall architecture of different versions of YOLOv5 is exactly the same. The main difference is that each submodule uses different depths and widths. Among them, YOLOv5n has 1.78M parameters and 4.28GFLOPs of calculation, which is more suitable for service robots.

[0004] However, there are still some shortcomings in the application of YOLOv5n to service robots. First, the network uses ordinary convolution, which has insufficient detection performance for tubular deformed objects with large deformation (such as wires), and deformed objects are one of the important targets in service robot target detection. Secondly, the network lacks good global modeling capabilities, and the extensive use of C3 modules leads to slow network convergence. Thirdly, the network has many redundant features and lacks the ability to refine and interact with important features, and cannot efficiently extract and identify features with high correlation with targets. Finally, the detection head of the network has a contradiction between classification and regression, requires large parameters and computing resources, and is prone to overfitting, resulting in low detection efficiency of targets. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the problems in the prior art that YOLOv5n has a large number of parameters, low detection efficiency, and insufficient detection performance when applied to service robots.

[0006] In order to solve the above technical problems, the present invention provides a lightweight service robot target detection method, comprising: Replace the second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction with MobileViT modules, and replace the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction with a combined architecture consisting of a C3_DSC module and a convolutional triplet attention module; replace the second C3 module in the neck network of YOLOv5n along the forward propagation direction with a MobileViT module, and replace the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction with C3_DSC modules; replace the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction with a deformable convolution module; The C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence; The lightweight service robot detection model is used to perform target detection on the collected images of the service robot.

[0007] Preferably, the lightweight service robot detection model includes a backbone network, a neck network and a head network; The backbone network includes: a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolutional triple attention module and an SPPF layer connected in sequence; The neck network includes: a sixth CBS module, a first upsampling module, a second C3 module, a seventh CBS module, a second upsampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence; The head network includes: a first detection head, a second detection head and a third detection head.

[0008] Preferably, the collected image is input into the backbone network of the lightweight service robot detection model, and a multi-scale feature map is output, including: The collected image is sequentially input into the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module along the forward propagation direction, and the first feature map is output; The first feature map is sequentially input into the fourth CBS module and the second MobileViT module along the forward propagation direction, and the second feature map is output; The second feature map is input into the fifth CBS module, the first C3_DSC module, the convolutional triplet attention module and the SPPF layer in sequence along the forward propagation direction, and the third feature map is output.

[0009] Preferably, the multi-scale feature map output by the backbone network is input into the neck network, and the multi-scale enhanced feature map is output, including: The third feature map is input to the sixth CBS module, and a third fused feature map is output; The third fused feature map is upsampled and concatenated with the second feature map, and then sequentially input into the second C3 module and the seventh CBS module along the forward propagation direction to output the second fused feature map; The second fused feature map is upsampled and concatenated with the first feature map to output the first fused feature map; The first fused feature map is input into the second MobileViT module, and the first enhanced feature map is output; The first enhanced feature map is spliced ​​with the second fused feature map after passing through the eighth CBS module, and then input into the second C3_DSC module to output the second enhanced feature map; The second enhanced feature map is spliced ​​with the third fused feature map after passing through the deformable convolution module, and then input into the third C3_DSC module to output the third enhanced feature map.

[0010] Preferably, an efficient decoupling head is used as the detection head of the head network, and the efficient decoupling head is optimized, including: replacing the 3×3 convolutions in the classification branch and the regression branch of the efficient decoupling head with depthwise separable convolutions.

[0011] Preferably, the C3_DSC module comprises: The input features of the C3_DSC module are sequentially passed through the first dynamic snake convolution and the BottleNeck module to obtain the first intermediate features, and the input features are passed through the second dynamic snake convolution to obtain the second intermediate features; the first intermediate features and the second intermediate features are concatenated and input into the CBS module to obtain the output features of the C3_DSC module; The BottleNeck module includes a CBS module and a deformable convolution module connected in sequence. The output features of the deformable convolution module are added to the input features of the BottleNeck module to obtain the output features of the BottleNeck module.

[0012] Preferably, the CBS module comprises a convolutional layer, a batch normalization layer and a SiLU activation function connected in sequence; The deformable convolution module includes a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0013] Preferably, the MobileViT module is further optimized, and the optimized MobileViT module includes: The input features of the MobileViT module pass through the n×n hole convolution layer and the first 1×1 convolution layer in sequence to obtain the local feature map; the local feature map passes through the expansion layer, multiple Transformer blocks and the folding layer to obtain the global feature; the global feature passes through the second 1×1 convolution layer, concatenated with the input feature, and then input into the n×n convolution layer to obtain the output feature of the MobileViT module.

[0014] Preferably, before constructing the lightweight service robot detection model, a network pruning method based on gradient amplitude adaptation is used to preprocess YOLOv5n, including: Sort the gradient values ​​of each network layer of the trained YOLOv5n, and use the average value of the gradient values ​​of all network layers as the threshold, and optimize the network layers whose gradient values ​​are lower than the threshold, including: If the network layer to be optimized is a neck layer with shortcut connections, layer pruning is used to remove the network layer; if the network layer to be optimized is a channel layer, the network layer is directly compressed.

[0015] The present invention also provides a lightweight service robot target detection device, comprising: Model building module, used to build a lightweight service robot detection model based on YOLOv5n, including: The second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction are replaced with MobileViT modules, and the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction is replaced with a combined architecture consisting of a C3_DSC module and a convolutional triplet attention module; the second C3 module in the neck network of YOLOv5n along the forward propagation direction is replaced with a MobileViT module, and the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction are replaced with C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction is replaced with a deformable convolution module; the C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence; The detection module is used to perform target detection on the collected images of the service robot using the lightweight service robot detection model.

[0016] The above technical solution of the present invention has the following beneficial effects compared with the prior art: The lightweight service robot target detection method described in the present invention improves the existing YOLOv5n from the backbone network and the neck network to construct a lightweight service robot detection model. In the lightweight service robot detection model, the C3 module is improved by dynamic snake convolution to form a new C3_DSC module, which replaces part of the C3 module in YOLOv5n to improve the accuracy of the detection model in extracting features of tubular deformable objects. A hybrid architecture consisting of a lightweight MobileViT module and a convolutional triple attention module is further introduced into the backbone network, and the Transformer block in the MobileViT module is used to solve the problem of insufficient global modeling ability of the convolutional neural network. The convolutional triple attention module interacts the features of the backbone network in different dimensions, and significantly improves the feature extraction performance of the detection model without almost increasing parameters. In the neck network, the MobileViT module is also introduced, combined with the C3_DSC module and the deformable convolution module, to further strengthen the extraction of deformation features and global features in the feature fusion stage. The lightweight service robot detection model constructed by the present invention combines convolutional neural networks with Transformer, which effectively improves the detection performance of multi-scale targets, especially deformed objects. It has high detection accuracy while ensuring the high real-time performance of the model, and is suitable for the target detection needs of service robots.

[0017] Furthermore, the lightweight service robot detection model adopts an efficient decoupling head as the detection head to solve the contradiction between classification and regression, and uses deep separable convolution to optimize the efficient decoupling head, further reducing the computational complexity of the detection model and improving the efficiency of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 This is the overall structure diagram of YOLOv5; Figure 2 This is the structure diagram of YOLOViT_SR; Figure 3 This is the block diagram of the C3_DSC module; Figure 4 It is the structural diagram of the MobileViT module; Figure 5 It is the structural diagram of the CTAM module; Figure 6 It is a structural diagram of the improved EDH as the first detection head; Figure 7 This is a comparison chart of the PR curve of YOLOViT_SR and the PR curve of the original model YOLOv5n, where: Figure 7 (a) in the figure is the PR curve of YOLOv5n. Figure 7 (a) is the PR curve of YOLOViT_SR; Figure 8 It is a visualization diagram of the detection results of the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1

[0020] This embodiment takes YOLOv5 (You Only Look Once, YOLO) as an example to briefly explain the structure and feature extraction process of the lightweight model.

[0021] YOLOv5 has the characteristics of multi-task, end-to-end, attention mechanism and multi-scale. Multi-task can be understood as completing the classification and regression of the target at the same time, realizing parameter sharing and avoiding overfitting; end-to-end means that the network model automatically learns and obtains features after receiving image information, and directly gives classification and regression prediction information; the attention mechanism focuses on the features of the target area to be detected, so as to quickly obtain effective features; multi-scale usually fuses up-sampled and down-sampled data, so as to detect targets of multiple scales. In addition, YOLOv5 uses methods such as Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling at the input end to improve the training performance of the model.

[0022] The overall structure of YOLOv5 refers to Figure 1 As shown. Combined Figure 1 , the feature extraction process of YOLOv5 includes: When the input image size is 640×640×3, after filtering with a convolution filter with a step size of 2 (6×6×3×64, where the first two parameters constitute 6×6, which is the kernel size of the convolution filter, the third parameter 3 is the number of channels, and corresponds to the number of channels of the input image, and the fourth parameter 64 is the number of convolution filters, which corresponds to the number of channels of the output features), the image size becomes 320×320×64. Similarly, after filtering with a convolution filter (3×3×64×128) with a stride of 2, the image size becomes 160×160×128; in the further processing process, the C3 module is introduced (superimposed 3 times), and combined with the convolution layer with a stride of 2 to obtain the output features, the size of which is 80×80×256; after two similar C3 modules repeatedly extract features, combined with the fast pyramid pooling layer (Spatial Pyramid Pooling Fast, SPPF) to obtain an output layer of size 20×20×1024; the output of the backbone network is combined with the neck structure (FPN structure + PAN structure) to obtain one of the network outputs of 20×20×1024, which is mainly used for the detection of large-scale objects; in order to improve the detection ability of multi-scale objects, the same neck structure is used to obtain two other outputs of 80×80×256 and 40×40×512; finally, the head structure detects the multi-scale feature map output by the neck structure.

[0023] YOLOv5n is a version of YOLOv5 suitable for service robots. The number of parameters and computational complexity of its network structure can be further reduced. The detection performance of deformed objects needs to be improved. At the same time, it is necessary to accelerate the convergence speed of the network and the ability to refine important features.

[0024] In order to solve the above problems, the present invention proposes a lightweight service robot target detection method, comprising: A lightweight service robot detection model is built based on YOLOv5n and named YOLOViT_SR (YOLO with Vision Transformer for Service Robots), including: Replace the second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction with MobileViT modules, and replace the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction with a combined architecture consisting of a C3_DSC module and a Convolutional Triplet Attention Module (CTAM); Replace the second C3 module in the neck network of YOLOv5n along the forward propagation direction with a MobileViT module, and replace the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction with C3_DSC modules; Replace the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction with a deformable convolution (DBS) module; The lightweight service robot detection model is used to perform target detection on the collected images of the service robot.

[0025] YOLOViT_SR structure reference Figure 2 shown.

[0026] YOLOViT_SR includes backbone network, neck network and head network.

[0027] The backbone network includes: a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolutional triplet attention module and an SPPF layer connected in sequence.

[0028] The neck network includes: a sixth CBS module, a first upsampling module, a second C3 module, a seventh CBS module, a second upsampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence.

[0029] The head network includes: a first detection head, a second detection head and a third detection head.

[0030] The captured image is input into the backbone network of the lightweight service robot detection model, and a multi-scale feature map is output, including: The collected image is sequentially input into the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module along the forward propagation direction, and the first feature map is output; wherein the size of the collected image used in this embodiment is 640×640×3, and the size of the first feature map is 80×80×256; The first feature map is sequentially input into the fourth CBS module and the second MobileViT module along the forward propagation direction, and the second feature map is output with a size of 40×40×512; The second feature map is input to the fifth CBS module, the first C3_DSC module, the CTAM module and the SPPF layer in the forward propagation direction, and the third feature map is output with a size of 20×20×1024.

[0031] The multi-scale feature map output by the backbone network is input into the neck network, and a multi-scale enhanced feature map is output, including: The third feature map is input to the sixth CBS module, and the third fused feature map is output with a size of 20×20×512; The third fused feature map is upsampled and concatenated with the second feature map, and then sequentially input into the second C3 module and the seventh CBS module along the forward propagation direction, and outputs the second fused feature map with a size of 40×40×256; The second fused feature map is upsampled and concatenated with the first feature map to output the first fused feature map with a size of 80×80×512; The first fused feature map is input to the second MobileViT module, and the first enhanced feature map is output with a size of 80×80×256; The first enhanced feature map is concatenated with the second fused feature map after passing through the eighth CBS module, and then input into the second C3_DSC module to output the second enhanced feature map with a size of 40×40×512; The second enhanced feature map is spliced ​​with the third fused feature map after passing through the deformable convolution module, and then input into the third C3_DSC module to output the third enhanced feature map with a size of 20×20×1024.

[0032] The multi-scale enhanced feature map output by the neck network is input into the head network to output the target recognition results, including: The first enhanced feature map, the second enhanced feature map and the third enhanced feature map are respectively input to the first detection head, the second detection head and the third detection head to obtain the first target recognition result, the second target recognition result and the third target recognition result.

[0033] The following is a detailed introduction to the internal structure of YOLOViT_SR.

[0034] In order to effectively extract the features of tubular deformable objects with large deformation, the present invention introduces dynamic snake convolution (DSC) and combines it with the C3 module to form the C3_DSC module. The structure of the C3_DSC module is shown in Figure 2. Figure 3 shown.

[0035] The C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence.

[0036] The input features of the C3_DSC module pass through the first dynamic snake convolution and the BottleNeck module in sequence to obtain the first intermediate features, and the input features pass through the second dynamic snake convolution to obtain the second intermediate features. After the first intermediate features and the second intermediate features are spliced, they are input into the CBS module to obtain the output features of the C3_DSC module.

[0037] The BottleNeck module includes a CBS module and a deformable convolution module connected in sequence. The output features of the deformable convolution module are added to the input features of the BottleNeck module to obtain the output features of the BottleNeck module.

[0038] The CBS module includes a convolutional layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0039] The deformable convolution module includes a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0040] In order to combine the advantages of convolutional neural networks and Transformer and combine local modeling with global modeling, the present invention introduces the MobileViT module.

[0041] Preferably, in order to increase the receptive field of the MobileViT module for extracting local information, this embodiment optimizes the MobileViT module by replacing the n×n convolutional layer used for local information extraction with an n×n hole convolutional layer. Figure 4 shown.

[0042] The optimized MobileViT modules include: The input features of the MobileViT module pass through the n×n hole convolution layer and the first 1×1 convolution layer in sequence to obtain the local feature map; the local feature map passes through the expansion layer, multiple Transformer blocks and the folding layer to obtain the global feature; the global feature passes through the second 1×1 convolution layer, concatenated with the input feature, and then input into the n×n convolution layer to obtain the output feature of the MobileViT module.

[0043] In order to further enhance the network's feature extraction of deformable objects, the present invention introduces a deformable convolution module in the 22nd layer of YOLOViT_SR.

[0044] CTAM module structure reference Figure 5As shown in Figure 2, CTAM is a hybrid attention mechanism with interactive dimensions, which consists of three parallel branches. The first two branches are responsible for capturing the cross-dimensional interaction between the channel and the height or width of the space, and the third branch is used to build a spatial attention mechanism. Finally, the outputs of the three branches are aggregated by averaging.

[0045] The lightweight design of the CTAM module can be seamlessly integrated into the neural network architecture without increasing the number of network parameters and computational complexity. By capturing cross-dimensions to calculate the weights of the attention mechanism, cross-dimensional interaction of features is achieved, significantly improving the feature extraction performance of the lightweight service robot detection model with almost no increase in parameters.

[0046] In order to solve the contradiction between classification and regression, the present invention adopts an efficient decoupled head (EDH) as the detection head, and replaces the three detection heads of the original network respectively. And in order to further reduce the amount of calculation, the present invention improves the existing efficient decoupled head and replaces the 3×3 convolution with a depth-separable convolution in the classification branch and the regression branch.

[0047] Since the detection head structures are similar, this embodiment takes the first detection head as an example, and its structure is referenced Figure 6 As shown, including: The first enhanced feature map is input into the first detection head, and after 1×1 convolution, it is input into the classification branch and regression branch respectively. In the classification branch and regression branch, they undergo depthwise separable convolution and 1×1 convolution respectively. The classification branch outputs the target prediction probability, and the regression branch outputs the regression value and target confidence score.

[0048] Preferably, in order to further reduce the network parameter amount and calculation amount of YOLOv5n, before constructing the lightweight service robot detection model, this embodiment first uses a network pruning method based on gradient amplitude adaptation to preprocess YOLOv5n, including: Sort the gradient values ​​of each network layer of the trained YOLOv5n, and use the average value of the gradient values ​​of all network layers as the threshold, and optimize the network layers whose gradient values ​​are lower than the threshold, including: If the network layer to be optimized is a neck layer with a shortcut connection, layer pruning is used to remove the network layer to ensure that the shortcut and residual layer can be added; if the network layer to be optimized is a channel layer, the network layer is directly compressed, for example, the number of channels in a channel layer is compressed from 256 to 78.

[0049] In summary, the present invention improves the existing YOLOv5n from the backbone network and the neck network to construct a lightweight service robot detection model. In the lightweight service robot detection model, the C3 module is improved by dynamic snake convolution to form a new C3_DSC module, which replaces part of the C3 module in YOLOv5n to improve the accuracy of the detection model in extracting features of tubular deformable objects. A hybrid architecture consisting of a lightweight MobileViT module and a convolutional triple attention module is further introduced into the backbone network, and the Transformer block in the MobileViT module is used to solve the problem of insufficient global modeling ability of the convolutional neural network. The convolutional triple attention module interacts the features of the backbone network in different dimensions, and significantly improves the feature extraction performance of the detection model without almost increasing the parameters. In the neck network, the MobileViT module is also introduced, combined with the C3_DSC module and the deformable convolution module, to further strengthen the extraction of deformation features and global features in the feature fusion stage. The lightweight service robot detection model constructed by the present invention combines convolutional neural networks with Transformer, which effectively improves the detection performance of multi-scale targets, especially deformed objects. It has high detection accuracy while ensuring the high real-time performance of the model, and is suitable for the target detection needs of service robots.

[0050] Furthermore, the lightweight service robot detection model adopts an efficient decoupling head as the detection head to solve the contradiction between classification and regression, and uses deep separable convolution to optimize the efficient decoupling head, further reducing the computational complexity of the detection model and improving the efficiency of target detection. Embodiment 2

[0051] Based on the lightweight service robot target detection method described in Example 1, this embodiment provides a lightweight service robot target detection device, including: Model building module, used to build a lightweight service robot detection model based on YOLOv5n, including: The second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction are replaced with MobileViT modules, and the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction is replaced with a combined architecture consisting of a C3_DSC module and a convolutional triplet attention module; the second C3 module in the neck network of YOLOv5n along the forward propagation direction is replaced with a MobileViT module, and the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction are replaced with C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction is replaced with a deformable convolution module; the C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence; The detection module is used to perform target detection on the collected images of the service robot using the lightweight service robot detection model. Embodiment 3

[0052] In order to illustrate the effect of a lightweight service robot target detection method described in Example 1 of the present invention, this example provides specific experimental results of the improved strategy.

[0053] First, this embodiment tests the effectiveness of each part of YOLOViT_SR through ablation experiments. The main parameter settings in the experiment are as follows: load pre-trained weights, train 400 epochs, set the block size to 16, the input image size is 640×640×3, the momentum is set to 0.9, the weight decay regularization coefficient is set to 0.0005, and the initial learning rate is set to 0.001. The data set used in the experiment is a self-collected benchmark data set for sweeping robots, totaling 8100 images, of which the training set accounts for 80%, totaling 6480 images, and the test set accounts for 20%, totaling 1620 images.

[0054] The specific implementation steps and results of the C3_DSC module ablation experiment include: 1. In the backbone network, the C3 module is replaced with the C3_DSC module using two methods. Replacing the 6th and 8th layers at the same time has the best effect, with the accuracy increased from 75.8% to 79.2%, the number of model parameters reduced by 0.02M, and the inference time almost unchanged.

[0055] 2. In the neck network, the C3 module is replaced with the C3_DSC module using two methods. Replacing the 21st and 24th layers at the same time has the best effect, with the accuracy increased from 75.8% to 79.0%, while the number of model parameters decreased by 0.01M and the inference time remained almost unchanged.

[0056] 3. By jointly replacing the backbone network with the neck network, the number of model parameters decreased by 0.02M, the accuracy was greatly improved by 4.7%, and the inference time only increased by 0.8ms.

[0057] The results of the C3_DSC module ablation experiment are shown in Table 1.

[0058] Table 1. C3_DSC module ablation experiment results Model Replace C3 layers mAP(%) #Params(M) GFLOPs Latency(ms) YOLOv5n - 75.8 0.45 1.2 1.5 8 77.4 0.44 1.1 1.5 Backbone network 6+8 79.2 0.43 1.1 2.0 Neck network 2421+24 77.479.0 0.440.44 1.11.1 1.51.7 Backbone + Neck Four floors 80.5 0.43 1.1 2.2 According to the ablation experiment results of the C3_DSC module and the MobileViT module, this embodiment further integrates the two modules, and finally obtains the model that can achieve the best balance between accuracy and speed when the C3 modules of the 4th, 6th, and 18th layers are replaced with the MobileViT modules, and the C3 modules of the 8th, 21st, and 24th layers are replaced with the C3_DSC modules. On this basis, the CTAM module and the improved EDH are added to form the final YOLOViT_SR network.

[0059] The ablation experiment results of YOLOViT_SR are shown in Table 2.

[0060] Table 2. Ablation results of YOLOViT_SR Model mAP(%) #Params GFLOPs Latency(ms) FPS YOLOv5n (before pruning) 76.8 1.78M 4.28 1.1 909 YOLOv5n (after pruning) 75.8 0.45M 1.2 1.5 652 +C3_DSC 80.1 0.37M 1.2 2.3 441 +MobileViT 84.2 0.94M 1.2 3.7 273 +EDH 84.7 1.4M 5.4 4.3 234 +CTAM 85.0 1.4M 5.5 4.5 223 As can be seen from Table 2, after pruning, the number of parameters of YOLOv5n decreased by 1.33M, the amount of calculation decreased by 3.08G, and the accuracy remained almost unchanged. On the basis of pruning, adding the C3_DSC module increased the accuracy to 80.1%, and adding the MobileViT module further increased the accuracy to 84.2%. Adding the CBAM module made the model accuracy reach 84.7%, and finally adding the improved EDH made the model accuracy reach 85.0%. Although the number of parameters and the amount of calculation of the model increased slightly, the accuracy increased by nearly 10%, and the speed remained at 143FPS, which fully met the real-time requirements of the service robot.

[0061] The comparative experimental results of YOLOViT_SR and other algorithms are shown in Table 3.

[0062] Table 3. Comparative experimental results of YOLOViT_SR and other algorithms Model mAP(%) #Params GFLOPs Latency(ms) FPS YOLOv5n (after pruning) 75.8 0.45M 1.2 2.3 435 YOLOv4-tiny 73.7 5.9M 16.2 1.3 743 YOLOX-tiny 89.8 5.04M 15.2 2.5 394 RT-DETR-R18 88.0 20.1M 60.2 6.2 161 YOLOViT_SR 85.0 1.4M 5.5 4.5 223 As can be seen from Table 3, YOLOViT_SR is compared with the current mainstream lightweight model. Compared with YOLOX-tiny, the number of parameters of YOLOViT_SR is about 1 / 4, the amount of calculation is about 1 / 3, and the accuracy difference is less than 5%; compared with RT-DETR-R18, the number of parameters and the amount of calculation of YOLOViT_SR are more obvious, which fully reflects the high efficiency of the present invention and achieves a better balance between accuracy and speed.

[0063] To further illustrate the effectiveness of the present invention, Figure 7 This is a comparison chart of the PR curve of YOLOViT_SR and the PR curve of the original model YOLOv5n, where Figure 7 (a) in the figure is the PR curve of YOLOv5n. Figure 7 (a) is the PR curve of YOLOViT_SR; Figure 8 This is a visualization diagram of the detection results of the method of the present invention.

[0064] Depend on Figure 7 It can be seen that the YOLOViT_SR proposed in the present invention not only improves the overall accuracy (mAP), but also the recognition accuracy of most objects is above 80%. Among them, wires are objects with large deformation, and the accuracy is improved from 56.2% to 70.3%. This shows the effectiveness of the C3_DSC module constructed in the present invention.

[0065] Figure 8 The figure shows the detection performance of the service robot in actual scenes. The bounding boxes of different colors in the figure indicate that objects of different categories have been identified. The numbers above the bounding boxes indicate the confidence level of the detection. The higher the confidence level, the greater the probability that it is a positive sample. Figure 8 It can be seen that the confidence of a single object in different scenes is very high, basically exceeding 0.5, and some objects can reach 0.9, such as the stool in the third row and fourth column. It can be seen that the YOLOViT_SR proposed in the present invention is fully suitable for target recognition of server robots.

[0066] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0067] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A lightweight service robot target detection method, characterized in that: include: A lightweight service robot detection model is built based on YOLOv5n, including: Replace the second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction with MobileViT modules, and replace the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction with a combined architecture consisting of a C3_DSC module and a convolutional triplet attention module; replace the second C3 module in the neck network of YOLOv5n along the forward propagation direction with a MobileViT module, and replace the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction with C3_DSC modules; replace the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction with a deformable convolution module; The C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence; The lightweight service robot detection model is used to perform target detection on the collected images of the service robot.

2. A lightweight service robot target detection method according to claim 1, characterized in that: The lightweight service robot detection model includes a backbone network, a neck network and a head network; The backbone network includes: a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolutional triple attention module and an SPPF layer connected in sequence; The neck network includes: a sixth CBS module, a first upsampling module, a second C3 module, a seventh CBS module, a second upsampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence; The head network includes: a first detection head, a second detection head and a third detection head.

3. A lightweight service robot target detection method according to claim 2, characterized in that: The captured image is input into the backbone network of the lightweight service robot detection model, and a multi-scale feature map is output, including: The collected image is sequentially input into the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module along the forward propagation direction, and the first feature map is output; The first feature map is sequentially input into the fourth CBS module and the second MobileViT module along the forward propagation direction, and the second feature map is output; The second feature map is input into the fifth CBS module, the first C3_DSC module, the convolutional triplet attention module and the SPPF layer in sequence along the forward propagation direction, and the third feature map is output.

4. A lightweight service robot target detection method according to claim 3, characterized in that: The multi-scale feature map output by the backbone network is input into the neck network, and a multi-scale enhanced feature map is output, including: The third feature map is input to the sixth CBS module, and a third fused feature map is output; The third fused feature map is upsampled and concatenated with the second feature map, and then sequentially input into the second C3 module and the seventh CBS module along the forward propagation direction to output the second fused feature map; The second fused feature map is upsampled and concatenated with the first feature map to output the first fused feature map; The first fused feature map is input into the second MobileViT module, and the first enhanced feature map is output; The first enhanced feature map is spliced ​​with the second fused feature map after passing through the eighth CBS module, and then input into the second C3_DSC module to output the second enhanced feature map; The second enhanced feature map is spliced ​​with the third fused feature map after passing through the deformable convolution module, and then input into the third C3_DSC module to output the third enhanced feature map.

5. A lightweight service robot target detection method according to claim 2, characterized in that: An efficient decoupling head is used as the detection head of the head network, and the efficient decoupling head is optimized, including: replacing the 3×3 convolutions in the classification branch and regression branch of the efficient decoupling head with depthwise separable convolutions.

6. A lightweight service robot target detection method according to claim 1, characterized in that: The C3_DSC module includes: The input features of the C3_DSC module are sequentially passed through the first dynamic snake convolution and the BottleNeck module to obtain the first intermediate features, and the input features are passed through the second dynamic snake convolution to obtain the second intermediate features; the first intermediate features and the second intermediate features are concatenated and input into the CBS module to obtain the output features of the C3_DSC module; The BottleNeck module includes a CBS module and a deformable convolution module connected in sequence. The output features of the deformable convolution module are added to the input features of the BottleNeck module to obtain the output features of the BottleNeck module.

7. A lightweight service robot target detection method according to claim 1, characterized in that: The CBS module includes a convolutional layer, a batch normalization layer and a SiLU activation function connected in sequence; The deformable convolution module includes a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

8. A lightweight service robot target detection method according to claim 1, characterized in that: The MobileViT module is further optimized, and the optimized MobileViT module includes: The input features of the MobileViT module pass through the n×n hole convolution layer and the first 1×1 convolution layer in sequence to obtain the local feature map; the local feature map passes through the expansion layer, multiple Transformer blocks and the folding layer to obtain the global feature; the global feature passes through the second 1×1 convolution layer, concatenated with the input feature, and then input into the n×n convolution layer to obtain the output feature of the MobileViT module.

9. A lightweight service robot target detection method according to claim 1, characterized in that: Before building the lightweight service robot detection model, a network pruning method based on gradient amplitude adaptation is used to preprocess YOLOv5n, including: Sort the gradient values ​​of each network layer of the trained YOLOv5n, and use the average value of the gradient values ​​of all network layers as the threshold, and optimize the network layers whose gradient values ​​are lower than the threshold, including: If the network layer to be optimized is a neck layer with shortcut connections, layer pruning is used to remove the network layer; if the network layer to be optimized is a channel layer, the network layer is directly compressed.

10. A lightweight service robot target detection device, characterized in that: include: Model building module, used to build a lightweight service robot detection model based on YOLOv5n, including: The second and third C3 modules in the backbone network of YOLOv5n along the forward propagation direction are replaced with MobileViT modules, and the fourth C3 module in the backbone network of YOLOv5n along the forward propagation direction is replaced with a combined architecture consisting of a C3_DSC module and a convolutional triplet attention module; the second C3 module in the neck network of YOLOv5n along the forward propagation direction is replaced with a MobileViT module, and the third and fourth C3 modules in the neck network of YOLOv5n along the forward propagation direction are replaced with C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n along the forward propagation direction is replaced with a deformable convolution module; the C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module includes a CBS module and a deformable convolution module connected in sequence; The detection module is used to perform target detection on the collected images of the service robot using the lightweight service robot detection model.

Images