Lightweight service robot target detection method and device

Abstract

The present application relates to the technical field of robot target detection, in particular to a lightweight service robot target detection method and device, comprising: constructing a lightweight service robot detection model, the lightweight service robot detection model is based on YOLOv5n, a MobileViT module is introduced in the backbone network, and a combination architecture composed of a C3_DSC module and a convolution triple attention module, a MobileViT module, a C3_DSC module and a deformable convolution module are introduced in the neck network, and then the lightweight service robot detection model is used for target detection on the collected images of the service robot. The lightweight service robot detection model constructed by the present application effectively improves the detection performance of multi-scale targets, especially deformed objects, has high detection accuracy while ensuring high real-time performance of the model, and is suitable for the target detection needs of service robots.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot target detection, in particular to a lightweight service robot target detection method and device. BACKGROUND

[0002] In recent years, target detection methods based on deep learning have gradually become the main solution in the field of robots and are also the basic perception ability required for robot intelligence. However, since robots use edge devices with limited resources, it is necessary to reasonably use computing resources while pursuing high precision. In order to apply high-performance target detection algorithms to service robots and expand the application range of deep neural network models, designing efficient lightweight network models has become a common research focus of academia and industry.

[0003] Current lightweight network research is mainly based on the design of convolutional neural networks. YOLOv5 is a typical end-to-end network model, which mainly has five versions, namely YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x. The overall architecture of different versions of YOLOv5 is completely consistent, and the main difference is the use of different depths and widths in each sub-module. Among them, the parameter amount of YOLOv5n is 1.78M, and the calculation amount is 4.28GFLOPs, which is more suitable for application in service robots.

[0004] However, there are still some deficiencies in the application of YOLOv5n in service robots. First, the network uses ordinary convolution, which has insufficient detection performance for tubular deformation objects with large deformation (such as power lines), and deformation objects are one of the important targets in the target detection of service robots. Second, the network lacks good global modeling capability, and the extensive use of C3 modules leads to slow network convergence. Third, the network has more redundant features, lacks the ability to refine and interact important features, and cannot efficiently extract and identify features with high target correlation. Finally, the detection head of the network has contradictions between classification and regression, which requires large parameters and computing resources, is prone to overfitting, and results in low detection efficiency of the target. SUMMARY

[0005] Therefore, the technical problem to be solved by the present application is to overcome the problems of large parameter amount, low detection efficiency and insufficient detection performance of YOLOv5n in service robots in the prior art.

[0006] To solve the above technical problems, the present application provides a lightweight service robot target detection method, comprising:

[0007] The second and third C3 modules in the backbone network of YOLOv5n along the positive propagation direction are replaced by MobileViT modules, the fourth C3 module in the backbone network of YOLOv5n along the positive propagation direction is replaced by a combined architecture composed of a C3_DSC module and a convolution triple attention module; the second C3 module in the neck network of YOLOv5n along the positive propagation direction is replaced by a MobileViT module, the third and fourth C3 modules in the neck network of YOLOv5n along the positive propagation direction are replaced by C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n along the positive propagation direction is replaced by a deformable convolution module;

[0008] The C3_DSC module comprises a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence;

[0009] The light service robot detection model is used for target detection on the collected image of the service robot.

[0010] Preferably, the light service robot detection model comprises a backbone network, a neck network and a head network.

[0011] The backbone network comprises a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolution triple attention module and an SPPF layer connected in sequence.

[0012] The neck network comprises a sixth CBS module, a first up-sampling module, a second C3 module, a seventh CBS module, a second up-sampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence.

[0013] The head network comprises a first detection head, a second detection head and a third detection head.

[0014] Preferably, the collected image is input into the backbone network of the light service robot detection model, and a multi-scale feature map is output, comprising:

[0015] The collected image is input into the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module along the positive propagation direction in sequence, and a first feature map is output.

[0016] The first feature map is sequentially input to the fourth CBS module and the second MobileViT module in the positive propagation direction, and a second feature map is output;

[0017] The second feature map is sequentially input to the fifth CBS module, the first C3_DSC module, the convolution triple attention module and the SPPF layer in the positive propagation direction, and a third feature map is output.

[0018] Preferably, the multi-scale feature map output by the backbone network is input to the neck network, and a multi-scale enhanced feature map is output, comprising:

[0019] The third feature map is input to the sixth CBS module, and a third fusion feature map is output;

[0020] The third fusion feature map is spliced with the second feature map after upsampling, and is sequentially input to the second C3 module and the seventh CBS module in the positive propagation direction, and a second fusion feature map is output;

[0021] The second fusion feature map is spliced with the first feature map after upsampling, and a first fusion feature map is output;

[0022] The first fusion feature map is input to the second MobileViT module, and a first enhanced feature map is output;

[0023] The first enhanced feature map is spliced with the second fusion feature map after passing through the eighth CBS module, and is input to the second C3_DSC module, and a second enhanced feature map is output;

[0024] The second enhanced feature map is spliced with the third fusion feature map after passing through the deformable convolution module, and is input to the third C3_DSC module, and a third enhanced feature map is output.

[0025] Preferably, the efficient decoupling head is used as the detection head of the head network, and the efficient decoupling head is optimized, comprising: replacing the 3x3 convolution in the classification branch and the regression branch of the efficient decoupling head with a depth separable convolution.

[0026] Preferably, the C3_DSC module comprises:

[0027] The input feature of the C3_DSC module sequentially passes through the first dynamic snake convolution and the BottleNeck module to obtain a first intermediate feature, and the input feature passes through the second dynamic snake convolution to obtain a second intermediate feature; after splicing the first intermediate feature and the second intermediate feature, the input is input to the CBS module to obtain the output feature of the C3_DSC module;

[0028] The BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence, and the output feature of the deformable convolution module is added to the input feature of the BottleNeck module to obtain the output feature of the BottleNeck module.

[0029] Preferably, the CBS module comprises a convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0030] The deformable convolution module comprises a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0031] Preferably, the MobileViT module is further optimized, and the optimized MobileViT module comprises:

[0032] The input feature of the MobileViT module passes through an n*n dilated convolution layer and a first 1*1 convolution layer in sequence to obtain a local feature map; the local feature map passes through an unfolding layer, a plurality of Transformer blocks and a folding layer to obtain a global feature; the global feature is spliced with the input feature after passing through a second 1*1 convolution layer, and then input to an n*n convolution layer to obtain the output feature of the MobileViT module.

[0033] Preferably, before constructing the lightweight service robot detection model, a network pruning method based on gradient amplitude self-adaption is used to pre-process the YOLOv5n, comprising:

[0034] The gradient values of each network layer of the trained YOLOv5n are sorted, and the average value of the gradient values of all network layers is taken as a threshold value, and the network layer with a gradient value lower than the threshold value is optimized, comprising:

[0035] If the network layer to be optimized is a neck layer with a shortcut connection, the layer pruning is used to remove the network layer; if the network layer to be optimized is a channel layer, the network layer is directly compressed.

[0036] The application also provides a lightweight service robot target detection device, comprising:

[0037] A model construction module is configured to construct a lightweight service robot detection model based on the YOLOv5n, comprising:

[0038] The second and third C3 modules in the backbone network of YOLOv5n in the positive propagation direction are replaced by MobileViT modules, the fourth C3 module in the backbone network of YOLOv5n in the positive propagation direction is replaced by a combined architecture composed of a C3_DSC module and a convolution triple attention module; the second C3 module in the neck network of YOLOv5n in the positive propagation direction is replaced by a MobileViT module, the third and fourth C3 modules in the neck network of YOLOv5n in the positive propagation direction are replaced by C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n in the positive propagation direction is replaced by a deformable convolution module; the C3_DSC module comprises a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence;

[0039] The detection module is used for target detection on the collected image of the service robot by using the lightweight service robot detection model.

[0040] The above technical scheme of the present application has the following beneficial effects compared with the prior art:

[0041] The lightweight service robot target detection method provided by the present application improves the existing YOLOv5n from the backbone network and the neck network, and constructs a lightweight service robot detection model. In the lightweight service robot detection model, the C3 module is improved by using a dynamic snake convolution to form a new C3_DSC module, which replaces part of the C3 modules in YOLOv5n, so as to improve the accuracy of feature extraction of tubular deformation objects by the detection model. In the backbone network, a hybrid architecture composed of a lightweight MobileViT module and a convolution triple attention module is further introduced, the Transformer block in the MobileViT module is used to solve the problem of insufficient global modeling ability of the convolutional neural network, and the convolution triple attention module interacts the features of part of the backbone network in different dimensions, which significantly improves the feature extraction performance of the detection model without increasing the parameters. In the neck network, a MobileViT module is also introduced, combined with a C3_DSC module and a deformable convolution module, to further strengthen the extraction of deformation features and global features in the feature fusion stage. The lightweight service robot detection model constructed by the present application combines the convolutional neural network with the Transformer, effectively improves the detection performance of multi-scale targets, especially deformation objects, has high detection accuracy while ensuring high real-time performance of the model, and is suitable for the target detection requirements of service robots.

[0042] Further, the lightweight service robot detection model adopts an efficient decoupling head as a detection head to solve the contradiction between classification and regression, and optimizes the efficient decoupling head by using a depth separable convolution, thereby further reducing the calculation amount of the detection model and improving the efficiency of target detection. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the drawings, in which:

[0044] Figure 1 is the overall structure diagram of YOLOv5;

[0045] Figure 2 is the structure diagram of YOLOViT_SR;

[0046] Figure 3 is the structure diagram of C3_DSC module;

[0047] Figure 4 is the structure diagram of MobileViT module;

[0048] Figure 5 is the structure diagram of CTAM module;

[0049] Figure 6 is the structure diagram of the improved EDH as the first detection head;

[0050] Figure 7 is the PR curve comparison diagram of YOLOViT_SR and the original model YOLOv5n, wherein, Figure 7 (a) in is the PR curve diagram of YOLOv5n, Figure 7 (a) in is the PR curve diagram of YOLOViT_SR;

[0051] Figure 8 is the detection result visualization diagram of the present application. DETAILED DESCRIPTION

[0052] The present application will be further described below in conjunction with the drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting the present application. Embodiment one

[0053] This embodiment takes YOLOv5 (You Only Look Once, YOLO) as an example to briefly describe the structure and feature extraction process of the lightweight model.

[0054] YOLOv5 has the characteristics of multi-task, end-to-end, attention mechanism and multi-scale. Multi-task can be understood as completing the classification and regression of the target at the same time, realizing parameter sharing and avoiding overfitting. End-to-end means that the network model automatically learns and obtains features after accepting image information, and directly gives the prediction information of classification and regression. Attention mechanism focuses on the target area features to be detected, so as to quickly obtain effective features. Multi-scale usually fuses the up-sampled and down-sampled data, so as to detect multi-scale targets. In addition, YOLOv5 uses Mosaic data enhancement, adaptive anchor box calculation, adaptive picture scaling and other methods at the input end to improve the training performance of the model.

[0055] The overall structure of YOLOv5 is shown in Figure 1 , the feature extraction process of YOLOv5 includes: Figure 1

[0056] When the input picture size is 640x640x3, the picture size becomes 320x320x64 after filtering by a convolution filter with a step of 2 (6x6x3x64, where the first two parameters constitute 6x6, which is the kernel size of the convolution filter, the third parameter 3 is the number of channel quantities, and corresponds to the number of channel quantities of the input picture, and the fourth parameter 64 is the number of convolution filters, which corresponds to the number of channel quantities of the output feature). By analogy, after filtering by a convolution filter with a step of 2 (3x3x64x128), the picture size becomes 160x160x128; in the further processing process, a C3 module (stacked 3 times) is introduced, and the output feature is obtained by combining the convolution layer with a step of 2, the size is 80x80x256; after repeatedly extracting features twice by similar C3 modules, the size of the output layer obtained by combining the Spatial Pyramid Pooling Fast (SPPF) is 20x20x1024; the output of the backbone network combined with the neck structure (FPN structure+PAN structure) is 20x20x1024, which is mainly used for detecting large-scale objects; in order to improve the detection ability of multi-scale objects, 80x80x256 and 40x40x512 are obtained by the same neck structure respectively; finally, the head structure detects the multi-scale feature maps output by the neck structure.

[0057] YOLOv5n is a version of YOLOv5 suitable for service robots, and the parameter quantity and calculation quantity of the network structure can be further reduced. The detection performance of the deformed object needs to be improved, and the convergence speed of the network and the refinement ability of important features need to be accelerated.

[0058] ​To solve the above problems, the application provides a lightweight service robot target detection method, comprising:

[0059] A lightweight service robot detection model is constructed based on YOLOv5n and named YOLOViT_SR (YOLO with Vision Transformer for Service Robots), which comprises:

[0060] The second and third C3 modules in the backbone network of YOLOv5n along the positive propagation direction are replaced by MobileViT modules, the fourth C3 module in the backbone network of YOLOv5n along the positive propagation direction is replaced by a combination architecture composed of a C3_DSC module and a convolutional triplet attention module (CTAM), the second C3 module in the neck network of YOLOv5n along the positive propagation direction is replaced by a MobileViT module, the third and fourth C3 modules in the neck network of YOLOv5n along the positive propagation direction are replaced by C3_DSC modules, and the fourth CBS module in the neck network of YOLOv5n along the positive propagation direction is replaced by a deformable convolution (DBS) module.

[0061] The collected image of the service robot is subjected to target detection by using the lightweight service robot detection model.

[0062] The structure of YOLOViT_SR is shown in Figure 2 .

[0063] YOLOViT_SR comprises a backbone network, a neck network and a head network.

[0064] The backbone network comprises a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolutional triplet attention module and an SPPF layer connected in sequence.

[0065] The neck network comprises a sixth CBS module, a first up-sampling module, a second C3 module, a seventh CBS module, a second up-sampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence.

[0066] The head network comprises a first detection head, a second detection head and a third detection head.

[0067] The image is input to the backbone network of the lightweight service robot detection model, and a multi-scale feature map is output, including:

[0068] The image is input to the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module in sequence along the positive propagation direction, and a first feature map is output; wherein the size of the image used in this embodiment is 640x640x3, and the size of the first feature map is 80x80x256;

[0069] The first feature map is input to the fourth CBS module and the second MobileViT module in sequence along the positive propagation direction, and a second feature map is output, with a size of 40x40x512;

[0070] The second feature map is input to the fifth CBS module, the first C3_DSC module, the CTAM module and the SPPF layer in sequence along the positive propagation direction, and a third feature map is output, with a size of 20x20x1024.

[0071] The multi-scale feature map output by the backbone network is input to the neck network, and a multi-scale enhanced feature map is output, including:

[0072] The third feature map is input to the sixth CBS module, and a third fusion feature map is output, with a size of 20x20x512;

[0073] The third fusion feature map is spliced with the second feature map after upsampling, and is input to the second C3 module and the seventh CBS module in sequence along the positive propagation direction, and a second fusion feature map is output, with a size of 40x40x256;

[0074] The second fusion feature map is spliced with the first feature map after upsampling, and a first fusion feature map is output, with a size of 80x80x512;

[0075] The first fusion feature map is input to the second MobileViT module, and a first enhanced feature map is output, with a size of 80x80x256;

[0076] The first enhanced feature map is spliced with the second fusion feature map after passing through the eighth CBS module, and is input to the second C3_DSC module, and a second enhanced feature map is output, with a size of 40x40x512;

[0077] The second enhanced feature map is spliced with the third fusion feature map after passing through the deformable convolution module, and is input to the third C3_DSC module, and a third enhanced feature map is output, with a size of 20x20x1024.

[0078] The multi-scale enhanced feature map output by the neck network is input to the head network, and a target recognition result is output, including:

[0079] The first, second and third enhanced feature maps are input into the first, second and third detection heads respectively to obtain the first, second and third target recognition results.

[0080] The internal structure of YOLOViT_SR is described in detail below.

[0081] In order to effectively extract the features of a tubular deformation object with large deformation, the application introduces a dynamic snake convolution (DSC) and combines it with a C3 module to form a C3_DSC module. Figure 3 The structure of the C3_DSC module is shown in

[0082] The C3_DSC module includes a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module.

[0083] The input features of the C3_DSC module are sequentially input into the first dynamic snake convolution and the BottleNeck module to obtain first intermediate features, and the input features are input into the second dynamic snake convolution to obtain second intermediate features.

[0084] The BottleNeck module includes a CBS module and a deformable convolution module connected in sequence.

[0085] The CBS module includes a convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0086] The deformable convolution module includes a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence.

[0087] In order to combine the advantages of convolutional neural networks and Transformers and combine local modeling and global modeling, the application introduces a MobileViT module.

[0088] Preferably, in order to increase the receptive field of the MobileViT module for extracting local information, the MobileViT module is optimized in this embodiment, and the n*n convolution layer for local information extraction is replaced by an n*n dilated convolution layer. Figure 4 The structure of the MobileViT module is shown in

[0089] The optimized MobileViT module comprises:

[0090] The input feature of the MobileViT module sequentially passes through an n*n cavity convolution layer and a first 1*1 convolution layer to obtain a local feature map; the local feature map passes through an unfolding layer, a plurality of Transformer blocks and a folding layer to obtain a global feature; the global feature is spliced with the input feature after passing through a second 1*1 convolution layer, and then input to an n*n convolution layer to obtain the output feature of the MobileViT module.

[0091] In order to further enhance the feature extraction of the network on the deformed object, the present application introduces a variability convolution module at the 22nd layer of YOLOViT_SR.

[0092] The structure of the CTAM module is shown in Figure 5 CTAM is a hybrid attention mechanism with cross-dimensional interaction, which is composed of three parallel branches. The first two branches are responsible for capturing cross-dimensional interaction between channels and height or width of space, and the third branch is used to build a spatial attention mechanism. The outputs of the three branches are aggregated by averaging.

[0093] The lightweight design of the CTAM module can be seamlessly integrated into the neural network architecture, almost without increasing the parameter quantity and computational quantity of the network. By capturing cross-dimensional interaction to calculate the weight of the attention mechanism, the cross-dimensional interaction of the features is realized, and the feature extraction performance of the lightweight service robot detection model is significantly improved without increasing the parameters.

[0094] In order to solve the contradiction between classification and regression, the present application adopts an efficient decoupled head (EDH) as a detection head, which replaces the three detection heads of the original network. And in order to further reduce the computational complexity, the present application improves the existing efficient decoupled head, replacing 3*3 convolution with depth separable convolution in the classification branch and the regression branch.

[0095] Since the structure of the detection head is similar, the first detection head is taken as an example in the embodiment, and the structure thereof is shown in Figure 6 , which comprises:

[0096] The first enhanced feature map is input to the first detection head, and after 1*1 convolution, it is input to the classification branch and the regression branch, respectively. In the classification branch and the regression branch, depth separable convolution and 1*1 convolution are sequentially performed, respectively. The classification branch outputs target prediction probability, and the regression branch outputs regression value and target confidence score.

[0097] Preferably, in order to further reduce the network parameter quantity and the calculation quantity of YOLOv5n, before constructing the lightweight service robot detection model, the embodiment first adopts a network pruning method based on gradient amplitude self-adaption to pre-process YOLOv5n, including:

[0098] The gradient values of each network layer of the trained YOLOv5n are sorted, and the average value of all network layer gradient values is taken as a threshold value, and the network layer with a gradient value lower than the threshold value is optimized, including:

[0099] If the network layer to be optimized is a neck layer with a shortcut connection, the layer pruning is used to remove the network layer, so as to ensure that the shortcut and the residual layer can be added together; if the network layer to be optimized is a channel layer, the network layer is directly compressed, such as compressing the channel number of a certain channel layer from 256 to 78.

[0100] In summary, the present application improves the existing YOLOv5n from the backbone network and the neck network to construct a lightweight service robot detection model. In the lightweight service robot detection model, the C3 module is improved by using dynamic snake convolution to form a new C3_DSC module, which replaces part of the C3 module in YOLOv5n, so as to improve the accuracy of feature extraction of tubular deformation objects by the detection model. In the backbone network, a hybrid architecture composed of a lightweight MobileViT module and a convolution triple attention module is further introduced, the Transformer block in the MobileViT module is used to solve the problem of insufficient global modeling ability of the convolutional neural network, and the convolution triple attention module interacts the features of part of the backbone network in different dimensions, which significantly improves the feature extraction performance of the detection model without increasing the parameters. In the neck network, the MobileViT module is also introduced, combined with the C3_DSC module and the deformable convolution module, to further strengthen the extraction of deformation features and global features in the feature fusion stage. The lightweight service robot detection model constructed by the present application combines convolutional neural networks and Transformers, effectively improves the detection performance of multi-scale targets, especially deformation objects, and has high detection accuracy while ensuring high real-time performance, which is suitable for the target detection needs of service robots.

[0101] Further, the lightweight service robot detection model adopts an efficient decoupling head as a detection head to solve the contradiction between classification and regression, and optimizes the efficient decoupling head by using a depth separable convolution, further reducing the calculation quantity of the detection model and improving the efficiency of target detection. Embodiment two

[0102] Based on the lightweight service robot target detection method described in embodiment one, the embodiment provides a lightweight service robot target detection device, comprising:

[0103] A model construction module is configured to construct a lightweight service robot detection model based on YOLOv5n, comprising:

[0104] The second and third C3 modules in the backbone network of YOLOv5n along the positive propagation direction are replaced by MobileViT modules, the fourth C3 module in the backbone network of YOLOv5n along the positive propagation direction is replaced by a combination architecture composed of a C3_DSC module and a convolution triple attention module; the second C3 module in the neck network of YOLOv5n along the positive propagation direction is replaced by a MobileViT module, the third and fourth C3 modules in the neck network of YOLOv5n along the positive propagation direction are replaced by C3_DSC modules; the fourth CBS module in the neck network of YOLOv5n along the positive propagation direction is replaced by a deformable convolution module; the C3_DSC module comprises a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence;

[0105] A detection module is configured to perform target detection on the collected images of the service robot by using the lightweight service robot detection model. Embodiment three

[0106] In order to illustrate the effect of the lightweight service robot target detection method described in embodiment one of the present application, the embodiment provides specific experimental results of the improvement strategy.

[0107] Firstly, the embodiment tests the effectiveness of each part in YOLOViT_SR through an ablation experiment. The main parameter settings in the experiment are as follows: loading pre-trained weights, training for 400 epochs, block size setting 16, input image size 640x640x3, momentum setting 0.9, weight decay regular coefficient setting 0.0005, initial learning rate setting 0.001. The data set used in the experiment is a self-collected benchmark data set for a sweeping robot, which contains a total of 8100 images, of which the training set accounts for 80%, a total of 6480 images, and the test set accounts for 20%, a total of 1620 images.

[0108] The specific implementation steps and results of the C3_DSC module ablation experiment include:

[0109] 1. In the backbone network, the C3 module is replaced by the C3_DSC module in two ways, among which the effect of replacing the 6th and 8th layers at the same time is the best, the precision is improved from 75.8% to 79.2%, the model parameter amount is reduced by 0.02M, and the inference time is almost unchanged.

[0110] 2. In the neck network, the C3 module is replaced by the C3_DSC module in two ways, among which the effect of replacing the 21st and 24th layers at the same time is the best, the precision is improved from 75.8% to 79.0%, and the model parameter amount is reduced by 0.01M, and the inference time is almost unchanged.

[0111] 3. The backbone network and the neck network are replaced jointly, the parameter amount of the model is reduced by 0.02M, the precision is greatly improved by 4.7%, and the inference time is only increased by 0.8ms.

[0112] The results of the ablation experiment of the C3_DSC module are shown in Table 1.

[0113] Table 1, results of the ablation experiment of the C3_DSC module

[0114] Model Replace C3 layer number mAP (%) #Params (M) GFLOPs Latency (ms) YOLOv5n - 75.8 0.45 1.2 1.5 8 77.4 0.44 1.1 1.5 Backbone network 6+8 79.2 0.43 1.1 2.0 Neck network 2421+24 77.479.0 0.440.44 1.11.1 1.51.7 Backbone + neck Total four layers 80.5 0.43 1.1 2.2

[0115] According to the ablation experiment results of the C3_DSC module and the MobileViT module, the two are further fused in this embodiment, and finally the C3 module of the 4th, 6th and 18th layers is replaced by the MobileViT module, and the C3 module of the 8th, 21st and 24th layers is replaced by the C3_DSC module. The model can achieve the best balance between precision and speed. On this basis, the CTAM module and the improved EDH constitute the final YOLOViT_SR network.

[0116] The ablation experiment results of YOLOViT_SR are shown in Table 2.

[0117] Table 2, ablation experiment results of YOLOViT_SR

[0118] Model mAP (%) #Params GFLOPs Latency (ms) FPS YOLOv5n (before pruning) 76.8 1.78M 4.28 1.1 909 YOLOv5n (after pruning) 75.8 0.45M 1.2 1.5 652 +C3_DSC 80.1 0.37M 1.2 2.3 441 +MobileViT 84.2 0.94M 1.2 3.7 273 +EDH 84.7 1.4M 5.4 4.3 234 +CTAM 85.0 1.4M 5.5 4.5 223

[0119] As can be seen from Table 2, the parameter amount of YOLOv5n after pruning is reduced by 1.33M, the calculation amount is reduced by 3.08G, and the precision is almost unchanged. On the basis of pruning, the C3_DSC module is added to improve the precision to 80.1%, the MobileViT module is added to further improve the precision to 84.2%, the CBAM module is added to make the model precision reach 84.7%, and finally the improved EDH is added to make the model precision reach 85.0%. Although the parameter amount and calculation amount of the model increase slightly, the precision improves by nearly 10%, and the speed remains at 143FPS, fully meeting the real-time requirements of service robots.

[0120] The comparison experimental results of YOLOViT_SR and other algorithms are shown in Table 3.

[0121] Table 3, comparison experimental results of YOLOViT_SR and other algorithms

[0122] Model mAP (%) #Params GFLOPs Latency (ms) FPS YOLOv5n (after pruning) 75.8 0.45M 1.2 2.3 435 YOLO4-tiny 73.7 5.9M 16.2 1.3 743 YOLOX-tiny 89.8 5.04M 15.2 2.5 394 RT-DETR-R18 88.0 20.1M 60.2 6.2 161 YOLOViT_SR 85.0 1.4M 5.5 4.5 223

[0123] As can be seen from Table 3, YOLOViT_SR is compared with the current mainstream lightweight model, compared with YOLOX-tiny, the parameter amount of YOLOViT_SR is about 1 / 4, the calculation amount is about 1 / 3, and the precision difference is less than 5%; compared with RT-DETR-R18, the parameter amount and calculation amount of YOLOViT_SR are more obvious, so the high efficiency of the present application is fully embodied, and a better balance between precision and speed is achieved.

[0124] In order to further illustrate the effectiveness of the present application, Figure 7 is a PR curve comparison diagram of YOLOViT_SR and the original model YOLOv5n, wherein Figure 7 (a) in (a) is the PR curve diagram of YOLOv5n, Figure 7 (a) in (a) is the PR curve diagram of YOLOViT_SR; Figure 8 is a detection result visualization diagram of the method of the present application.

[0125] As can be seen from Figure 7 , the YOLOViT_SR proposed in the present application not only improves the overall precision (mAP), but also the recognition precision of most objects is more than 80%, among which the wire belongs to an object with large deformation, and the precision is improved from 56.2% to 70.3%, so the effectiveness of the C3_DSC module constructed in the present application can be seen.

[0126] Figure 8 shows the detection performance of the service robot in the actual scene, the different colored bounding boxes in the figure represent that different categories of objects are recognized, and the numbers above the bounding boxes represent the confidence of detection, the higher the confidence, the greater the probability of being a positive sample. As can be seen from Figure 8 , the confidence of a single object in different scenes is very high, basically more than 0.5, and some objects can reach 0.9, such as the stool in the fourth column of the third row. It can be seen that the YOLOViT_SR proposed in the present application is completely suitable for target recognition of server robots.

[0127] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a software embodiment, the methods can be tangibly embodied in a machine-readable storage medium having stored thereon instructions that can be used to program a computer to perform any of the operations described herein. The software implementation can be for example, in the form of a computer program product. The software implementation can be implemented as a computer program product on a computer readable medium.

[0128] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in the flowchart block or blocks.

[0129] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps in the flowchart block or blocks.

[0130] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 Figure 1 means for functionally implementing the steps in the flowchart block or blocks.

[0131] Obviously, the embodiments described above are only examples and are not intended to limit the present application. Other variations or modifications can be made to the embodiments described above based on the principles of the present application. The scope of the present application is defined by the appended claims.

Claims

1.A lightweight service robot target detection method, characterized in that, The application relates to a lightweight service robot detection model based on YOLOv5n. The second and third C3 modules in the backbone network of YOLOv5n are replaced by MobileViT modules in the positive propagation direction, the fourth C3 module in the backbone network of YOLOv5n is replaced by a combined architecture composed of a C3_DSC module and a convolution triple attention module, the second C3 module in the neck network of YOLOv5n is replaced by a MobileViT module in the positive propagation direction, the third and fourth C3 modules in the neck network of YOLOv5n are replaced by C3_DSC modules in the positive propagation direction, and the fourth CBS module in the neck network of YOLOv5n is replaced by a deformable convolution module. The C3_DSC module comprises a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence; the input feature of the C3_DSC module sequentially passes through the first dynamic snake convolution and the BottleNeck module to obtain a first intermediate feature, and the input feature passes through the second dynamic snake convolution to obtain a second intermediate feature; after the first intermediate feature and the second intermediate feature are spliced, the CBS module is input to obtain the output feature of the C3_DSC module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence, and the output feature of the deformable convolution module is added to the input feature of the BottleNeck module to obtain the output feature of the BottleNeck module; The lightweight service robot detection model comprises a backbone network, a neck network and a head network. The backbone network comprises a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolution triple attention module and an SPPF layer connected in sequence. The neck network comprises a sixth CBS module, a first up-sampling module, a second C3 module, a seventh CBS module, a second up-sampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence. The head network comprises a first detection head, a second detection head and a third detection head. The MobileViT module is further optimized, and the optimized MobileViT module comprises ​ The input features of the MobileViT module are sequentially subjected to an n*n cavity convolution layer and a first 1*1 convolution layer to obtain a local feature map; the local feature map is subjected to an unfolding layer, a plurality of Transformer blocks and a folding layer to obtain a global feature; the global feature is subjected to a second 1*1 convolution layer and then spliced with the input features, and then input to an n*n convolution layer to obtain output features of the MobileViT module; The acquisition image of the service robot is input into the lightweight service robot detection model. 2.The method of claim 1, wherein, The acquisition image is input into the backbone network of the lightweight service robot detection model to output a multi-scale feature map, including: The acquisition image is input into the first CBS module, the second CBS module, the first C3 module, the third CBS module and the first MobileViT module in sequence along the positive propagation direction to output a first feature map; The first feature map is input into the fourth CBS module and the second MobileViT module in sequence along the positive propagation direction to output a second feature map; The second feature map is input into the fifth CBS module, the first C3_DSC module, the convolution triple attention module and the SPPF layer in sequence along the positive propagation direction to output a third feature map. 3.The method of claim 2, wherein, The multi-scale feature map output by the backbone network is input into the neck network to output a multi-scale enhanced feature map, including: The third feature map is input into the sixth CBS module to output a third fusion feature map; The third fusion feature map is spliced with the second feature map after upsampling, and then input into the second C3 module and the seventh CBS module in sequence along the positive propagation direction to output a second fusion feature map; The second fusion feature map is spliced with the first feature map after upsampling to output a first fusion feature map; The first fusion feature map is input into the second MobileViT module to output a first enhanced feature map; The first enhanced feature map is spliced with the second fusion feature map after passing through the eighth CBS module, and then input into the second C3_DSC module to output a second enhanced feature map; The second enhanced feature map is spliced with the third fusion feature map after passing through the deformable convolution module, and then input into the third C3_DSC module to output a third enhanced feature map. 4.The method of claim 1, wherein, An efficient decoupling head is used as the detection head of the head network, and the efficient decoupling head is optimized, including: replacing the 3*3 convolution in the classification branch and the regression branch of the efficient decoupling head with a depth separable convolution. 5.The method of claim 1, wherein, The CBS module includes a convolution layer, a batch normalization layer and a SiLU activation function connected in sequence; The deformable convolution module includes a dynamic snake convolution layer, a batch normalization layer and a SiLU activation function connected in sequence. 6.The method of claim 1, wherein, Before constructing the lightweight service robot detection model, a network pruning method based on gradient amplitude self-adaption is used to pre-process YOLOv5n, including: The gradient values of each network layer of the trained YOLOv5n are sorted, and the average value of all network layer gradient values is taken as a threshold, and the network layer with a gradient value lower than the threshold is optimized, including: If the network layer to be optimized is a neck layer with a shortcut connection, the layer pruning is used to remove the network layer; if the network layer to be optimized is a channel layer, the network layer is directly compressed. 7.A lightweight service robot target detection device, characterized by comprising: including: The model construction module is used for constructing a lightweight service robot detection model based on YOLOv5n, and the lightweight service robot detection model comprises: The second and third C3 modules in the backbone network of YOLOv5n along the positive propagation direction are replaced by MobileViT modules, the fourth C3 module in the backbone network of YOLOv5n along the positive propagation direction is replaced by a combined architecture composed of a C3_DSC module and a convolution triple attention module, the second C3 module in the neck network of YOLOv5n along the positive propagation direction is replaced by a MobileViT module, the third and fourth C3 modules in the neck network of YOLOv5n along the positive propagation direction are replaced by C3_DSC modules, and the fourth CBS module in the neck network of YOLOv5n along the positive propagation direction is replaced by a deformable convolution module; the C3_DSC module comprises a first dynamic snake convolution, a BottleNeck module, a second dynamic snake convolution and a CBS module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence; the input feature of the C3_DSC module sequentially passes through the first dynamic snake convolution and the BottleNeck module to obtain a first intermediate feature, and the input feature passes through the second dynamic snake convolution to obtain a second intermediate feature; after the first intermediate feature and the second intermediate feature are spliced, the spliced feature is input into the CBS module to obtain the output feature of the C3_DSC module; the BottleNeck module comprises a CBS module and a deformable convolution module connected in sequence, and the output feature of the deformable convolution module is added to the input feature of the BottleNeck module to obtain the output feature of the BottleNeck module; The lightweight service robot detection model comprises a backbone network, a neck network and a head network; The backbone network comprises a first CBS module, a second CBS module, a first C3 module, a third CBS module, a first MobileViT module, a fourth CBS module, a second MobileViT module, a fifth CBS module, a first C3_DSC module, a convolution triple attention module and an SPPF layer connected in sequence; The neck network comprises a sixth CBS module, a first up-sampling module, a second C3 module, a seventh CBS module, a second up-sampling module, a second MobileViT module, an eighth CBS module, a second C3_DSC module, a deformable convolution module and a third C3_DSC module connected in sequence; The head network comprises a first detection head, a second detection head and a third detection head; The MobileViT module is further optimized, and the optimized MobileViT module comprises: The input feature of the MobileViT module is sequentially subjected to an n*n cavity convolution layer and a first 1*1 convolution layer to obtain a local feature map; the local feature map is subjected to an unfolding layer, a plurality of Transformer blocks and a folding layer to obtain a global feature; the global feature is subjected to a second 1*1 convolution layer and then spliced with the input feature, and then input to an n*n convolution layer to obtain an output feature of the MobileViT module; The detection module is configured to perform target detection on the collected image of the service robot by using the lightweight service robot detection model.

Images