A lightweight object detection network design method for indoor semantic SLAM system

Abstract

The present invention discloses a lightweight target detection network design method for indoor semantic SLAM systems, belonging to the fields of computer vision and deep learning. Based on the YOLOv8n lightweight target detection framework, the present invention enhances the feature extraction capability of the backbone network by adopting a C2f module based on a deformable convolutional network structure in the backbone network; enhances the feature representation capability of the backbone network in generating feature maps by using a deformable attention module; and comprehensively enhances the network's detection capability for multi-scale and multi-deformation targets in indoor dynamic environments by using a dynamic target detection head in combination with scale self-attention, spatial self-attention, and channel self-attention. Experimental results show that the present invention can quickly and accurately identify common targets in indoor scenes, can provide the required semantic information for indoor semantic SLAM systems, and improve the accuracy and operating speed of semantic SLAM systems.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision and deep learning, and in particular relates to a lightweight target detection network design method for an indoor semantic SLAM system. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) involves a mobile robot using its onboard sensors to build a map of its surroundings and estimate its own motion during movement, without prior knowledge of the environment. SLAM encompasses both localization and mapping and is considered a key issue in achieving robot autonomy. It holds significant research implications for areas such as robot navigation, control, and mission planning.

[0003] Semantic SLAM technology combines SLAM technology with deep learning technology. It uses the powerful feature representation and data fitting capabilities of neural networks in deep learning to act on existing VSLAM modules such as feature extraction, data association, depth estimation, optical flow estimation, and loop detection to comprehensively improve system performance. In addition, it can also focus on scene understanding and enhance the perception capabilities of the VSLAM system.

[0004] As a major component of semantic SLAM systems, the performance of the target detection network used for semantic information acquisition has a significant impact on the performance of semantic SLAM systems. Because SLAM systems need to be deployed on mobile robot platforms, they place high demands on real-time performance. Therefore, a lightweight target detection network for semantic information acquisition is necessary. Furthermore, the accuracy of semantic information acquisition directly impacts the positioning accuracy and mapping performance of semantic SLAM systems. Current research on target detection networks for indoor semantic SLAM systems primarily focuses on increasing the variety of detected targets to enhance the network's versatility. However, there is a lack of targeted network design for specific usage scenarios, resulting in poor overall semantic SLAM performance and difficulty achieving the desired results. Therefore, for semantic SLAM systems operating in dynamic indoor scenarios, a specialized target detection network must be designed that can accurately identify the dynamic, multi-scale, and multi-deformation targets commonly found in indoor semantic SLAM systems, thereby improving the positioning accuracy and environmental perception of indoor semantic SLAM systems. Summary of the Invention

[0005] The present invention provides a lightweight target detection network design method for an indoor semantic SLAM system, which can provide fast and accurate semantic information for the indoor semantic SLAM system, thereby improving the positioning accuracy, mapping effect and operation efficiency of the semantic SLAM system.

[0006] In order to achieve the above objectives, the present invention adopts the following technical solutions:

[0007] A method for designing a target detection network for an indoor semantic SLAM system includes the following steps:

[0008] Step 1: Establish a lightweight target detection network based on YOLOv8n as the basic framework;

[0009] Step 2: Design a feature extraction module based on deformable convolution and a deformable attention feature enhancement module;

[0010] Step 3: Design the DyHead-based target detection head network.

[0011] In the above steps, the lightweight target detection framework based on YOLOv8n in step 1 includes the following steps:

[0012] (a) Lightweight backbone network for extracting image features

[0013] The backbone network is used to extract image features and obtain the overall feature map of the image. YOLOv8n's backbone network design adopts the design concept of CSPNet (Cross Stage Partial Network). By dividing the underlying feature map into two parts, operating on one part, and then combining it with the remaining unchanged part across layers, it optimizes the network structure and improves speed and accuracy.

[0014] (b) Neck feature fusion network

[0015] The Neck network is located between the Backbone network and the Head target detection head. YOLOv8n's Neck network uses an FPN+PAN network structure to integrate features extracted by the backbone network, improving the network's ability to detect multi-scale targets.

[0016] (c) Object detection head network

[0017] The object detection head network performs final object detection based on the input feature map. YOLOv8n uses an anchor-free architecture and decouples it, splitting the loss function into classification and position losses. The regression branch is used to identify objects and locate the object frame, while the classification branch is used to classify the objects. Finally, object detection is achieved based on the results of these two branches.

[0018] Step 2 is achieved by modifying and optimizing the backbone network structure in step 1, which specifically includes the following steps: for the last C2f feature extraction module in the backbone network, deformable convolution is used instead of the two-dimensional convolution operation. The deformable convolution can adjust the convolution kernel structure according to the different detection targets, so as to better extract the features of the target. By using the deformable convolution kernel, the network's sensitivity to dynamic targets is improved, the receptive field is expanded, and the robustness of the network is enhanced; then, the 20×20 feature map output by the backbone network SPPF module is used as the input of the deformable attention module (DeformableAttention, DA) to improve the representation ability of the backbone network feature map, and the feature-enhanced output is passed to the neck network, which propagates from bottom to top through the neck network to improve the network's recognition and positioning accuracy for small and dynamic targets.

[0019] Step 3 is achieved by replacing the target detection head network in step 1 with the DyHead structure. The DyHead module consists of three parts π L ,π S ,π C They correspond to scale self-attention, spatial self-attention, and channel self-attention respectively; first, the three feature maps of different scales obtained by the neck network are connected to obtain the feature tensor of the entire image Among them, L represents the scale dimension of the feature; S is the product of the width and height of the feature map, representing the spatial dimension; C represents the number of channels, then the attention function of the image feature tensor is:

[0020]

[0021] DyHead significantly improves the representation capability of the object detection head by coherently combining multi-head self-attention mechanisms within the scale-aware feature layer, space-aware spatial positions, and task-aware output channels.

[0022] Beneficial effects: The present invention provides a target detection network design method for an indoor semantic SLAM system, and provides an accurate and fast semantic information acquisition method for the indoor semantic SLAM system. Based on the YOLOv8n lightweight target detection framework, the feature extraction capability of the backbone network is enhanced by adopting a C2f module based on a deformable convolutional network structure in the backbone network; the feature representation capability of the backbone network in generating feature maps is enhanced by using a deformable attention module; and the network's detection capability for multi-scale and multi-deformation targets in indoor dynamic environments is comprehensively improved by using a dynamic target detection head, combined with scale self-attention, spatial self-attention, and channel self-attention. The experimental results of the VOC dataset show that the present invention can realize accurate and fast recognition of common multi-scale and multi-deformation targets in indoor environments, and provide fast and accurate semantic information for indoor semantic SLAM systems, thereby improving the positioning accuracy, mapping effect, and operating efficiency of the semantic SLAM system. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a diagram showing the structure of a lightweight target detection network for an indoor semantic SLAM system designed in an embodiment of the present invention;

[0024] Figure 2 This is a network structure diagram of the SPPF module in an embodiment of the present invention;

[0025] Figure 3 This is a network structure diagram of a deformable convolution module in an embodiment of the present invention;

[0026] Figure 4 This is a diagram of the DyHead network structure in an embodiment of the present invention;

[0027] Figure 5 This is the scene target detection result of the indoor semantic SLAM system in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments:

[0029] A method for designing a target detection network for an indoor semantic SLAM system includes the following steps: 1. Designing a lightweight target detection framework based on YOLOv8n

[0030] The designed target detection network structure is as follows Figure 1 As shown in the figure, it is based on the YOLOv8n lightweight target detection network as the basic framework; the entire network structure is divided into three parts: the backbone network, the neck feature fusion network, and the target detection head network;

[0031] 1.1 Backbone Network

[0032] The backbone network consists of five CBS modules, three C2f modules, one C2f-DCN module, and one SPPF module. The CBS module includes convolution (Conv), batch normalization (BN), and the SiLU activation function. CBS has two parameters, S and K, representing the step size and kernel size in the convolution calculation, respectively, and is used for image downsampling and feature extraction. The Bottleneck module is a special residual structure that uses multiple small convolution kernels instead of large ones, deepening the network layer while reducing the number of network parameters. The C2f module consists of one CBS module and several parallel Bottleneck modules, which are used to enhance the image features extracted by the CBS. The C2f module uses the gradient splitting principle to effectively avoid slow convergence when the network is too deep. At the same time, different scale models are scaled and the number of channels is adjusted. While optimizing the network structure to achieve lightweightness, the C2f module can also obtain rich gradient flow information, which can improve the overall performance of the network.

[0033] SPP, known as spatial pyramid pooling, extracts and fuses multi-scale features from feature maps by performing parallel max-pooling operations of different sizes, addressing the multi-scale problem and improving detection accuracy. SPPF uses three 5×5 max-pooling layers connected in series. This allows two 5×5 pooling layers to be connected in series to create the equivalent of a 9×9 pooling layer, and three 5×5 pooling layers to be connected in series to create the equivalent of a 13×13 pooling layer. By replacing large convolution kernels with multiple small convolution kernels connected in series, the network's computational load is effectively reduced, improving detection efficiency.

[0034] 1.2 Neck feature fusion network

[0035] The neck feature fusion network is located between the backbone network and the target detection head network, and its function is to fuse the features extracted by the backbone network. The neck feature fusion network of the present invention uses the FPN+PAN network structure; the function of the Feature Pyramid Network (FPN) is to obtain multi-scale feature information of the image by multiple upsampling of the feature map, and to obtain feature maps of three different scales by fusing semantic information of different levels for subsequent detection; the essence of FPN is to transfer the rich semantic information of the high-level layer to the low-level layer through upsampling, thereby enhancing the semantic information of the bottom layer. In order to enable the high-level layer to also obtain the rich details of the low-level layer, the Path Aggregation Network (PAN) adds a downsampling process on the basis of the original FPN structure, so as to strengthen the position information of the high-level feature map and further improve the detection accuracy.

[0036] 1.3 Object Detection Head Network

[0037] The target detection head network realizes the final target detection based on the input feature map. The present invention adopts the center point-based Anchor Free method to replace the anchor box-based method to generate the prediction box. The idea is to directly learn the prediction box shape of various objects during the training process, and generate the prediction box by first finding the center point during reasoning, and then predicting the distance from the center point to the four sides of the prediction box. Compared with the Anchor Based method, this method generates fewer prediction boxes and does not rely on prior knowledge, which improves the network's ability to predict the target shape and has stronger generalization performance. In addition, it can also improve the detection ability of dynamic objects, and is more flexible when detecting occluded objects. It is very suitable for the indoor dynamic environment faced by the indoor semantic SLAM system.

[0038] 2. Feature extraction module based on deformable convolution and deformable attention feature enhancement module

[0039] In the dynamic indoor environment targeted by indoor semantic SLAM systems, there are moving targets that need to be detected, and their morphology and image size within the camera may change with movement. Traditional convolution operations, due to the use of fixed-size convolution kernels, have poor feature extraction performance when facing these dynamic targets. To address these issues, the present invention designs a feature extraction module based on deformable convolution and a deformable attention feature enhancement module to enhance the performance of the target detection network.

[0040] 2.1 Feature Extraction Module Based on Deformable Convolution

[0041] In order to improve the ability of neural networks to detect dynamic objects, the present invention introduces a deformable convolution network (DCN); DCN modifies the convolution kernel so that the shape and size of the convolution kernel are no longer fixed. DCN adds a learnable offset to each point on the convolution kernel so that the convolution kernel can learn different convolution kernel structures according to the different features of the target to be detected.

[0042] By performing a standard two-dimensional convolution operation on a certain sampling point p0, its eigenvalue can be obtained:

[0043]

[0044] Among them, R is the convolution area near the sampling point, p n is the position element in R, w(p n ) is p n The convolution kernel weight value at x(p0+p n ) represents the sampling position of the feature map to be tested;

[0045] Deformable convolution adds an offset Δp to the sampling point based on the standard two-dimensional convolution operation. n , so the eigenvalue of the sampling point p0 after deformable convolution is:

[0046]

[0047] The overall structure of the DCN module after the introduction of deformable convolution is as follows Figure 3 As shown in the figure, first, the standard two-dimensional convolution is used to extract features from the image, just like the standard convolutional neural network, to obtain the initial feature map of the image; then the initial feature map is convolved and input into the bias domain to obtain the offset (x, y) of each pixel on the two-dimensional plane, and then the Δp in the deformable convolution kernel is obtained. n , and then use Δp on the initial input feature map n Get the feature map output by DCN.

[0048] The present invention uses the DCN module to replace the convolution part in the C2f module. The deformable convolution can adjust the convolution kernel structure according to the different detection targets, so as to better extract the features of the target. By using the deformable convolution kernel, the network's sensitivity to dynamic targets is improved, the receptive field is expanded, and the robustness of the network is enhanced. Since the introduction of the DCN module will increase the number of parameters that the system needs to learn, it will affect the convergence speed of the system. At the same time, if the DCN module is used too much in the backbone network, the potential semantic information will be lost during the continuous downsampling of the image, resulting in the loss of features in the final feature map, affecting the detection accuracy. Therefore, the present invention only replaces the last C2f module in the backbone network, and the standard convolution kernel is still used in the remaining CBS modules and C2f modules, thereby improving the backbone network's feature extraction capability for dynamic targets, extracting more detailed features of the image, and improving the representation capability of the feature map; thereby improving the network's detection capability for multi-scale and multi-deformation targets.

[0049] 2.2 Deformable Attention Feature Enhancement Module

[0050] While the self-attention mechanism can significantly improve network performance, the standard self-attention mechanism needs to process all pixels in the image, which requires a large amount of computation. This paper adopts a deformable attention mechanism, which enhances the representation capability of the sparse attention mechanism while reducing the demand for video memory.

[0051] In the deformable attention mechanism (DA), for the input feature map X, there is a linear mapping q, which is passed through the bias network θ offset Obtain the offset of the sampling reference point, then add the offset to the reference point to obtain the coordinates of the deformation reference point; use bilinear interpolation to sample the feature map to obtain the sampling feature point On this basis, according to the mapping matrix W v and W k Obtained after deformation and Then according to q, And the relative position deviation R is used to obtain the final multi-head attention output.

[0052] DA learns a set of offsets for each input image, moving the key and value to a position where the image features are more prominent, ultimately achieving a deformable attention mechanism. In the object detection network designed by this invention, considering the attention mechanism's demand for video memory, a DA attention module is added after the SPPF module, which performs multi-scale feature extraction and fusion. The 20×20 feature map obtained by SPPF is used as the input of the DA attention module, and the feature-enhanced output is passed to the neck network, which propagates from bottom to top, improving the network's recognition and positioning accuracy for small and dynamic targets.

[0053] 3. DyHead-based target detection head network

[0054] The Neck network ultimately generates feature maps of 20×20, 40×40, and 80×80 sizes. The object detection head performs classification and localization tasks on these three feature maps. To further enhance the network's ability to detect objects at multiple scales, this paper uses a dynamic object detection head (DyHead) to enhance the capabilities of the object detection head.

[0055] like Figure 4 As shown, the DyHead module consists of three parts π L ,π S ,π C Before using DyHead, the three feature maps of different scales obtained by the Neck layer are connected to obtain the feature tensor of the entire image. Among them, L represents the scale dimension of the feature; S = H × W, is the product of the width and height of the feature map, representing the spatial dimension; C represents the number of channels, then the attention function of the tensor is

[0056]

[0057] Among them, the scale-aware self-attention module π L Different weights are assigned to feature maps of different scales, so that the network can adaptively fuse according to the importance of the scale map and enhance the scale perception ability of the target detection head. The formula is:

[0058]

[0059] represents the average pooling layer in the network module; f(x) is a linear function, representing the 1×1 convolution layer in the network; σ(x) is the hard-sigmoid function, and its specific formula is

[0060]

[0061] Spatial perception attention module π S , used to enhance the network's ability to discriminate different spatial locations. This module first uses DCN for sparse sampling, and then aggregates features of each scale at the same spatial location. The specific formula is

[0062]

[0063] Where K is the number of sparse sampling locations, Δp k is the position offset, Δm k It is about position p k The weight factor of

[0064] Channel self-attention module π C By adaptively opening / closing feature channels to cope with various tasks, the generalization of the network is enhanced:

[0065]

[0066] Among them, [α 1 , α 2 , β 1 , β 2 ] is a hyperparameter used to learn and control the activation threshold.

[0067] DyHead significantly improves the representation capability of the object detection head by coherently combining the multi-head self-attention mechanism within the scale-aware feature layer, the spatial position of space-awareness, and the task-aware output channel. It can be used for different tasks such as classification and center / bounding box regression, and can be well integrated into the neural network proposed in this article.

[0068] VOC dataset training and experimental verification

[0069] We trained and experimented on the VOC dataset to verify whether the object detection network we designed meets the requirements of an indoor semantic SLAM system. We used a workstation with an Intel Xeon W-2255 @ 3.7 GHz CPU and an NVIDIA GeForce RTX 3080 GPU as the training and testing platform. The comparative experimental results are shown in Tables 1 and 2.

[0070] Table 1 Model performance comparison experiment

[0071]

[0072] As can be seen from the table, the network designed by the present invention has a significant lead in overall accuracy compared to the classic target detection networks listed. At the same time, in terms of lightweight, YOLOv5n has fewer model parameters and computational complexity. However, the parameter count of the present invention is also at a very low level, only 3.7M, and the number of floating-point operations is only 9.8G, which can meet the real-time requirements of the semantic SLAM system for the target detection network.

[0073] Table 2 Comparison results of indoor target accuracy

[0074]

[0075] In addition to comparing the overall model accuracy, Table 2 also shows the mAP@.5% target detection accuracy for indoor targets in the VOC dataset. It can be seen that the network designed by this invention achieves the best accuracy for both dynamic targets such as people, cats, and dogs, and static targets such as bottles, chairs, and monitors, fully demonstrating the excellent detection effect of the designed system for indoor targets.

[0076] Figure 5 The target detection results of the network designed in the present invention in the high-dynamic environment faced by the indoor semantic SLAM system are demonstrated, and its detection results of multi-scale and multi-deformation targets in this high-dynamic environment are demonstrated; the experimental results show that the target detection network designed in the present invention has a performance improvement of 2.5% and 3.5% in terms of overall mAP@.5 / % and mAP@.5:.95 / % accuracy compared with YOLOv8; in addition, the target detection network proposed in the present invention has a higher recognition accuracy rate for common dynamic and static targets indoors, and the target detection network designed in the present invention can meet the target detection requirements of indoor semantic SLAM systems.

[0077] In summary, the target detection network design method for indoor semantic SLAM system designed in the present invention performed well in experiments, and can better meet the target detection requirements of indoor semantic SLAM system, provide fast and accurate semantic information for semantic SLAM system, and improve the positioning accuracy and mapping effect of semantic SLAM system while ensuring the real-time operation of the system.

[0078] The above are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. For those skilled in the art, within the scope of their knowledge and without departing from the principles of the present invention, they can make several improvements to the present invention, which are also considered to be within the scope of protection of the present invention.

Claims

1. A target detection network design method for indoor semantic SLAM system, characterized in that: The following steps are involved: Step 1: Establish a lightweight target detection network framework based on YOLOv8n; Step 2: By modifying and optimizing the backbone network structure in step 1, we design a feature extraction module based on deformable convolution and a deformable attention feature enhancement module. Step 3: Design a DyHead-based target detection head network by replacing the target detection head network in step 1 with the DyHead module; The DyHead module consists of three parts 、 、 They correspond to the scale-aware self-attention module, the spatial-aware self-attention module, and the channel-aware self-attention module respectively; The DyHead module coherently combines multi-head self-attention mechanisms within the scale-aware feature layer, the spatial position of the space-aware, and the task-aware output channel; The scale-aware self-attention module Different weights are assigned to feature maps of different scales, so that the network can adaptively fuse according to the importance of the scale map and enhance the scale perception ability of the target detection head. The formula is: ， in, Represents the average pooling layer in the network module; is a linear function, representing 1 in the network 1 convolutional layer; is the hard-sigmoid function, and the specific formula is: ； The spatial perception attention module , used to enhance the network's ability to discriminate different spatial locations. First, DCN is used for sparse sampling, and then the features of each scale are aggregated at the same spatial position. The specific formula is: ， in, is the number of sparsely sampled locations, is the position offset, It's about location The weight factor of The channel self-attention module By adaptively opening / closing feature channels to cope with various tasks, the generalization of the network is enhanced. The specific formula is: ， in, is a hyperparameter used to learn and control the activation threshold.

2. The target detection network design method for indoor semantic SLAM system according to claim 1, wherein The entire network structure of the target detection network in step one is divided into three parts: a backbone network for image feature extraction, a neck feature fusion network for feature fusion, and a target detection head network for outputting target detection results; the neck feature fusion network is located between the backbone network and the target detection head network, and is used to fuse the features extracted by the backbone network.

3. The target detection network design method for indoor semantic SLAM system according to claim 2, wherein The backbone network consists of 5 CBS modules, 3 C2f modules, 1 C2f-DCN module and one SPPF module.

4. The target detection network design method for indoor semantic SLAM system according to claim 3, wherein In step 2, a deformable convolutional network is introduced. Deformable convolution adds an offset to the sampling point based on the standard two-dimensional convolution operation. , using the sampling points after deformable convolution The eigenvalues ​​of are: ， in, is the convolution area near the sampling point, for mid-position element, for The convolution kernel weight value at , Indicates the sampling position of the feature map to be tested.

5. The target detection network design method for indoor semantic SLAM system according to claim 4, wherein The convolution part in the C2f module is replaced by the deformable convolutional network module. Only the last C2f module in the backbone network is replaced, and the standard convolution kernel is still used in the remaining CBS modules and C2f modules.

6. The target detection network design method for indoor semantic SLAM system according to claim 3, wherein In step 2, the deformable attention module is introduced. The deformable attention module will be added after the SPPF module that performs multi-scale feature extraction and fusion. The feature map obtained by SPPF is used as the input of the deformable attention module, and the output after feature enhancement is passed into the neck network and propagated from bottom to top through the neck network.

Images