Obstacle detection method of visual impaired group guiding waistcoat system based on improved YOLOv11

Abstract

The invention discloses an obstacle detection method for a visual impaired group guiding waistcoat system based on improved YOLOv11, and the method comprises the steps: 1, obtaining an obstacle data set, and completing the marking work of the data set; 2, a backbone network of the YOLOv11 is replaced by a ShuffleNetV2 (ShuffleNetV2); thirdly, a lightweight detection head is redesigned, and subsequent deployment is facilitated; 4, completing the training of the improved YOLOv11 model by using the data set; 5, deploying the improved YOLOv11 detection model into embedded hardware equipment for sensing; and 6, performing model reasoning to realize real-time obstacle detection of the camera on the waistcoat. According to the invention, the real-time performance and accuracy of obstacle detection on the waistcoat can be ensured, a reliable basis is provided for guiding a visual impaired group to make an obstacle avoidance decision, and the travel safety of the visual impaired group is ensured.

Description

Technical Field

[0001] The present invention belongs to the field of visual target detection, and in particular to an obstacle detection method for a visually impaired group guiding vest system based on an improved YOLOv11. Background Art

[0002] Vision is one of the five most important senses of human beings. As a vulnerable group in society, due to visual defects, the quality of life and life experience of the visually impaired group are very different from those of ordinary people, and their travel safety has not been guaranteed. The rapid development of artificial intelligence technology provides a new solution to ensure the travel safety of the visually impaired group. In real life scenarios, obstacle detection on the route of the visually impaired group is crucial to ensure the safety of the visually impaired group's travel. However, the current target detection technology is affected by many factors, such as lighting and occlusion, resulting in low detection accuracy, which has become a problem that needs to be solved urgently.

[0003] As deep learning technology continues to mature, researchers have begun to apply it to the field of target detection, which has brought revolutionary changes to the advancement of technology. The emergence of convolutional neural networks can effectively extract image features, avoiding the limitations of traditional manual feature extraction methods. What is exciting is that convolutional neural networks integrate feature extraction and target classification, significantly improving the accuracy and speed of detection. However, the current target detection model based on convolutional neural networks still has certain shortcomings in terms of real-time performance and accuracy, and needs further improvement and optimization. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide an obstacle detection method for a visually impaired group guiding vest system based on improved YOLOv11. Combining the advantages of the YOLOv11 model, by deploying the model on the vests worn by the visually impaired group, obstacles in complex traffic scenes can be quickly and accurately identified and located, so as to improve the safety and real-time performance of obstacle avoidance detection, and inform the visually impaired group where to avoid obstacles through vibration sensors, thereby ensuring the safety of the visually impaired group's travel.

[0005] Technical solution: The obstacle detection method of the visually impaired group guidance vest system based on the improved YOLOv11 of the present invention can achieve high-precision and high-real-time detection of obstacles, and feedback the obstacle avoidance information to the visually impaired group through the vibration sensor. It includes the following steps:

[0006] Step 1: Obtain the obstacle dataset of the visually impaired group guidance vest system, complete the dataset annotation work, and divide the dataset into training set, test set and validation set;

[0007] Step 2: Build the framework of the YOLOv11 target detection algorithm in the Pytorch environment and replace its backbone network with the ShuffleNetV2 network;

[0008] Step 3: Redesign a lightweight detection head to reduce the model size for subsequent deployment;

[0009] Step 4: Use the data set to design a data enhancement method, train the improved YOLOv11 network, obtain the optimal model parameters and save them;

[0010] Step 5: Deploy the improved YOLOv11 algorithm to the embedded hardware device;

[0011] Step 6: Import the trained network model parameters, perform model inference, and realize real-time obstacle detection of the mobile collaborative robot.

[0012] Furthermore, the obstacle dataset in step 1 is divided into a training set, a test set and a validation set; the training set is used for model training and optimization of network parameters; the test set is used for model testing during training and comparison of training effects; the validation set is used for model verification after training to verify the effectiveness and generalization of the model; each obstacle dataset contains images and labels, and the labels contain the category and location information of each target object in the image, wherein the location information is the coordinates of the center point of the annotation box and the width and height of the annotation box.

[0013] Furthermore, the YOLOv11 target detection algorithm in step 2 is a one-stage target detection algorithm based on deep learning, and its network structure includes an input end, a backbone network Backbone, a neck network NECK, and a detection head Head; the input end performs data preprocessing and data enhancement tasks to provide high-quality input data for the model. At the input end, YOLOv11 performs a series of preprocessing operations on the original image, including image scaling and normalization; so that the model can learn image features. In order to improve the generalization ability of the model, YOLOv11 uses data enhancement technology at the input end;

[0014] The ShuffleNetV2 network is a lightweight convolutional neural network, which is an improved version of ShuffleNet and is mainly used on embedded devices with limited memory and computing power resources. The ShuffleNetV2 network uses grouped convolution and channel shuffling to reduce the amount of calculation on the feature channel, solve the problem of feature communication closure, and improve the expressiveness of the model. In addition, ShuffleNetV2 also uses depthwise separable convolution to replace the traditional convolution operation. The depthwise separable convolution decomposes the standard convolution into two independent steps: depthwise convolution and pointwise convolution. This decomposition method can significantly reduce the amount of calculation and parameters while maintaining good performance. By combining these strategies, ShuffleNetV2 can achieve efficient feature extraction and detection performance while maintaining lightweight. It uses more mapping channels to enrich channel information and help the model obtain more spatial feature information. This makes ShuffleNetV2 perform well in a variety of computer vision tasks, especially in resource-constrained scenarios with broad application prospects.

[0015] Furthermore, the data enhancement technology includes Mosaic data enhancement and adaptive anchor frame calculation; the Mosaic data enhancement enriches the background and small targets of the detected object by splicing four pictures by random scaling, random cropping and random arrangement; this data enhancement method helps the model learn more scene and object changes during training and improves detection performance. At the input end, YOLOv11 uses adaptive anchor frame calculation technology; the anchor frame is a predefined set of rectangular frames of fixed size and proportion, which is used to predict the position of the object in the target detection task. The adaptive anchor frame calculation technology automatically adjusts the size and proportion of the anchor frame according to the size and proportion of the object in the input image to improve the accuracy of object detection.

[0016] Furthermore, the backbone network Backbone is used for feature extraction. It is a convolutional neural network. The backbone network of YOLOv11 adopts CSPNet structure, which mainly includes CBL structure, CSP structure and SPPF structure. The CBL structure is composed of convolution layer, BN layer and activation function, which simplifies the model structure and improves the speed of model reasoning. The CSP structure is composed of 3 CBL structures and multiple Bottleneck modules with shortcut structures, which reduces the use of video memory and speeds up the reasoning speed during the forward inference period, while solving the problem of gradient data disappearance and explosion in the training stage and enhancing the ability of convolution to extract image feature information. The rich feature information obtained by the CSP structure is further input into the SSPF structure. As the last layer of Backbone, the SSPF structure is used to perform multi-scale feature fusion on the output after input and convolution at different scales. Through multi-scale feature fusion, SSPF can capture contextual information at different levels in the image and improve the network's ability to locate and identify targets. After completing the preliminary feature fusion, the information is passed to the next stage of the network to provide support for subsequent target detection tasks. This design enables YOLOv11 to more accurately detect objects of different sizes and shapes, improving the accuracy and robustness of object detection.

[0017] Furthermore, the role of the neck network Neck is feature enhancement fusion, which is responsible for further processing and fusion of the features extracted from the Backbone network to improve detection accuracy. The design purpose of the Neck network is to better utilize the features extracted by the Backbone network and reprocess and reasonably utilize these features at different stages. The Neck network adopts the structure of the classic feature pyramid network FPN and the path aggregation network PANet to extract and fuse features at different scales, thereby improving the network's detection capabilities for targets of different sizes.

[0018] Furthermore, the detection head is a key part specifically responsible for target detection and classification. It receives three feature maps of different scales output from the previous feature enhancement fusion network. These feature maps have sizes of 20×20, 40×40 and 80×80 respectively. They are sent to the detection layer for coordinate regression prediction and classification tasks. In the detection layer, the network will perform target detection on each feature map and predict the bounding box coordinates and category of the target. Subsequently, the non-maximum suppression algorithm is used to eliminate redundant detection boxes to ensure that each target is detected only once, so that YOLOv11 can effectively predict large, medium and small targets of different types, improve the network's detection performance for dense objects, and make target detection more accurate and reliable.

[0019] Furthermore, in step three, the lightweight detection head greatly reduces the number of parameters by sharing convolutions to make the model lighter, and uses the Scale layer to scale the features to solve the problem of inconsistent scales of the targets detected by each detection head, thereby reducing the loss of model accuracy with fewer parameters and less calculation.

[0020] Furthermore, in step 4, the training adopts a training strategy of training preheating, cosine annealing to adjust the learning rate, and adaptive anchor frame calculation to improve the training effect and performance of the model;

[0021] The training sets the loss function, which includes the target confidence loss Classification Loss and bounding box regression loss The total loss is the weighted sum of the three parts, and the expression is:

[0022]

[0023] Among them, the weights of target confidence loss, classification loss and bounding box regression loss are λ1, λ2, λ3, L conf and L cls The binary cross entropy function is used for calculation, and the calculation formula is as follows:

[0024] L n =-(y n log(δ(x n ))+(1-y n )log(1-δ(x n )))

[0025] Where: x n To predict the score of the nth sample as a positive example, y n represents the label of the nth sample, and δ represents the sigmoid function;

[0026] L reg Use the CloU Loss function. CloU is a bounding box regression loss calculation method. The calculation formula is as follows:

[0027]

[0028] Where: parameter A represents the real box, B represents the predicted box, IoU is called intersection over union, which represents the overlap between the predicted box and the real box, ρ 2 (·) represents the Euclidean distance, b represents the coordinates of the center point of the prediction box, and b gt represents the coordinates of the center point of the real box, a represents the balance parameter, v represents the consistency of the aspect ratio of the real box and the predicted box, and c represents the diagonal length of the minimum area between the predicted box and the real box.

[0029] Furthermore, in step five, the embedded hardware device uses NVIDIA's AI edge hardware platform Jestonnano; the platform system is Ubuntu18.04, equipped with 4GB of running memory, four ARM Cortex-A57 cores and 128 Nvidia Maxwell GPU cores.

[0030] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0031] The obstacle detection method of the visually impaired group guiding vest system based on the improved YOLOv11 proposed in the present invention applies the YOLOv11 model to the obstacle detection task of the visually impaired group during travel for the first time. Taking into account the subsequent edge deployment problem, in order to reduce the amount of calculation, the present invention adopts a lightweight ShuffleNetV2 structure in the backbone network, which significantly reduces the computational complexity while improving the detection speed and effectiveness. In addition, the present invention redesigns the detection head and designs a lightweight detection head to reduce computational redundancy while enhancing the network's detection ability for dense objects. After data set training and model optimization, the improved YOLOv11 network performs well in obstacle detection tasks, which not only improves the accuracy and robustness of target detection, but also realizes efficient deployment and real-time detection in embedded hardware devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a structural diagram of the overall obstacle detection method of the guided vest system;

[0033] Figure 2 This is the structural diagram of the redesigned lightweight detection head;

[0034] Figure 3 It is a work flow chart of the present invention;

[0035] Figure 4 It is a working effect diagram of the present invention. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is fully and clearly described below in conjunction with the accompanying drawings. This embodiment is only used to explain the present invention and does not constitute a limitation on the protection scope of the present invention.

[0037] Figure 1 This is the overall structure diagram of the vest guidance system, which includes the following modules:

[0038] 1. Guidance System Perception Module

[0039] The perception module consists of a camera and the AI ​​edge hardware device Jeston nano.

[0040] Use the camera to collect image information in real time and transmit the image information to the host computer Jeston nano on the vest.

[0041] Jeston nano is used to process and analyze the collected external environment information. The improved YOLOv11 target detection algorithm is deployed in Jeston nano to perform relevant calculations and reasoning for obstacle detection.

[0042] The TensorRT model acceleration framework is used in the process of model reasoning. As a high-performance deep learning software development toolkit launched by NVIDIA, TensorRT's core functions are mainly focused on optimizing the reasoning process of deep learning. This toolkit not only includes an inference optimizer, but also provides the operating environment required for reasoning. TensorRT has a variety of features, such as quantization accuracy control, layer fusion, and kernel tuning. These features enable models optimized by the TensorRT platform to achieve better reasoning performance.

[0043] After obtaining the obstacle category and location information, the information is transmitted to the lower computer through a specific communication protocol to provide decision-making information for the control module.

[0044] 2. Boot system control module

[0045] The control module consists of MCU and vibration sensor.

[0046] The MCU uses the STM32F4 series, which is used to control the vibration sensors on the left and right sides of the vest. Combined with the data transmitted by the host computer, the corresponding vibration sensors are controlled.

[0047] Replacing the backbone network of YOLOv11 with the lightweight network ShffleNetV2 is conducive to achieving real-time detection under limited computing resources.

[0048] The main structure of the network is divided into four parts: Input, Backbone, Neck and Prediction.

[0049] The Input module is the input part consisting of images collected by the camera;

[0050] The Backbone part implements the feature extraction of the image. This part has been replaced by the ShffleNetV2 network, which lightweights the network structure without having much impact on the feature extraction capability.

[0051] ShuffleNetV2 uses group convolution to reduce the amount of computation on feature channels, but this will cause the problem of "feature communication closure". Therefore, channel shuffling is used. Channel shuffling is to recombine the results of convolution of different groups so that the features between groups can be integrated and communicated with each other, thus solving the problem of "feature communication closure" between groups.

[0052] In addition, ShuffleNetV2 also uses depthwise separable convolution to replace the traditional convolution operation. Depthwise separable convolution decomposes the standard convolution into two independent steps: depthwise convolution and pointwise convolution. This decomposition method can significantly reduce the amount of calculation and parameters while maintaining good performance.

[0053] The original detection head of YOLOv11 is replaced with a redesigned lightweight detection head. By using shared convolution, the number of parameters is greatly reduced, making the model lighter. The Scale layer is used to scale the features to solve the problem of inconsistent scales of the targets detected by each detection head. This minimizes the loss of model accuracy with fewer parameters and less computation.

[0054] The Neck part realizes the feature enhancement and fusion of the image.

[0055] The Prediction part realizes the final obstacle target prediction.

[0056] Figure 3 The workflow diagram of obstacle target detection for the vest guidance system for the visually impaired group includes the following specific steps:

[0057] Step 1: The camera collects images;

[0058] The camera and Jeston nano are connected with a data cable. First, the hardware and software need to be initialized, and then the camera is turned on and the data is read.

[0059] Step 2: Image preprocessing;

[0060] The operation of preprocessing each frame of image captured by the camera. Image preprocessing includes color space conversion, image scaling, normalization, and Mosaic data enhancement.

[0061] Color space conversion is to convert an image from RGB color space to other color spaces;

[0062] Image scaling generally involves reducing the input image to a certain pixel size to meet the input requirements of the model.

[0063] Normalization is the process of scaling image data to a specific range, such as [-1, 1]. This is done by subtracting the minimum value (usually 0) from each pixel value and then dividing by the range (usually 255). Normalization is very important for object detection tasks because it helps improve the performance and stability of the model, reduces computation and memory usage, improves the generalization ability of the model, and speeds up the training process.

[0064] Mosaic data enhancement enriches the background and small targets of the detected objects by randomly scaling, cropping and arranging four pictures. This data enhancement method helps the model learn more scene and object changes during training and improves detection performance.

[0065] Step 3: Input the image information into the improved YOLOv11 network;

[0066] The improved YOLOv11 network includes a backbone network (Backbone) ShuffleNetV2, a neck network (Neck) and a detection head (Head / Prediction).

[0067] Step 4: Get the output result of the network;

[0068] The output result includes the category information, confidence information and target box information of the target object.

[0069] In order to verify the effectiveness of our adopted model and improved modules, we conducted ablation experiments on the original YOLOv5 network, the YOLOv5 network with the backbone network replaced by ShuffleNetV2, the original YOLOv8 network, and the YOLOv11 network with the backbone network replaced by ShuffleNetV2, and verified the effectiveness of the improved module by comparing the mean average precision value mAP. Among them, mAP0.5 means that when IoU is set to 0.5, the AP of all images in each category is calculated, and then all categories are averaged, that is, mAP. GFLOPs is a key indicator to measure the computational amount of a deep learning model. It reflects the number of floating-point operations required by the model in a forward propagation or back-propagation process. By calculating GFLOPs, we can understand the computational complexity of the model, which is crucial for evaluating the performance and efficiency of the model. After the experiment, the results shown in Table 1 are obtained:

[0070] Table 1

[0071]

[0072] Conclusion: Under the same data set, ShufleNetV2 network has higher accuracy, is more in line with the requirements of lightweight model, and has lower computational complexity. Considering the average precision and floating point computing power, the improved YOLOv11 network is the most suitable network for this task.

[0073] Through experiments, we can see that each module improvement in this invention improves the accuracy of the model, and each improvement is meaningful.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An obstacle detection method for a visually impaired group guidance vest system based on improved YOLOv11, characterized in that: The steps include: Step 1: Obtain the obstacle dataset of the visually impaired group guidance vest system, complete the dataset annotation work, and divide the dataset into training set, test set and validation set; Step 2: Build the framework of the YOLOv11 target detection algorithm in the Pytorch environment and replace its backbone network with the ShuffleNetV2 network; Step 3: Redesign a lightweight detection head to reduce the model size for subsequent deployment; Step 4: Use the data set to design a data enhancement method, train the improved YOLOv11 network, obtain the optimal model parameters and save them; Step 5: Deploy the improved YOLOv11 algorithm to the embedded hardware device; Step 6: Import the trained network model parameters, perform model inference, and realize real-time obstacle detection of the mobile collaborative robot.

2. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 1, characterized in that: The obstacle dataset in step 1 is divided into a training set, a test set and a validation set; the training set is used for model training and network parameter optimization; the test set is used for model testing during training and comparison of training effects; the validation set is used for model verification after training to verify the effectiveness and generalization of the model; each obstacle dataset includes images and labels, and the labels include the category and location information of each target object in the image, wherein the location information is the coordinates of the center point of the annotation box and the width and height of the annotation box.

3. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 1, characterized in that: The YOLOv11 target detection algorithm in step 2 is a one-stage target detection algorithm based on deep learning, and its network structure includes an input end, a backbone network Backbone, a neck network NECK, and a detection head Head; the input end performs data preprocessing and data enhancement tasks to provide high-quality input data for the model. At the input end, YOLOv11 performs a series of preprocessing operations on the original image, including image scaling and normalization; so that the model can learn image features. In order to improve the generalization ability of the model, YOLOv11 uses data enhancement technology at the input end; The ShuffleNetV2 network adopts grouped convolution and channel shuffling to reduce the amount of computation on feature channels, solve the problem of feature communication closure, and improve the expressiveness of the model. In addition, ShuffleNetV2 also adopts depthwise separable convolution to replace traditional convolution operations. Depthwise separable convolution decomposes standard convolution into two independent steps: depthwise convolution and pointwise convolution.

4. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 3 is characterized in that: The data enhancement technology includes Mosaic data enhancement and adaptive anchor frame calculation; the Mosaic data enhancement enriches the background and small targets of the detected object by splicing four pictures by random scaling, random cropping and random arrangement; at the input end, YOLOv11 adopts adaptive anchor frame calculation technology; the anchor frame is a predefined set of rectangular frames of fixed size and proportion, which are used to predict the position of the object in the target detection task. The adaptive anchor frame calculation technology automatically adjusts the size and proportion of the anchor frame according to the size and proportion of the object in the input image, thereby improving the accuracy of object detection.

5. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 3 is characterized in that, The backbone network Backbone is used for feature extraction. It is a convolutional neural network. The backbone network of YOLOv11 adopts CSPNet structure, which mainly includes CBL structure, CSP structure and SPPF structure. The CBL structure is composed of convolution layer, BN layer and activation function, which simplifies the model structure and improves the speed of model reasoning. The CSP structure is composed of 3 CBL structures and multiple Bottleneck modules with shortcut structures, which reduces the use of video memory and speeds up the reasoning speed during the forward inference period, while solving the problem of gradient data disappearance and explosion in the training stage and enhancing the ability of convolution to extract image feature information. The rich feature information obtained by the CSP structure is further input into the SSPF structure. As the last layer of Backbone, the SSPF structure is used to perform multi-scale feature fusion on the output after input and convolution at different scales. Through multi-scale feature fusion, SSPF can capture contextual information at different levels in the image and improve the network's ability to locate and identify targets. After completing the preliminary feature fusion, the information is passed to the next stage of the network to provide support for subsequent target detection tasks.

6. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 3 is characterized in that: The role of the neck network Neck is feature enhancement fusion. It is responsible for further processing and fusing the features extracted from the Backbone network to improve detection accuracy. The Neck network adopts the structure of the classic feature pyramid network FPN and path aggregation network PANet to extract and fuse features at different scales to improve the network's detection ability for targets of different sizes.

7. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 3 is characterized in that: The detection head is a key part responsible for target detection and classification. It receives three feature maps of different scales output by the previous feature enhancement fusion network. These feature maps have sizes of 20×20, 40×40 and 80×80 respectively. They are sent to the detection layer for coordinate regression prediction and classification tasks. In the detection layer, the network performs target detection on each feature map and predicts the bounding box coordinates and category of the target. Subsequently, the non-maximum suppression algorithm is used to eliminate redundant detection boxes to ensure that each target is detected only once, so that YOLOv11 can effectively predict large, medium and small targets of different types, improve the network's detection performance for dense objects, and make target detection more accurate and reliable.

8. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 1, characterized in that: In step three, the lightweight detection head greatly reduces the number of parameters by sharing convolutions to make the model lighter, and uses the Scale layer to scale the features to solve the problem of inconsistent target scales detected by each detection head, thereby reducing the loss of model accuracy with fewer parameters and less calculation.

9. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 1, characterized in that: In step 4, the training adopts training preheating, cosine annealing to adjust the learning rate, and adaptive anchor frame calculation training strategy to improve the training effect and performance of the model; The training sets the loss function, which includes the target confidence loss Classification Loss and bounding box regression loss The total loss is the weighted sum of the three parts, and the expression is: Among them, the weights of target confidence loss, classification loss and bounding box regression loss are λ1, λ2, λ3, L conf and L cls The binary cross entropy function is used for calculation, and the calculation formula is as follows: L n =-(y n log(δ(x n ))+(1-y n )log(1-δ(x n ))) Where: x n To predict the score of the nth sample as a positive example, y n represents the label of the nth sample, and δ represents the sigmoid function; L reg Use the CloU Loss function. CloU is a bounding box regression loss calculation method. The calculation formula is as follows: Where: parameter A represents the real box, B represents the predicted box, IoU is called intersection over union, which represents the overlap between the predicted box and the real box, ρ 2 (·) represents the Euclidean distance, b represents the coordinates of the center point of the prediction box, and b gt represents the coordinates of the center point of the real box, a represents the balance parameter, v represents the consistency of the aspect ratio of the real box and the predicted box, and c represents the diagonal length of the minimum area between the predicted box and the real box.

10. The obstacle detection method for the visually impaired group guiding vest system based on improved YOLOv11 according to claim 1, characterized in that: In step 5, the embedded hardware device uses NVIDIA's AI edge hardware platform Jestonnano; the platform system is Ubuntu 18.04, equipped with 4GB of running memory, four ARM Cortex-A57 cores and 128 Nvidia Maxwell GPU cores.

Images