An armor plate detection tracking method for mobile robots

Abstract

The present application relates to the technical field of image processing, and more particularly to an armor plate detection and tracking method for a mobile robot, comprising collecting mobile robot armor plate pictures; improving the YOLOv5s network for small armor plate targets; and constructing a DeepSort model to predict and track the detected armor plate. The present application adds a CA attention module in the backbone network of YOLOv5s, so that the network obtains information of a larger area, strengthens feature extraction, and improves armor plate detection accuracy; deletes the large target detection layer in the YOLOv5s network, while maintaining detection accuracy and improving detection speed; uses the DeepSort algorithm to track the detected targets, uses Kalman filtering and the Hungarian algorithm to solve the association problem between the current frame detection frame and the track, improves the accuracy of data association, and effectively improves the tracking effect of the armor plate in the case of being blocked and complex background light.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an armor plate detection and tracking method for a mobile robot. BACKGROUND

[0002] In recent years, under the environment of rapid development of artificial intelligence, mobile robots are widely used in industries such as industry, agriculture, and medical treatment. The main research technologies of mobile robots include environment perception, motion control, and path planning. In practical applications, target tracking technology has a great influence on the performance of mobile robots. Due to the rapid movement and complex and changeable working environment, real-time and accurate target tracking technology has become the focus of mobile robot research.

[0003] Taking the global robot competition RoboMaster as the research background, the target tracking technology of the mobile robot is studied. In the RoboMaster competition, whether the armor plate can be correctly identified is the key to winning the competition. In the competition, the images obtained by the camera are usually simply preprocessed, and then a matching contour method is used to identify the armor plate and perform attacks. Although this method can meet the real-time requirements of the competition, the on-site environment is complex, the light is chaotic, and there are many occlusions, which can easily lead to incorrect identification of the armor plate. SUMMARY

[0004] In view of the shortcomings of the existing algorithm, the improved algorithm is applied to the mobile robot, and the armor plate of the mobile robot can be stably tracked under the conditions of being occluded and complex background light.

[0005] The technical scheme adopted by the present application is: an armor plate detection and tracking method for a mobile robot comprises the following steps:

[0006] Step 1: collect the armor plate pictures of the mobile robot and obtain the data set;

[0007] Step 2: improve the YOLOv5s network for small targets of the armor plate, input the training set into the improved YOLOv5s network for training, obtain a YOLOv5s detection model, and use the YOLOv5s detection model for detection of the armor plate of the mobile robot;

[0008] Further, step 2 specifically comprises:

[0009] Step 2.1: add a CA attention module before the SPPF layer of the YOLOv5s network, and embed the position information into the channel attention;

[0010] Step 2.2: delete the large target detection layer in the YOLOv5s network, delete the anchors of the large target detection layer, and delete the network layer corresponding to the large target detection layer;

[0011] Further, delete the anchors of the large target detection layer as (116, 90), (156, 198) and (373, 326), and get 6 anchor boxes of the anchor module as (10, 13), (16, 30), (33, 23), (30, 61), (62, 45) and (59, 119);

[0012] Delete the 8th layer set convolution layer, the 9th layer C3 module, the 12th layer upsampling layer, the 22nd-24th layer in the head and the large target detection head in the detection layer in the original YOLOv5s network.

[0013] Step 2.3. Constructing the improved YOLOv5s network;

[0014] Further, specifically comprising:

[0015] 1. Initializing the YOLOv5s network parameters, normalizing the image, and setting the batch size, epochs, initial learning rate, IOU threshold, momentum, weight decay, confidence threshold and non-maximum overlap rate;

[0016] 2. Preprocessing the armor plate data set, including Mosaic data enhancement, adaptive anchor box calculation and adaptive image scaling;

[0017] 3. Inputting the image into the backbone module fused with CA attention to generate feature maps of multiple sizes, and then performing upsampling, downsampling and Concat splicing operations to generate feature maps of different scales;

[0018] 4. Inputting the feature maps of different scales into the Detect layer of the head part to obtain the detection box position, class and confidence of the target to be detected, deleting the detection box with a confidence less than the set threshold, calculating the IOU between the detection box and the real box, and then using the positioning loss, classification loss and confidence loss for back propagation, and updating and adjusting the gradient in the YOLOv5s network to adjust the weight parameters.

[0019] Step three, constructing a DeepSort model to predict and track the detected armor plate;

[0020] Further, specifically comprising:

[0021] Step 3.1, creating a corresponding tracking box according to the detection box of the first frame target, initializing the parameters of the Kalman filter using the detection box information of the first frame target, and predicting the target position information of the next frame through the Kalman filter;

[0022] Step 3.2, the detection frame and the prediction frame of the current frame target are matched by IOU, the cost matrix is calculated by integrating motion information and appearance information through the Mahalanobis distance and the feature cosine distance of the target frame, the matching successful tracking frame and the detection frame, the matching failed detection frame and the matching failed tracking frame are obtained by taking the cost matrix as the input of the Hungarian algorithm for linear matching, and the step is executed until the confirmation frame or the tracking ends;

[0023] Further, step 3.2 includes: updating the Kalman filter parameters of the matching successful prediction frame; the matching failed detection frame is considered as a new target, and a new prediction frame is newly built for the matching failed detection frame; at this time, the matching failed tracking frame is a tracking frame in an unconfirmed state, and the tracking frame is deleted; the tracking frame in the unconfirmed state is matched successfully for three consecutive frames to become a tracking frame in a confirmed state; and the tracking frame in the confirmed state is matched unsuccessfully for 30 consecutive frames, so that the target is considered to be lost, and the tracking frame is deleted.

[0024] Step 3.3, the detection frame of the current frame and the prediction frame of the last frame confirmation frame are matched in cascade, priority is given to the target object with high frequency of occurrence, the Kalman filter parameters of the matching successful prediction frame are updated, the mismatched prediction frame, the previous unconfirmed prediction frame and the mismatched detection frame are matched by IOU and the cost matrix is calculated, and steps 3.2 and 3.3 are repeated until the tracking ends.

[0025] Advantages of the present application:

[0026] 1. A CA attention module is added in the backbone network of YOLOv5s, the position information is embedded into the channel attention, the network obtains more regional information, the feature extraction is strengthened, and the armor plate detection precision is improved.

[0027] 2. The large target detection layer in the YOLOv5s network is deleted for small targets of the armor plate, the detection speed is improved while the detection precision is maintained, the DeepSort algorithm is used to track the detected targets, the Kalman filter and the Hungarian algorithm are used to solve the association problem of the current frame detection frame and the track, the accuracy of data association is improved, and the tracking effect of the armor plate under the conditions of being occluded and complex background light is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is the improved YOLOv5s network structure diagram of the present application;

[0029] Figure 2 is the P-R curve diagram of the present application;

[0030] Figure 3 is the visualized parameter schematic diagram of the YOLOv5s model training of the present application;

[0031] Figure 4It is an improved YOLOv5s target detection comparison chart of the application. DETAILED DESCRIPTION

[0032] The application will be further described below in conjunction with the accompanying drawings and examples, which are simplified schematic diagrams and only schematically show the basic structure of the application, and thus only show the components related to the application.

[0033] As shown in the drawings, Figure 1 An armor plate detection and tracking method for a mobile robot includes the following steps:

[0034] Step 1, collect mobile robot armor plate pictures to obtain a data set;

[0035] Collect 2500 mobile robot armor plate pictures under different scenes, divide the armor plates in the pictures into two classes of red and blue, use the labeling tool LabelImg to label the pictures, write the labeling information into an xml file, divide the training set and the test set according to the ratio of 8:2, and convert the xml format into a txt file. Finally, the training set is 2000 pictures, the test set is 500, the corresponding targets in the pictures are segmented out using the labeled xml file, the pictures are renamed, the size is unified, and the training set and the validation set are divided to obtain a DeepSort tracking model.

[0036] Step 2, improve the YOLOv5s network for small armor plate targets, input the training set into the improved YOLOv5s network for training, obtain a YOLOv5s detection model, and use it for detection of mobile robot armor plates;

[0037] Further, step 2 specifically includes:

[0038] Step 2.1: add a CA attention module before the SPPF layer of the YOLOv5s network, embed the position information into the channel attention, so that the mobile network obtains information of a larger area, strengthens feature extraction, and improves the armor plate detection accuracy;

[0039] Step 2.2, delete the large target detection layer in the YOLOv5s network, and delete the anchors of the large target detection layer and the network layer corresponding to the large target detection layer;

[0040] 1. The anchors of the large target detection layer are (116, 90), (156, 198) and (373, 326), and 6 anchor boxes of the anchor module are obtained, which are (10, 13), (16, 30), (33, 23), (30, 61), (62, 45) and (59, 119);

[0041] 2. Delete the 8th layer set convolution layer, the 9th layer C3 module, the 12th layer up-sampling layer, the 22-24th layer in the head and the large target detection head in the detection layer in the original YOLOv5s network.

[0042] Step 2.3, the YOLOv5s network mainly includes an input end, a backbone layer, a neck layer and a head layer, specifically including:

[0043] 1. Initialize the model parameters, the operating system used in the application is Ubuntu 18.04, the learning framework of the whole experiment is Pytorch 1.9.1, the experimental environment is Python 3.7, the GPU acceleration software is CUDA 11.3 and CUDNN 8.4; In the experiment, the batch size is 8, the epochs is set to 100, the image normalization size is 640x640, the initial learning rate is 0.01, the IOU threshold is 0.5, the momentum is 0.937, the weight decay is 0.0005, the confidence threshold is 0.3, and the non-maximum overlap rate is 0.5;

[0044] 2. Preprocess the armor plate data set, wherein the preprocessing includes Mosaic data enhancement, adaptive anchor frame calculation and adaptive picture scaling; wherein the Mosaic data enhancement: four pictures are spliced by random scaling, cropping and arrangement; adaptive anchor frame calculation: different anchor frames need to be set for different data sets, and the kmeans algorithm and genetic algorithm are used to obtain the preset anchor frame suitable for the target in the data set; adaptive picture scaling: since the picture length and width are not the same, the picture is usually scaled to a fixed size, the length-width ratio is kept as much as possible, and the missing edge is filled with gray;

[0045] 3. Then the image is input into the backbone part fused with CA attention to generate feature maps of multiple sizes, and then the up-sampling operation, down-sampling operation and Concat splicing operation are performed to complete multi-feature fusion, and feature maps of different scales are generated.

[0046] 4. The feature maps of different scales are input into the Detect layer of the head part, the detection frame position, class and confidence of the target to be detected can be obtained, the detection frame with a confidence less than a set threshold is deleted, the IOU between the detection frame and the real frame is calculated, then the positioning loss, classification loss and confidence loss are used for back propagation, and the gradient in the YOLOv5s network is updated to adjust the weight parameters, finally the model in each stage is verified by using the verification set, and the YOLOv5s detection model is obtained.

[0047] The application mainly uses precision (P), recall rate (R) and average precision mean mAP as evaluation indexes to verify the performance of the improved YOLOv5s algorithm, and the specific calculation formula is:

[0048]

[0049]

[0050]

[0051] In the formula, TP refers to the number of positive samples predicted by positive samples, FP is the number of negative samples predicted by positive samples, FN is the number of positive samples predicted by negative samples; Precision refers to the ratio of correct prediction in all predicted positive samples; Recall refers to the ratio of correct prediction in all positive samples; AP is the integral area of the P-R curve, indicating the good and bad of the trained model in the current category, and the curve diagram is as shown in Figure 2 num_classes refers to the total number of classes, and mAP refers to the good and bad in all classes. The larger the mAP, the better the network performance.

[0052] To verify the superiority of the improved YOLOv5s algorithm performance in the present application, the YOLOv5s original model, the YOLOv5s model fused with CA attention and the model of the present application are respectively trained and compared under the collected armor plate data set. A large number of parameters are displayed in the visual training as shown in Figure 3 The comparison data are as shown in the following table:

[0053] Algorithm Precision Recall mAP YOLOv5s 88.2％ 75.7％ 82.2％ YOLOv5s+CA 89.0％ 76.7％ 84.1％ Algorithm 91.5％ 75.8％ 84.3％

[0054] The comparison chart of the detection results is as shown in FIG. 4. It can be seen that the performance of the algorithm of the present application is generally optimal.

[0055] Step 3, constructing a DeepSort model to predict and track the detected armor plate;

[0056] Step 3 specifically includes:

[0057] Step 3.1, creating a corresponding tracking box according to the detection box of the first frame target, initializing the parameters of the Kalman filter using the detection box information of the first frame target, and predicting the target position information of the next frame through the Kalman filter;

[0058] Step 3.2, matching the detection box and the predicted box of the current frame target through IOU, integrating motion information and appearance information through Mahalanobis distance and feature cosine distance to calculate a cost matrix, taking the cost matrix as the input of the Hungarian algorithm to perform linear matching, obtaining the matching successful tracking box and detection box, the matching failed detection box, and the matching failed tracking box, and executing this step until the confirmation box or the tracking ends;

[0059] Step 3.2 further comprises: updating the Kalman filter parameters of the matched successful prediction box; the detection box that fails to match is considered to be a new target, and a new prediction box is created for it; at this time, the tracking box that fails to match is a tracking box in an unconfirmed state, which is deleted; the tracking box in the unconfirmed state is matched successfully for three consecutive frames to become a tracking box in a confirmed state; the tracking box in the confirmed state is matched unsuccessfully for 30 consecutive frames, and the target is considered to be lost, and the tracking box is deleted.

[0060] Step 3.3, the detection box of the current frame and the prediction box of the last frame confirmed box are matched, the target object with high frequency is given priority, the Kalman filter parameters of the matched successful prediction box are updated, the unmatched prediction box, the previous unconfirmed prediction box and the unmatched detection box are matched and the cost matrix is calculated, and steps 3.2 and 3.3 are repeated until the tracking is completed.

[0061] Experimental process:

[0062] The YOLOv5s model and the DeepSort model are used for real-time detection and tracking of the robot armor plate.

[0063] Specifically, it comprises:

[0064] 1. The picture is read from the camera and imported into the network.

[0065] 2. First, enter the YOLOv5s model, extract features through the backbone layer, fuse the extracted features through the neck layer, finally output the fused features for detection, and set a confidence threshold to filter the detection box, and delete the detection box with a confidence lower than the threshold.

[0066] 3. The Kalman filter is initialized using the position information of the first frame armor plate, and then the armor plate is matched, tracked, predicted and updated according to the detection box of YOLOv5s and the tracking box of Kalman filter.

[0067] With the above ideal embodiments according to the present application as inspiration, through the above description, relevant personnel can make various changes and modifications without deviating from the scope of the technical idea of the present application. The technical scope of the present application is not limited to the contents in the specification, and must be determined according to the scope of the claims.

Claims

1. A method for detecting and tracking armor plates for mobile robots, characterized in that, Includes the following steps: Step 1: Collect images of the mobile robot's armor plates to obtain a dataset; the armor plates are divided into two categories: red and blue. Step 2: Improve the YOLOv5s network for small targets on the armor plate. Input the training set into the improved YOLOv5s network for training to obtain the YOLOv5s detection model, which is used to detect the armor plate of the mobile robot. Step 2.1: Add a CA attention module before the SPPF layer of the YOLOv5s network to embed the location information into the channel attention; Step 2.2: Delete the large object detection layer in the YOLOv5s network, and delete the anchors and the network layer corresponding to the large object detection layer. The anchors of the large object detection layer are (116,90), (156,198) and (373,326), resulting in 6 anchor boxes in the anchors module, which are (10,13), (16,30), (33,23), (30,61), (62,45), and (59,119). Remove the convolutional layer at layer 8, the C3 module at layer 9, the upsampling layer at layer 12, layers 22-24 in the head, and the large target detection head in the detection layer from the original YOLOv5s network; Step 2.3 Construct an improved YOLOv5s network; Step 3: Construct a DeepSort model to predict and track the detected armor plates.

2. The armor plate detection and tracking method for a mobile robot according to claim 1, characterized in that, Step 2.3 specifically includes: Initialize the YOLOv5s network parameters, normalize the image, and set the batch size, epochs, initial learning rate, IOU threshold, momentum, weight decay, confidence threshold, and non-maximum overlap rate. Preprocessing of the armor plate dataset includes: Mosaic data augmentation, adaptive anchor box calculation, and adaptive image scaling; The image is input into the backbone module that integrates CA attention to generate feature maps of various sizes. Then, upsampling, downsampling, and Concat stitching operations are performed to generate feature maps of different scales. Feature maps of different scales are input into the Detect layer of the head part to obtain the location, category and confidence of the detection box of the target to be detected. Detection boxes with confidence less than a set threshold are deleted. The IOU between the detection box and the ground truth box is calculated. Then, backpropagation is performed using localization loss, classification loss and confidence loss, and the gradient in the YOLOv5s network is updated to adjust the weight parameters.

3. The armor plate detection and tracking method for a mobile robot according to claim 1, characterized in that, Step three specifically includes: Step 3.1: Create a corresponding tracking box based on the target detection box in the first frame, initialize the parameters of the Kalman filter using the target detection box information in the first frame, and predict the target position information in the next frame using the Kalman filter. Step 3.2: Perform IOU matching on the detection box and prediction box of the target in the current frame. Integrate motion information and appearance information by using the Mahalanobis distance and feature cosine distance of the target box to calculate the cost matrix. Use the cost matrix as input to the Hungarian algorithm for linear matching to obtain the successfully matched tracking box and detection box, the unmatched detection box, and the unmatched tracking box. Execute this step until a confirmation box appears or tracking ends. Step 3.3: Perform concatenated matching of the detection boxes in the current frame and the predicted boxes of the confirmed boxes in the previous frame. Prioritize targets that appear frequently. Update the Kalman filter parameters of the successfully matched predicted boxes. Perform IOU matching of the mismatched predicted boxes, the unconfirmed predicted boxes and the mismatched detection boxes and calculate the cost matrix. Repeat steps 3.2 and 3.3 until the tracking ends.

4. The armor plate detection and tracking method for a mobile robot according to claim 3, characterized in that, Step 3.2 also includes: updating the Kalman filter parameters of the successfully matched prediction boxes; for the failed detection boxes, it is assumed that a new target has appeared, and a new prediction box is created for the failed detection boxes; the failed tracking boxes are in an unconfirmed state and are deleted; if an unconfirmed tracking box is successfully matched for three consecutive frames, it becomes a confirmed tracking box; if a confirmed tracking box fails to match for 30 consecutive frames, it is assumed that the target has been lost and the tracking box is deleted.

Images