Lightweight target positioning method based on improved YOLOv5

Abstract

The invention discloses a lightweight target positioning method based on improved YOLOv5, and the method comprises the steps: replacing a C3 part of a backbone module in a YOLOv5s network with a MobileNetV3 module, reducing the calculation amount of a model through depth separable convolution, and improving the feature extraction efficiency; replacing an original CIoU loss function with an EIoU loss function to enhance the stability and precision of target bounding box regression; lightweight processing is carried out on a stereo matching algorithm, a sparse algebraic reconstruction method is adopted to simplify a cost aggregation algorithm, and calculation complexity and memory consumption are reduced; model deployment optimization is carried out through ONNX Runtime, the reasoning time is shortened, and memory occupation is reduced; three-dimensional coordinates of a target relative to a coordinate system are calculated by combining stereo matching depth information of a binocular camera and YOLOv5 target detection information, so that light-weight target positioning is realized; according to the method, while the target detection and positioning precision is ensured, the operation efficiency of the binocular vision system in the resource-constrained equipment is remarkably improved, and efficient lightweight target positioning is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of target positioning, relates to a lightweight target positioning system, and specifically to a lightweight target positioning method based on improved YOLOv5. Background Art

[0002] With the rapid development of drone technology, three-dimensional positioning technology is widely used in unmanned systems, robots, autonomous navigation, augmented reality and other fields.

[0003] Common target positioning uses visual SLAM synchronous positioning, laser radar positioning, etc. For unmanned systems, deep learning and binocular vision are usually combined to develop positioning technology. Common deep learning models include YOLO (You Only Look Once) and Faster R-CNN (Region-based Convolutional Neural Networks).

[0004] In the unmanned system environment, the existing deep learning models may be limited by factors such as the scale, occlusion, and illumination changes of the target during target positioning, which seriously affects its stability and reliability in practical applications. In addition, the existing technology has certain limitations in the integration of network structure and algorithm optimization when realizing lightweight target positioning of binocular vision systems, and fails to give full play to the advantages of binocular vision systems in resource-constrained environments. There are defects such as insufficient accuracy in target detection and target positioning, and large number of algorithm parameters and resource consumption during target positioning. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a lightweight target positioning method based on improved YOLOv5, to improve the accuracy of target detection and target positioning, and to reduce the number of parameters and resource consumption of the algorithm when performing target positioning.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A lightweight target positioning method based on improved YOLOv5 includes the following steps:

[0008] Step 1: Use the camera calibration tool in MATLAB to calibrate the black and white grid images collected by the binocular camera to obtain the internal and external parameters of the camera;

[0009] Step 2: Improve the YOLOv5s model by modifying the network feature extraction structure of YOLOv5s and the regression box loss function to improve the model's positioning accuracy for the target and reduce the model parameters and computational complexity.

[0010] Step 3: Deploy the improved model into the target positioning algorithm, and use the reasoning method of ONNX Runtime to perform target detection on the binocular images collected by the binocular camera in real time;

[0011] Step 4: Calculate the disparity of the left and right images obtained by the binocular camera through a lightweight stereo matching algorithm using sparse algebraic reconstruction;

[0012] Step 5: Use triangulation to convert the two-dimensional coordinates on the image into three-dimensional space coordinates relative to the binocular camera, and obtain the depth information of each pixel to complete the lightweight positioning of the target.

[0013] The present invention also has the following technical features:

[0014] Preferably, the internal and external parameters of the camera described in step 1 include focal length, principal point coordinates, distortion parameters and baseline.

[0015] Preferably, the improvement to the YOLOv5s model described in step 2 includes:

[0016] S1 replaces the standard convolutional layer in the C3 module in the original Backbone network layer with the depth-separable convolutional layer in MobileNetV3;

[0017] S2 replaces the original bounding box regression loss function CIoU with EIoU to obtain the improved YOLOv5s model structure;

[0018] S3 uses the improved model training structure for model training.

[0019] Furthermore, the C3 module described in S1 is replaced by the MobileNetV3 module, including the following steps:

[0020] 1) Perform convolution on each input channel independently and output the same number of channels;

[0021] 2) Combine the output channels of the depthwise convolution using 1x1 convolution to get the desired number of output channels.

[0022] 3) Make sure the number of input and output channels of the MobileNetV3 module is consistent with the corresponding parts of the C3 module in YOLOv5s.

[0023] Furthermore, the formula for combining the output channels of the depth convolution is:

[0024] Y=X′*K 1×1

[0025] Where Y represents the output feature map, X′ represents the output of the deep convolution, and K 1×1 is a 1×1 convolution kernel.

[0026] Furthermore, the MobileNetV3 module composition includes a lightweight hardened activation function H-Swish. The H-Swish formula is:

[0027]

[0028] Among them, HSwish represents the hardening activation function H-Swish, x represents the value input to the activation function, usually the output of the neurons in the previous layer, and ReLU6(x) is defined as ReLU6(x)=max(0,min(6,x)).

[0029] Preferably, the sparse algebraic reconstruction in step 4 includes:

[0030] S1 uses the FAST feature detection algorithm to calculate the response value of each feature point. For each feature point, its response is calculated. The calculation formula is as follows:

[0031] R i =f(I, x i ,y i )

[0032] Among them, R i is the response value of each feature point, (x i ,y i ) represents the coordinates of the feature points, f is the feature detection function, and I is the image.

[0033] S2 sets a response value threshold and compares it with the response values ​​of pixels around the feature point to select feature points with response values ​​higher than the threshold.

[0034] S3 only matches the selected sparse feature points, uses descriptors for matching, and defines the matching cost formula as follows:

[0035] C(P i , P j )=||D i -D j || 2

[0036] Among them, D i and D j is the feature point P i and P j The descriptor of .

[0037] S4 filters low-quality matching pairs through ratio testing to ensure the reliability of matching.

[0038] Preferably, the calculation process of the three-dimensional coordinates in step 5 includes:

[0039] S1 collects the depth information of the pedestrian target collected after target recognition, and calculates the depth information D of the target through the SGBM algorithm and the triangle similarity principle. The calculation formula is as follows:

[0040]

[0041] Where L is the baseline distance, H is the camera focal length, and d is the parallax of the left and right cameras;

[0042] S2 calculates the pixel coordinate information of the pedestrian center point through the coordinates of the pedestrian target detection frame. The calculation formula is as follows:

[0043]

[0044]

[0045] Among them, x left ,x right ,y top ,y bottom are the x and y coordinates of the upper left and lower right corners of the target detection box respectively;

[0046] S3 converts the pixel coordinate information of the target object into three-dimensional coordinates. The calculation formula is as follows:

[0047]

[0048]

[0049] Z=D

[0050] Among them, (c x ,c y ) is the pixel coordinate of the target, H is the focal length of the camera, and D is the depth information of the target, so the three-dimensional coordinates of the target object are (X, Y, Z).

[0051] Compared with the prior art, the present invention has the following technical effects:

[0052] The present invention performs target positioning based on YOLOv5, and improves the network structure of its model training structure YOLOv5s, replaces the backbone network convolution mode with depth separable convolution, replaces the regression box loss function with EIoU Loss, uses the images of the left and right perspectives acquired by the binocular camera, and calculates the depth information of the pixel points through the lightweight stereo matching technology, so as to achieve lightweight positioning of the target. The present invention combines the binocular vision technology, the improved YOLOv5 target detection model, and the improved stereo matching method, can improve the accuracy of target detection and target positioning, greatly reduce the computing resource consumption when the unmanned system performs target positioning, and compared with the traditional target positioning method, the model training efficiency is improved by about 7.6%, and the measurement positioning error is reduced by about 1.8%. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is an overall flow chart of a lightweight target positioning method based on improved YOLOv5 of the present invention;

[0054] Figure 2 It is the improved YOLOv5s network structure. DETAILED DESCRIPTION

[0055] The specific contents of the present invention are further explained in detail below in conjunction with embodiments.

[0056] Reference Figure 1 and Figure 2 This embodiment provides a high-precision pedestrian positioning and tracking method based on vehicle-mounted binocular vision, including the following steps:

[0057] Step 1: Collect the black and white grid calibration plate photo data through the binocular camera, organize the data into a data set, import it into MATLAB, delete the data with large errors, calibrate the binocular camera to obtain the internal and external parameters of the camera, including focal length, principal point coordinates, distortion parameters and baseline, etc., to provide an accurate camera parameter basis for subsequent target positioning;

[0058] Step 2: Improve the YOLOv5s model by modifying the network layer and replacing the loss function to improve the positioning efficiency of the model in a resource-constrained environment;

[0059] Improvements to the YOLOv5s model include:

[0060] S1 replaces the C3 module in the Backbone layer of YOLOv5s version 6.0 with the MobileNetV3 module proposed by the Google team. By combining technologies such as deep separable convolution and linear bottleneck, the computational complexity of the model is significantly reduced.

[0061] S2 replaces the original bounding box regression loss function CIoU with EIoU to obtain the improved YOLOv5s model structure; EIoU further improves the stability and accuracy of target box regression based on CIoU;

[0062] S3 uses self-collected photos of pedestrians and fruits for model training, including 1,028 training sets and 131 test sets, totaling 1,159 images and their labels, including targets in different weather, lighting, and backgrounds. Through unsupervised learning training, a high-precision pedestrian detection model is trained to provide reliable support for subsequent target recognition.

[0063] Step 3: Deploy the improved model into the target positioning algorithm, and use the reasoning method of ONNX Runtime to perform target detection on the binocular images collected by the binocular camera in real time.

[0064] The reasoning process using ONNX Runtime includes:

[0065] S1 exports the original model format to ONNX format;

[0066] S2 sets up the ONNX Runtime inference environment and imports ONNX Runtime and other necessary libraries;

[0067] S3 uses the prepared input data for inference, parses the inference results according to the output format of YOLOv5, and extracts the target detection information.

[0068] Step 4: Calculate the disparity of the left and right images obtained by the binocular camera using a lightweight stereo matching algorithm with sparse algebraic reconstruction.

[0069] The sparse algebraic reconstruction process includes:

[0070] S1 uses the FAST feature detection algorithm to calculate the response value of each feature point. For each feature point, its response is calculated. The calculation formula is as follows:

[0071] R i =f(I, x i ,y i )

[0072] Among them, R i is the response value of each feature point, (x i ,y i ) represents the coordinates of the feature points, f is the feature detection function, and I is the image;

[0073] S2 sets a response value threshold and compares it with the response values ​​of the pixels around the feature point to select the feature point with a response value higher than the threshold;

[0074] S3 only matches the selected sparse feature points, uses descriptors for matching, and defines the matching cost formula as follows:

[0075] C(P i , P j )=||D i -D j || 2

[0076] Among them, D i and D j is the feature point P i and P j Descriptor of ;

[0077] S4 filters low-quality matching pairs through ratio testing to ensure the reliability of matching.

[0078] Step 5: Use triangulation to convert the two-dimensional coordinates on the image into three-dimensional space coordinates relative to the binocular camera, and obtain the depth information of each pixel to complete the lightweight positioning of the target;

[0079] The calculation process of three-dimensional coordinates includes:

[0080] S1 collects the depth information of the pedestrian target collected after target recognition, and calculates the depth information D of the target through the lightweight stereo matching algorithm and the triangle similarity principle. The calculation formula is as follows:

[0081]

[0082] Where L is the baseline distance, H is the camera focal length, and d is the parallax of the left and right cameras;

[0083] S2 calculates the pixel coordinate information of the pedestrian center point through the coordinates of the pedestrian target detection frame. The calculation formula is as follows:

[0084]

[0085]

[0086] Among them, x left , x right ,y top ,y bottom are the x and y coordinates of the upper left and lower right corners of the target detection box respectively;

[0087] S3 converts the pixel coordinate information of the target object into three-dimensional coordinates. The image size is 640×480. The image pixel coordinates obtained by the code are based on the upper left vertex as the origin, so the center coordinates of the image are (320,240). The calculation formula is as follows:

[0088]

[0089]

[0090] Z=D

[0091] Among them, (c x ,c y ) is the pixel coordinate of the target, H is the focal length of the camera, and D is the depth information of the target, so the three-dimensional coordinates of the target object are (X, Y, Z).

Claims

1. A lightweight target positioning method based on improved YOLOv5, characterized in that: The following steps are involved: Step 1: Use the camera calibration tool in MATLAB to calibrate the black and white grid images collected by the binocular camera to obtain the internal and external parameters of the camera; Step 2: Improve the YOLOv5s model by modifying the network feature extraction structure of YOLOv5s and the regression box loss function to improve the model's positioning accuracy for the target and reduce the model parameters and computational complexity. Step 3: Deploy the improved model into the target positioning algorithm, and use the reasoning method of ONNX Runtime to perform target detection on the binocular images collected by the binocular camera in real time; Step 4: Calculate the disparity of the left and right images obtained by the binocular camera through a lightweight stereo matching algorithm using sparse algebraic reconstruction; Step 5: Use triangulation to convert the two-dimensional coordinates on the image into three-dimensional space coordinates relative to the binocular camera, and obtain the depth information of each pixel to complete the lightweight positioning of the target.

2. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 1, characterized in that: The intrinsic and extrinsic parameters of the camera described in step 1 include focal length, principal point coordinates, distortion parameters and baseline.

3. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 1, characterized in that: The improvements to the YOLOv5s model described in step 2 include: S1 replaces the standard convolutional layer in the C3 module in the original Backbone network layer with the depth-separable convolutional layer in MobileNetV3; S2 replaces the original bounding box regression loss function CIoU with EIoU to obtain the improved YOLOv5s model structure; S3 uses the improved model training structure for model training.

4. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 3, characterized in that: Replacing the C3 module described in S1 with the MobileNetV3 module includes the following steps: 1) Perform convolution on each input channel independently and output the same number of channels; 2) Combine the output channels of the depthwise convolution using 1x1 convolution to obtain the desired number of output channels; 3) Ensure that the number of input and output channels of the MobileNetV3 module is consistent with YOLO v The corresponding part of C3 module in 5s is consistent.

5. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 4, characterized in that: The formula for combining the output channels of the depth convolution using 1x1 convolution is: Y=X′*K 1×1 Among them, Y represents the output feature map, X′ represents the output of the deep convolution, and K 1×1 is a 1×1 convolution kernel.

6. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 4, characterized in that: The MobileNetV3 module composition includes a lightweight hardened activation function H-Swish, and the H-Swish formula is: Among them, HSwish represents the hardening activation function H-Swish, x represents the value input to the activation function, usually the output of the previous layer of neurons, and ReLU6(x) is defined as ReLU6(x)=max(0,min(6,x)).

7. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 1, characterized in that: The sparse algebraic reconstruction described in step 4 includes: S1 uses the FAST feature detection algorithm to calculate the response value of each feature point. For each feature point, its response is calculated. The calculation formula is as follows: R i =f(I,x i ,y i ) Among them, R i is the response value of each feature point, (x i ,y i ) represents the coordinates of the feature points, f is the feature detection function, and I is the image; S2 sets a response value threshold and compares it with the response values ​​of the pixels around the feature point to select the feature point with a response value higher than the threshold; S3 only matches the selected sparse feature points, uses descriptors for matching, and defines the matching cost formula as follows: C(P i ,P j )=||D i -D j || 2 Among them, D i and D j is the feature point P i and P j Descriptor of ; S4 filters low-quality matching pairs through ratio testing to ensure the reliability of matching.

8. The lightweight target positioning method based on improved YOLOv5 as claimed in claim 1, characterized in that: The calculation process of the three-dimensional coordinates described in step 5 includes: S1 collects the depth information of the pedestrian target collected after target recognition, and calculates the depth information D of the target through the SGBM algorithm and the triangle similarity principle. The calculation formula is as follows: Where L is the baseline distance, H is the camera focal length, and d is the parallax of the left and right cameras; S2 calculates the pixel coordinate information of the pedestrian center point through the coordinates of the pedestrian target detection frame. The calculation formula is as follows: Among them, x left ,x right ,y top ,y bottom are the x and y coordinates of the upper left and lower right corners of the target detection box respectively; S3 converts the pixel coordinate information of the target object into three-dimensional coordinates. The calculation formula is as follows: Z=D Among them, (c x ,c y ) is the pixel coordinate of the target, H is the focal length of the camera, and D is the depth information of the target, so the three-dimensional coordinates of the target object are (X, Y, Z).

Images