A dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms

Abstract

The present invention discloses a dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms. First, an image captured by a loaded camera is used as input, and then a convolutional neural network is used to perform nonlinear processing on the image. Finally, a rectangular bounding box is used to select and label the identified content, and the desired target is identified and detected. YOLOv5s is a model based on a deep neural network, which predicts the position and category of the desired target by using a convolutional neural network. The CSRT algorithm tracks the target by extracting image features and superimposing machine learning concepts, and updates features online in real time according to changes in the target and environment. Specifically, it includes: constructing a spatial constraint correlation filter; constructing a spatial reliable area; channel reliability estimation; and tracking channel and spatial reliability. The present invention ensures the accuracy and real-time performance of mobile target recognition and tracking by fusing the two major visual algorithms, YOLOv5s and CSRT.

Description

Technical Field

[0001] The present invention belongs to the technical field of target recognition, and in particular relates to a dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms. Background Art

[0002] In recent years, the combined application of drones and machine vision has become increasingly widespread, extending beyond simple applications like aerial photography. Using a drone's camera, an onboard computer can read the camera image and simultaneously perform parallax calculations on the image information to generate a depth map, thereby obtaining information about the camera's relative distance to the target. The camera's sensitivity to moving targets also facilitates their identification, location, and tracking. The use of visual information to identify and locate specific targets has been extensively researched and applied in the robotics field both domestically and internationally, yielding considerable scientific research results.

[0003] After a drone recognizes a marker, it needs to use this feature to obtain its relative position information. Pose solution refers to the conversion between the world coordinate system and the onboard camera coordinate system based on image coordinates. Currently, the commonly used method is the PnP (Perspective-n-Point) pose solution method; to achieve higher solution accuracy, GPS (Global Positioning System) or RTK is often used to obtain more precise target position information. Methods based on image pixel information are the simplest and most effective, but due to image resolution issues, position information errors are large, and there is also the problem that position errors vary with the distance between the onboard camera and the moving target. PnP-based pose solution methods are relatively accurate, but in the absence of accurate depth information, fixed-rotor drones can easily lose track of the target. Estimation methods that integrate GPS or RTK offer the highest accuracy, but if GPS is lost or RTK fails, the drone will lose the target. These methods are relatively complex and require a lot of computing power.

[0004] In summary, mobile target recognition and positioning and tracking algorithms still face problems such as low positioning accuracy, poor real-time tracking, and easy target loss. However, enabling drones to achieve precise mobile landing on moving ships places higher demands on existing technologies. Not only does it require the onboard vision system to accurately identify the moving ship platform, but it also requires real-time, stable, and high-precision positioning methods to estimate the position of the moving ship. Furthermore, drones must be able to handle issues such as target recognition being affected by lighting and obscured, resulting in unclear recognition features. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this paper proposes a dual-redundancy vision algorithm based on the fusion of YOLOv5s and the CSRT algorithm. This paper seeks to explore a more accurate and real-time vision algorithm under conditions such as high-speed movement of ship landing platforms, significant environmental impact of visual navigation, and real-time communication link constraints of RTK navigation.

[0006] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0007] A dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms. The YOLOv5s is a model based on a deep neural network and predicts the position and category of a desired target by using a convolutional neural network. First, an image captured by a loaded camera is used as input, and then a convolutional neural network is used to perform nonlinear processing on the image. Finally, a rectangular bounding box is used to select and label the recognized content, and the desired target is identified and detected. The CSRT algorithm tracks the target by extracting image features and superimposing machine learning concepts, and updates features online in real time according to changes in the target and environment. The algorithm is specifically divided into four steps: Step S1: constructing a spatial constraint correlation filter; Step S2: constructing a spatially reliable region; Step S3: estimating channel reliability; and Step S4: tracking channel and spatial reliability.

[0008] Furthermore, the overall structure of YOLOv5s consists of four parts: input end, backbone system, Neck network layer, and Head output end;

[0009] The input end mainly performs preprocessing operations on the input image, including data enhancement and image filling. In addition, the input end also integrates adaptive anchor box calculation, which can automatically set the size of the initial anchor box when the data set is changed;

[0010] The backbone system uses deep convolution operations, bottleneck cross-stage local structure BottleneckCSP and spatial pyramid pooling SPP to extract features of different scales from the same feature map, and extract target features of different levels from the image;

[0011] The Neck network layer includes a path aggregation structure (PAN) and a feature pyramid (FPN). PAN transmits positioning information from bottom to top in the network, fusing information from different network layers in the Backbone. FPN transmits semantic information from top to bottom.

[0012] The head output is the final detection part, which predicts objects of different sizes on feature maps of different sizes.

[0013] Furthermore, step S1 of the CSRT algorithm specifically includes the following steps:

[0014] Step S11: Collect training samples: First, a set of training sample images similar to the target needs to be collected; these sample images should contain different poses, scales, and lighting conditions of the target;

[0015] Step S12: Calculate the target template: select one from the training samples as the target template; a grayscale image or a feature image of the target template, such as a gradient direction histogram, may be used;

[0016] Step S13: Calculate the response map: perform correlation operation on the target template and the image to be tracked to obtain a response map; each pixel value in the response map represents the similarity between the position and the target template;

[0017] Step S14: Apply spatial constraints: In order to improve the accuracy of tracking, spatial constraints need to be applied; information such as the position, scale, and orientation of the target can be used to constrain the response map;

[0018] Step S15: New target template: Using the spatially constrained response map, the position with the largest response is selected as the new position of the target, and the image around this position is used as the target template for the next frame;

[0019] Step S16: Iterative tracking: Repeat steps 3 to 5, continuously iteratively update the target template to achieve continuous tracking of the target.

[0020] Furthermore, step S2 of the CSRT algorithm specifically includes the following steps:

[0021] Step S21: Determine the position and scale information of the target: In each frame, the position and scale of the target in the current frame are predicted based on the target position and scale information of the previous frame;

[0022] Step S22: Determine the search area: Determine the search area of the target based on the location and scale information of the target; the search area should contain the target and be large enough to accommodate the target's motion range;

[0023] Step S23: Limiting the search area: In order to improve the accuracy and efficiency of tracking, the search area can be limited to construct a spatially reliable area;

[0024] Step S24: spatial reliability estimation: within the spatially reliable region, the channel reliability of each pixel is estimated based on the characteristic information of the image;

[0025] Step S25: Limiting the tracking range: Based on the estimated results of the channel reliability, the tracking range is limited; pixels with lower channel reliability can be excluded from the tracking range, and target tracking is performed only in areas with higher channel reliability.

[0026] Furthermore, step S3 of the CSRT algorithm specifically includes the following steps:

[0027] Step S31: Feature extraction: extracting features from the image for estimating channel reliability; commonly used features include brightness, texture, gradient, etc.;

[0028] Step S32: Feature normalization: normalize the extracted features to make them have the same range and scale; this can ensure that the contributions of different features to the channel reliability estimation are relatively balanced;

[0029] Step S33: Building a model: Building a model based on the extracted features to estimate channel reliability; commonly used models include statistical models and gradient models;

[0030] Step S34: training the model: using the labeled training data to train the model; the training data includes the features of the image and the corresponding channel reliability; by minimizing the error between the features and the channel reliability, an optimal model can be obtained;

[0031] Step S35: Channel reliability estimation: For a given image, use the trained model to estimate the channel reliability at each pixel; according to the value of the feature, the corresponding channel reliability estimation result is obtained through the model.

[0032] Furthermore, step S4 of the CSRT algorithm specifically includes the following steps:

[0033] Step S41: Target initialization: In the first frame, a target area is selected as the initial tracking target through user input or target detection algorithm;

[0034] Step S42: Channel reliability estimation: Calculate the channel reliability of each pixel based on the target initialization area. Channel reliability can be estimated using methods such as image features, statistical models, or gradient information. The higher the channel reliability, the greater the contribution of the pixel's color or texture information to target tracking.

[0035] Step S43: Spatial reliability estimation: Based on the channel reliability estimation results, the tracking range is limited. Pixels with low channel reliability can be excluded from the tracking range, and target tracking is performed only in areas with high channel reliability. This can reduce the impact of noise and interference and improve tracking accuracy and robustness.

[0036] Step S44: Target tracking: Use a target tracking algorithm to track the target within a limited tracking range. A commonly used target tracking algorithm is a Kalman filter, which can estimate the position and state of the target based on the target's motion model and observation information.

[0037] Step S45: Update the target model: As the target tracking progresses, the target model can be continuously updated; online learning methods can be used to integrate new observation information into the target model to adapt to changes in the target's appearance and motion pattern.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. This invention uses the YOLOv5s target recognition algorithm based on deep learning, which can learn the characteristic markers placed in advance on the ship from the data. The model has few parameters and runs fast, meeting the target recognition requirements of high-speed moving ships. The effect and speed of target recognition are not affected by the speed of the ship. It has the advantages of fast detection speed and high accuracy.

[0040] 2. This invention uses the CSRT tracker algorithm, which implements target tracking by extracting image features and superimposing machine learning concepts. It can update features online in real time according to changes in the target and the environment, and can adapt well to changes in the environment, thereby improving the robustness of tracking.

[0041] 3. The present invention integrates the YOLO and CSRT algorithms to overcome the shortcomings of the two algorithms and take into account the real-time and accuracy of mobile target detection and tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a YOLOv5s network structure based on the dual-redundancy vision algorithm fused with YOLOv5s and CSRT algorithms;

[0043] Figure 2 Flowchart for constructing spatially reliable regions from training regions;

[0044] Figure 3 Flowchart for minimizing reliability constraints for channel learning;

[0045] Figure 4 Schematic diagram of the tracking initialization and update process;

[0046] Figure 5 This is the framework diagram of the dual-redundancy vision algorithm that integrates YOLOv5s and CSRT;

[0047] Figure 6 This is the effect diagram of feature marker recognition and tracking based on the fusion of YOLOv5s and CSRT algorithms. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] To address the difficulty in identifying targets on mobile platforms at high speeds, this paper proposes a dual-redundancy vision algorithm based on the fusion of the YOLOv5s and CSRT algorithms. YOLOv5s is a deep neural network-based model that uses convolutional neural networks to predict the location and category of desired targets. It first loads an image captured by a camera as input, then uses a convolutional neural network to perform nonlinear processing on the image. Finally, it selects the identified content with a rectangular bounding box, assigns a classification label to the identified content, and detects the desired target.

[0050] Considering that fixed-rotor drones have some inevitable lighting and occlusion problems when tracking moving targets on moving ships, such as interference objects such as reflected light from the sea surface and birds flying on the sea, drones can easily lose target identification information, which ultimately leads to failure in tracking moving ships. Therefore, the present invention is based on the CSRT tracker algorithm, which implements target tracking through image feature extraction and superimposed machine learning concepts. It can update features online in real time according to changes in the target and environment, and can adapt well to changes in the environment, thereby improving the robustness of tracking. It is specifically divided into four steps: (1) constructing a spatial constraint correlation filter; (2) constructing a spatial reliable area, and the process is as follows: Figure 2 As shown; (3) Channel reliability estimation, channel learning reliability constraint minimization process is as follows Figure 3 As shown; (4) Tracking of channel and space reliability, its initialization and update process is as follows Figure 4 shown.

[0051] like Figure 1 As shown in the figure, the overall structure of YOLOv5s consists of four parts: input end, backbone system, Neck network layer, and Head output end;

[0052] The input end mainly performs preprocessing operations on the input image, including data augmentation and image padding, so that it can be applied to different data sets. In addition, the input end also integrates adaptive anchor box calculation, which can automatically set the size of the initial anchor box when the data set is changed;

[0053] The backbone system uses deep convolution operations, bottleneck cross-stage local structure BottleneckCSP and spatial pyramid pooling SPP to extract features of different scales from the same feature map, and extract target features of different levels from the image;

[0054] The Neck network layer includes a path aggregation structure (PAN) and a feature pyramid (FPN). PAN transmits positioning information from bottom to top in the network, fusing information from different network layers in the Backbone. FPN transmits semantic information from top to bottom.

[0055] The head output is the final detection part, which predicts targets of different sizes on feature maps of different sizes.

[0056] The mobile target recognition algorithm based on YOLO has the advantage of high real-time performance, but its disadvantage is that it is easily affected by factors such as lighting and is prone to losing the target in a changing environment. The mobile target tracking algorithm based on CSRT updates the detection model based on the previous frame tracking. It has strong resistance to environmental disturbances. However, when the identified target has large pixel displacement, occlusion, or size changes, the tracking will be lost or offset due to incorrect previous frame sampling and model update, and the tracking speed is slower than the recognition speed. Based on the fusion of the above two algorithms, the real-time and accuracy of mobile target detection and tracking can be taken into account. The algorithm framework is as follows: Figure 5 Press Figure 5 The algorithm framework shown is after algorithm fusion:

[0057] (1) Once the fixed-rotor drone detects a feature marker on a high-speed moving ship, it will automatically help the CSRT tracker to select the feature marker.

[0058] (2) If tracking fails, YOLO will also identify the moving target and select it, and then restart the CSRT-based tracker to track the moving target again. The feature identification and tracking effect is as follows: Figure 6 shown.

[0059] (3) Finally, under the premise of maintaining the real-time and accuracy of moving target tracking, the position information of the moving target in the camera coordinate system is output, providing a basis for solving the position and posture of the moving target.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms, characterized by: The YOLOv5s is a model based on a deep neural network. It predicts the position and category of the desired target by using a convolutional neural network. First, the image captured by the loaded camera is used as input. Then, the convolutional neural network is used to perform nonlinear processing on the image. Finally, a rectangular bounding box is used to select the recognized content, and a classification label is assigned to the recognized content. The desired target is identified and detected. The CSRT algorithm tracks the target by extracting image features and superimposing machine learning concepts. It updates features online in real time according to changes in the target and environment. It is divided into four steps: Step S1: Constructing a spatial constraint correlation filter; Step S2: Constructing a spatially reliable area; Step S3: Estimating channel reliability; Step S4: Tracking channel and spatial reliability.

2. The dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms according to claim 1, characterized in that: The overall structure of YOLOv5s consists of four parts: input end, backbone system, Neck network layer, and Head output end; The input end mainly performs preprocessing operations on the input image, including data enhancement and image filling. In addition, the input end also integrates adaptive anchor box calculation, which can automatically set the size of the initial anchor box when the data set is changed; The backbone system uses deep convolution operations, bottleneck cross-stage local structure BottleneckCSP and spatial pyramid pooling SPP to extract features of different scales from the same feature map, and extract target features of different levels from the image; The Neck network layer includes a path aggregation structure (PAN) and a feature pyramid (FPN). PAN transmits positioning information from bottom to top in the network, fusing information from different network layers in the Backbone. FPN transmits semantic information from top to bottom. The head output is the final detection part, which predicts objects of different sizes on feature maps of different sizes.

3. A dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms according to claim 1, characterized in that: Step S1 of the CSRT algorithm specifically includes the following steps: Step S11: Collect training samples: First, a set of training sample images similar to the target needs to be collected; these sample images should contain different poses, scales, and lighting conditions of the target; Step S12: Calculate the target template: select one from the training samples as the target template; a grayscale image or a feature image of the target template, such as a gradient direction histogram, may be used; Step S13: Calculate the response map: perform correlation operation on the target template and the image to be tracked to obtain a response map; each pixel value in the response map represents the similarity between the position and the target template; Step S14: Apply spatial constraints: In order to improve the accuracy of tracking, spatial constraints need to be applied; information such as the position, scale, and orientation of the target can be used to constrain the response map; Step S15: New target template: Using the spatially constrained response map, the position with the largest response is selected as the new position of the target, and the image around this position is used as the target template for the next frame; Step S16: Iterative tracking: Repeat steps 3 to 5, continuously iteratively update the target template to achieve continuous tracking of the target.

4. The dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms according to claim 1, characterized in that: Step S2 of the CSRT algorithm specifically includes the following steps: Step S21: Determine the position and scale information of the target: In each frame, the position and scale of the target in the current frame are predicted based on the target position and scale information of the previous frame; Step S22: Determine the search area: Determine the search area of the target based on the location and scale information of the target; The search area should contain the target and be large enough to accommodate the target's range of motion; Step S23: Limiting the search area: In order to improve the accuracy and efficiency of tracking, the search area can be limited to construct a spatially reliable area; Step S24: spatial reliability estimation: within the spatially reliable region, the channel reliability of each pixel is estimated based on the characteristic information of the image; Step S25: Limiting the tracking range: Based on the estimated results of the channel reliability, the tracking range is limited; pixels with lower channel reliability can be excluded from the tracking range, and target tracking is performed only in areas with higher channel reliability.

5. The dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms according to claim 1 is characterized in that: Step S3 of the CSRT algorithm specifically includes the following steps: Step S31: Feature extraction: extracting features from the image for estimating channel reliability; commonly used features include brightness, texture, gradient, etc.; Step S32: Feature normalization: normalize the extracted features to make them have the same range and scale; this can ensure that the contributions of different features to the channel reliability estimation are relatively balanced; Step S33: Building a model: Building a model based on the extracted features to estimate channel reliability; commonly used models include statistical models and gradient models; Step S34: training the model: using the labeled training data to train the model; the training data includes the features of the image and the corresponding channel reliability; by minimizing the error between the features and the channel reliability, an optimal model can be obtained; Step S35: Channel reliability estimation: For a given image, use the trained model to estimate the channel reliability at each pixel; According to the value of the feature, the corresponding channel reliability estimation result is obtained through the model.

6. The dual-redundancy vision algorithm based on the fusion of YOLOv5s and CSRT algorithms according to claim 1, characterized in that: Step S4 of the CSRT algorithm specifically includes the following steps: Step S41: Target initialization: In the first frame, a target area is selected as the initial tracking target through user input or target detection algorithm; Step S42: Channel reliability estimation: Calculate the channel reliability of each pixel based on the target initialization area. Channel reliability can be estimated using methods such as image features, statistical models, or gradient information. The higher the channel reliability, the greater the contribution of the pixel's color or texture information to target tracking. Step S43: Spatial reliability estimation: Based on the channel reliability estimation results, the tracking range is limited. Pixels with low channel reliability can be excluded from the tracking range, and target tracking is performed only in areas with high channel reliability. This can reduce the impact of noise and interference and improve tracking accuracy and robustness. Step S44: Target tracking: using a target tracking algorithm to track the target within a limited tracking range; The commonly used target tracking algorithm is the Kalman filter, which can estimate the position and state of the target based on the target's motion model and observation information; Step S45: Update the target model: As the target tracking progresses, the target model can be continuously updated; online learning methods can be used to integrate new observation information into the target model to adapt to changes in the target's appearance and motion pattern.

Images