Unmanned aerial vehicle formation cooperative target tracking method, device and equipment based on monocular vision and medium

Abstract

The invention discloses an unmanned aerial vehicle formation cooperative target tracking method, device and equipment based on monocular vision, and a medium, and relates to the technical field of unmanned control and visual target recognition tracking, and the method comprises the steps: detecting and recognizing a static or dynamic target, marking a target position, and processing the target through a kernel correlation filtering algorithm; tracking of the piloting unmanned aerial vehicle on the target is achieved, tracking precision is improved, formation members share formation information and position information of the piloting unmanned aerial vehicle, the expected position of each following unmanned aerial vehicle at the next moment is calculated through a position calculation algorithm, and a formation is formed to achieve cooperative tracking of the target by the formation. After a static or dynamic target is recognized through a monocular vision sensor, cooperative tracking of the target under the condition that the unmanned aerial vehicle formation keeps a predetermined formation or changes the formation is achieved. According to the invention, an improved formation control method is combined with a target identification tracking method based on monocular vision, so that the system performance is improved, and the formation is ensured to safely and efficiently complete tasks.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent formation control and visual target recognition and tracking technology, and in particular to a method, device, equipment and medium for cooperative target tracking of an unmanned aerial vehicle formation based on monocular vision. Background Art

[0002] With the development of sensor and electronic technology, drone formations have been widely used in military and civilian fields. Target recognition and tracking of drones refers to the technology of real-time detection and recognition of ground or air targets through onboard vision, radar and other sensors, and real-time tracking of target positions and motion trajectories through continuous image frame tracking. This technology involves computer vision, machine learning and aviation, and has significant application prospects in today's security and monitoring, border patrol, search and rescue and other fields.

[0003] Currently, vision-based UAV formation collaborative target tracking technology is one of the research hotspots in the field of UAVs. In the existing technology, the process of UAVs identifying targets through visual sensors will inevitably be affected by complex environmental interference and algorithm complexity. When tracking targets in real time, how to improve algorithm accuracy and save computing resources is also a major challenge. In addition, since UAV formations often need to maintain or adjust the formation according to actual needs when tracking targets, and the target's movement direction and speed are uncertain, how to calculate and control the position and speed of formation members in real time has also become a technical problem that needs to be solved in this field.

[0004] Based on the above, the present invention proposes a technical solution for cooperative target tracking of UAV formation based on monocular vision. While realizing target tracking of UAV formation based on monocular vision, the established formation formation is maintained or the formation formation is adjusted according to actual needs. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, equipment and medium for cooperative target tracking of UAV formation based on monocular vision. The specific plan is as follows: First, according to the current initialization stage of the system, the leader and follower agents are selected, and the communication relationship between each agent is established. Among them, a drone is selected as the leader agent, and a monocular camera is installed on the drone; The leader agent obtains image information through a monocular camera, inputs the image information into the first link, then inputs the output result of the first link into the second link, and determines the target recognized by the system according to the output result of the second link; the first link is edge detection algorithm calculation, and the second link is template matching algorithm calculation; The leader agent inputs the target position information into the third link, and decides the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The relative positions of the follower agents and the leader agent are solved, and the motion decisions of the leader agent are broadcast to the follower agents.

[0006] Furthermore, the leader agent obtains image information through a monocular camera, inputs the image information into the first link, then inputs the output result of the first link into the second link, and determines the target recognized by the system according to the output result of the second link, specifically, Input the target edge line template image to the system, obtain the image in real time through the monocular camera, execute the first link edge detection algorithm, filter the original image containing the target, calculate the gradient amplitude and direction, use the adaptive threshold method to obtain the edge feature information in the image, and obtain the binary image containing the target edge line; Execute the second stage template matching algorithm, and match the multi-scale and multi-angle template image generated by the input target template image with the extracted binary image containing the target edge line to determine the similarity. After a successful match, the bounding box area and center point coordinates are obtained.

[0007] Furthermore, the third step includes: The improved kernel correlation filtering algorithm is executed, and the target bounding box and center coordinates output in the second stage are used as the two parameters of the target initial size and position respectively. The tracker is initialized, and the kernel correlation filter for tracking is generated and trained. The target position is searched in each frame thereafter, thereby controlling the movement of the leader intelligent agent and displaying the tracking status in real time on the ground station.

[0008] Furthermore, the follower intelligent agent calculates its expected position at the next moment and controls its own motion trajectory. When the formation needs to be changed according to the actual task, the follower intelligent agent calculates its expected position in the formation through a position calculation algorithm based on the change in its own formation position, which is the expected angle and distance from the leading intelligent agent, to achieve formation changes when the formation tracks the target.

[0009] Furthermore, an edge detection algorithm is executed to filter the original image containing the target and calculate the gradient amplitude and direction, and an adaptive threshold method is used to obtain edge feature information in the image. Specifically, Perform a Gaussian filter on the input image; wherein the value of each pixel represents the brightness or color of the image, and when applying the Gaussian filter, each pixel in the image is convolved with a two-dimensional Gaussian function; For the filtered image, the Sobel operator is used to calculate the gradient and intensity of each pixel in the image. The convolution kernel template used for calculation is expanded to , , 45° and 135°, and obtain the gradient amplitudes corresponding to the four directions by performing convolution operations with the gradient templates in the four directions; Divide the image into background and foreground, and maximize the inter-class variance so as to maximize the difference between the background and foreground after threshold segmentation; The grayscale level is divided according to the grayscale information of the image, and the image is divided into a first part image and a second part image, and a first probability and a second probability of any pixel in the image in the first part image and the second part image are obtained, and the first pixel average grayscale value and the second pixel average grayscale value are calculated according to the first probability and the second probability, and the image average grayscale value and the inter-class variance are calculated according to the first pixel average grayscale value and the second pixel average grayscale value. When the inter-class variance is maximum, the current optimal threshold is obtained, and the optimal threshold is used to distinguish between strong and weak edge lines, thereby connecting the strong and weak edge lines, and finally processing the image into a binary image that only retains the edge line features.

[0010] Furthermore, the second stage template matching algorithm is executed to match the multi-scale and multi-angle template image generated by the input target template image with the extracted binary image containing the target edge line to determine the similarity. After the match is successful, the target is marked with a red box as the target's bounding box. At this time, the system will output the area size of the bounding box and the coordinates of the center point of the bounding box, which respectively represent the size and position of the target in the system's field of view at the current moment, specifically, For the target template image input to the system, the odd rows and columns of the target image are sampled through the Gaussian convolution kernel to reduce the image resolution layer by layer. The image resolution after each sampling is 1 / 4 of the image in the previous layer, so as to generate an image pyramid consisting of a collection of target template images with different resolutions; different scaling factors are set for template images in different levels (such as 0.25, 0.5, 1.5, etc.), and different rotation angles are set for the template images in each layer (such as 45°, 90°, 180°, etc.).

[0011] The binary image containing the target edge line output by the first stage is subjected to normalized cross-correlation operation with the template image of the same size to determine the matching similarity between the current target image and the template image, and judge whether the match is successful. If the match is successful, it means that the target is successfully identified, and the target bounding box area and the coordinates of the center point of the bounding box are output, and the current time is set as the starting frame of the tracking process.

[0012] Furthermore, the improved kernel correlation filtering algorithm is executed, and the target bounding box size and center coordinates output in the second step are used as the two parameters of the target initial size and position respectively, the tracker is initialized, and the correlation filter for tracking is generated and trained, and the target position is searched in each frame thereafter, thereby controlling the movement mode of the leader intelligent body, and the tracking status is displayed in real time on the ground station, specifically, The target bounding box output in the second stage is used as the initial tracking area of ​​the kernel correlation filter algorithm, and the center coordinates are used as the initial position of the target. The features are extracted to generate the relevant classifier for tracking. The target and non-target areas are used as input samples for training. The samples are mapped to the high-dimensional space through the mapping function. After the classifier training step is completed, the next frame image features are input into the classifier to determine the target position. The maximum value of the response is the predicted current target position. After the new target position is determined, the current target area is used as a training sample to update the classifier to obtain a new detection model, so as to correct the tracking frame in real time. Through the maximum response value and the average peak correlation energy, the threshold is set to determine whether the target is lost. The target is lost in real time. When the target is lost, the update is stopped and the tracking status is displayed in real time on the ground terminal. The output results determine the movement mode of the leading intelligent agent.

[0013] In the second aspect, the present application discloses a UAV formation collaborative target tracking device based on monocular vision, comprising: The initialization module is used to select the leader and follower agents according to the current initialization stage of the system and establish the communication relationship between each agent. Among them, a drone is selected as the leader agent and a monocular camera is mounted on the drone. A matching module, used for the leader agent to obtain image information through a monocular camera, input the image information into the first link, then input the output result of the first link into the second link, and determine the target recognized by the system according to the output result of the second link; the first link is edge detection calculation, and the second link is template matching calculation; A decision module is used for the leader agent to input the target position information into the third link, and decide the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The solution module is used to solve the relative positions of the follower agents and the leader agent, and broadcast the movement decision of the leader agent to the follower agents.

[0014] In a third aspect, the present application discloses an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the aforementioned monocular vision-based UAV formation collaborative target tracking method disclosed above.

[0015] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned monocular vision-based UAV formation collaborative target tracking method disclosed above are implemented.

[0016] As can be seen from the above, the present invention discloses a method, device, equipment and medium for cooperative target tracking of unmanned aerial vehicle formation based on monocular vision, which relates to the technical field of multi-rotor unmanned aerial vehicle formation control and visual target recognition and tracking. The pilot-following algorithm and the position solution algorithm are used to optimize the formation control effect. For the pilot unmanned aerial vehicle equipped with a monocular camera, the adaptive edge detection algorithm and the adaptive template matching algorithm are used to detect and identify static or dynamic targets and mark the target position. The target is processed by the kernel correlation filter algorithm to realize the tracking of the target by the pilot unmanned aerial vehicle and improve the tracking accuracy. The formation formation information and the position information of the pilot unmanned aerial vehicle are shared among the formation members. The expected position of each follower unmanned aerial vehicle at the next moment is solved by the position solution algorithm to form a formation to realize the cooperative tracking of the formation target. After the monocular vision sensor recognizes the static or dynamic target, the unmanned aerial vehicle formation realizes the cooperative tracking of the target while maintaining the predetermined formation or changing the formation. The present invention combines the improved formation control method with the target recognition and tracking method based on monocular vision to improve the system performance and ensure that the formation completes the task safely and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a quad-rotor drone system provided by an embodiment of the present invention; Figure 2 A schematic diagram of a target recognition and tracking algorithm based on monocular vision provided by an embodiment of the present invention; Figure 3 A schematic diagram of an image pyramid strategy provided by an embodiment of the present invention.

[0019] Figure 4 A two-dimensional plane schematic diagram of the formation expected position solution provided by an embodiment of the present invention; Figure 5A schematic diagram of a UAV formation collaborative target tracking device based on monocular vision provided by an embodiment of the present invention; Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] An embodiment of the present invention provides a method for cooperative target tracking of a UAV formation based on monocular vision, which solves the problem that a multi-agent formation is difficult to identify and track targets under environmental interference, and realizes vision-based formation target recognition and tracking through an improved target recognition and tracking method and a formation member position solution algorithm. In response to problems in actual mission environments, such as changes in lighting and field of view angles, the leading UAV obtains image information through a monocular vision sensor, implements an improved adaptive Canny edge detection algorithm and an adaptive template matching algorithm to complete target recognition and mark the target position. The leading agent runs an improved kernel correlation filter (KCF) algorithm based on the target recognition result to enter the tracking state of the target, and broadcasts the result to each follower agent. The system combines the formation information and controls the formation through a position solution algorithm, thereby realizing cooperative tracking of the target by the formation. Figure 1 As shown, multiple drones and the ground end form a system communication network. Among them, the drone adopts a quad-rotor drone with a variable wheelbase and model. The intelligent bodies establish communication through the equipped ZigBee communication module to transmit data such as the position and angle of each intelligent body; the intelligent body establishes communication with the ground end through the Wi-Fi module to detect image information and target recognition results in real time from the ground end.

[0022] The present invention proposes a UAV formation cooperative target tracking method based on monocular vision. Figure 2 As shown, the method includes: Step S100, according to the current initialization stage of the system, a leader and a follower agent are selected, and a communication relationship between each agent is established, wherein a drone is selected as the leader agent, and a monocular camera is mounted on the drone; In this embodiment, the current initialization stage of the system includes selecting a leading intelligent agent, a following intelligent agent and setting initial formation parameters; establishing communication relationships between each intelligent agent and the ground station, wherein the ground station saves the IPv4 address of the leading intelligent agent in the Wi-Fi network and saves the MAC address of each intelligent agent in the Zigbee network as the initial parameters of the formation; the leading intelligent agent turns on the monocular camera and transmits images to the ground station in real time; In this embodiment, a drone is selected as the leading intelligent agent, and a monocular camera is mounted on the drone. Specifically, the pilot drone is equipped with a monocular camera and an onboard computer for real-time acquisition and processing of image data. In the system initialization phase, the pilot-follower method is adopted to set the initial formation of the system according to the actual situation and task requirements, select a drone as the leading intelligent agent, and determine the positions of the remaining follower intelligent agents and the positional relationship with the leading intelligent agent. The Zigbee-based communication method is used to establish a communication relationship between the intelligent agents to achieve real-time information exchange. The Wi-Fi-based communication method is used to establish a communication relationship between the intelligent agent and the ground terminal, and the image data information is displayed in real time on the ground terminal.

[0023] Step S200, the leader agent acquires image information through a monocular camera, inputs the image information into a first link, inputs the output result of the first link into a second link, and determines the target recognized by the system according to the output result of the second link; the first link is edge detection calculation, and the second link is template matching calculation; In this embodiment, an improved adaptive Canny edge detection algorithm is implemented on the leading intelligent agent to obtain a binary image containing the target edge line features, and the input template image is subjected to an improved adaptive template matching algorithm. The obtained result is used as the result of target recognition to provide reliable perception data for the leading intelligent agent and the formation to perform tasks, and the recognition result is displayed in real time on the ground end.

[0024] In order to enhance the anti-interference ability of the system and achieve performance optimization in this embodiment, the target recognition process of the leading intelligent agent is divided into two stages. After the image information is acquired in real time by the monocular camera and the onboard computer, the improved adaptive Canny edge detection algorithm is first implemented to filter the original image that may contain the target and calculate the gradient amplitude and direction. The edge feature information in the image is obtained by the adaptive threshold method, and the binary image containing the target edge line features is output. Then, the improved adaptive template matching algorithm is used to match the target image with the generated multi-scale and multi-angle template image. After the match is successful, the target is marked with a red box as the target's bounding box, and the size and center point coordinates of the target bounding box are output, which represent the size and position of the target in the system's field of view, respectively. This method can effectively deal with environmental interference to achieve system performance optimization.

[0025] Step S300, the leader agent inputs the target's location information into the third link, and decides the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The leader agent implements the improved KCF algorithm in the onboard computer to track the target's motion trajectory in the image frame in real time. The algorithm is used to predict the target's motion trajectory and convert the result into a control variable to control the movement of the leader agent and display the tracking status in real time on the ground terminal.

[0026] In this example, in order to save the computing resources of the system, the target recognition result in step S200 is used as a reliable input, and the moment when the target recognition is judged to be successful is used as the starting frame. The leader agent implements the improved KCF algorithm, uses the target bounding box as the initial tracking area of ​​the kernel correlation filter algorithm, and the center coordinates as the initial position of the target. It extracts features to generate a kernel correlation filter for tracking and trains it to search for the target position in each frame thereafter. The maximum response value and the average peak correlation energy are added to the algorithm as a method of confirming the target confidence. When the target is judged to be lost during the tracking process, the detection update step in the algorithm is stopped, so as to save the computing resources of the system while tracking the target in real time.

[0027] Step S400, calculating the relative positions of the follower agents with respect to each other and with respect to the leader agent, and broadcasting the movement decision of the leader agent to the follower agents.

[0028] Among them, the movement mode of the leading intelligent body depends on the movement mode of the target. Each follower intelligent body calculates its expected position at the next moment in real time based on the movement trajectory of the leading intelligent body through the position solution algorithm. In this embodiment, when the leading intelligent body enters the target tracking state, its own position information is broadcast to each follower intelligent body through Zigbee. Each follower intelligent body calculates its expected position at the next moment based on the information of the leading intelligent body and the position settlement algorithm, and controls its own movement trajectory. When the formation needs to be changed according to the actual task, each follower intelligent body combines its own formation position change with the position solution algorithm to realize the formation change when the formation tracks the target.

[0029] This embodiment adopts the pilot-follow method to realize formation control, and any formation member is selected as the leader agent (leader), and the rest are the following agents (followers). The agents establish a communication relationship through Zigbee, and exchange information such as position and angle in real time. The agents establish a communication relationship with the ground terminal through Wi-Fi, and the motion status of the formation is displayed in real time on the ground terminal. The leading agent obtains image information in real time through the equipped monocular camera and onboard computer, and implements the target recognition algorithm and target tracking algorithm based on monocular vision to control its own real-time tracking of the target's motion trajectory. In this embodiment, the leading agent runs the improved adaptive Canny edge detection algorithm, detects and extracts the edge feature information of the image after Gaussian filtering the image, and implements the improved adaptive template matching algorithm. The target recognition result is obtained from the matching result, and the target size and position are output. On this basis, the improved KCF algorithm is run, thereby realizing stable tracking of the target, and the tracking status is displayed in real time on the ground terminal. According to the actual task requirements, the leading agent sends its own position information to each following agent, and the expected position of each following agent at the next moment is calculated through the position solution algorithm and the formation needs. As a result, the motion state information of the leading intelligent agent is shared with each follower intelligent agent, which enables the entire formation system to track the target and effectively improves the autonomy and performance of the system.

[0030] In this embodiment, after the leading intelligent agent obtains image information through a monocular camera, the Canny edge detection algorithm is improved to improve the efficiency and accuracy of edge detection of the target, as the process of obtaining images using the camera during flight of the drone will be affected by changes in lighting and interference from other objects. The improved adaptive Canny edge detection algorithm can adaptively adjust the threshold according to the actual environment to obtain more accurate target edge feature information, and effectively reduce the impact of lighting changes and interference from other objects on edge feature extraction, providing more reliable data for subsequent target identification. Specifically, in step S200, the first processing link includes: extracting edge features of the image through the improved adaptive Canny edge detection algorithm, and the specific operations are as follows: The input image is first subjected to Gaussian filtering. In an image, the value of each pixel represents the brightness or color of the image. When applying a Gaussian filter, each pixel in the image is convolved with a two-dimensional Gaussian function. The form of the two-dimensional Gaussian function is as follows: (1) In the formula represents the weight of the Gaussian function, is the standard deviation of the Gaussian function, and Indicates the distance from a pixel to the center pixel. Gaussian filters can reduce noise in images, blur images and reduce details, which is beneficial for subsequent image processing steps. Its smoothing effect depends on the standard deviation of the Gaussian function. The larger the standard deviation, the more obvious the smoothing effect of the filter.

[0031] For the filtered image, the Sobel operator is used to calculate the gradient and intensity of each pixel in the image. The convolution kernel template used for calculation is expanded to , , 45° and 135°, and the gradient amplitude is obtained by convolution operation with the gradient template in the four directions and is defined as , , and The gradient amplitude of the entire image and the gradient direction The calculation formulas are shown in formula (2) and formula (3): (2) (3) After obtaining the gradient amplitude and direction from equations (2) and (3), non-maximum suppression is performed along the gradient direction to effectively remove non-edge points, and then a threshold is selected to connect the edge lines.

[0032] The image is divided into two categories (background and foreground) based on the maximum inter-class variance method (OSTU), and the inter-class variance is maximized, that is, the difference between the two categories after threshold segmentation is maximized. According to the grayscale information of the image, the size of The images are divided into The gray level, The gray level is The number of pixels, then the gray level is The probability of the pixel corresponding to is . Let the selected threshold be , , and divide the image into and Two parts, including By gray value in All pixels within By gray value in All pixels in the range. Then we can get the value of any pixel in the image and The probability of and for (4) From (4), we can get and The average gray value of pixels in and for (5) (6) From formula (5) and (6), we can get the size of The average gray value of the image It can be expressed as (7) By calculating formula (7) and substituting it into formula (4), we can get the between-class variance for (8) So the between-class variance Maximum The value is the optimal threshold, let it be If there is When it is not unique, the average of all values ​​is taken as the optimal threshold. On this basis, the result determined by the threshold is used to connect the weak edge pixels with the strong edge pixels to form a complete edge, and finally a binary image containing the target edge line features is obtained.

[0033] The above operations can obtain the edge feature information of the target, which to a certain extent overcomes problems such as illumination changes and provides more efficient and reliable data for subsequent target recognition.

[0034] In this embodiment, in view of the change in flight altitude and camera angle during the flight of the drone, which causes the deformation of the target in scale and angle, an adaptive template matching algorithm is used to generate multi-scale and multi-angle template matching to match the target, thereby improving the accuracy of target recognition. Specifically, in step S200, the second processing link includes: realizing target recognition through an improved adaptive template matching algorithm, and outputting the target position and size, for example: For the target template image input to the system, the Gaussian convolution kernel is used to sample the odd rows and columns of the template image to reduce the image resolution layer by layer. The image resolution after each sampling is 1 / 4 of the previous layer image, such as Figure 3 As shown, an image pyramid consisting of a set of target template images of different resolutions is generated, wherein different scaling factors are set for the template images in different levels (such as 0.25, 0.5, 1.5, etc.), and different rotation angles are set for the template images in each layer (such as 45°, 90°, 180°, etc.).

[0035] For size and The target image and template image , perform normalized cross-correlation operation to measure the similarity of images. The calculation formula is as follows: (9) In the formula and are the pixel values ​​in the target image and the template image respectively, and are the means of the target image and the template image, respectively. Represents the image with coordinates is the matching degree of the starting position. According to the calculation result of the normalized cross-correlation, the matching score between the current target and the template image is determined to determine whether the matching is successful. If the matching is successful, it means that the target is successfully identified.

[0036] The above operations can accurately complete target recognition. At the same time, the target is marked with a red box as the target's bounding box, and the size and center point coordinates of the target bounding box are output, representing the size and position of the target in the system's field of view, respectively, as the input for the next stage.

[0037] In this embodiment, the KCF algorithm is improved to solve the target loss problem when the UAV tracks a dynamic target. During the tracking process, the target confidence is determined to confirm whether the target is lost, which effectively improves the tracking accuracy and saves computing resources. Specifically, in step S300, the third processing link includes: the leading agent implements the improved KCF algorithm to track the target's motion trajectory in real time. For example: Using the target recognition results in S200, the target and non-target areas are used as input samples, and the ridge regression model is used to train the kernel correlation filter, that is, to train the classifier ( is the classifier weight coefficient, is the basic sample), so that the training sample and regression labels The average error is the smallest: (10) In the formula It represents the regularization parameter introduced to prevent overfitting. The filter weights that can match the target are obtained through training. They are used to match the image block with the target template and output the matching score.

[0038] Since target tracking is a nonlinear problem, the samples are passed through the mapping function Mapping to a high-dimensional space can make nonlinear problems linearly separable, and the weight coefficients of the classifier It can be expressed as a linear combination of sample mapping as shown in formula (11): (11) In the formula is the linear combination coefficient, which can be substituted into the classifier and formula (10) to obtain Vector The expression of is as follows (12): (12) In the formula is the kernel function, For regression label The column vector is composed of. Perform Fourier transform on equation (12): (13) In the formula for The Fourier form of is the kernel function The Fourier form of the first row. After the training classifier step is completed, the next frame image features are input into the classifier to determine the target position. is the current training sample, For the next frame test sample, use express and The kernel function of is used to solve the response in the frequency domain: (14) In the formula Represents the convolution operator. The maximum value of the response is the predicted current target position. After determining the new target position, the current target area is used as a training sample to update the classifier to obtain a new detection model, so as to correct the tracking frame in real time.

[0039] When the previous frame of the tracking process is partially blocked or the target is lost, resulting in tracking failure in the next frame, the maximum response value and the average peak correlation energy are used as a method to confirm the target confidence and to determine whether the target is lost. The calculation formula is as follows: (15) In the formula, Indicates horizontal translation of the target , vertical translation The detection response value after and Indicates the maximum and minimum responses of the current frame; Indicates arithmetic mean operation; Indicates the current frame Line The response value of the column pixel. By setting a suitable threshold, it can be determined whether the target is lost. When the target is lost, stop updating.

[0040] Through the above operations, the leading intelligent agent can track the target with high accuracy and determine in real time whether the target is lost, while displaying the tracking status in real time on the ground end.

[0041] In this embodiment, in order to meet the real-time requirements of formation control, a classic pilot-follower control algorithm is used. According to the desired formation setting, the follower agent follows the leader agent at a specific distance and angle. After the follower agent learns the position information of the leader agent, it calculates its own desired position and yaw angle through the position calculation algorithm, thereby maintaining the preset formation. Specifically, in S400, it includes: like Figure 4 As described above, when a multi-agent formation is performing a task, when the leading agent enters the target tracking state, each follower agent in the formation needs to calculate its own expected position. After knowing the expected position point, it controls itself to reach the expected position to maintain the formation. is the world coordinate system, is the coordinate system of the leader drone, and the distance of the follower ,angle Follow the leader, , Is the following distance In the navigator body coordinate system The amount of , are the positions of the leader and follower in the world coordinate system, The angle between the body coordinate system and the world coordinate system. The expected following distance in the body coordinate system , It can be expressed as: (16) Through coordinate system transformation, the actual following distance between the follower and the leader , It can be expressed as: (17) From formula (17), we can deduce the expected position coordinates of the UAV in the world coordinate system: , for: (18) The follower UAV in the formation system can calculate its expected position by the above formula, and on this basis, control the motion trajectory of each follower UAV to maintain the formation when the system performs the target tracking task.

[0042] It should be noted that when the formation needs to be changed according to the actual task, the formation change information is sent from the ground end to each intelligent agent. The follower intelligent agent combines its own formation position change with the position solution algorithm to realize the formation change when the formation tracks the target.

[0043] The embodiment of the present invention discloses a UAV formation cooperative target tracking method based on monocular vision, which relates to the field of multi-rotor UAV formation control and visual target recognition and tracking, and can realize formation cooperative tracking after identifying a static or dynamic target. The present invention includes: using a pilot-following algorithm and an improved position solution algorithm to optimize the formation control effect; considering interference problems such as light and field of view angle changes, for a pilot UAV equipped with a monocular camera, an improved adaptive Canny edge detection algorithm and an adaptive template matching algorithm are implemented, and an adaptive threshold method is used to extract edge feature information of the target in the field of view, and it is matched with the generated multi-scale multi-angle template image to improve the accuracy and real-time performance of target recognition in a dynamic environment; for the problem of target tracking loss in a dynamic scene, after completing target recognition, the pilot UAV runs an improved KCF algorithm to track the detected static or moving target in real time, and sends information such as position and movement speed to other members in the formation in real time, and each follower UAV combines the formation formation information and the position information of the pilot UAV, and solves its expected position at the next moment through the position solution algorithm to realize formation cooperative tracking of the target. The present invention combines an improved formation control method with a target recognition and tracking method based on monocular vision to improve system performance and ensure that the formation completes its mission safely and efficiently.

[0044] The embodiment of the present invention provides a UAV formation cooperative target tracking method based on monocular vision, and the method is used in a system, and the system includes: a multi-agent and a ground terminal. The multi-agent includes at least two UAVs. Figure 1 As shown, multiple drones and the ground end form a system communication network. Among them, the drone adopts a quad-rotor drone with a variable wheelbase and model. The intelligent bodies establish communication through the equipped ZigBee communication module to transmit data such as the position and angle of each intelligent body; the intelligent body establishes communication with the ground end through the Wi-Fi module to detect image information and target recognition results in real time from the ground end.

[0045] The present invention also proposes a UAV formation cooperative target tracking device based on monocular vision, as shown in the attached Figure 5 The device 10 includes an initialization module, a matching module, a decision module, and a solution module.

[0046] The initialization module is used to select the leader and follower agents according to the current initialization stage of the system and establish the communication relationship between each agent. Among them, a drone is selected as the leader agent and a monocular camera is mounted on the drone. A matching module, used for the leader agent to obtain image information through a monocular camera, input the image information into the first link, then input the output result of the first link into the second link, and determine the target recognized by the system according to the output result of the second link; the first link is edge detection calculation, and the second link is template matching calculation; A decision module is used for the leader agent to input the target position information into the third link, and decide the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The solution module is used to solve the relative positions of the follower agents and the leader agent, and broadcast the movement decision of the leader agent to the follower agents.

[0047] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the monocular vision-based UAV formation collaborative target tracking method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0048] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0049] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0050] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0051] Among them, the operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the massive data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the monocular vision-based UAV formation collaborative target tracking method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to data transmitted from an external device received by the electronic device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0052] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the monocular vision-based UAV formation collaborative target tracking method disclosed in any of the aforementioned embodiments are implemented.

[0053] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0054] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0055] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0056] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0057] The above is a detailed introduction to the UAV formation collaborative target tracking method, device, equipment and storage medium based on monocular vision provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A UAV formation cooperative target tracking method based on monocular vision, characterized in that: include: According to the current initialization stage of the system, the leader and follower agents are selected, and the communication relationship between each agent is established. Among them, a drone is selected as the leader agent, and a monocular camera is installed on the drone; The leader agent acquires image information through a monocular camera, inputs the image information into the first link, then inputs the output result of the first link into the second link, and determines the target recognized by the system according to the output result of the second link; The first step is edge detection algorithm calculation, and the second step is template matching algorithm calculation; The leader agent inputs the target position information into the third link, and decides the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The relative positions of the follower agents and the leader agent are solved, and the motion decisions of the leader agent are broadcast to the follower agents.

2. The UAV formation collaborative target tracking method based on monocular vision according to claim 1 is characterized in that: The leader agent obtains image information through a monocular camera, inputs the image information into the first link, then inputs the output result of the first link into the second link, and determines the target recognized by the system according to the output result of the second link, specifically, Input the target edge line template image to the system, obtain the image in real time through the monocular camera, execute the first link edge detection algorithm, filter the original image containing the target, calculate the gradient amplitude and direction, use the adaptive threshold method to obtain the edge feature information in the image, and obtain the binary image containing the target edge line; Execute the second stage template matching algorithm, and match the multi-scale and multi-angle template image generated by the input target template image with the extracted binary image containing the target edge line to determine the similarity. After a successful match, the target bounding box area and center point coordinates are output.

3. The UAV formation collaborative target tracking method based on monocular vision according to claim 1 is characterized in that: The third step includes: The improved kernel correlation filtering algorithm is executed, and the target bounding box area and center point coordinates output in the second stage are used as the size and position of the target at the current moment, respectively. The tracker is initialized, a classifier for tracking is generated and trained, and the target position is searched in each frame. The movement of the leading intelligent agent is controlled, and the tracking status is displayed in real time on the ground station.

4. The UAV formation collaborative target tracking method based on monocular vision according to claim 1 is characterized in that: The follower agent calculates its expected position at the next moment and controls its own motion trajectory. When the formation needs to be changed according to the actual task, the follower agent calculates its expected position in the formation through a position calculation algorithm based on the change in its own formation position, which is the expected angle and distance from the leading agent, to achieve formation changes when the formation tracks the target.

5. The UAV formation collaborative target tracking method based on monocular vision according to claim 2 is characterized in that: Execute the edge detection algorithm, filter the original image containing the target, calculate the gradient amplitude and direction, and use the adaptive threshold method to obtain the edge feature information in the image. Specifically, Perform a Gaussian filter on the input image; wherein the value of each pixel represents the brightness or color of the image, and when applying the Gaussian filter, each pixel in the image is convolved with a two-dimensional Gaussian function; For the image after filtering, the Sobel operator is used to calculate the gradient and intensity of each pixel in the image; the convolution kernel template used for calculation is expanded to , , 45° and 135°, and obtain the gradient amplitudes corresponding to the four directions by performing convolution operations with the gradient templates in the four directions; Divide the image into background and foreground, and maximize the inter-class variance so as to maximize the difference between the background and foreground after threshold segmentation; The grayscale level is divided according to the grayscale information of the image, and the image is divided into a first part image and a second part image, and a first probability and a second probability of any pixel in the image in the first part image and the second part image are obtained, and the first pixel average grayscale value and the second pixel average grayscale value are calculated according to the first probability and the second probability, and the image average grayscale value and the inter-class variance are calculated according to the first pixel average grayscale value and the second pixel average grayscale value. When the inter-class variance is maximum, the current optimal threshold is obtained, and the optimal threshold is used to distinguish between strong and weak edge lines, thereby connecting the strong and weak edge lines, and finally processing the image into a binary image that only retains the edge line features.

6. The UAV formation collaborative target tracking method based on monocular vision according to claim 2 is characterized in that: The second stage template matching algorithm is executed to match the multi-scale and multi-angle template image generated by the input target template image with the extracted binary image containing the target edge line to determine the similarity, and output the current size and position of the target after the match is successful, specifically, For the target template image input to the system, the Gaussian convolution kernel is used to sample the odd rows and columns of the template image to reduce the image resolution layer by layer. The image resolution after each sampling is 1 / 4 of the image in the previous layer, thereby generating an image pyramid consisting of a collection of target template images with different resolutions; Different scaling factors are set for the template images in different levels, and different rotation angles are set for the template images in each layer; The binary image containing the target edge line output by the first stage is subjected to normalized cross-correlation operation with the template image of the same size to determine the matching similarity between the current target image and the template image, and judge whether the match is successful. If the match is successful, it means that the target is successfully identified, and the target bounding box area and the coordinates of the center point of the bounding box are output, and the current time is set as the starting frame of the tracking process.

7. The UAV formation cooperative target tracking method based on monocular vision according to claim 3 is characterized in that: The improved kernel correlation filtering algorithm is executed, and the target bounding box and center coordinates output in the second link are used as the two parameters of the initial size and position of the target respectively, the tracker is initialized, and the correlation filter for tracking is generated and trained, and the target position is searched in each frame thereafter, thereby controlling the movement mode of the leader intelligent body, and the tracking status is displayed in real time on the ground station, specifically, The target bounding box output in the second stage is used as the initial tracking area of ​​the kernel correlation filter algorithm, and the center coordinates are used as the initial position of the target. The features are extracted to generate the kernel correlation filter for tracking. The target and non-target areas are used as input samples for training. The samples are mapped to high-dimensional space through a mapping function. When the classifier training step is completed, the next frame image features are input into the classifier to determine the target position. The maximum value of the response is the predicted current target position. After the new target position is determined, the current target area is used as a training sample to update the classifier to obtain a new detection model, so as to correct the tracking frame in real time. Through the maximum response value and the average peak correlation energy, the threshold is set to determine whether the target is lost. The target is lost in real time. When the target is lost, the update is stopped and the tracking status is displayed in real time on the ground terminal. The output results determine the movement mode of the leading intelligent agent.

8. A UAV formation cooperative target tracking device based on monocular vision, characterized in that: include: The initialization module is used to select the leader and follower agents according to the current initialization stage of the system and establish the communication relationship between each agent. Among them, a drone is selected as the leader agent and a monocular camera is mounted on the drone. A matching module, used for the leader agent to obtain image information through a monocular camera, input the image information into the first link, then input the output result of the first link into the second link, and determine the target recognized by the system according to the output result of the second link; the first link is edge detection calculation, and the second link is template matching calculation; A decision module is used for the leader agent to input the target position information into the third link, and decide the movement mode of the leader agent according to the output result of the third link; the third link is a single target tracking calculation; The solution module is used to solve the relative positions of the follower agents and the leader agent, and broadcast the movement decision of the leader agent to the follower agents.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the UAV formation collaborative target tracking method based on monocular vision as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the monocular vision-based UAV formation collaborative target tracking method as described in any one of claims 1 to 7 are implemented.

Images