Fixed-wing target pursuit method combining improved SiamFc network and gimbal control

Abstract

The present invention discloses a method for fast target pursuit of fixed-wing aircraft that combines improved SiamFc and gimbal control. An improved SiamFc model is obtained by pre-training the improved SiamFc network with data; the ground end receives the image captured by the target detection module, selects the target and obtains the target features, and transmits them back to the target detection module; the target detection module runs the improved SiamFc model, and obtains the target pixel position based on the real-time image and target feature input; the gimbal control module uses the difference between the target pixel position and the center pixel position of the camera image as the error, and controls the gimbal movement to track the target; the gimbal obtains the gimbal attitude based on the gyroscope data, and transmits it to the fixed-wing control module; the fixed-wing control module adjusts the fixed-wing attitude based on the gimbal attitude to pursue the ground target. The present invention controls the flight attitude of the fixed-wing aircraft by controlling the gimbal camera attitude, thereby overcoming the shortcoming of poor flexibility of the fixed-wing UAV and achieving fast tracking of ground targets by the fixed-wing aircraft.

Description

Technical Field

[0001] The present invention relates to a target tracking system and method in the fields of unmanned aerial vehicles (UAVs) and graphics, and in particular to a fixed-wing target tracking method combining an improved SiamFc network and pan / tilt control. Background Art

[0002] When fixed-wing aircraft fly at high speeds, it is difficult to achieve fast tracking, which can easily lead to target loss and other problems. Due to the current lack of research on fixed-wing target pursuit, no relevant research data has been found.

[0003] Therefore, the current target detection algorithm cannot quickly and accurately lock the target when the fixed-wing aircraft is flying at high speed, which easily leads to target loss. At the same time, the fixed-wing aircraft is subject to dynamic constraints and cannot quickly lock the target's position. Summary of the Invention

[0004] In order to solve the problems existing in the background technology, the present invention proposes a fixed-wing target pursuit method combining an improved SiamFc network and pan-tilt control.

[0005] The present invention controls the flight attitude of the fixed-wing through the attitude of the gimbal camera, and the fixed-wing control module controls the fixed-wing UAV according to the three-axis attitude of the gimbal, thus overcoming the shortcoming of the poor flexibility of the fixed-wing UAV and realizing the fixed-wing to quickly track ground targets.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: Figure 2 As shown, the following steps are included:

[0007] S1. Use known large-scale data to pre-train the improved SiamFc network to obtain an improved SiamFc model, and store it in the target detection module;

[0008] S2. The ground terminal receives image information collected by the target detection module on the fixed-wing aircraft through the wireless bridge transmission module, selects the target from the image information, frames the target in the image information, processes the target to obtain the target features, and transmits the target features and the target back to the target detection module through the wireless bridge transmission module;

[0009] S3, the target detection module runs the improved SiamFc model, and calculates the pixel coordinates of the target in the camera image as the target pixel position based on the real-time image and target feature input;

[0010] S4, the gimbal control module uses the difference between the target pixel position collected in S3 and the center pixel position of the camera image as the error, and uses the PID control algorithm to control the fixed-wing gimbal movement to track the target so that the target is in the center of the camera image;

[0011] S5. The gimbal determines its own attitude based on the gyroscope data and transmits the attitude to the fixed-wing control module via serial communication combined with the MavLink communication protocol.

[0012] S6. The fixed-wing control module controls the fixed-wing to adjust its attitude according to the gimbal attitude transmitted in step S5, follows the gimbal attitude change, and then pursues the ground target.

[0013] The improved SiamFc network in step S1 is to replace the second convolutional layer of the SiamFc network with a depthwise separable network. This can reduce the number of parameters of the SiamFC network during convolution operations and improve tracking speed.

[0014] In step S1, the onboard computer TX2 runs the improved SiamFc algorithm in the target detection module, and the camera mounted on the fixed-wing aircraft collects image information of the target area in real time. The improved SiamFc algorithm uses a depthwise separable convolutional network to replace the second layer of conventional convolution in the original network. This replacement reduces the number of parameters to 1 / 25 of the original, thereby improving target detection speed.

[0015] The step S2 is specifically as follows: the ground end establishes a communication connection with the onboard computer TX2 on the fixed-wing through the wireless bridge transmission module on the fixed-wing aircraft, the camera captures the image of the target detection module and transmits it to the ground end through the bridge transmission module and the UDP communication protocol, the target object is selected on the ground end, and the target features are extracted through the image algorithm, and UDP communication is performed with the onboard computer TX2 in the target detection module of the fixed-wing through the wireless bridge transmission module, and the selected target object and its target features are transmitted to the target detection module again.

[0016] In step S3, specifically, the following steps are performed: inputting the real-time target image to be detected, the target features and the corresponding target area into the trained improved SiamFc model, comparing the similarity between the target image to be detected and the target area corresponding to the target features, returning the detection target area with higher similarity in the target image to be detected, and calculating the pixel coordinate value of the target in the detection target area.

[0017] In step S4, the target tracking method of the pan-tilt head is as follows: the onboard computer TX2 and the pan-tilt head are connected through the serial port and the mavlink communication protocol to form a pan-tilt head control module. The onboard computer TX2 calculates the target pixel position by improving the SiamFc model, and subtracts the error from the center pixel coordinates of the camera image. The PID control method is used to generate the control amount and send the control signal to the pan-tilt head through the serial port to track the target, so as to keep the target in the center of the camera image. The control formula is as follows:

[0018] error x (t) = center x(t)-target y (t)

[0019] error y (t) = center y (t)-target y (t)

[0020]

[0021]

[0022] Among them: center x (t), center y (t) are the x and y coordinates of the camera center point at time t, target y (t), target y (t) are the x and y pixel coordinates of the detection target at time t, error x (t), error y (t) are the x and y coordinate errors between the target position and the camera center at time t, U x (t), U y (t) represents the gimbal x-axis control amount and gimbal y-axis control amount, K p is the proportional coefficient, T I is the integration time constant, T D is the differential time constant.

[0023] In step S5, the gimbal acquires attitude information in real time through the gyroscope and transmits the gimbal attitude to the fixed-wing control module through the serial port and the MavLink communication protocol. The present invention utilizes MavLink to communicate with the drone and the gimbal at the same time, thereby simplifying the amount of code for reading the gimbal attitude.

[0024] In step S6, the fixed-wing control module controls the fixed-wing to adjust its attitude according to the gimbal attitude transmitted in step S5, and follows the gimbal attitude change, specifically:

[0025] The fixed-wing control module calculates the gimbal's attitude data based on the attitude quaternion data sent back by the gimbal. The formula is as follows:

[0026]

[0027] In the above formula, q0, q1, q2, q3 are the attitude quaternion data transmitted from the gimbal in step S5; yaw y Indicates the yaw attitude data of the gimbal, pitch y Indicates the pitch attitude data of the gimbal, roll yIndicates the roll attitude data of the gimbal, y represents the gimbal, and atan2 represents the inverse tangent function;

[0028] Convert quaternion to yaw by the above formula y , pitch y , roll y The yaw, pitch and roll attitude information of the gimbal respectively.

[0029] According to the current attitude of the fixed wing, the current attitude of the fixed wing is directly obtained by the flight control, and the error between the attitude of the fixed wing and the attitude of the gimbal is obtained. yaw 、error pitch 、error roll , error yaw 、error pitch 、error roll They are the yaw attitude difference, pitch attitude difference, and roll attitude difference between the fixed-wing and gimbal respectively. The roll value of the gimbal is always kept consistent with that of the fixed-wing drone. roll = 0. Adding a differential link to speed up the fixed-wing reaction speed. Based on the attitude data of the gimbal and the yaw attitude difference and pitch attitude difference between the fixed-wing and the gimbal, the desired attitude control value of the fixed-wing is obtained. The calculation formula is as follows:

[0030]

[0031] Among them, T Dy is the yaw differential coefficient, T Dp is the pitch differential coefficient, yaw sp 、pitch sp 、roll sp are the control quantities of desired yaw, desired pitch and desired roll, respectively, and error yaw 、error pitch They are the yaw attitude difference and pitch attitude difference between the fixed wing and the gimbal, respectively.

[0032] In this way, the fixed-wing control module sends the desired attitude control value to the flight control through the drone control operating system Mavros based on the attitude data sent back by the gimbal, controlling the fixed-wing drone to quickly pursue the target, thereby achieving rapid target tracking.

[0033] The present invention adopts the improved SiamFc algorithm as the core of the target detection module, and uses a deep separable network to replace the second layer of conventional convolution of the original network, which improves the response speed of the SiamFc algorithm. The target detection module is designed with this as the core.

[0034] The present invention transmits the ground frame-selected target features to the onboard computer TX2 in the target detection module through the bridge transmission module connection and the UDP communication protocol. The onboard computer TX2 runs the improved SiamFc algorithm to perform similarity matching between the target image and the camera image to obtain the target pixel position information.

[0035] At the same time, taking advantage of the fast and flexible gimbal tracking speed, a gimbal camera is configured under the fixed-wing, and a gimbal control module is designed. The PID control method is used to control the gimbal to track the target as the core, keeping the target in the pixel center of the camera in real time. At the same time, the gimbal attitude information is obtained in real time and sent to the fixed-wing control end through the UAV communication protocol MavLink to control the fixed-wing to track the target. The new form of gimbal and fixed-wing joint tracking is adopted to improve the tracking speed, reduce the probability of target loss, and improve the robustness of the system.

[0036] The beneficial effects of the present invention are:

[0037] The present invention adopts an improved SiamFc network model and replaces the second layer conventional volume of the original network with a deep separable network to actively improve the response speed of the SiamFc network model. At the same time, it takes advantage of the fast and flexible tracking speed of the gimbal, configures the gimbal and camera under the fixed wing, controls the gimbal and camera to track the target through PID control, and sends the posture information to the fixed wing. It adopts a new form of gimbal and fixed wing joint tracking, which improves the tracking speed, reduces the probability of target loss, and increases the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0039] Figure 1 It is a schematic diagram of the architecture of the present invention;

[0040] Figure 2 Schematic diagram of the architecture of the SiamFc method;

[0041] Figure 3 This is a diagram of the network bridge communication between the ground terminal and the fixed-wing aircraft terminal;

[0042] Figure 4 This is the airborne control flow chart. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0044] like Figure 1 As shown, the system specifically implemented by the present invention includes a ground terminal, a target detection module, a gimbal control module and a fixed-wing control module. The ground terminal is placed on the ground, and the target detection module, the gimbal control module and the fixed-wing control module are all on the fixed wing. The target detection module, the gimbal control module and the fixed-wing control module are connected in sequence by wires, and the target detection module is wirelessly connected to the ground terminal through a wireless bridge transmission module.

[0045] The ground terminal is specifically implemented using a ground computer.

[0046] The fixed-wing control module is used to control the flight attitude of the fixed-wing, and the gimbal control module is used to control the attitude of the gimbal installed on the fixed-wing. A camera is fixed on the gimbal, and a gyroscope is installed at the gimbal motion joint.

[0047] The target detection module includes a camera, which is fixedly mounted on the pan-tilt platform and is used to control the camera on the pan-tilt platform to capture images.

[0048] In a specific implementation, the ground terminal is used to receive image data transmitted back by the onboard computer TX2 through the wireless bridge transmission module combined with the UDP communication protocol, and after selecting the target in the image, the target features are transmitted back to the onboard computer TX2 through the wireless bridge transmission module combined with the UDP protocol.

[0049] The wireless bridge transmission module uses a 5.8G industrial-grade bridge module with a transmission distance of more than 2 kilometers. Figure 3 As shown, a relatively lightweight airborne bridge is placed on the fixed-wing UAV, and the other end of the bridge is placed on the ground. After pairing, data transmission is carried out, and the communication between the fixed-wing target detection module and the ground end is completed through the UDP communication protocol.

[0050] The target detection module uses the improved SiamFc algorithm. The SiamFc algorithm flow chart is as follows: Figure 2 As shown in Figure 2, replacing the second layer of Conv-layers conventional volume of the original network with a deep separable network actively improves the response speed of the SiamFc algorithm.

[0051] Depthwise Separable Convolution DSC consists of two parts: Depthwise Convolution and Pointwise Convolution.

[0052] The size of the second-layer conventional convolution kernel in the original network is 5×5, and the number of convolution kernels is 256. After replacing the original network with depthwise separable convolution, the number of parameters is reduced to 1 / 25 of the original, thereby improving the target detection speed.

[0053] The improved SiamFc model is pre-trained using large-scale data known in advance to obtain the detection model of the target detection module.

[0054] The image information collected by the camera of the fixed-wing target detection module is connected through the wireless bridge transmission module and transmitted to the ground end through the UDP communication protocol.

[0055] The pursuit target is framed out from the image sent back by the target detection module. The ground end will frame the target and obtain the feature information and send it to the onboard computer TX2 through the wireless bridge transmission module and the UDP communication protocol. The improved SiamFc in the target detection module of the present invention will run in the onboard computer TX2. After testing, the performance of the onboard computer TX2 is sufficient to complete the rapid detection of the target.

[0056] The onboard computer TX2 inputs the target feature information and the current image data of the gimbal camera into the improved SiamFc model in the target detection module for target detection.

[0057] The improved SiamFc model in the target detection module generates a response map through correlation operation (i.e., convolution) after inputting the target feature data and the current frame image data. The position with the highest response value is the target position, and the current pixel coordinates of the target are calculated.

[0058] The onboard computer TX2 and the gimbal are connected via the serial port and the MavLink communication protocol to form a gimbal control module. The onboard computer TX2 calculates the target pixel position through the improved SiamFc model and subtracts the error from the center pixel coordinates of the camera image. PID control is used to generate the control variable and send the control signal to the gimbal through the serial port to track the target, keeping the target in the center of the camera image. The control formula is as follows:

[0059] error x (t) = center x (t)-target y (t)

[0060] error y (t) = center y (t)-target y (t)

[0061]

[0062]

[0063] Among them: center x (t), center y (t) are the x and y coordinates of the camera center point at time ty (t), target y (t) are the x and y pixel coordinates of the detection target at time t, error x (t), error y (t) are the x and y coordinate errors between the target position and the camera center at time t, U x (t), U y (t) represents the gimbal x-axis control amount and gimbal y-axis control amount, K p is the proportional coefficient, T D is the differential time constant.

[0064] The gimbal obtains its own attitude through the gyroscope and sends the attitude information to the fixed-wing control module through the serial port connection and the MavLink drone communication protocol.

[0065] like Figure 4 The fixed-wing control module shown in the lower half consists of the open-source flight controller PX4 and the Mavros operating system running on the onboard computer TX2. After receiving the gimbal attitude, the onboard computer TX2 calculates the desired attitude control value and sends it to the flight controller. The desired attitude control value is calculated as follows: The fixed-wing control module calculates the gimbal attitude data based on the attitude quaternion data sent back by the gimbal using the following formula:

[0066]

[0067] Adding a differential link speeds up the fixed-wing response. The desired attitude control value of the fixed-wing is obtained based on the gimbal attitude data and the yaw attitude difference and pitch attitude difference between the fixed-wing and the gimbal. The calculation formula is as follows:

[0068]

[0069] After calculating the desired control amount of the fixed wing, the desired attitude control amount is sent to the flight control to control the fixed wing to transform to the desired attitude, thereby achieving the purpose of rapid target pursuit.

[0070] This design has made innovations in target detection algorithm and tracking form. It uses the improved SiamFc algorithm to improve the target detection speed, and uses the gimbal-assisted fixed-wing target tracking method to overcome the inflexibility of the fixed-wing, greatly improving the accuracy of fixed-wing UAV target pursuit.

Claims

1. A fixed-wing target pursuit method combining an improved SiamFc network and pan-tilt control, characterized in that: The method comprises the following steps: S1. Use known data to pre-train the improved SiamFc network to obtain an improved SiamFc model, and store it in the target detection module; S2. The ground terminal receives the image information collected by the target detection module through the wireless bridge transmission module, selects the target from the image information, processes the image information to obtain the target features, and transmits the target features back to the target detection module through the wireless bridge transmission module. S3, the target detection module runs the improved SiamFc model, and calculates the pixel coordinates of the target in the camera image as the target pixel position based on the real-time image and target feature input; S4, the pan-tilt control module uses the difference between the target pixel position collected in S3 and the center pixel position of the camera image as the error, and uses the PID control algorithm to control the pan-tilt movement to track the target so that the target is in the center of the camera image; S5. The gimbal determines its own attitude based on the gyroscope data and transmits the attitude to the fixed-wing control module. S6, the fixed-wing control module controls the fixed-wing to adjust its attitude according to the gimbal attitude transmitted in step S5, follows the gimbal attitude change, and then pursues the ground target; The improved SiamFc network in step S1 is to replace the second convolutional layer of the SiamFc network with a depth-wise separable network; In step S5, the gimbal acquires attitude information in real time through the gyroscope and transmits the gimbal attitude to the fixed-wing control module through the serial port and the MavLink communication protocol.

2. The fixed-wing target pursuit method combining an improved SiamFc network and pan / tilt control according to claim 1, characterized in that: In step S1, the onboard computer TX2 runs the improved SiamFc algorithm in the target detection module, and the camera installed on the fixed wing collects image information of the target area in real time.

3. The fixed-wing target pursuit method combining an improved SiamFc network and pan / tilt control according to claim 1, characterized in that: The step S2 is specifically as follows: the ground end and the onboard computer on the fixed-wing establish a communication connection through the wireless bridge transmission module on the fixed-wing aircraft, the camera of the target detection module collects images and transmits them to the ground end through the bridge transmission module and the UDP communication protocol, the target object is selected on the ground end, and the target features are extracted through the image algorithm, and UDP communication is performed with the onboard computer TX2 in the target detection module of the fixed-wing through the wireless bridge transmission module, and the selected target object and its target features are transmitted to the target detection module again.

4. The fixed-wing target pursuit method combining an improved SiamFc network and pan / tilt control according to claim 1, characterized in that: In step S3, specifically, the following steps are performed: inputting the real-time target image to be detected, the target features and the corresponding target area into the trained improved SiamFc model, comparing the similarity between the target image to be detected and the target area corresponding to the target features, returning the detection target area with higher similarity in the target image to be detected, and calculating the pixel coordinate value of the target in the detection target area.

5. The fixed-wing target pursuit method combining an improved SiamFc network and pan / tilt control according to claim 1, characterized in that: In step S4, the target tracking method of the pan-tilt head is as follows: the onboard computer TX2 and the pan-tilt head are connected through the serial port and the mavlink communication protocol to form a pan-tilt head control module. The onboard computer TX2 calculates the target pixel position by improving the SiamFc model, and subtracts the error from the center pixel coordinates of the camera image. The PID control method is used to generate the control amount and send the control signal to the pan-tilt head through the serial port to track the target, so as to keep the target in the center of the camera image. The control formula is as follows: error x (t)=center x (t)-target y (t) errot y (t)=center y (t)-target y (t) Among them: center x (t), center y (t) are the x and y coordinates of the camera center point at time t, target y (t), target y (t) are the x and y pixel coordinates of the detection target at time t, error x (t), error y (t) are the x and y coordinate errors between the target position and the camera center at time t, U x (t), U y (t) represents the gimbal x-axis control amount and gimbal y-axis control amount, K p is the proportional coefficient, T I is the integration time constant, T D is the differential time constant.

6. The fixed-wing target pursuit method combining an improved SiamFc network and pan / tilt control according to claim 1, characterized in that: In step S6, the fixed-wing control module controls the fixed-wing to adjust its attitude according to the gimbal attitude transmitted in step S5, and follows the gimbal attitude change, specifically: The fixed-wing control module calculates the gimbal's attitude data based on the attitude quaternion data sent back by the gimbal. The formula is as follows: In the above formula, q0, q1, q2, q3 are the attitude quaternion data transmitted from the gimbal in step S5; yaw y Indicates the yaw attitude data of the gimbal, pitch y Indicates the pitch attitude data of the gimbal, roll y Indicates the roll attitude data of the gimbal, y represents the gimbal, and atan2 represents the inverse tangent function; According to the current attitude of the fixed wing, the error value between the fixed wing attitude and the gimbal attitude is obtained. yaw 、error pitch 、error roll , error yaw 、error pitch 、errot roll The yaw attitude difference, pitch attitude difference, and roll attitude difference between the fixed wing and the gimbal are respectively added. A differential link is added to speed up the fixed wing's reaction speed. The desired attitude control value of the fixed wing is obtained based on the gimbal's attitude data and the yaw attitude difference and pitch attitude difference between the fixed wing and the gimbal. The calculation formula is as follows: Among them, T Dy is the yaw differential coefficient, T Dp is the pitch differential coefficient, yaw sp 、pitch sp 、roll sp are the control quantities of desired yaw, desired pitch and desired roll, respectively, and error yaw 、error pitch They are the yaw attitude difference and pitch attitude difference between the fixed wing and the gimbal, respectively.

Images