Fixed-wing unmanned aerial vehicle autonomous carrier landing guiding method based on vision

Abstract

The invention discloses a vision-based fixed-wing unmanned aerial vehicle autonomous carrier landing guiding method, which breaks through the defects of traditional active vision guiding, realizes robust extraction of a runway in a carrier landing process by introducing a data-driven artificial intelligence algorithm, and then realizes the unmanned aerial vehicle landing guidance by constructing a special optimization method. According to the method, a dense feature point optimization method is used, the guide precision of the algorithm is improved, and in general, the safety and reliability of the autonomous carrier landing process of the fixed-wing unmanned aerial vehicle are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image measurement, and particularly relates to a vision-based autonomous landing guidance method for fixed-wing unmanned aerial vehicles (UAVs). Technical Background

[0002] Compared with manned carrier-based aircraft, future unmanned carrier-based aircraft will play a greater role. However, unmanned carrier-based aircraft have a very high dependence on the automatic landing system, which also makes the autonomous landing technology of UAVs a key and difficult technology that countries urgently need to solve at present.

[0003] There are two key technologies for UAV autonomous landing technology. One is the high-precision UAV landing guidance technology, and the other is the highly stable UAV landing control technology. The former mainly provides the target path or target points for the UAV during the landing process, and the latter mainly realizes controlling the UAV to land on the moving platform according to the target path or target points.

[0004] Currently, in the field of UAV landing guidance technology, the mainly adopted method is the vision guidance method based on active cooperation markers. However, the cooperation marker method has a high system complexity, low robustness, and requires the transformation of the carrier-based deck. Summary of the Invention

[0005] The purpose of the present invention is: In the passive vision guidance technology, the present invention introduces an artificial intelligence algorithm and proposes a vision-based autonomous landing guidance method for fixed-wing UAVs to improve the robustness and accuracy of UAV landing guidance.

[0006] The technical solution of the present invention: In order to achieve the above-mentioned invention purpose, a vision-based autonomous landing guidance method for fixed-wing UAVs is proposed, including the following steps:

[0007] Step 1: Obtain the true three-dimensional coordinate information of the runway at the carrier-based end, and select the guiding target point O as the coordinate origin tar , and save it in the form of a point cloud. The point cloud set is denoted as ψ;

[0008] Step 2: Calibrate the internal and external parameters of the camera to obtain the model and parameters of the camera;

[0009] In a possible embodiment, the specific steps of the said Step 2 are as follows:

[0010] 2.1 Select the pinhole imaging principle as the camera perspective projection model. Assume that the internal parameter of the camera is K and the external parameter is T, then there is:

[0011]

[0012] A 3D point P in space w , and its projection on the camera pixel plane is p, then there is:

[0013] p = λ·K·T·P w

[0014] 2.2 Prepare planar calibration boards such as checkerboard boards, Apriltag boards, circular boards, etc., and move the camera to image the calibration board at different distances and angles;

[0015] 2.3 Extract the feature points p on the calibration board in the image calib , as well as the spatial coordinates P of the feature points of the calibration board wcalib , then there is:

[0016] p calib = λ·K·[r1 r2 r3 t]·P wcalib = H·P wcalib

[0017] Set B = K -T , K -1 , and the matrix B can be solved through multiple calibration board images, and then the internal parameter matrix K can be obtained:

[0018] 2.4 Assume that the camera has radial distortion and tangential distortion, and the distortion expression formula is as follows:

[0019] Radial distortion:

[0020]

[0021] Tangential distortion:

[0022]

[0023] Through multiple observation points, the distortion parameters k1, k2, k3, p1, p2 in the above formula can be solved using the least squares method

[0024] Step 3: Collect the image data of the entire process of the aircraft landing on the ship, annotate the runway in each image, and obtain the pixel-level runway segmentation result as the ground truth;

[0025] Make a shipboard runway semantic segmentation dataset for the UAV to land on the ship, which is used to train the semantic segmentation network;

[0026] In a possible embodiment, in the said step 3, the two sides of the runway can be segmented using a block diagram, and an accurate pixel-level runway instance area can be obtained, which can reduce the noise error in the neural network training process and improve the semantic segmentation accuracy of the neural network after training.

[0027] Step 4: Establish a semantic segmentation network for the shipboard runway, and use the shipboard runway semantic segmentation dataset obtained in Step 3 to train the semantic segmentation network to obtain the trained network parameters. Thus, the preliminary preparation work is completed;

[0028] In a possible embodiment, in Step 4, the semantic segmentation network for the shipboard runway is constructed based on the yolov8 neural network model, specifically as follows:

[0029] 4.1.1 For the real-time operating environment, select a suitable yolov8 backbone network, including yolov8n, yolov8s, yolov8m, yolov8l, and yolov8x. If the computing power of the real-time operating hardware environment is low, you can choose network models with lower parameter quantities such as yolov8n and yolov8s. When the computing power of the hardware environment is sufficient, you can choose models with more parameter quantities to improve the accuracy of the runway semantic segmentation results;

[0030] 4.1.2 Yolov8 is usually applicable to object detection tasks. For the shipboard runway semantic segmentation task, the yolov8 model needs to be modified. On the basis of the detection head, a semantic segmentation head is added. The semantic segmentation head can output a feature map of N×W×H using a 3×3 convolutional neural network and an upsampling network. N is the number of detected objects, W is the width of the image, and H is the height of the image. The semantic distribution information of the runway is contained in this feature map;

[0031] 4.1.3 Construct a training loss function so that the semantic segmentation network based on yolov8 can converge quickly according to the loss function. The semantic segmentation network based on yolov8 is semantic segmentation based on the detection results.

[0032] When designing the loss function, it is necessary to consider both the object detection loss function and the semantic segmentation loss function at the same time; 4.1.4 Use the shipboard runway semantic segmentation dataset in Step 3 to train the semantic segmentation network. First, adjust the image size to make the input image size of the modified image consistent with the input image size in the real-time operating hardware environment, and input the dataset image into the semantic segmentation network to obtain the output Q;

[0033] 4.1.5 Use the shipboard runway semantic segmentation dataset in Step 3 to obtain the expected output of the network for the training image Then, according to the loss function, using the principle of backpropagation, adjust the model parameters of the semantic segmentation network to complete one training;

[0034] 4.1.6 Repeat the training until the output Q of the semantic segmentation network is consistent with the expected output, and stop the training.

[0035] In a possible embodiment, in step 4, the semantic segmentation network for the shipborne runway is specifically as follows:

[0036] 4.2.1 Designing the semantic segmentation network for the shipborne runway includes selecting a suitable main feature extraction network, designing an accurate semantic segmentation head network, and designing an efficient loss function;

[0037] 4.2.2 The main feature extraction network of the semantic segmentation network for the shipborne runway adopts an encode-decode network, which can increase the feature extraction ability of the feature extraction network and retain the position information of the network output features in the image. The Encode part uses a fully convolutional network to downsample the image to increase the feature dimension, and the Decode part uses a deconv network and an unpooling network to upsample the features to increase the feature size;

[0038] 4.2.3 The semantic segmentation head network of the semantic segmentation network for the shipborne runway adopts a 3×3 or 1×1 fully convolutional network layer, which compresses and fuses the feature dimension while retaining the feature size, so that the final output is a feature map of C×W×H, where C is 2 times the number of semantic segmentation categories, which is 2 in this task, W is the width of the image, and H is the height of the image;

[0039] 4.2.4 The loss function of the semantic segmentation network for the shipborne runway needs to transfer the ground truth data in the shipborne runway semantic segmentation dataset in step 3 to the network training process, and the loss function selects BCEloss4.2.5 Use the shipborne runway semantic segmentation dataset in step 3 to train the semantic segmentation network. First, adjust the image size to make the input image size consistent with the input image size in the real-time running hardware environment, and input the dataset image into the semantic segmentation network to obtain the output Q;

[0040] 4.2.6 Use the shipborne runway semantic segmentation dataset in step 3 to obtain the expected output of the network for the training image Then, according to the loss function and using the principle of backpropagation, the model parameters of the semantic segmentation network can be adjusted to complete one training;

[0041] 4.2.7 Repeat the training until the output Q of the shipborne runway semantic segmentation network is consistent with the expected output and

[0042] stop the training.

[0043] In a possible embodiment, in step 4, the loss function during the training process is as follows, where Q is the predicted output value of the network, and

[0044]

[0045] Step 5: In actual application, use the semantic segmentation network obtained in Step 4 to extract the runway image of the shipboard end captured by the UAV in real time, obtain the runway area in the image, and find the contour set Ω of this area;

[0046] In a possible embodiment, in the said Step 5, the specific process of finding the contour set Ω of this area according to the runway area includes:

[0047] 5.1 Use the semantic segmentation network obtained in Step 4 to obtain the runway area set in the image, construct a binary image, where the pixel value of the runway area is 255 and that of the non-runway area is 0;

[0048] 5.2 Remove the noise areas in the binary image, such as noise points or noise lines, to avoid incorrect sets during the contour extraction process. Gaussian filters or median filters can be used to smooth the image;

[0049] 5.3 Use contour detection algorithms, such as Canny edge detection, Sobel operator, Laplacian operator, etc. to extract the contours in the binary image;

[0050] 5.4 Use the runway edge line features to remove the possible noise contour sets. The area, perimeter, aspect ratio, etc. of the runway can be used for screening to obtain the final required runway area contour set.

[0051] Step 6: Use the runway point cloud set ψ in Step 1 and the runway contour set Ω in Step 5 to construct the following optimization equation to realize the relative pose measurement of the UAV and the target guiding point at the image capture moment.

[0052]

[0053] Among them, Ω is the runway contour set in Step 5, K represents the internal parameters of the camera, R and t are the rotation matrix and translation vector from the camera to the runway target at the shipboard end at the camera shooting moment, that is, the target to be optimized, and P w is the point in the runway point cloud set ψ in Step 1. Indicates that the imaging position of the point in the runway point cloud set ψ in the image is not inside the contour set obtained in Step 5.

[0054] In a possible embodiment, the specific steps in the said Step 6 are as follows:

[0055] 6.1 Based on the runway contour set Ω, a two-dimensional value table graph is constructed during the optimization process. The query result of the value table is the loss function value. By constructing the two-dimensional value table in advance, repetitive calculations during the optimization process can be reduced, the optimization speed can be increased, and numerical differentiation can be achieved by interpolating the two-dimensional value table, thereby completing the relative pose optimization process of the UAV and the target guiding point at the image shooting moment;

[0056] 6.2 According to the camera model and parameters obtained in step 2, combined with the perspective projection process, project the runway point cloud set ψ onto the current image to obtain the pixel set Π. Set the initial rotation matrix R init

[0057] and the translation vector t init , then the pixel set Π can be expressed as:

[0058] Π = K·[R init , t init ·ψ

[0059] 6.3 For each pixel point in the set Π, query in the two-dimensional value table graph to obtain the loss function of this point, and add up the loss functions of all points to obtain the loss function of the current optimization;

[0060] 6.4 Using the method of numerical differentiation, minimize the loss function, and update the initial rotation matrix R init and the translation vector t init , and use the central difference method to obtain the numerical differentiation formula as follows:

[0061]

[0062] where, represents the gradient of f(R init , t init ) at the target point (R init , t init ), represents the partial derivative of f(R, t) with respect to the independent variable r1. (R init , t init ) has 6 unknowns, and the current gradient value is obtained using this formula in combination with the two-dimensional value table Use the gradient descent method to update the independent variable, and the update formula is as follows:

[0063]

[0064] 6.5 Repeat 6.2 - 6.4 to obtain the final solution result of this image, that is, the relative pose R cam-tar and t cam-tar of the camera relative to the target guiding point. When the gradient value is less than the set threshold, the final optimization result is obtained:

[0065] R cam-tar = R new

[0066] t cam-tar = t new

[0067] 6.6 Combining the installation declination and lever arm of the camera on the UAV, the relative pose R uva-tar and t uva-car :

[0068] R uva-tar = R uav-cam · R cam-tar

[0069] t uva-tar = R uav-cam · t cam-tar + t uva-cam

[0070] Step 7: Output R uva-tar and t uva-tar to the UAV's rear-end flight control system to complete the autonomous landing guidance for the fixed-wing UAV.

[0071] Advantages of the present invention:

[0072] The present invention provides a vision-based autonomous landing guidance method for fixed-wing UAVs. This method overcomes the shortcomings of traditional active vision guidance. By introducing data-driven artificial intelligence algorithms, it realizes the robust extraction of the runway during the landing process. Then, by constructing a special optimization method, it realizes the measurement of the relative pose between the UAV and the target guidance point at the image capture moment. This method uses the method of optimizing dense feature points to improve the guidance accuracy of the algorithm. Generally speaking, the present invention improves the safety and reliability of the autonomous landing process of fixed-wing UAVs. Description of the drawings

[0073] Figure 1 is a schematic diagram of the coordinate system of the preferred embodiment of the present invention;

[0074] Figure 2 is a flowchart of the method of Embodiment 1 of the present invention;

[0075] Figure 3 is a schematic diagram of the semantic segmentation of the aircraft landing runway in Embodiment 1 of the present invention;

[0076] Figure 4 is the Cantor set Ω of the runway area in Embodiment 1 of the present invention;

[0077] Figure 5 is a schematic diagram of the two-dimensional value table diagram in Embodiment 1 of the present invention;

[0078] Figure 6 This is a schematic diagram of the principle of Embodiment 1 of the present invention. Detailed implementation manners

[0079] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0080] The features and illustrative embodiments of various aspects of the present invention will be described in detail below. In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be practiced without some of these specific details. The following description of the embodiments is only intended to provide a better understanding of the present invention by showing examples of the present invention. The present invention is in no way limited to any specific arrangement and method set forth below, but covers any improvement, substitution and modification of structures, methods and devices without departing from the spirit of the present invention. Well-known structures and technologies are not shown in the drawings and the following description to avoid unnecessarily obscuring the present invention.

[0081] It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other, and the various embodiments may refer to and cite each other. The present invention will be described in detail below with reference to the drawings and in combination with the embodiments.

[0082] Figure 1 This is a schematic diagram of the coordinate system involved in the present invention, where the carrier-based landing runway coordinate system O tar takes the target guiding point as the origin, the horizontal right direction as the x-axis, the downward direction as the y-axis, and the forward direction as the z-axis. The camera coordinate system O cam takes the center of the camera imaging plane as the origin, the horizontal right direction as the x-axis, the downward direction as the y-axis, and the forward direction as the z-axis. The aircraft coordinate system O pla takes the aircraft center of mass as the origin, the horizontal right direction as the x-axis, the downward direction as the y-axis, and the forward direction as the z-axis. The aircraft coordinate system O pla and the carrier-based landing runway coordinate system O tar have a rotation matrix R and a translation vector t between them, and R and t are the required UAV autonomous landing guidance information.

[0083] Embodiment 1

[0084] As Figure 2 shown, the process is as follows:

[0085] Step 1: Obtain the true three-dimensional coordinate information of the runway on the shipborne side, and select the guiding target point O as the coordinate origin tar , and save it in the form of a point cloud. The point cloud set is denoted as ψ;

[0086] Step 2: Calibrate the camera to obtain the camera model and parameters. The camera is a pinhole imaging model. The principal point, focal length, tangential distortion, and radial distortion parameters of the camera can be obtained using the checkerboard calibration method;

[0087] 2.1 Select the pinhole imaging principle as the camera perspective projection model. Assume that the internal parameters of the camera are K and the external parameters are T. Then there are:

[0088]

[0089] A 3D point P in space w , and its projection on the camera pixel plane is p. Then there are:

[0090] p = λ·K·T·P w

[0091] 2.2 Prepare a 15x15 checkerboard calibration board, move the camera to image the calibration board at different distances and angles, and obtain 30 sets of calibration board images with different perspectives, ensuring that the calibration board is imaged completely and the edges are clear in each image;

[0092] 2.3 Extract the checkerboard corner points in the calibration board image as feature points p calib , and at the same time extract the spatial coordinates P of the checkerboard feature points of the calibration board wcalib , then there are:

[0093] p calib = λ·K·[r1 r2 r3 t]·P wcalib = H·P wcalib

[0094] Set E = K -T ·K -1 , through 30 calibration board images, combined with the least squares method, the matrix B can be solved, and then the internal parameter matrix K can be obtained:

[0095]

[0096] 2.4 Use radial distortion and tangential distortion as the camera distortion parameters. The distortion expression formula is as follows:

[0097] Radial distortion:

[0098]

[0099] Tangential distortion:

[0100]

[0101] Through multiple observation points, the distortion parameters k1, k2, k3, p1, and p1 in the above formula can be solved by least squares.

[0102] Step 3: Collect image data of the entire process of the aircraft landing on the ship, covering different weather and lighting conditions as much as possible. Manually annotate the runway in each image to obtain a pixel-level runway segmentation result as the ground truth, and create a shipboard runway semantic segmentation dataset for the UAV to land on the ship, which is used to train the semantic segmentation network. The schematic diagram of the runway semantic segmentation is as Figure 3 shown;

[0103] Step 4: Design and implement a semantic segmentation network for the shipboard runway based on the yolov8 model, and use the dataset obtained in Step 3 to train this network to obtain the trained network parameters.

[0104] 4.1 On the real-time operation platform of RK3588, the NPU computing power is about 6 TOPS. Due to the low computing power of the operating hardware environment, the yolov8s model is used, which has fewer network layers and about 11.2M parameters.

[0105] 4.2 The yolov8 network is suitable for object detection tasks. For the shipboard runway semantic segmentation task, the yolov8 model needs to be modified. A semantic segmentation head is added on the basis of the detection head. The semantic segmentation head can output a feature map of N×W×H using a 3×3 convolutional neural network and an upsampling network. N is the number of detected targets, W is the width of the image, and H is the height of the image. This feature map contains the semantic distribution information of the runway.

[0106] 4.3 Construct the training loss function so that the yolov8-based semantic segmentation network can converge quickly according to the loss function. The yolov8-based semantic segmentation network is semantic segmentation based on the detection result. When designing the loss function, the object detection loss function and the semantic segmentation loss function need to be considered simultaneously. The loss function is as follows;

[0107] loss = DFLoss + CIOULoss + BCE Loss + MASK Loss

[0108] 4.4 Use the shipboard runway semantic segmentation dataset in Step 3 to train the semantic segmentation network. First, adjust the image size to make the input image size consistent with the input image size in the real-time operation hardware environment, and input the dataset image into the semantic segmentation network to obtain the output Q.

[0109] 4.5 Use the shipboard runway semantic segmentation dataset in Step 3 to obtain the expected output of the network for the training image. According to the loss function, the model parameters of the semantic segmentation network can be adjusted using the principle of backpropagation to complete one training;

[0110] 4.6 Repeat the training until the output Q of the semantic segmentation network is consistent with the expected output and stop the training. Step 5: In actual application, use the network model obtained in step 4 to process the images taken by the drone in real time, extract the runway area in the images, and obtain the contour set Ω of this area, as Figure 4 shown;

[0111] 5.1 Use the trained yolov8-based semantic segmentation network in step 4 to obtain the set of runway areas in the real-time captured images, and construct a binary image based on the set. In the binary image, the pixel value of the runway area is 255, and the non-runway area is 0;

[0112] 5.2 Remove the noise areas in the binary image, such as noise points and noise lines, to avoid incorrect sets during the contour extraction process. Gaussian filters or median filters can be used to smooth the image;

[0113] Gaussian filter formula:

[0114]

[0115] Median filter formula:

[0116] V(x, y) = mediax{V(x - k, y - l)|(k, l) ∈ S}

[0117] 5.3 Use the contour detection algorithm, such as the Sobel operator, to extract the contours in the binary image. The Sobel operator is a first-order derivative edge detection operator. The operator contains two 3x3 matrices. Convolving the matrices with the image respectively can obtain the horizontal and vertical gradient values. Gradient values greater than 50 are listed as edge points in the contour set;

[0118]

[0119] where A is the original binary image.

[0120] 5.4 Use the runway edge line features to remove the possible noise contour set. The aspect ratio attribute can be used for screening. If the aspect ratio is less than 10, it is deleted as noise to obtain the final required runway area contour set.

[0121] Step 6: Construct an optimization equation to achieve the relative pose measurement of the UAV and the target guiding point at the image shooting moment. One of the key points of the present invention is the construction of the optimization equation. By using the runway point cloud set ψ and the runway contour set Ω, and combining the camera imaging process, an optimization equation without a matching relationship is constructed, so as to optimize the pose of the camera from the camera to the runway target at the carrier end at the shooting moment. The optimization equation is as follows:

[0122]

[0123] where Ω is the runway contour set in Step 5, K represents the internal parameter of the camera, R and t are the rotation matrix and translation vector from the camera to the runway target at the carrier end at the camera shooting moment, that is, the target to be optimized, and P w is the point in the runway point cloud set ψ in Step 1. means that the imaging position of the point in the runway point cloud set ψ in the image is not inside the contour set obtained in Step 5.

[0124] 6.1 Based on the runway contour set Ω, construct a two-dimensional value table graph in the optimization process. The query result of the value table is the loss function value. Since the accurate matching relationship between the image points and the three-dimensional points cannot be known in the present invention, the distance from the point to the Cantor set Ω is used as the query result of the value table. The schematic diagram is as Figure 5 shown. The query values of point A and point B on the two-dimensional value table in the figure are the Euclidean distances d1 from point A to the contour set Ω and d2 from point B to the contour set Ω, respectively;

[0125] 6.2 Based on the camera model, combine the perspective projection process to project the runway point cloud set ψ onto the current image to obtain the pixel set Π. Set the initial rotation matrix R init and translation vector T init during the projection;

[0126] Π = K · [R init , t init · ψ

[0127] 6.3 For each pixel point in the set Π, query in the two-dimensional value table graph to obtain the loss function of this point. Add up the loss functions of all points to obtain the loss function of the current optimization. The principle of the optimization process of the present invention is as Figure 6 shown. The optimization direction is to move all the runway projection point sets Π into the contour set Ω; 6.4 Use the method of numerical differentiation to minimize the loss function and update the initial rotation matrix R init and translation vector T init ;

[0128] The numerical differentiation formula is obtained by using the central difference method as follows:

[0129]

[0130] Among them, represents the gradient of f(R init , t init ) at the target point (R init , t init ); represents the partial derivative of f(R, t) with respect to the independent variable r1. (R init , t init ) has a total of 6 unknowns. Using this formula in combination with a two-dimensional value table to obtain the current gradient value Update the independent variable using the gradient descent method. The update formula is as follows:

[0131]

[0132] 6.5 Repeat 6.2 - 6.4 to obtain the final solution result of this image, that is, the relative pose R cam-tar and T cam-tar ;

[0133] The gradient value is less than 10 -6 to obtain the final optimization result:

[0134] R cam-tar = R new

[0135] t cam-tar = t new

[0136] 6.6 Combine the installation declination and lever arm of the camera on the UAV to obtain the relative pose R uva-tar and T uva-tar of the UAV and the target guiding point at the image shooting moment;

[0137] R uva-tar = R uav-cam ·R cam-tar

[0138] t uva-tar = R uav-cam ·t cam-tar + t uva-cam

[0139] Step 7: Output R uva-tar and T uva-tar to the UAV's rear - end flight control system to complete the autonomous landing guidance for the fixed - wing UAV.

[0140] Example 2

[0141] In Embodiment 2, the method of the present invention can design a semantic segmentation network for the carrier-based runway.

[0142] Steps 1 to 3 are the same as those in Embodiment 1.

[0143] Step 4. Designing a semantic segmentation network for the carrier-based runway includes selecting a suitable main feature extraction network, designing an accurate semantic segmentation head network, and designing an efficient loss function. On this basis, the network is trained using the dataset obtained in Step 3 to obtain the trained network parameters;

[0144] 4.1 The main feature extraction network of the semantic segmentation network for the carrier-based runway adopts an encode-decode network, which can increase the feature extraction ability of the feature extraction network and retain the position information of the network output features in the image. The Encode part uses 9 fully convolutional networks to downsample the image to increase the feature dimension, and the Decode part uses 9 deconv networks to upsample the features to increase the feature size;

[0145] 4.2 The semantic segmentation head network of the semantic segmentation network for the carrier-based runway adopts 3 3×3 and 1 1×1 fully convolutional network layers, which fuse the depth of the features while retaining the feature size, so that the final output is a feature map of C×W×H, where C is 2 times the number of semantic segmentation categories, which is 2 in this task, W is the width of the image,

[0146] H is the height of the image;

[0147] 4.3 The loss function of the semantic segmentation network for the carrier-based runway needs to transfer the data ground truth in the carrier-based runway semantic segmentation dataset in Step 3 to the network training process, and the loss function selects BCEloss

[0148]

[0149] where Q is the predicted output of the network, is the true label in the carrier-based runway semantic segmentation dataset, and q and respectively represent a pixel in the network predicted output and the true label.

[0150] 4.4 Use the carrier-based runway semantic segmentation dataset in Step 3 to train the semantic segmentation network. First, adjust the image size to make the input image size consistent with the input image size in the real-time running hardware environment, and input the dataset image into the semantic segmentation network to obtain the output Q;

[0151] 4.5 Use the carrier-based runway semantic segmentation dataset in Step 3 to obtain the expected output of the network for the training image Based on the loss function, the model parameters of the semantic segmentation network can be adjusted using the principle of backpropagation to complete one training session.

[0152] 4.6 Repeat the training until the output Q of the runway semantic segmentation network on the shipborne terminal is consistent with the expected output, and then stop the training.

[0153] Steps 5 to 7 are the same as those in Example 1.

Claims

1. A vision-based autonomous landing guidance method for fixed-wing UAVs, characterized in that, It includes the following steps: Step 1: Obtain the true three-dimensional coordinate information of the runway on the shipborne side, and select the guiding target point O as the coordinate origin tar , save it in the form of point cloud, and denote the point cloud set as ψ; Step 2: Calibrate the internal and external parameters of the camera to obtain the camera model and parameters; Step 3: Collect image data of the entire process of the aircraft landing on the ship, annotate the runway in each image, obtain the pixel-level runway segmentation result and use it as the ground truth; form a shipboard runway semantic segmentation dataset for the UAV to land on the ship, which is used to train the semantic segmentation network; Step 4: Establish a semantic segmentation network for the shipboard runway, and use the shipboard runway semantic segmentation dataset obtained in Step 3 to train the semantic segmentation network to obtain the trained network parameters; Step 5: Use the semantic segmentation network obtained in Step 4 to extract the shipboard runway image captured by the UAV in real time, obtain the runway area in the image, and calculate the contour set Ω of this area; Step 6: Use the runway point cloud set ψ in Step 1 and the runway contour set Ω in Step 5 to construct the following optimization equation to realize the relative pose measurement of the UAV and the target guiding point at the image capture moment, and use the relative pose measurement result to complete the autonomous landing guidance of the fixed-wing UAV: Among them, Ω is the runway contour set in step 5, K represents the internal parameters of the camera, R and t are the rotation matrix and translation vector from the camera to the runway target at the shipboard end at the moment of camera shooting, that is, the target to be optimized, and P w is the point in the runway point cloud set ψ in step 1; means that the imaging position of the point in the runway point cloud set ψ in the image is not inside the contour set obtained in step 5.

2. The autonomous landing guidance method for a fixed-wing UAV based on vision according to claim 1, characterized in that In Step 2, the specific process of camera internal and external parameter calibration includes: 2.1 Select the pinhole imaging principle as the camera perspective projection model. Assume that the internal parameter of the camera is K and the external parameter is T, then there is: A 3D point P in space w , and its projection on the camera pixel plane is p, then there is: p = λ·K·T·P w 2.2 Prepare a planar calibration board such as a checkerboard board, an Apriltag board, a circular board, etc., and move the camera to image the calibration board at different distances and angles; 2.3 Extract the feature points p on the calibration board in the image calib , and the spatial coordinates P of the feature points on the calibration board wcalib , then there is: p calib = λ·K·[r1 r2 r3 t]·P wcalib = H·P wcalib Set B = K -T ·K -1 , the matrix B can be solved by multiple calibration plate images, and then the internal parameter matrix K can be obtained: 2.4 Assume that the camera has radial distortion and tangential distortion, and the distortion expression formula is as follows: Radial distortion: Tangential distortion: Through multiple observation points, the distortion parameters k1, k2, k3, p1, and p2 in the above formula can be solved by the least squares method.

3. A vision-based autonomous landing guidance method for a fixed-wing UAV according to claim 1, characterized in that, In Step 3, use a block diagram to segment both sides of the runway.

4. A vision-based autonomous landing guidance method for a fixed-wing UAV according to claim 1, wherein In Step 4, the semantic segmentation network for the shipboard runway is constructed based on the yolov8 neural network model.

5. A vision-based autonomous landing guidance method for fixed-wing UAVs according to claim 4, characterized in that, In Step 4, for the shipboard runway semantic segmentation task, modify the yolov8 model, add a semantic segmentation head on the basis of the detection head. The semantic segmentation head uses a 3×3 convolutional neural network and an upsampling network, and outputs a feature map of N×W×H, where N is the number of detected targets, W is the width of the image, and H is the height of the image. This feature map contains the semantic distribution information of the runway.

6. A vision-based autonomous landing guidance method for fixed-wing UAVs according to claim 4, characterized in that In the step 4, the training process includes: First, adjust the image size to make the input size of the image consistent with the input image size in the real-time running hardware environment, and input the dataset image into the semantic segmentation network to obtain the output Q; Use the shipborne runway semantic segmentation dataset in step 3 to obtain the expected output of the network for the training image According to the loss function, use the principle of backpropagation to adjust the model parameters of the semantic segmentation network to complete one training; Repeat the training to make the output Q of the semantic segmentation network consistent with the expected output and stop the training.

7. A vision-based autonomous landing guidance method for a fixed-wing unmanned aerial vehicle according to claim 1, characterized in that In Step 4, the main network for feature extraction of the semantic segmentation network for the shipboard runway uses an encode-decode network.

8. A vision-based autonomous landing guidance method for fixed-wing UAVs according to claim 1, characterized in that In Step 5, the specific process of calculating the contour set Ω of this area according to the runway area includes: Use the semantic segmentation network obtained in Step 4 to obtain the runway area set in the image, construct a binary image, where the pixel value of the runway area is 255 and the non-runway area is 0; Remove the noise areas in the binary image, such as noise points or noise lines, to avoid incorrect sets during the contour extraction process. Gaussian filters or median filters can be used to smooth the image; Use the contour detection algorithm, such as Canny edge detection, Sobel operator, Laplacian operator, etc. to extract the contour in the binary image; Use the runway edge line features to remove the possible noise contour set. The runway area, perimeter, aspect ratio and other attributes can be used for screening to obtain the final required runway area contour set.

9. A vision-based autonomous landing guidance method for fixed-wing UAVs according to claim 1, characterized in that, In step 6, the specific steps are as follows: 6.1 Based on the runway contour set Ω, construct a two-dimensional value table graph during the optimization process. The query result of the value table is the loss function value. By constructing the two-dimensional value table in advance, the repeated calculations during the optimization process can be reduced, the optimization speed can be improved, and numerical differentiation can be realized by using two-dimensional value table interpolation, so as to complete the relative pose optimization process of the UAV and the target guiding point at the image shooting moment; 6.2 According to the camera model and parameters obtained in step 2, combined with the perspective projection process, project the runway point cloud set ψ onto the current image to obtain the pixel set П. Set the initial rotation matrix R init and the translation vector t init . At this time, the pixel set П can be expressed as: Π = K·[R init , t init ·ψ 6.3 For each pixel point in the set П, query in the two-dimensional value table graph to obtain the loss function of this point. Add up the loss functions of all points to obtain the loss function of the current optimization. 6.4 Using the method of numerical differentiation to minimize the loss function and update the initial rotation matrix R init and the translation vector t init , the numerical differentiation formula is obtained by using the central difference method as follows: Among them, Denote the gradient of f(R init , t init ) at the target point (R init , t init ). Denote the partial derivative of f(R, t) with respect to the independent variable r1. (R init , t init ) has a total of 6 unknowns. Use this formula in combination with the two-dimensional value table to obtain the current gradient value Update the independent variable using the gradient descent method. The update formula is as follows: 6.5 Repeat 6.2 - 6.4 to obtain the final solution result of this image, that is, the relative pose R of the camera with respect to the target guiding point cam-tar and t cam-tar , when the gradient value is less than the set threshold, the final optimization result is obtained: R cam-tar = R new t cam-tar = t new 6.6 Combining the installation declination and lever arm of the camera on the UAV, the relative pose R uva-tar and t uva-tar : R uva-tar = R uav-cam ·R cam-tar t uva-tar = R uav-cam ·t cam-tar + t uva-cam .

Images