Binocular vision air refueling taper sleeve real-time tracking and auxiliary docking method

Abstract

The invention provides a binocular vision air refueling taper sleeve real-time tracking and auxiliary docking method. The binocular vision air refueling taper sleeve real-time tracking and auxiliary docking method comprises the following steps that S1, a real-time video or image of a refueling taper sleeve is obtained through a binocular camera; s2, using artificial intelligence to complete taper sleeve model training, and using YOLOV11 to complete taper sleeve identification and tracking on this basis; s3, auxiliary information such as the approaching speed, the axial deviation, the plane deviation, the butt joint distance and the taper sleeve swing amplitude of the taper pipe and the taper sleeve is calculated; and S4, controlling the frequency of sound broadcasting according to the distance, and performing auxiliary prompt on the alarm information through red, green and yellow colors. Auxiliary butt joint of the taper sleeve and the taper pipe is carried out through videos or images obtained through binocular vision, the precision and safety of refueling butt joint are remarkably improved in the complex air environment, and the method has wide application prospects.

Description

Technical Field

[0001] The invention relates to the technical field of visual image processing, and in particular to a binocular vision aerial refueling drogue real-time tracking and auxiliary docking method. Background Art

[0002] Aerial refueling technology is a technology that uses a tanker to refuel other aircraft or helicopters in flight. It can significantly improve the endurance of fighter jets and plays an extremely important supporting role in the operations of strategic or tactical aviation units. Dredgers are generally used in hose-type aerial refueling, and the drogues need to be tracked and assisted in docking in real time, so a binocular vision aerial refueling drogue real-time tracking and assisted docking method is set up.

[0003] When binocular vision is used for real-time tracking and assisted docking of the aerial refueling drogue, the drogue is docked randomly during the aerial refueling process in the prior art, making it difficult to accurately track and assist in docking in real time. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies in the prior art, the present invention provides a binocular vision aerial refueling drogue real-time tracking and auxiliary docking method, which solves the problem that the aerial refueling drogue is difficult to track and assist in docking.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A binocular vision aerial refueling drogue real-time tracking and auxiliary docking method, comprising the following steps:

[0008] S1. Obtain real-time video or image of the refueling drogue through a binocular camera;

[0009] S2. Use artificial intelligence to complete the training of the cone sleeve model, and on this basis, use YOLOV11 to complete the cone sleeve recognition and tracking;

[0010] S3, calculate the auxiliary information such as the approach speed, axial deviation, plane deviation, docking distance, swing amplitude of the cone sleeve and the like of the cone tube;

[0011] S4. The frequency of sound broadcast is controlled according to the distance, and the warning information is assisted by red, green and yellow colors.

[0012] Preferably, the step S1 further comprises the following steps:

[0013] S1A, binocular camera configuration: Install a high-resolution binocular camera in the refueling docking area of ​​the receiving aircraft; the binocular camera has real-time data acquisition capabilities, can provide high frame rate and high-resolution video streams, and support work in adverse weather conditions;

[0014] S1B, Image acquisition process: Use the binocular camera to obtain the front scene video of the aerial refueling drogue in real time; use the synchronous acquisition mechanism to ensure the frame alignment of the left and right camera images to obtain depth information;

[0015] S1C, image preprocessing: denoising, enhancing and histogram equalization are performed on the collected images; background noise is filtered out through a specific filter to retain the significant features of the cone sleeve area.

[0016] Preferably, the step S2 further comprises the following steps:

[0017] S2A, collect a large number of cone sleeve image data under different lighting, angles, distances and backgrounds;

[0018] S2B, use deep learning frameworks (such as PyTorch or TensorFlow) to train the YOLOv11 model; during model training, use data augmentation techniques (such as rotation, scaling, and cropping) to improve the robustness of recognition;

[0019] S2C, real-time cone sleeve recognition and tracking: Deploy the trained YOLOv11 model to the edge computing device to achieve real-time cone sleeve recognition; track the cone sleeve and extract its three-dimensional coordinates based on the depth information of the binocular camera;

[0020] S2D, recognition and tracking optimization: Use optical flow algorithm and Kalman filter technology to optimize tracking accuracy; for the high-speed movement of the cone sleeve, improve system stability through multi-frame information fusion.

[0021] Preferably, the step S3 further comprises the following steps:

[0022] S3A, Approach speed calculation: The approach speed between the receiving aircraft and the drogue is calculated by combining the change in the position of the drogue in consecutive frames with the time difference. The approach speed refers to the relative speed between the drogue and the drogue in three-dimensional space. The calculation formula is as follows:

[0023] S3B, Calculation of axial deviation and plane deviation: Axial deviation and plane deviation are the relative position deviations between the cone sleeve and the cone tube in three-dimensional space, which are used to guide the adjustment operation. Calculate the difference between the center point of the cone sleeve and the target point of the cone tube in three-dimensional space; decompose the difference into axial deviation and plane deviation, which respectively reflect the offset direction and amplitude of the cone sleeve.

[0024] S3C, docking distance measurement: using the parallax principle of the binocular camera, accurately measure the three-dimensional distance between the cone sleeve and the cone tube; the basic formula is:

[0025] S3D, depth information calculation process: Matching feature points is to find the same feature points (such as corner points, edge points, etc.) in the left and right images, usually achieved through feature point extraction algorithms (such as SIFT, SURF), and calculate the pixel coordinate difference of the corresponding points in the left and right images through the feature matching algorithm, that is, the disparity d

[0026] S3E, calculation of the depth of the cone sleeve center point: The target detection of the cone sleeve is to use the target detection algorithm (such as YOLOv11) to detect the bounding box B of the cone sleeve t , and obtain the center point coordinates (x t ,y t ); extracting the center point depth is to extract the depth corresponding to the center point of the cone sleeve from the depth map z(x,y);

[0027] S3F, drogue swing amplitude analysis: extract the historical trajectory of the drogue center point and analyze the movement pattern of the drogue; the drogue swing amplitude is to analyze the deviation of the drogue relative to its trajectory during operation, which is used to guide the stability adjustment of the operation. Calculate the swing amplitude and frequency to prompt the pilot to adjust the operation strategy; the formula for the swing amplitude is:

[0028] Preferably, the step S4 further comprises the following steps: sound broadcast, color auxiliary prompt, graded prompt rules, warning information display, according to the docking distance D between the cone sleeve and the cone tube distance and the deviation D distance and D distance Set the color hint scheme.

[0029] (III) Beneficial effects

[0030] The present invention provides a binocular vision aerial refueling drogue real-time tracking and auxiliary docking method. It has the following beneficial effects:

[0031] The use of videos or images acquired through binocular vision to assist in the docking of the cone-in-cone and cone-tube significantly improves the accuracy and safety of refueling docking in complex aerial environments and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the step structure of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Example:

[0035] like Figure 1 As shown, the embodiment of the present invention provides a binocular vision aerial refueling drogue real-time tracking and auxiliary docking method, including the following steps:

[0036] S1. Obtain real-time video or image of the refueling drogue through a binocular camera;

[0037] S2. Use artificial intelligence to complete the training of the cone sleeve model, and on this basis, use YOLOV11 to complete the cone sleeve recognition and tracking;

[0038] S3, calculate the auxiliary information such as the approach speed, axial deviation, plane deviation, docking distance, swing amplitude of the cone sleeve and the like of the cone tube;

[0039] S4. The frequency of sound broadcast is controlled according to the distance, and the warning information is assisted by red, green and yellow colors.

[0040] Step S1 further comprises the following steps:

[0041] S1A, binocular camera configuration: Install a high-resolution binocular camera in the refueling docking area of ​​the receiving aircraft; the binocular camera has real-time data acquisition capabilities, can provide high frame rate and high-resolution video streams, and support work in adverse weather conditions;

[0042] S1B, Image acquisition process: Use the binocular camera to obtain the front scene video of the aerial refueling drogue in real time; use the synchronous acquisition mechanism to ensure the frame alignment of the left and right camera images to obtain depth information;

[0043] S1C, image preprocessing: denoising, enhancing and histogram equalization are performed on the collected images; background noise is filtered out through a specific filter to retain the significant features of the cone sleeve area.

[0044] Step S2 further comprises the following steps:

[0045] S2A, collect a large number of cone sleeve image data under different lighting, angles, distances and backgrounds;

[0046] S2B, use deep learning frameworks (such as PyTorch or TensorFlow) to train the YOLOv11 model; during model training, use data augmentation techniques (such as rotation, scaling, and cropping) to improve the robustness of recognition;

[0047] S2C, real-time cone sleeve recognition and tracking: Deploy the trained YOLOv11 model to the edge computing device to achieve real-time cone sleeve recognition; track the cone sleeve and extract its three-dimensional coordinates based on the depth information of the binocular camera;

[0048] S2D, recognition and tracking optimization: Use optical flow algorithm and Kalman filter technology to optimize tracking accuracy; for the high-speed movement of the cone sleeve, improve system stability through multi-frame information fusion.

[0049] Step S3 further comprises the following steps:

[0050] S3A, Approach speed calculation: The approach speed between the receiving aircraft and the drogue is calculated by combining the change in the position of the drogue in consecutive frames with the time difference. The approach speed refers to the relative speed between the drogue and the drogue in three-dimensional space. The calculation formula is as follows:

[0051] (x1, y1, z1): 3D coordinates of the cone sleeve in the current frame;

[0052] (x2, y2, z2): 3D coordinates of the cone sleeve in the previous frame;

[0053] Δt: the time interval between the current frame and the previous frame;

[0054] x1,y1,z1,x2,y2,z2,Δt updates the speed value in real time through binocular depth information. Determine the three-dimensional coordinates of the cone sleeve through the depth information of the binocular camera; calculate the relative speed based on the coordinate changes of consecutive frames;

[0055] S3B, Calculation of axial deviation and plane deviation: Axial deviation and plane deviation are the relative position deviations between the cone sleeve and the cone tube in three-dimensional space, which are used to guide the adjustment operation. Calculate the difference between the center point of the cone sleeve and the target point of the cone tube in three-dimensional space; decompose the difference into axial deviation and plane deviation, which respectively reflect the offset direction and amplitude of the cone sleeve.

[0056] Axial deviation: The calculation formula for axial deviation is: D axial =z t -z c

[0057] z t : Depth coordinate of the taper sleeve;

[0058] z c : Depth coordinate of target cone sleeve;

[0059] Plane deviation: The plane deviation calculation formula is:

[0060] (x t ,y t ): coordinates of the cone sleeve on the plane;

[0061] (x c ,y c ): plane coordinates of the target cone sleeve;

[0062] The spatial deviation between the cone sleeve and the cone tube is calculated in real time through a binocular camera; multimodal prompts (such as sound and color) are generated based on the deviation data;

[0063] S3C, docking distance measurement: the parallax principle of the binocular camera is used to accurately measure the three-dimensional distance between the cone sleeve and the cone tube; the docking distance is the straight-line distance between the center of the cone sleeve and the target point of the cone tube. The binocular vision depth calculation uses the images captured by the left and right cameras to estimate the depth of the midpoint of the scene through triangulation. The basic formula is:

[0064] z: the depth information of the object, that is, the distance from the camera;

[0065] f: focal length of the camera (in pixels, obtained through camera calibration);

[0066] B: baseline distance between the left and right cameras (i.e., the fixed distance between the two cameras);

[0067] d: disparity, which indicates the difference in pixel position of the same object in the left and right camera images;

[0068] S3D, calculation process of depth information: matching feature points is to find the same feature points (such as corner points, edge points, etc.) in the left and right images, usually achieved through feature point extraction algorithms (such as SIFT, SURF), and calculate the pixel coordinate difference of the corresponding points in the left and right images through the feature matching algorithm, that is, the disparity d;

[0069] Depth information calculation is that the disparity d reflects the position difference of the object in the left and right images, and the depth z is calculated according to the formula Directly calculated;

[0070] Depth map generation is to generate a dense depth map of the scene by calculating the disparity pixel by pixel. The depth z of each pixel can be expressed as

[0071] S3E, calculation of the center depth of the taper sleeve: the center depth of the taper sleeve z t It is obtained by detecting the depth value of the center point coordinates of the cone sleeve in the depth map. The calculation steps are as follows: The target detection of the cone sleeve is to use the target detection algorithm (such as YOLOv11) to detect the bounding box B of the cone sleeve t , and obtain the center point coordinates (x t ,y t ); Extracting the center point depth is to extract the depth corresponding to the center point of the cone sleeve from the depth map z(x,y), z t = z(x t ,y t );

[0072] The estimation of the depth morphology of the cone sleeve outer contour is to more comprehensively describe the spatial position of the cone sleeve, and it is necessary to calculate the depth distribution and average depth of its outer contour; the outer contour point set extraction is from the bounding box B of the target detection algorithm t Extract the pixel point set P of the outer contour t ={(x i ,y i )}; The outer contour depth distribution is for each pixel point (x i ,y i ) Calculate the depth value; Z t ={z(x i ,y i )Ⅰ(x i ,y i )∈P t};

[0073] The average depth calculation is to calculate the average depth z of the outer contour of the cone sleeve avg ; Where n is the number of pixels in the outer contour point set;

[0074] The depth standard deviation is the standard deviation of the depth of the outer profile of the taper sleeve. z , used to estimate the morphological change: δz ＝

[0075] The distance between the center point of the cone sleeve and the target position is calculated by binocular vision. t ,y t , z t ) and the target cone center position (x c ,y c , z c ) can calculate the Euclidean distance D between the two distance , the calculation formula is as follows: Provides real-time distance data during the docking phase. Distance data is used to control the frequency of sound prompts and set color warnings;

[0076] S3F, drogue swing amplitude analysis: extract the historical trajectory of the drogue center point and analyze the movement pattern of the drogue; the drogue swing amplitude is to analyze the deviation of the drogue relative to its trajectory during operation, which is used to guide the stability adjustment of the operation. Calculate the swing amplitude and frequency to prompt the pilot to adjust the operation strategy; T = {(x 1， y 1， z1), (x 2， y 2， z2), ...., (x n， y n， z n)}; The formula for the swing amplitude is:

[0077] A amplitude＝max(Doffset)

[0078] Where D offset Represents the distance from the center line at each time point: The average center position of the drogue within a specified time window. When the drogue swing exceeds the threshold, the pilot is prompted to make adjustments through an audible or visual alarm.

[0079] Step S4 further comprises the following steps:

[0080] Voice broadcast; according to the real-time distance between the cone sleeve and the cone tube, set up multi-level audio prompts: when the distance is far, the sound prompt frequency is low; as the distance approaches, the sound prompt frequency gradually increases; when the docking distance is reached, the sound prompt is converted to a continuous sound; the sound prompt frequency, f, is related to the distance D from the cone sleeve to the cone tube distance Inversely proportional; k is the adjustment coefficient, which is used to control the range of sound frequency variation.

[0081] Color auxiliary prompts; set three-color warning signs according to different distance ranges:

[0082] Red: Indicates that the cone sleeve deviates from the target range and needs to be adjusted immediately;

[0083] Yellow: Indicates that the drogue is close to the target range but has not reached the docking condition;

[0084] Green: indicates that the cone sleeve is in the correct position and can be docked;

[0085] Grading prompt rules;

[0086] D distance >5m; low frequency prompt (about 1 time / second);

[0087] 2m<D distance ≤5m; medium frequency prompt (about 3 times / second);

[0088] D distance ≤2m; high frequency prompt (about 5 times / second);

[0089] D distance =0; continuous tone prompt;

[0090] Warning information display; real-time display of current parameters (speed, deviation, distance, etc.) on the system interface; providing a visual prompt interface to help pilots quickly understand the current status;

[0091] According to the butt distance D between the cone sleeve and the cone tube distance and the deviation Ddistance and D distance Set the color hint scheme.

[0092] Red: D distance >5m or D axial >0.5m or D planar >0.5m; prompts that the deviation is too large and needs to be adjusted immediately;

[0093] Yellow: 2m<D distance ≤5m and D axial ≤0.5m and D planar ≤0.5m; prompts that the target range has been reached, but the docking conditions have not yet been met;

[0094] Green: D distance ≤2m and D axial ≤0.1m and D planar ≤0.1m; indicates that the cone sleeve is in the correct position and the docking can be completed.

[0095] The software features high-precision detection and tracking, target recognition algorithm based on YOLOv11, high detection accuracy and real-time performance, multi-dimensional auxiliary information: providing various auxiliary parameters such as speed, deviation, distance, etc., and fully supporting refueling docking operations. Multimodal interactive prompts: combining sound and visual prompts to help pilots quickly grasp the current status and improve operational efficiency. High robustness: adapt to different lighting conditions, bad weather and complex situations of rapid movement of the drogue.

[0096] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A binocular vision aerial refueling drogue real-time tracking and auxiliary docking method, characterized in that: The following steps are involved: S1. Obtain real-time video or image of the refueling drogue through a binocular camera; S2. Use artificial intelligence to complete the training of the cone sleeve model, and on this basis, use YOLOV11 to complete the cone sleeve recognition and tracking; S3, calculate the auxiliary information such as the approach speed, axial deviation, plane deviation, docking distance, swing amplitude of the cone sleeve and the like of the cone tube; S4. The frequency of sound broadcast is controlled according to the distance, and the warning information is assisted by red, green and yellow colors.

2. A binocular vision aerial refueling drogue real-time tracking and assisted docking method according to claim 1, characterized in that: The step S1 further comprises the following steps: S1A, binocular camera configuration: Install a high-resolution binocular camera in the refueling docking area of ​​the receiving aircraft; the binocular camera has real-time data acquisition capabilities, can provide high frame rate and high-resolution video streams, and support work in adverse weather conditions; S1 B, Image acquisition process: Use the binocular camera to obtain the front scene video of the aerial refueling drogue in real time; use the synchronous acquisition mechanism to ensure the frame alignment of the left and right camera images to obtain depth information; S1 C. Image preprocessing: De-noise, enhance and histogram equalize the collected images; filter out background noise through specific filters and retain the significant features of the cone sleeve area.

3. The binocular vision aerial refueling drogue real-time tracking and assisted docking method according to claim 1, characterized in that: The step S2 further comprises the following steps: S2A, collect a large number of cone sleeve image data under different lighting, angles, distances and backgrounds; S2B, use deep learning frameworks (such as PyTorch or TensorFlow) to train the YOLOv11 model; during model training, use data augmentation techniques (such as rotation, scaling, and cropping) to improve the robustness of recognition; S2C, real-time cone sleeve recognition and tracking: Deploy the trained YOLOv11 model to the edge computing device to achieve real-time cone sleeve recognition; track the cone sleeve and extract its three-dimensional coordinates based on the depth information of the binocular camera; S2D, recognition and tracking optimization: Use optical flow algorithm and Kalman filter technology to optimize tracking accuracy; for the high-speed movement of the cone sleeve, improve system stability through multi-frame information fusion.

4. A binocular vision aerial refueling drogue real-time tracking and assisted docking method according to claim 1, characterized in that: The step S3 further comprises the following steps: S3A, Approach speed calculation: The approach speed between the receiving aircraft and the drogue is calculated by combining the change in the position of the drogue in consecutive frames with the time difference. The approach speed refers to the relative speed between the drogue and the drogue in three-dimensional space. The calculation formula is as follows: S3B, Calculation of axial deviation and plane deviation: Axial deviation and plane deviation are the relative position deviations between the cone sleeve and the cone tube in three-dimensional space, which are used to guide the adjustment operation. Calculate the difference between the center point of the cone sleeve and the target point of the cone tube in three-dimensional space; decompose the difference into axial deviation and plane deviation, which respectively reflect the offset direction and amplitude of the cone sleeve. S3C, docking distance measurement: using the parallax principle of the binocular camera, accurately measure the three-dimensional distance between the cone sleeve and the cone tube; the basic formula is: S3D, depth information calculation process: Matching feature points is to find the same feature points (such as corner points, edge points, etc.) in the left and right images, usually achieved through feature point extraction algorithms (such as SIFT, SURF), and calculate the pixel coordinate difference of the corresponding points in the left and right images through the feature matching algorithm, that is, the disparity d S3E, calculation of the depth of the cone sleeve center point: The target detection of the cone sleeve is to use the target detection algorithm (such as YOLOv11) to detect the bounding box B of the cone sleeve t , and obtain the center point coordinates (x t ,y t ); extracting the center point depth is to extract the depth corresponding to the center point of the cone sleeve from the depth map z(x,y); S3F, drogue swing amplitude analysis: extract the historical trajectory of the drogue center point and analyze the movement pattern of the drogue; the drogue swing amplitude is to analyze the deviation of the drogue relative to its trajectory during operation, which is used to guide the stability adjustment of the operation. Calculate the swing amplitude and frequency to prompt the pilot to adjust the operation strategy; the formula for the swing amplitude is:

5. The binocular vision aerial refueling drogue real-time tracking and assisted docking method according to claim 1, characterized in that: The step S4 further includes the following steps: sound broadcast, color auxiliary prompt, graded prompt rules, warning information display, according to the docking distance D between the cone sleeve and the cone tube distance and the deviation D distance and D distance Set the color hint scheme.

Images