Real-time identification and positioning method and device for all-weather unmanned aerial vehicle in limited airspace based on deep learning

Abstract

The invention provides a real-time identification and positioning method and device for an all-weather unmanned aerial vehicle in a limited airspace based on deep learning. The real-time identification and positioning method is based on at least two cameras and an infrared sensor which are mounted on a holder; the method comprises the following steps: firstly, ensuring that a limited airspace image is stably captured by utilizing holder control based on adaptive Kalman; then, splicing a plurality of captured images into a complete high-definition panoramic image based on an infrared and optical panoramic fusion technology; the unmanned aerial vehicle target in the high-definition panoramic image is detected and recognized in real time based on target detection; and finally, by adopting an aerial target positioning technology based on photogrammetry, according to the identified position of the unmanned aerial vehicle target in the high-definition panoramic image, and by combining a photogrammetry principle, calculating a specific position coordinate of the unmanned aerial vehicle target in the space, thereby realizing real-time identification and positioning of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle positioning and identification, and in particular to a real-time identification and positioning method and device for an all-weather unmanned aerial vehicle in a limited airspace based on deep learning. Background Art

[0002] The risk factors of drones in restricted airspace to the operating power system are as follows:

[0003] Collision risk: During flight, the drone may collide with power system cables, towers and other facilities, causing damage to the facilities or crashing of the drone.

[0004] Interference risk: The electromagnetic signals of drones may interfere with the communication and control signals of the power system, causing failure or loss of control of the power system.

[0005] Intrusion risk: Drones may intrude into the no-fly zones of the power system, posing a threat to the safety of the power system.

[0006] Malicious attack risk: Hackers or malicious users may use drones to attack power systems, such as carrying explosives, destructive weapons, etc., causing damage to the power system.

[0007] In view of the above problems, the main means currently adopted in the prior art are as follows:

[0008] Establish an electromagnetic shielding area: Establish an electromagnetic shielding area around the power system and prohibit drones from entering. This method requires the use of professional electromagnetic shielding equipment and materials, which is costly.

[0009] Use radio spectrum monitoring equipment: Use radio spectrum monitoring equipment around the power system to monitor the radio signals of drones. Once the drone signal is detected, it can be interfered with or shielded to prevent it from interfering with and damaging the power system. This method requires the use of professional radio spectrum monitoring equipment and jammers, which is costly.

[0010] Use radar technology: Use radar technology around the power system to monitor the location and trajectory of drones. Once a drone is found to enter a restricted airspace, alarms or jammers can be used to prevent it from interfering with and damaging the power system. This method requires the use of professional radar equipment and software, which is costly.

[0011] In addition to the high cost, the above solutions may still have problems such as misidentification, missed identification or missed processing due to illegal modification of drones, and cannot completely eliminate the interference risk of drones. Especially in the field deployment of airport environments, existing airport cameras and visual monitoring devices should be used to realize the all-weather positioning and monitoring function of drones. Summary of the invention

[0012] Therefore, considering the defects and shortcomings of the above-mentioned prior art, the present invention proposes an innovative all-weather UAV real-time identification and positioning method, which is suitable for UAVs within a limited airspace. The core of this method is the integration of adaptive Kalman control pan-tilt technology, panoramic fusion technology of infrared and optical images, target training and recognition technology based on deep learning attention mechanism and Transformer model, and precise photogrammetry positioning technology. Especially under all-weather conditions, the present invention can cope with various meteorological and lighting changes and identify UAV targets. Through the adaptive Kalman pan-tilt control technology, precise attitude control of the pan-tilt is achieved; the panoramic fusion technology of infrared and optical images is used to achieve clear imaging in complex all-weather environments; the deep learning technology of the attention mechanism and the Transformer model is combined to improve the accuracy and real-time performance of target recognition; and the positioning technology based on photogrammetry is used to achieve precise positioning and identification of UAVs in the air.

[0013] The present invention can realize all-weather positioning monitoring of the status of UAVs within a limited space through simple dual-camera deployment, thereby ensuring the safety of flight operations and preventing the risk of invasion by foreign flying objects.

[0014] The present invention specifically adopts the following technical solutions:

[0015] A real-time identification and positioning method for all-weather UAVs in a limited airspace based on deep learning, the method comprising:

[0016] Based on at least two cameras and an infrared sensor mounted on a pan-tilt, and utilizing adaptive Kalman-based pan-tilt control to capture images in a limited airspace;

[0017] The captured multiple images are stitched into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology;

[0018] Real-time detection and recognition of drone targets in high-definition panoramic images based on target detection;

[0019] The aerial target positioning technology based on photogrammetry is adopted. According to the position of the identified UAV target in the high-definition panoramic image and combined with the principle of photogrammetry, the specific position coordinates of the UAV target in space are calculated.

[0020] Furthermore, the adaptive Kalman-based gimbal control uses a Kalman filter algorithm to predict and update the flight state of the UAV on the basis of establishing the gimbal differential state equation and the observation equation.

[0021] Furthermore, the adaptive Kalman-based PTZ control represents the process of obtaining the PTZ rotation angle as an offline process system, which is expressed by a linear stochastic differential equation and a rotation angle observation equation. The linear stochastic differential equation is expressed as:

[0022] ε(k)=Aε(k-1)+Bμ(k)+ω(k)

[0023] The observation equation of the rotation angle is expressed as:

[0024] Z(k)=Hε(k)+ν(k)

[0025] Among them, ε(k) is the predicted value of the rotation angle at time k; μ(k) is the influencing factor on the rotation angle at time k; A and B are the influencing factors on the rotation angle observation, in the form of a matrix; Z(k) is the observed value of the rotation angle at time k; H is the acquired characteristic parameter; ω(k) and ν(k) represent the process and measurement noise;

[0026] The Kalman filter algorithm is used to obtain the rotation angle data of the control process to realize the pan / tilt rotation;

[0027] The recursive processing of the Kalman filter algorithm is as follows:

[0028] Predict the current state based on the system's previous state:

[0029] ε(k|k-1)=Aε(k-1|k-1)+Bμ(k)

[0030] Among them, ε(k|k-1) is the result predicted by the previous state, ε(k-1|k-1) is the optimal result of the previous state, μ(k) is the factor affecting the rotation angle at time k, corresponding to the covariance of ε(k|k-1), expressed as P:

[0031] P(k|k-1)=AP(k-1|k-1)A'+Q

[0032] Where P(k|k-1) is the covariance corresponding to ε(k|k-1), P(k-1|k-1) is the covariance corresponding to the moment ε(k-1|k-1), A' is the transpose of A, and Q is the covariance of the system process;

[0033] Adaptively adjust the covariance Q of the system process;

[0034] Collect the measured values ​​of the current state, combine the predicted value and the measured value, and get the optimal estimate ε(k|k) of the current state k:

[0035] ε(k|k)=ε(k|k-1)+Kg(k)(Z(k)-Hε(k|k-1))

[0036] Where Kg is the Kalman gain:

[0037]

[0038] Where H is the observation matrix and R is the covariance matrix of measurement noise;

[0039] Combined with the above formula, the optimal optimization estimate ε(k|k) under the K state is calculated;

[0040] The search for drone targets is regarded as a continuous observation process. The corresponding filtering process needs to be run continuously, and the covariance of ε(k|k) in the K state is continuously updated as follows:

[0041] P(k|k)=(I-Kg(k)H)P(k|k-1)

[0042] Among them, the rotation angle observation is a single modulus, and I is 1;

[0043] Through continuous autoregressive calculation, the optimal parameters of the rotation angle of the pan-tilt control process are obtained; then the optimal parameters of the rotation angle are compared with the center position of the image to identify whether the current pan-tilt adjustment has reached the predetermined center position.

[0044] Furthermore, in the process of stitching the captured multiple images into a complete high-definition panoramic image based on the infrared and optical panoramic fusion technology:

[0045] fusing an infrared image and a visible light image acquired by at least an infrared sensor and a camera arranged on a gimbal;

[0046] Perform preprocessing on the fusion results including de-distortion, registration and image enhancement;

[0047] Extract and match the multi-view spatial image feature information from the pre-processed fusion results;

[0048] Based on the extracted and matched multi-scene spatial image feature information, multi-scene image data is fused and stitching traces are eliminated.

[0049] Furthermore, the pre-processing filtering process adopts Gaussian filtering;

[0050] In the process of extracting feature information of multi-view spatial domain images, the Hessian matrix is ​​used to detect image feature points to obtain stable edge points of image data, and a box filter is used to approximately replace the Gaussian filter to convert the filtering of two-dimensional data into a problem of calculating addition and subtraction operations of points between different regions;

[0051] Feature point matching uses RANSAC to perform correlation matching on feature points and obtain the corresponding homography matrix as the transformation matrix between the two images;

[0052] The multi-scene image data fusion is specifically as follows: after matching the feature points, several pairs of feature matching points are selected, substituted into the homography transformation matrix, the parameter values ​​of the homography matrix are solved, and one image data is mapped to another image data according to the homography matrix to realize the data fusion operation.

[0053] Furthermore, the real-time detection and identification of drone targets in high-definition panoramic images based on target detection is specifically: real-time detection and identification of drone targets in high-definition panoramic images based on attention mechanism and Transformer;

[0054] The steps of object training and recognition based on attention mechanism and Transformer include:

[0055] Data collection and preprocessing: Collect drone image and video data of various target objects, scenes and lighting conditions, and perform preprocessing on the data including size normalization, color space conversion and data enhancement;

[0056] Feature extraction: Use convolutional neural networks to extract image features, combined with attention mechanisms, so that the model focuses on the areas in the image that are most relevant to target recognition;

[0057] Transformer model construction: Build a Transformer model, including multiple self-attention layers and feed-forward networks, to process sequence data and divide the image into multiple patches as sequence input;

[0058] Model testing and validation: Label the object locations in the dataset, assign a unique identifier to each object, train the Transformer model to recognize and locate objects in the image, and use the cross entropy loss function and position regression loss to jointly optimize the classification and localization tasks.

[0059] Furthermore, the aerial target positioning technology based on photogrammetry is adopted, and the position of the identified drone target in the high-definition panoramic image is specifically as follows:

[0060] Determine the parameters of the two cameras, including focal length, viewing angle and distance, use a calibration device to calibrate each camera to obtain internal parameters including focal length and principal point coordinates and external parameters including camera position relationship, record the focal length f, viewing angle fov, baseline distance b between cameras and rotation and translation matrix of the camera relative to the world coordinate system of each camera, and perform denoising, correction and scaling on the image under the premise of ensuring synchronous acquisition;

[0061] Obtain the pixel coordinates (u, v) of the drone targets in the images of the two cameras through target detection;

[0062] According to the obtained pixel coordinates and camera parameters, pixel coordinate conversion is performed to convert the pixel coordinates (u, v) into normalized coordinates (x_n, y_n), and the coordinates of the drone target in the camera coordinate system are calculated;

[0063] x_n=(u-c_x) / f

[0064] y_n=(v-c_y) / f

[0065] Among them, (c_x, c_y) is the coordinate of the principal point of the image;

[0066] On the basis of obtaining the coordinates of the UAV target in the camera coordinate system, the distance from the UAV target to the camera is estimated according to the parallax of the UAV target in the two camera images; then the position of the UAV target in the world coordinate system is calculated by using the principle of similar triangles and the baseline distance of the camera.

[0067] And, a real-time identification and positioning device for all-weather UAVs in a limited airspace based on deep learning, comprising: at least two cameras and an infrared sensor mounted on a gimbal;

[0068] and, an adaptive Kalman control module for capturing images in a limited airspace using a pan / tilt control based on adaptive Kalman;

[0069] Infrared and optical panoramic fusion module, used to stitch multiple captured images into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology;

[0070] Target detection module, used for real-time detection and identification of drone targets in high-definition panoramic images;

[0071] The aerial target positioning module based on photogrammetry is used to calculate the specific position coordinates of the UAV target in space according to the position of the identified UAV target in the high-definition panoramic image and combined with the principle of photogrammetry.

[0072] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the program.

[0073] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0074] Compared with the prior art, the present invention and its preferred solution are applicable to all types of drones and flight scenarios in a limited airspace all day and night, with high recognition rate and positioning accuracy, and can realize the fusion of infrared and optical panoramic image data in a limited airspace. At the same time, it can greatly improve the monitoring accuracy and efficiency of drones, and provide data support for the real-time identification and positioning of humans and machines in a limited airspace. Once a drone is found to have entered a limited airspace, it can be prevented from interfering with and damaging the power system by means of alarms or jammers. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0076] Figure 1 A schematic diagram of a flow chart of a method for real-time identification and positioning of a drone provided by an embodiment of the present invention;

[0077] Figure 2 A schematic diagram of data fusion provided by an embodiment of the present invention;

[0078] Figure 3 A schematic diagram of real-time identification of drones using drone target training and identification technology based on attention mechanism and Transformer provided in an embodiment of the present invention;

[0079] Figure 4 A schematic diagram of an aerial target positioning technology based on photogrammetry provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] Hereinafter, the specific embodiments of the present application will be described in detail with reference to the accompanying drawings. According to these detailed descriptions, those skilled in the art can clearly understand the present application and can implement the present application. Without violating the principles of the present application, the features in the various embodiments can be combined to obtain new implementations, or certain features in certain embodiments can be replaced to obtain other preferred implementations.

[0081] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are specifically described in detail with reference to the accompanying drawings:

[0082] like Figure 1As shown, in the basic scheme provided by the embodiment of the present invention: based on at least two cameras and one infrared sensor installed on the gimbal; first, the gimbal control based on adaptive Kalman is used to ensure the stable capture of limited airspace images; then, the captured multiple images are spliced ​​into a complete high-definition panoramic image through infrared and optical panoramic fusion technology; and the UAV target in the high-definition panoramic image is detected and identified in real time based on target detection; finally, the aerial target positioning technology based on photogrammetry is adopted, and according to the position of the identified UAV target in the high-definition panoramic image, combined with the principle of photogrammetry, the specific position coordinates of the target in space are calculated, thereby realizing real-time recognition and positioning of the UAV.

[0083] The following is a further demonstration and introduction of several main functional modules and algorithm implementations of the present invention:

[0084] (1) PTZ control based on adaptive Kalman control

[0085] The control accuracy of the pan / tilt in the embodiment of the present invention adopts adaptive Kalman filtering to improve the control accuracy and reliability.

[0086] Considering that the gimbal is easily affected by the external environment during operation, which affects the accuracy and reliability of horizontal and vertical movement, the process of obtaining the gimbal rotation angle is represented as an offline process system, which is expressed by a linear stochastic differential equation and a rotation angle observation equation.

[0087] The linear stochastic differential equation is expressed as follows:

[0088] ε(k)=Aε(k-1)+Bμ(k)+ω(k) (1)

[0089] The observation equation of the rotation angle is expressed as:

[0090] Z(k)=Hε(k)+ν(k) (2)

[0091] In the above two formulas (1) and (2):

[0092] ε(k) is the predicted value of the rotation angle at time k; μ(k) is the factor affecting the rotation angle at time k; A and B are the factors affecting the observation of the rotation angle. In the complex flight control environment, A and B are matrices; Z(k) is the observed value of the rotation angle at time k; H is the acquired characteristic parameter; ω(k) and ν(k) represent the process and measurement noise;

[0093] For the linear stochastic differential equation that meets the above conditions, the process and the measurement are Gaussian white noise, so Kalman filtering is used to obtain the rotation angle data of the accurate control process.

[0094] The Kalman filter recursive processing process is as follows:

[0095] The recursive process uses the process model of the offline process system to predict the next state of the system. Assuming that the current system state is k, according to the system model, the current state can be predicted based on the previous state of the system:

[0096] ε(k|k-1)=Aε(k-1|k-1)+Bμ(k) (3)

[0097] In the above formula, ε(k|k-1) is the result predicted by the previous state, ε(k-1|k-1) is the optimal result of the previous state, μ(k) is the factor affecting the rotation angle at time k, and the covariance corresponding to ε(k|k-1) is represented by P:

[0098] P(k|k-1)=AP(k-1|k-1)A'+Q (4)

[0100] Where P(k|k-1) is the covariance corresponding to ε(k|k-1), P(k-1|k-1) is the covariance corresponding to the moment ε(k-1|k-1), A' is the transpose of A, and Q is the covariance of the system process;

[0101] Then the covariance Q of the system process is adaptively adjusted;

[0102] The above formula (3) and formula (4) are the processing process of the pan / tilt rotation using the adaptive Kalman filter, that is, the prediction of the system;

[0103] Collect the measured values ​​of the current state, combine the predicted value and the measured value, and get the optimal estimated value ε(k|k) of the current state k.

[0104] ε(k|k)=ε(k|k-1)+Kg(k)(Z(k)-Hε(k|k-1)) (5)

[0105] In formula (5), Kg is the Kalman gain

[0106]

[0107] Combining the above formula, the optimal optimization estimate ε(k|k) under state K is calculated.

[0108] Considering that in limited control patrol, searching for drone targets is a continuous observation process, the above filtering process needs to be run continuously to continuously update the covariance of ε(k|k) in the K state.

[0109] P(k|k)=(I-Kg(k)H)P(k|k-1) (7)

[0110] Among them, the rotation angle observation is a single modulus, and I is 1;

[0111] When the system enters the k+1 moment, P(k|k) is the formula (4), and this process continues to be calculated by autoregression;

[0112] Through the above processing, the optimal parameters of the rotation angle of the pan / tilt control process can be obtained;

[0113] By comparing the optimized parameters of the rotation angle with the center position of the image, it is identified whether the current adjustment of the pan / tilt head reaches the predetermined center position.

[0114] (2) Panoramic fusion of infrared and optical airspace images

[0115] In order to cope with various weather and lighting changes, this embodiment considers performing feature detection and extraction on the received multi-scene infrared and optical image data of a limited airspace, and then using key points to stitch the regional image data together, and finally forming infrared and optical panoramic image data for the limited area.

[0116] like Figure 2 As shown, the steps of implementing panoramic fusion and stitching of infrared and optical airspace images in this embodiment are as follows:

[0117] Step 1: Preprocessing of panoramic fusion and stitching of infrared and optical images in the airspace is mainly the processing process before the panoramic fusion and stitching of the airspace images. The purpose of preprocessing is to pre-solve some undesirable factors that may appear in the stitching process of multi-view data, and provide good conditions for extracting feature points of multi-view airspace image data. The filtering process of this embodiment adopts Gaussian filtering. The two-dimensional Gaussian filtering process is as follows:

[0118]

[0119] Step 2: Extract feature information from multi-view infrared and optical spatial images

[0120] The extraction of multi-view spatial image feature information is the collection of essential characteristics. In data fusion and splicing, point features are mainly used. The Hessian matrix is ​​used to detect image feature points.

[0121] The Hessian matrix is ​​a matrix composed of partial derivatives of a binary function. If the coordinates of a point are set to X(x,y), the form of the Hessian matrix is ​​as follows:

[0122]

[0123] The purpose of constructing the Hessian matrix is ​​to obtain stable edge points of image data and lay a good foundation for feature extraction. The Hessian matrix of spot detection is used to detect feature points. Its determinant value indicates the amount of change around the data point. The feature point needs to take the maximum and minimum values ​​of the determinant value. At the same time, in order to achieve the scale-invariant feature, the determinant value of the scale σ is used for feature point detection. Set the coordinates of a point as X(x, y), L xx (X,σ) is the Gaussian second-order partial derivative Convolution with image data X(x,y) produces the following results:

[0124] L(X,σ)=G(x,y,σ)*X(x,y) (10)

[0125] The Hessian matrix H(X,σ) at scale σ is as follows:

[0126]

[0127] Among them, X(x,y) is a two-dimensional function, and the determinant of the Hessian matrix can effectively determine the extreme points of the two-dimensional function.

[0128] Its determinant det(H) is defined as:

[0129]

[0130] In order to improve the operation speed, a box filter is used to approximate the Gaussian filter. The box filter transforms the filtering of two-dimensional data into the problem of calculating the addition and subtraction of points between different regions, which can be completed by simply looking up the integral graph. The approximate value of the Hessian matrix determinant of each point is:

[0131] det(H(x,y,σ)=L xx (X,σ)*L yy (X,σ)-(0.9*L xy (X,σ) 2 (13)

[0132] Preferably, this embodiment uses a weighting coefficient of 0.9 for multiplication in order to balance the deviation caused by using the box filter approximation.

[0133] Step 3: To match the features of multi-scene infrared and optical fusion image data, first, the feature information of the image data must be extracted, and then the feature information of the two image data must be matched. Since feature points are critical in the entire process of image data matching, the extraction of feature points is very important. When extracting feature points, two points must be noted: one is that as many feature points as possible need to be extracted; the other is that during the extraction process, the features of the feature points must be judged to extract feature points with typical image data features. The present invention uses RANSAC to perform association matching on the feature points and obtain the corresponding homography matrix. The specific process is as follows:

[0134] Random sampling is repeated a number of times, corresponding to the number of iterations.

[0135] The number of random sampling is determined by the following calculation:

[0136] 1-p=(1-w n ) k (14)

[0137] Among them, 1-p represents the probability that at least one of the selected n points is an outlier, which can generally be taken as 0.95, and w is the ratio of the inliers to all the data points. n The probability that the n points are all inside the game is 1-w n It represents the probability that at least one of the selected n points is an outlier. (1-w n ) k There is a probability that all the points selected will not be points within the game.

[0138] After calculation, the selected points are different:

[0139]

[0140] Randomly select matching pairs and then calculate the transformation matrix H parameters of the two image data;

[0141] When the transformation matrix is ​​H, calculate the distance between each pair of matching feature points, set the matching feature typical point (x, y) to the associated point H (x', y') after the change matrix H, and the Euclidean distance of the matching point is as follows:

[0142]

[0143] Set a fixed threshold and compare each distance with the threshold. When the distance does not exceed the threshold, this point can be regarded as an inlier. If the number of inliers exceeds the total number, then this matching point can be directly used as the correct and optimal matching point. The transformation matrix H between the last two images can be directly calculated based on the inliers and outliers at this time.

[0144] Step 4: Data fusion of multiple images. After matching the feature points, several pairs of feature matching points with good quality can be selected. These point pairs are substituted into the homography transformation matrix to solve the parameter values ​​of the homography matrix. Then, one image data can be mapped to another image data according to the homography matrix to realize data fusion operation.

[0145] Assume f1(x, y) and f2(x, y) are the two scene image data, and f(x, y) is the result of fusion data. The expression after fusion is as follows:

[0146]

[0147] (3) UAV target training and recognition based on attention mechanism and Transformer

[0148] The attention mechanism is a mechanism that simulates human visual attention. It highlights important information by assigning different weights to different parts of input data. Transformer is based on the self-attention mechanism. By stacking multiple self-attention layers and feedforward networks, it achieves efficient sequence-to-sequence learning. In target training and recognition technology, Transformer can capture long-distance dependencies and learn rich feature representations through end-to-end training. The steps of target training and recognition based on attention mechanism and Transformer are as follows: Figure 3 As shown:

[0149] Step 1: Data collection and preprocessing. By collecting a large amount of drone image and video data, which should contain a variety of target objects, scenes, and lighting conditions, the data is preprocessed, including size normalization, color space conversion, data enhancement, etc., to improve the generalization ability of the model.

[0150] Step 2: Feature extraction, using a convolutional neural network (CNN) to extract image features, combined with an attention mechanism, enables the model to focus on the areas in the image that are most relevant to target recognition.

[0151] Step 3: Transformer model construction: Build a Transformer-based model, which includes multiple self-attention layers and feedforward networks to process sequence data. It is suitable for dividing images into multiple parts (patches) as sequence input.

[0152] Step 4: Model testing and validation, label the object locations in the dataset, assign a unique identifier to each object, train the Transformer model to identify and locate objects in the image, and use the cross entropy loss function and position regression loss to jointly optimize the classification and positioning tasks.

[0153] (4) Aerial target positioning based on photogrammetry

[0154] The positioning of aerial targets in limited airspace mainly uses stereo vision technology to obtain the coordinates of the target in three-dimensional space by matching and calculating the images of two cameras. It is necessary to obtain the camera's resolution, focal length, viewing angle, distance and other parameters in advance. The specific steps are as follows: Figure 4 As shown:

[0155] 1. Determine the parameters of the two cameras, including focal length, viewing angle, and distance, and use a calibration device to calibrate each camera to obtain its internal parameters (focal length, principal point coordinates) and external parameters (camera position relationship). Record the focal length (f), viewing angle (fov), baseline distance between cameras (b), and rotation and translation matrices of the cameras relative to the world coordinate system of each camera to ensure synchronous acquisition, and perform denoising, correction, scaling, and other processing on the image.

[0156] 2. Determine the pixel coordinates of the target in the two images by using the object detection algorithm to detect the target in the images of the two cameras and obtain its pixel coordinates (u, v).

[0157] 3. According to the pixel coordinates and camera parameters, perform pixel coordinate conversion, convert the pixel coordinates (u, v) into normalized coordinates (x_n, y_n), and calculate the coordinates of the target in the camera coordinate system.

[0158]

[0159] Among them, (c_x, c_y) are the coordinates of the principal point of the image.

[0160] First, the distance from the target to the camera is estimated based on the disparity of the target in the two camera images.

[0161] 4. Using the principle of similar triangles and the baseline distance of the camera, calculate the position of the target in the world coordinate system.

[0162] In addition, corresponding to the method provided in the above embodiment, an embodiment of the present invention further provides a real-time identification and positioning device for an all-weather UAV in a limited airspace based on deep learning, comprising: at least two cameras and an infrared sensor installed on a gimbal;

[0163] and, an adaptive Kalman control module for capturing images in a limited airspace using a pan / tilt control based on adaptive Kalman;

[0164] Infrared and optical panoramic fusion module, used to stitch multiple captured images into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology;

[0165] Target detection module, used for real-time detection and identification of drone targets in high-definition panoramic images;

[0166] The aerial target positioning module based on photogrammetry is used to calculate the specific position coordinates of the UAV target in space according to the position of the identified UAV target in the high-definition panoramic image and combined with the principle of photogrammetry.

[0167] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0168] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0171] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0172] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0173] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0174] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0175] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

[0176] The present invention is not limited to the above-mentioned optimal implementation mode. Under the inspiration of the present invention, anyone can derive various other forms of a real-time identification and positioning method and device for all-weather drones in a limited airspace based on deep learning. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A real-time identification and positioning method for all-weather UAVs in a limited airspace based on deep learning, characterized in that: The method comprises: Based on at least two cameras and an infrared sensor mounted on a pan-tilt, and utilizing adaptive Kalman-based pan-tilt control to capture images in a limited airspace; The captured multiple images are stitched into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology; Real-time detection and recognition of drone targets in high-definition panoramic images based on target detection; The aerial target positioning technology based on photogrammetry is adopted. According to the position of the identified UAV target in the high-definition panoramic image and combined with the principle of photogrammetry, the specific position coordinates of the UAV target in space are calculated.

2. According to claim 1, a real-time identification and positioning method for all-weather UAVs in a limited airspace based on deep learning is characterized in that: The adaptive Kalman-based gimbal control uses the Kalman filter algorithm to predict and update the flight state of the UAV on the basis of establishing the gimbal differential state equation and observation equation.

3. The real-time identification and positioning method of an all-weather UAV in a limited airspace based on deep learning according to claim 2 is characterized in that: The adaptive Kalman-based PTZ control represents the process of obtaining the PTZ rotation angle as an offline process system, which is expressed by a linear stochastic differential equation and a rotation angle observation equation. The linear stochastic differential equation is expressed as: ε(k)=Aε(k-1)+Bμ(k)+ω(k) The observation equation of the rotation angle is expressed as: Z(k)=Hε(k)+ν(k) Among them, ε(k) is the predicted value of the rotation angle at time k; μ(k) is the influencing factor on the rotation angle at time k; A and B are the influencing factors on the rotation angle observation, in the form of a matrix; Z(k) is the observed value of the rotation angle at time k; H is the acquired characteristic parameter; ω(k) and ν(k) represent the process and measurement noise; The Kalman filter algorithm is used to obtain the rotation angle data of the control process to realize the pan / tilt rotation; The recursive processing of the Kalman filter algorithm is as follows: Predict the current state based on the system's previous state: ε(k|k-1)=Aε(k-1|k-1)+Bμ(k) Among them, ε(k|k-1) is the result predicted by the previous state, ε(k-1|k-1) is the optimal result of the previous state, μ(k) is the factor affecting the rotation angle at time k, corresponding to the covariance of ε(k|k-1), expressed as P: P(k|k-1)=AP(k-1|k-1)A'+Q Where P(k|k-1) is the covariance corresponding to ε(k|k-1), P(k-1|k-1) is the covariance corresponding to the moment ε(k-1|k-1), A' is the transpose of A, and Q is the covariance of the system process; Adaptively adjust the covariance Q of the system process; Collect the measured values ​​of the current state, combine the predicted value and the measured value, and get the optimal estimate ε(k|k) of the current state k: ε(k|k)=ε(k|k-1)+Kg(k)(Z(k)-Hε(k|k-1)) Where Kg is the Kalman gain: Where H is the observation matrix and R is the covariance matrix of measurement noise; Combined with the above formula, the optimal optimization estimate ε(k|k) under the K state is calculated; The search for drone targets is regarded as a continuous observation process. The corresponding filtering process needs to be run continuously, and the covariance of ε(k|k) in the K state is continuously updated as follows: P(k|k)=(I-Kg(k)H)P(k|k-1) Among them, the rotation angle observation is a single modulus, and I is 1; Through continuous autoregressive calculation, the optimal parameters of the rotation angle of the pan-tilt control process are obtained; then the optimal parameters of the rotation angle are compared with the center position of the image to identify whether the current pan-tilt adjustment has reached the predetermined center position.

4. The method for real-time identification and positioning of all-weather UAVs in a limited airspace based on deep learning according to claim 1, characterized in that: In the process of stitching the captured multiple images into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology: fusing an infrared image and a visible light image acquired by at least an infrared sensor and a camera arranged on a gimbal; Perform preprocessing on the fusion results including de-distortion, registration and image enhancement; Extract and match the multi-view spatial image feature information from the pre-processed fusion results; Based on the extracted and matched multi-scene spatial image feature information, multi-scene image data is fused and stitching traces are eliminated.

5. The method for real-time identification and positioning of all-weather UAVs in a limited airspace based on deep learning according to claim 4, characterized in that: The pre-processing filtering process adopts Gaussian filtering; In the process of extracting feature information of multi-view spatial domain images, the Hessian matrix is ​​used to detect image feature points to obtain stable edge points of image data, and a box filter is used to approximately replace the Gaussian filter to convert the filtering of two-dimensional data into a problem of calculating addition and subtraction operations of points between different regions; Feature point matching uses RANSAC to perform correlation matching on feature points and obtain the corresponding homography matrix as the transformation matrix between the two images; The multi-scene image data fusion is specifically as follows: after matching the feature points, several pairs of feature matching points are selected, substituted into the homography transformation matrix, the parameter values ​​of the homography matrix are solved, and one image data is mapped to another image data according to the homography matrix to realize the data fusion operation.

6. The method for real-time identification and positioning of all-weather UAVs in a limited airspace based on deep learning according to claim 1, characterized in that: The real-time detection and identification of drone targets in high-definition panoramic images based on target detection is specifically: real-time detection and identification of drone targets in high-definition panoramic images based on attention mechanism and Transformer; The steps of object training and recognition based on attention mechanism and Transformer include: Data collection and preprocessing: Collect drone image and video data of various target objects, scenes and lighting conditions, and perform preprocessing on the data including size normalization, color space conversion and data enhancement; Feature extraction: Use convolutional neural networks to extract image features, combined with attention mechanisms, so that the model focuses on the areas in the image that are most relevant to target recognition; Transformer model construction: Build a Transformer model, including multiple self-attention layers and feed-forward networks, to process sequence data and divide the image into multiple patches as sequence input; Model testing and validation: Label the object locations in the dataset, assign a unique identifier to each object, train the Transformer model to recognize and locate objects in the image, and use the cross entropy loss function and position regression loss to jointly optimize the classification and localization tasks.

7. The method for real-time identification and positioning of all-weather UAVs in a limited airspace based on deep learning according to claim 1, characterized in that: Using aerial target positioning technology based on photogrammetry, the positions of the identified drone targets in the high-definition panoramic image are as follows: Determine the parameters of the two cameras, including focal length, viewing angle and distance, use a calibration device to calibrate each camera to obtain internal parameters including focal length and principal point coordinates and external parameters including camera position relationship, record the focal length f, viewing angle fov, baseline distance b between cameras and rotation and translation matrix of the camera relative to the world coordinate system of each camera, and perform denoising, correction and scaling on the image under the premise of ensuring synchronous acquisition; Obtain the pixel coordinates (u, v) of the drone targets in the images of the two cameras through target detection; According to the obtained pixel coordinates and camera parameters, pixel coordinate conversion is performed to convert the pixel coordinates (u, v) into normalized coordinates (x_n, y_n), and the coordinates of the drone target in the camera coordinate system are calculated; x_n=(u-c_x) / f y_n=(v-c_y) / f Among them, (c_x, c_y) is the coordinate of the principal point of the image; On the basis of obtaining the coordinates of the drone target in the camera coordinate system, the distance from the drone target to the camera is estimated according to the parallax of the drone target in the two camera images; Then, the position of the UAV target in the world coordinate system is calculated using the principle of similar triangles and the baseline distance of the camera.

8. A real-time identification and positioning device for all-weather drones in a limited airspace based on deep learning, characterized in that: include: At least two cameras and one infrared sensor mounted on a gimbal; and, an adaptive Kalman control module for capturing images in a limited airspace using a pan / tilt control based on adaptive Kalman; Infrared and optical panoramic fusion module, used to stitch multiple captured images into a complete high-definition panoramic image based on infrared and optical panoramic fusion technology; Target detection module, used for real-time detection and identification of drone targets in high-definition panoramic images; The aerial target positioning module based on photogrammetry is used to calculate the specific position coordinates of the UAV target in space according to the position of the identified UAV target in the high-definition panoramic image and combined with the principle of photogrammetry.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of a real-time identification and positioning method for an all-weather drone in a limited airspace based on deep learning are implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a real-time identification and positioning method for an all-weather UAV in a limited airspace based on deep learning are implemented as described in any one of claims 1 to 7.

Images